Blackboard Down! A Retrospective Look at Crashing

May
2006

In 1982, I was forced to take a workshop in "appleworks for teachers." I didn't look forward to spending a week with computer geeks, and went along nearly kicking and screaming. It turned out to be the most important week of my teaching career! The instructor started the workshop with three important pieces of advice worth sharing:

Back up your work. There are only two kinds of computer users; those who have lost data and those who will.

Save often. There is such a thing as a "Karmic Editor". The KE attacks documents in progress, no one is exempt and you may never know why.

Move on. You may have something go wrong and while you are trying everything you can think of to fix the problem, it seems to fix itself. It just might be that you have no clue about how it got fixed. You have to be okay with that, count your blessings, and move on. Trying to figure out how you fixed the problem may drive you crazy.

I have carried, and honored, these three pieces of advice and they have served me well, especially in the last month.

Imagine that you have been talking up the uses of a Course Management System (CMS) to your faculty for a couple of years and finally most people have discovered the benefit of using the CMS for keeping a grade book, posting assignments and class announcements. In addition, you have a ton of online course sections running and have had no problems with anything for five wonderful years. Would you be depending on the system, maybe to a point of complacency? What if your CMS failed, suddenly, mid semester? Just such a catastrophe happened to us during our spring break. We all came back to school to face the complete loss of our online system (we use Blackboard and host our own), which ironically happened due to a major "glitch," while our technicians were installing a much needed redundant back-up system. There were lots of reasons and circumstances that necessitated our doing this mid-semester, and we have wonderful technical folks, so I won't go into the details. Suffice it to say, that it can happen in the best of situations to really good people. This article is about what happened and lessons learned. (Remember Katrina?) Alfred E. Neuman, in Mad Magazine, back in 1960-something, said, "Learn from the mistakes of others, because you will never live long enough to make them all yourself."

We spent a week trying to recover the data that had been lost (content entered directly into the CMS) and during that week we provided instructors with web space and assistance in posting content for students while we were in recovery mode. Our Blackboard Network Coordinator worked with our Web Master to immediately post a page, replacing our blackboard portal, which informed students that they should check their college email. Unfortunately, it took us a couple of days to figure out that students thought their classes were postponed for a few days. We then added a note to the original "check your email" post informing them that their courses had not been cancelled.

As the faculty distance education coordinator, I felt it was important to keep everyone informed about what was going on every step of the way. I sent at least three updates per day, stayed in phone and email contact with affected faculty, and assisted with securing web space and posting content as needed. I also tried to assess what people would need, should our data not be recoverable.

The data was not readily recoverable. We weren't sure, at first, how much we had lost.

After we found out the data was very likely gone, our immediate concern was to get blackboard back up, with or without all of the course content. Students were out of touch because they couldn't access Blackboard the way they were accustomed to. We have college email for all of our students, but they don't use it much for a variety of reasons. Because we expect them to use college email, we don't have their private email addresses and have to communicate with them either through the CMS or hope that they check their email.

Prior to this, as a regular practice, some instructors demanded that students check their email several times a week and they send out announcements in the email to force students to establish access. (This is a good idea, if you rely on college email for students.)

We found out that most faculty, including me, had not consistently used the college email with students.

So, we used our automated phone system to contact all the online students reminding them to watch their email. Unfortunately, that showed us just how many students had out-dated phone information on record!

While we were waiting to hear about the data, I sent out an email suggesting that in the worst case scenario, we should, as a faculty, if we lost their work or grades, plan to do nothing to penalize students,. Whatever policy that was developed beyond that, would depend on each individual class situation and what the instructor had backed up. The reason it was only a suggestion on my part was, first I wasn't sure if anyone had lost anything, and second, we didn't have a formal policy established for the worst case scenario!

After a meeting of faculty, technicians and administrators, we decided (against my better judgment) to hold forums with faculty BEFORE we created such a policy. I was concerned that the forum process would take too long, and we would all be doing different things. My actual worry was that some instructors would give blanket credit for all work lost, and others would require students to make up everything which would result in messy conflicts. This did happen in some cases. (By the way, just to put your mind at ease, we have since recovered much of what was originally thought to be lost and were not in as horrible shape as we originally feared. Students had a lot of saved work and faculty had almost all of their data, either saved locally or institutionally archived.)

It took a little over a week to get back up and running and it took a few days to resume the rhythm of teaching and learning that is the nature of distance education. The students were, for the most part, fabulous and patient. The instructors, for the most part, were also wonderful, understanding, incredibly hard-working and committed to student success. We did receive a call from the press, which I'll elaborate on in the recommendations portion of this article.

Recommendations to Keep you Safe: Lessons Learned

Here are some practical suggestions to take into consideration now!

1. Institutional Responsibility: Develop a plan with the technicians at your college that consists of a protocol for regular back-up of your course management system. Send the plan (along with a budget pointing out the FTE generated by your program) that delineates the cost of the back up system, to your budget committee, to all of your instructional administrators, and anyone else you can think of. Then make sure the plan is implemented.

2. Faculty Responsibility: Develop a protocol for instructors, which includes a schedule for regular back up of faculty developed course content, grade book items, test banks, and anything else that may have been entered directly into the CMS. (Documents and web pages that were uploaded into a CMS are likely saved on someone's hard drive prior to up-load, but may also need to be mentioned in the faculty back-up protocol.) Train people how to accomplish content and grade book back up.

3. Senate and Administration Responsibility: It is important to develop and publish a lost work/grades policy that instructors can follow consistently and that the college will support.

4. Student Responsibility: In the syllabus and anywhere else where directions for how to be a distance education student may be found (orientation, class schedule, etc.,), describe how students will be expected to keep copies of their work and anything else that they upload into a CMS. (In my syllabus, I have directions for discussion board participation that requires students to write their posts into Word, then spell check and edit, then copy and paste into the discussion forum online. They are then advised to save all of their postings and assignments in a folder on their own computers as back-up.)

5. Use of College Sponsored Email: Make sure that you state in your syllabus that you will be using college email to contact them, and that in the event of an interruption of the CMS, you will be contacting them via that email system. Also develop a policy that is stated in the syllabus that covers lost time due to system interruption and loss of work due to any technology failure. Develop and include in the syllabus, two policies: one that covers technology failure that is at the institutional level and another for a student's technology problem.

The first two seem like obvious safety measures for anyone who uses technology; however, if your system has been stable for quite a while, you may not be paying enough attention to these issues. If you do nothing else, revisit backing up your own work! The student responsibility issue may be off your radar.

Your syllabus is the appropriate place to tell students about backing up their own work as one of their responsibilities as students in the course.

No matter what the reason, when dealing with the press, make sure you have a well thought out statement of what happened and be honest. I waited until I could be on a conference call with my vice President of Instruction and we talked to the press together. The resulting articles served to help our students stay informed and were really well done.

Backing up data is really everyone's responsibility from the institution to the instructors to the students. We made two especially good decisions in the midst of this situation: we established one point of contact for information to the teachers, which was the faculty distance education coordinator, and the vice President of Instruction became the point administrator who stayed in close communication with the coordinator and the technicians.

Communication was critical. The academic senate presidents from both campuses were closely involved in the planning meetings and implementation of strategies for recovery. The forums that we held gave us a lot of good information about the extent of the problems, and provided some great solutions. The forums and frequent email communication kept rumor out of the mix and kept everyone focused on solutions.

I have to say, that what seemed to be a horrible mess, turned out pretty well in the end. Our Vice President of Instruction was a great help throughout the process of recovery and we wouldn't have made it without his confidence in us and his willingness to provide us with whatever support we requested. It took the cooperation of faculty, staff, students and administrators to turn around what could have been a total disaster. We are all bound to have problems. It is the nature of technology. Reflection, planning and diligence will keep technology issues from becoming disasters. All told, there are only two kinds of technology users, after all.

For comments and questions contact Pat at pjames [at] msjc.edu

Please check out the information about the Online Teaching Conference coming in June at http://www.cccone.org!