When things happen, they happen at once

I don't know why this is, but there's probably some probability model that explains it. We're humming along for months without problems. Anyway, when things happen, then tend to happen all at once. So, here's the events that happened Friday June 22nd... it always seems to happen on Friday!

Anyway, here's what happened. I awake to paging on my cell phone. Not an unusual occurance as we have lots of equipment and several pieces like to page. So, I picked up the phone and glanced. Well, for some reason, the mail server was acting up. So, I walked over to my computer and VPNed to work. It was ping failing, so I decided to power cycle it. I connected to the power cycler and forced a power off (not the best thing to do, but it's usually a last resort anyway). That didn't work. That meant I'd have to drive up to work. Corporate mail was down (as well as several other mailboxes).

So, I call the office to see who's there. Long story short, they couldn't get it back up either. But, he suspected the main boot drive has having partially crashed. So, I stop by Best Buy and buy a new hard drive. We clone and replace it (which, in and of itself is kind of a pain and took excessively long to do because of Windows). I digress. After the boot drive was cloned and replaced, we tried booting it again. Once again, there was at least one other drive (in an array) that also had gone bad.

So, we hop in my boss's car and go back to Best Buy to get yet another drive. Good thing I didn't try to drive my car there, some idiot decided to park his semi right in front of a large segment of cars at work blocking us ALL in. We replace that one and start the rebuild. Then, yet another drive was bad. Back to Best Buy, but this time I bought 3 more drives (to replace the last 3 just in case). No more going to Best Buy that day.

So, right about the time we got the second drive in the unit and it was successfully rebuilding, we had lunch. I turned on the projector (to give a small demo) and the bulb exploded (literally)! Hot pieces of glass came careening out of the unit and burned the very wood conference table (smoked and everything). So, not only did I have to clean up glass shards from the projector bulb, I had to go grab the other projector and move it into the main conference room until I could get a bulb replacement.

About the time I got done with lunch and sat down at my desk, I decided to call Dell about the projector bulb. Dialed the number and heard 'fast busy'. Hmmm.. I dialed another number. Fast busy... Every number I dialed (except our local circuit numbers) resulted in fast busy. I rebooted the phone system. That didn't work. But, I knew that our PRI connectivity to the AT&T switch was up and functioning. So, I called AT&T to report trouble. What a nightmare that was. I must have called about 5 different 8xx numbers before I found one that would actually help me. The first number, the one on the bill for service, couldn't help me. They transferred me into a system that said, "Sorry, we don't handle your account, call this 800 number". So, I call that number. This time I hear, "Welcome to AT&T's residential customer service center". We're not a residence! Click! I hang up. So, I dial the number listed on the demark (jack on the wall) and they also tell me they can't help me, but they give me yet another number. I call it. Finally, I get someone who can help.

After the AT&T telephone nightmare, they tell me there was an outage going on in San Jose (not that close to where I work, but close enough). Great. So, we have no inbound or outbound phone service either. No email, no phone. Thankfully, I had just implemented Live Help, but it hadn't yet been announced. It was about 3:30PM by the time I found out about the phones, but this had apparently been happening since about 2PM. The AT&T guy gives me a trouble ticket number and I hang up. About 2 minutes after I hang up, I receive an automated call on my cell phone (from AT&T) that tells me the exact same thing that the rep just told me. Then, I didn't receive any other calls from anyone else regarding that ticket. At about 5:30, I noticed the phone service was back up. Still no status updates from AT&T.

Of course, my mind wasn't so much concerned with the phone system as trying to repair the mail server. So, while ignoring the fact that AT&T hadn't call me to give status updates, I was still trying to make sure the email server would come back up properly. Well, it did and didn't. NTFS is still giving me chkdsk problems on the volume. If we try to repair it, it could take hours and totally corrupt the volume. So, we're trying to get what data off we can and then format and restore. This is about the fastest way to deal with a volume this corrupted. Chkdsk, under these conditions, could take hours (and I mean like 12-24 hours) to complete. We can't be offline that long. So, we skip the disk check operation for now and hope we can get the data off to a new volume where we can bring the mail server back up. Then, I can format that old volume in a proper way. Then, move it all back.

So, after we get the system back up around 5:20 PM (serving email), I sit down to call Dell. I realize the phone system may still be down, but I try anyway. Sure enough, the call goes through. Yay. I get some guy in India who proceeds to give me the run around about what they can do about the projector (nothing apparently). A projector bulb explodes, burns a table and they do NOTHING other than tell me to buy a new bulb. A $315 bulb! I did buy the bulb, though. We need the projector. But, what about the burn on the table? The old bulb wasn't that used. Anyway... that's the ramblings from hell Friday.