Tweet
I feel for our poor IT staff here. They have not been having a good week. Of course, none of the rest of us have been either.
We've been experiencing a number of network performance and uptime issues over the last few days. Servers going down repeatedly, access to any number of apps dying (like email), even our phones (VoIP phones, of course) have gone out a fair bit. Our customers haven't been able to get to our client site. Nobody's been able to get much of anything done (kind of hard to check in code changes when source control is offline). We couldn't even surf the web because all outside connections were down at one point.
Last night, one of the corner office management team comes around and tells everyone to go home. They're going to shut everything but power down in about 15 minutes, etc.... Ok, fine with me. I'm outta here.
I came in this morning and happened to catch one of the IT guys as he was coming inside (might have been outside slugging a bottle of JD, at this point. I'd forgive him.) Turns out there were several failures in the Storage Area Network. Between a drive controller that was eating itself and some bad configurations, all hell broke loose.
They had managed to get most of the critical functionality back online by this morning, but it was still touchy, and we were running rather tight on space. Our office has almost everything virtualized (doesn't matter whether I agree with it or not. That's the way it is.) When the SAN went down, we from 40 TERABYTES of storage down to about 10 overnight.
I'm not in IT, and I want to go home for the week.
We've been experiencing a number of network performance and uptime issues over the last few days. Servers going down repeatedly, access to any number of apps dying (like email), even our phones (VoIP phones, of course) have gone out a fair bit. Our customers haven't been able to get to our client site. Nobody's been able to get much of anything done (kind of hard to check in code changes when source control is offline). We couldn't even surf the web because all outside connections were down at one point.
Last night, one of the corner office management team comes around and tells everyone to go home. They're going to shut everything but power down in about 15 minutes, etc.... Ok, fine with me. I'm outta here.
I came in this morning and happened to catch one of the IT guys as he was coming inside (might have been outside slugging a bottle of JD, at this point. I'd forgive him.) Turns out there were several failures in the Storage Area Network. Between a drive controller that was eating itself and some bad configurations, all hell broke loose.
They had managed to get most of the critical functionality back online by this morning, but it was still touchy, and we were running rather tight on space. Our office has almost everything virtualized (doesn't matter whether I agree with it or not. That's the way it is.) When the SAN went down, we from 40 TERABYTES of storage down to about 10 overnight.
I'm not in IT, and I want to go home for the week.
Comment