So, we had a contractor in today to upgrade the firmware in our UPS. Sounds simple, right? It’s a gigantic Emerson thing; just put it in bypass and have at it.
Anyway, the guy forgot to put it in bypass. So, at 3pm, the whole datacenter went dark.
I am not making this up.
Everything seems back to normal, now. I shudder to think how many man-hours have gone into restoring affected servers, given that it took down everything on campus. 😦