I never signed up for this… or did I? I guess being in the IT world means working with things no one else wants to touch. Office printer stops working? Oh get Steve from accounting in there, he likes tech stuff. You get the picture. That’s been my life for the past 7 months or so, which is both good and bad. I did sign up for an IT oriented job, but taking up the role of “junior” systems administrator when I’m not busy with my primary job title gets exhausting.

First some context, because it’s always relevant. We were in the process of moving into an entirely new building – bigger, better, more advanced systems, the whole lot. Unfortunately I was still the “new guy” around town, so my opinion on how the site should be setup was never on management’s radar. Great, fine! I’m not being paid for my opinions anyway! Wrong; once all of the contract installers left, there was a lot to be desired. Naturally we didn’t (and don’t) have a full time sysadmin to manage the site’s network and IT infrastructure. I was approached one day while visiting the new site by the owner of the company to help figure out how we might finish the site setup. Me? I wasn’t going to say no to the guy who signs my check, and plus, this is the responsibility I’ve really wanted for a while. Oh to be young and innocent again…

Security. Always important, always ignored. The company had spent a significant amount of money on new security systems for the site – the smart entrance lock system alone was $1,400 PER DOOR. Not to mention the 4k IP cameras overlapping every surface inside and out of the building. Great, right? Can you guess what was running all of this expensive equipment? A single DL380 running Windows Server 2008 R2 on a 5 year old UPS (with a broken front screen). It’s not a bad server by any means, and with a 9TB RAID tucked inside it works nicely for what we need. But it begs the question as to why a corner need be cut here, the brains of the whole system. Not to mention the time it took to get this thing installed in our rack… I will never buy anything other than quick-install rails ever again.

Worse than just the system’s performance on paper, the actual problems I’ve faced keeping this thing running have been less than ideal. 4 months into 24/7 operation a primary fan failed in the middle of the night (on a damned weekend). Because of the inefficiency of the old fan system, this meant replacing the bad one immediately before it can be operated again. No boot for you! Fine, it was pulled and fixed/back online the next day. Not before I had a minor panic attack because it could no longer find the boot device on the RAID, of course. I still have no earthly idea why the system did this, but the raid controller set the originally configured boot partition to OFF when I booted it. WHY? Fine, it’s fixed now; but it’s also 8am on a Monday and I am already behind on work. I don’t have time for this mess. There was also a week period where our county experienced nightly brownouts for a few minutes at a time, but that shouldn’t be a problem since we have everything on a series of UPS’, right? Wrong again, they only have 2 managed ports total! WHY? Why would we use these things? Because of the sporadic power outages our entire security system kept getting killed at 2am every morning – so I went through with the effort of building another unused Windows server into a PRTG monitor. At least when I woke up every morning I could see what was still down instead of having to guess.

The latest happy-fun-time security management issue came in the form of “poor” quality from the cameras. “We paid a premium for high definition 4k cameras!” Well, you didn’t, because most of the cameras are standard 1080p. The one camera that was actually 4k? It’s watching the utility closet. This is why you absolutely need to consult with someone, ANYONE, in the industry about how your IT systems are designed and deployed before actually doing so. Even just someone who normally installs security systems could have been more helpful. Overall it hasn’t been the worst deployment of an IT system I’ve been a part of or read about, but all of the hassle could have been avoided. It is absolutely worth the money to avoid problems like this. Critical downtime I should add. Remember the power issues I described a moment ago? Yeah, we had no camera system when our building was being vandalized one time because of that. Oops!


Leave a Reply

Your email address will not be published. Required fields are marked *