First Day Back

First day back after a break is always hard. By the last day of vacation, I'm actually sleeping a little later and shifting the body clock back is rather harsh.

It's tough enough getting started again, but it's even worse when you're thrown a curve ball.

I got in at my usual 7:00, made my coffee, and started getting my lessons ready for the day. At about 7:30 we lost power in half of the room. Unfortunately, it was the half with the CS servers. The machines that provide log in and file services as well as svn, our web server, mail server, wiki and other services.

Now, on top of my teaching duties (four classes of 32 students and another of about 20), I basically run the computer services for our CS program. I used to do the whole network, but I stopped that a few years ago. I receive some help from colleagues, but it's still mostly me.

For our CS program we have two Linux labs of about 31 computers each (all running Linux) and a bunch of servers running the same.

After spending about 20 minutes to find the circuit breaker, most services came back up. Terror struck 10 minutes later when we saw that all of the students home directories had disappeared!!!!!

After a brief period of panic and a few stressful minutes of scouring file systems for evidence of the missing directories I found the problem. The answer, as with so many other problems, is that I was being an idiot. About a month earlier, I had installed a new drive for the student directories. I had mounted it, but neglected to change the fstab file. This is what happens when you have to do all your sysadmin work on live systems in brief periods of time between classes. When we lost power today, the old drive was mounted, not the new one. It's fixed now.

It would be nice if there we actually had people to run our systems, but since that's not going to happen, I've spend time over the years to try to make the whole thing somewhat do able. Using common tools such as NFS, NIS, Apache and the like help, and we've currently moved to using kvm for virtual machines so we can easily experiment with new systems and cleanly and have a one "machine" per service mentality, but system administration still can't really be done as a part time job.

Any one out there have any war stories or suggestions as part time sysadmins?