View Full Version : Ahh crap.

06-19-2010, 06:37 PM
Quick background: I'm a sysadmin for a large web hosting company. We have 2 data centers, one in the main office, one in an old word processing company's facility. I work by myself most shifts in the latter, as a caretaker for about 500 servers. We're in the middle of a LOT of upgrades, software and hardware. </background>

We had a meeting the other day for all lower level admins and level 3 techs with the CEO. There's a WTF involved with the CEO that I'll have to leave for another day, as I'm still somewhat miffed about the issue, and can't write well when miffed. To allow the day shift junior admin to attend the meeting, a senior admin I don't particularly like and have griped about before (the one who likes to make a public example of everyone who does anything remotely wrong) took his place.

After the meeting, I met up with the senior admin here at work. Right as I get in, the server the CEO's website is on crashes, and doesn't come back up after a reboot.

Senior admin takes a look, determines that /var is totally kaput - an fsck starts throwing all files in the partition into the lost+found directory. This means the partition is just gone, with no hope of saving it.

His answer to this problem is "well...have fun with this backup restore. I'm going home." >_<

I spent 9 hours that day JUST working with this box. The issues lasted for another 12 hours, which the grave shift got to finish handling.

Another box just up and died in the same manner. I'm in the process of trying to repair the damage now. Figures it goes tits up right as I'm walking out the door for a caffeine run. :-(

The cause for this? We just migrated /var**** and a lot of other partitions over to ext3* from reiserfs** (affectionately called murderfs in some circles) In the migration, the fstab*** was not updated to show the correct options that should be there to reduce filesystem damage over time. This means we have about 800 ticking time bombs between all data centers of boxes that may or may not come back from a reboot.

Geek to english key:
*: ext3 is the default filesystem in linux. it's a great filesystem, and handles files well. It's older, and is showing it's age. There are newer and better out there, but ext3 is most stable and reliable, and therefore default.
**: reiser was developed by Hans Reiser, recently convicted of murdering his wife, thus murderfs. It has a lot of neat features, and is best suited for small filesystems, but in practice tends to go timebomb in a large environment and just eat all your data. For this reason, we're getting off reiser.
***: /etc/fstab is the file in linux where all filesystems are recorded. you can mount a volume without it being in fstab, but for filesystems that must be mounted at boot, like in a server environment, this is where everything goes.
****: /var is a system partition, and is where log files, temporary files, and a few other generally dynamic items go. As a side note, mysql databases go to /var/lib/mysql. This is its own drive in most cases, fortunately, and would be unaffected by /var going kablooie.

Der Cute
06-19-2010, 09:53 PM
So, for some reason, people didn't plan ahead and think: hmm. this file size MIGHT get bigger than X gb, which will then result in totally hosed files.
Even though .ext3 works great with this file now, the file will get bigger and need different handling....meaning we should NOW migrate to something else....

I hate it when people cut nose off like that.


06-20-2010, 05:35 PM
I hate it when people cut nose off like that.


actually, our problem was with not doing writeback caching and stuff that reiser does by default, but ext3 doesn't. ext3 is fine for our homedirs, but our massive backup boxes, which I'll have to take a photo of later, use xfs. The backup boxen are 5 (with an optional 6th) 36 port 4u chassis hooked up via eSAS, loaded with 2TB drives. That's right. Before the RAID6, that's 360TB in one box, with the option of another 72TB, for 432TB of storage. We split the monster stack of drives into 20TB segments, which allow us to back up up to 5 home directories (daily and weekly backups of 2TB home directories.)

Re the FS issues, that's what ZFS in Solaris, and btrfs in linux are for. They have some great new features like dynamic volume growing, fault tolerance, and max volume and file size in the EiB range - specifically 16 EiB for both volume and file size.