BrianJarrett.com

Several months back I had a catastrophic system failure on my webserver. I was, at the time, running Mandrake Linux and I was using tar to archive all my file, then backing them up on my fileserver. I noticed one day that I was getting core dumps and segmentation faults all over the place and when I tried to reboot the system it died completely. I wouldn’t even boot.

My first thought was that it was a memory problem. I ran a program to test all the memory and it all checked out. I then thought maybe it was a motherboard problem and maybe it was somehow corrupting files. After moving the site to a totally separate box I noticed that the same thing occurred. This time, however, I received a message from cron telling me that there were segmentation faults during my archive job.

Turns out that I was exceeding the max filesize limit for tar. According to the documentation tar can only handle a file up to 2 gigs. My file was 6 gigs. tar handled it okay until the file grew to over 6 gigs, at that point things went very wrong and important system files were destroyed, resulting in the complete failure of the system.

I then restored the system and stopped the tar job from running. I now use a program called Beyond Compare to compare the contents of my webserver to my backup drive (on Windows) and copy over anything that’s changed. I lose file ownership and permissions but it’s not difficult to restore that anyway. (Of course the same thing could be done in an all Linux environment too; it’s really a matter of preference.) I copy all these files over a Samba share from the webserver to the fileserver.

The problem has now been resolved and I’ve had no repeat occurrances for five or six months now. This problem was particularly difficult to track down because the cause was not apparent and the symptoms at first pointed to a memory or motherboard problem, rather than a problem with tar.