« March 6, 2005 | Main | March 11, 2005 »
Tuesday, March 8, 2005
Linux: ext3 File Systems on Heavily Loaded Servers
Linux ext3 file systems use a "journal" to permit quick recovery in case of a system crash. Each pending change to the file system is written to a journal area from which changes can be applied if the system crashes before they're written to the actual file system structure. Under heavy load, it is possible for the journal area to overflow, which causes the journal daemon to crash, after which you get an incessant stream of journal failure messages on the console and all I/O to the file system fails. However, as long as the file system is not critical to system operation, the system appears to be running, so applications may not fail over to a backup server as you'd expect. I first encountered this when I made a 4 Gb tar file backup of a file system in /tmp and then tried to copy it to another file system exported with NFS to transfer to another machine. OK, that was really pushing things, I'll admit. However, about an hour ago, my production server, running on Fedora Core 3 kernel 2.6.10-1.766_FC3smp went blooie with this problem and had to be power cycled to recover. Neither an init 0 nor /sbin/reboot in an active console window sufficed to cleanly stop the system. The server was sufficiently responsive in this mode to fool the load balancer into thinking it was up, so the automatic fail-over to the backup server did not occur until I powered down the failed machine. This problem appears to be known, with a patch submitted to correct it, and this patch along with a couple of other related ones is in the 2.6.11-rc4-mm1 release candidate kernel which has not yet reached Fedora Core users. If you're running a heavily loaded mission critical server with ext3 file systems, you might consider falling back to ext2 until the fix for this problem arrives in kernel update for the distribution you're using. As this message is addressed solely to up-to-their-elbows system administrators, I'll not describe how to make that change nor why you may prefer to continue to take your chances with ext3.Posted at
21:51
