Linux filesystem benchmark extravaganza – including Deadline vs CFQ schedulers and ext4 instability

I have some spare time these days so I figured I’d finally test as many filesystems on Linux as I could…

The new ext4 is an option with modern kernels so I loaded Ubuntu 9.04 and tried postmark and bonnie++ on the same partition using various filesystems and switching between the CFQ and Deadline schedulers.

Switching schedulers permanently can be achieved by changing the boot options and appending, say, elevator=deadline, but you can also switch them on the fly by running the following:

echo deadline > /sys/block/sda/queue/scheduler

You can check what’s currently selected by simply typing

cat /sys/block/sda/queue/scheduler

You’ll get back something like:

noop anticipatory [deadline] cfq

The scheduler in brackets is the currently selected one.

Reader beware: Running postmark on ext4 locked up the system repeatedly during the transaction phase of the benchmark, using either my own compiled version and the one from the repository, so obviously there is some issue there and I cannot at this time recommend ext4no other filesystem caused lockups. I did run bonnie++ as well since that didn’t crash with ext4.

The objective of this exercise wasn’t to show which filesystem is fastest, but rather to illustrate that, depending on what you want to do, you may want to re-examine the choice of filesystem and scheduler with your application if you’re running Linux. BTW the current recommendation for Databases and fast intelligent external arrays, and ubuntu’s default in the server edition is the Deadline scheduler, and not CFQ. However, all other distrubutions at the moment use CFQ!

So, without further ado, some benchmarks… (I’m not including the entire postmark output since it would be far too large, I just kept the most important metrics, anyone that wants the entire results is more than welcome to send me an email and I’ll hook you up).

Postmark MB/s:

FilesystemRead MB/sWrite MB/sIOPS
Reiser CFQ4.8510.25227
Reiser Deadline5.3811.35246
XFS CFQ2.334.93109
XFS Deadline2.354.97105
XFS Tuned2.735.76120
JFS CFQ1.753.6978
JFS Deadline1.733.6576
Ext3 CFQ2.715.73115
Ext3 Deadline2.866.03122

 

MBPS

Postmark IOPS:

iops

Bonnie++ write speed:

FilesystemIOPSBlock writes KB/sRewrite KB/s
Reiser CFQ4283165718199
Reiser Deadline4623229018154
XFS CFQ4713990118557
XFS Deadline4833984019653
XFS Tuned5924060420746
JFS CFQ4333165118528
JFS Deadline4523910618755
Ext3 CFQ4033110817235
Ext3 Deadline3383180317885
Ext4 CFQ4513926518519
Ext4 Deadline4463925718221

bonnieMBPS

Bonnie++ IOPS:

bonnieiops

 

Observations:

The Deadline scheduler seems to be consistently better for anything that’s not ext-based! A lot of work has been done on the Linux kernel to optimize it for the ext2-3-4 filesystems, and that shows. However, depending on what you want to do, ext3 may not be the best option (I don’t know yet about ext4 for postmark-type loads but based on the bonnie++ results it’s solid).

Here’s a list of some considerations:

  • Will the filesystem host many many small files or a few large ones? Reiser still rules the “many small files” use case, by far. The rest are fairly close, and JFS seriously lags. For large files, XFS is great.
  • Do you care if the filesystem takes a long time to fsck? Ext3 still takes quite long, whereas something like XFS doesn’t. Ext4 should remedy this.
  • Do you care for something that’s still actively being maintained? In this case only ext3-4 and XFS are the options.
  • Do you want defrag tools? Choose wisely since few filesystems do (XFS and ext4).

My current overall recommendation is XFS since it’s mature and also very tunable. For reference, here’s how I got the better results for XFS (the results in the graphs for tuned XFS were with the deadline scheduler):

mkfs.xfs -f -d agcount=4 -l lazy-count=1 -l size=64m /dev/sda7

mount -o nobarrier,noatime,nodiratime,logbufs=8 /dev/sda7 /test

Don’t just follow the above blindly, normally mkfs tries to auto-adjust those (i.e. the agcount) but the important ones to look for are the log size and the mount options, especially the nobarrier and logbufs. Remember though that nobarrier is only recommended if you have battery backup.

D

3 Replies to “Linux filesystem benchmark extravaganza – including Deadline vs CFQ schedulers and ext4 instability”

  1. Great article! Thanks for taking the time to benchmark these systems.

    It’s interesting how strong Reiser still is technically but, I just don’t see it going anywhere. Since Hans went to jail for murder and Namesys vaporized Reiser has pretty much stopped development.

    I’m very disappointed in the OSS community for not taking up Reiser or at least incorporating some of its technology into the other file systems. They could all benefit but, it seems that the not invented here(NIH) mentality remains strong.

    In my opinion the worst offender is EXT-4. Yet another forked implementation that yet again lacks so many features already available in other file systems. Look at Reiser, look at ext3cow look at… You get he idea.

  2. Hi,

    just found your post…doing some postmark benmarking myself right now and wonder if you could share sure settings used in those tests?

    e.g.

    set size?
    set numbers?
    set transactions?

    or any other settings you used…

    I would like to run the same test on my box to compare results…

    thanks,
    Rene’

  3. I have just used a recovery tool called Stellar Phoenix Linux Recovery when i lost my ext2 file on my Linux system.I don’t have any idea with ext4 file system but thanks for this wonderful information…

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.