Linux filesystem benchmark extravaganza – including Deadline vs CFQ schedulers and ext4 instability

I have some spare time these days so I figured I’d finally test as many filesystems on Linux as I could…

The new ext4 is an option with modern kernels so I loaded Ubuntu 9.04 and tried postmark and bonnie++ on the same partition using various filesystems and switching between the CFQ and Deadline schedulers.

Switching schedulers permanently can be achieved by changing the boot options and appending, say, elevator=deadline, but you can also switch them on the fly by running the following:

echo deadline > /sys/block/sda/queue/scheduler

You can check what’s currently selected by simply typing

cat /sys/block/sda/queue/scheduler

You’ll get back something like:

noop anticipatory [deadline] cfq

The scheduler in brackets is the currently selected one.

Reader beware: Running postmark on ext4 locked up the system repeatedly during the transaction phase of the benchmark, using either my own compiled version and the one from the repository, so obviously there is some issue there and I cannot at this time recommend ext4 – no other filesystem caused lockups. I did run bonnie++ as well since that didn’t crash with ext4.

The objective of this exercise wasn’t to show which filesystem is fastest, but rather to illustrate that, depending on what you want to do, you may want to re-examine the choice of filesystem and scheduler with your application if you’re running Linux. BTW the current recommendation for Databases and fast intelligent external arrays – and ubuntu’s default in the server edition – is the Deadline scheduler, and not CFQ. However, all other distrubutions at the moment use CFQ!

So, without further ado, some benchmarks… (I’m not including the entire postmark output since it would be far too large, I just kept the most important metrics, anyone that wants the entire results is more than welcome to send me an email and I’ll hook you up).

Postmark MB/s:

Filesystem

Read MB/s

Write MB/s

IOPS

Reiser CFQ

4.85

10.25

227

Reiser Deadline

5.38

11.35

246

XFS CFQ

2.33

4.93

109

XFS Deadline

2.35

4.97

105

XFS Tuned

2.73

5.76

120

JFS CFQ

1.75

3.69

78

JFS Deadline

1.73

3.65

76

Ext3 CFQ

2.71

5.73

115

Ext3 Deadline

2.86

6.03

122

 

MBPS

Postmark IOPS:

iops

Bonnie++ write speed:

Filesystem

IOPS

Block writes KB/s

Rewrite KB/s

Reiser CFQ

428

31657

18199

Reiser Deadline

462

32290

18154

XFS CFQ

471

39901

18557

XFS Deadline

483

39840

19653

XFS Tuned

592

40604

20746

JFS CFQ

433

31651

18528

JFS Deadline

452

39106

18755

Ext3 CFQ

403

31108

17235

Ext3 Deadline

338

31803

17885

Ext4 CFQ

451

39265

18519

Ext4 Deadline

446

39257

18221

bonnieMBPS

Bonnie++ IOPS:

bonnieiops

Observations:

The Deadline scheduler seems to be consistently better for anything that’s not ext-based! A lot of work has been done on the Linux kernel to optimize it for the ext2-3-4 filesystems, and that shows. However, depending on what you want to do, ext3 may not be the best option (I don’t know yet about ext4 for postmark-type loads but based on the bonnie++ results it’s solid).

Here’s a list of some considerations:

  • Will the filesystem host many many small files or a few large ones? Reiser still rules the “many small files” use case, by far. The rest are fairly close, and JFS seriously lags. For large files, XFS is great.
  • Do you care if the filesystem takes a long time to fsck? Ext3 still takes quite long, whereas something like XFS doesn’t. Ext4 should remedy this.
  • Do you care for something that’s still actively being maintained? In this case only ext3-4 and XFS are the options.
  • Do you want defrag tools? Choose wisely since few filesystems do (XFS and ext4).

My current overall recommendation is XFS since it’s mature and also very tunable. For reference, here’s how I got the better results for XFS (the results in the graphs for tuned XFS were with the deadline scheduler):

mkfs.xfs -f -d agcount=4 -l lazy-count=1 -l size=64m /dev/sda7

mount -o nobarrier,noatime,nodiratime,logbufs=8 /dev/sda7 /test

Don’t just follow the above blindly, normally mkfs tries to auto-adjust those (i.e. the agcount) but the important ones to look for are the log size and the mount options, especially the nobarrier and logbufs. Remember though that nobarrier is only recommended if you have battery backup.

D

3 Replies to “Linux filesystem benchmark extravaganza – including Deadline vs CFQ schedulers and ext4 instability”

  1. Great article! Thanks for taking the time to benchmark these systems.

    It’s interesting how strong Reiser still is technically but, I just don’t see it going anywhere. Since Hans went to jail for murder and Namesys vaporized Reiser has pretty much stopped development.

    I’m very disappointed in the OSS community for not taking up Reiser or at least incorporating some of its technology into the other file systems. They could all benefit but, it seems that the not invented here(NIH) mentality remains strong.

    In my opinion the worst offender is EXT-4. Yet another forked implementation that yet again lacks so many features already available in other file systems. Look at Reiser, look at ext3cow look at… You get he idea.

  2. Hi,

    just found your post…doing some postmark benmarking myself right now and wonder if you could share sure settings used in those tests?

    e.g.

    set size?
    set numbers?
    set transactions?

    or any other settings you used…

    I would like to run the same test on my box to compare results…

    thanks,
    Rene’

  3. I have just used a recovery tool called Stellar Phoenix Linux Recovery when i lost my ext2 file on my Linux system.I don’t have any idea with ext4 file system but thanks for this wonderful information…

Leave a comment for posterity...