It’s been a while since I checked the status of Linux-land regarding filesystems and CPU and I/O schedulers, so I thought I’d post some results.
A bunch of new distributions are coming out with Linux kernel 2.6.32 as standard (Ubuntu 10.04LTS one of them), and one distribution (PC Linux OS) will have Con Kolivas’ BFS as default process scheduler.
Since I’m a scheduler and filesystem aficionado, I was intrigued by the simplicity of the BFS and wanted to see how it might affect I/O performance. I don’t believe that there’s ever a single scheduling algorithm that fits all possible use cases, and, furthermore, I think the default Linux CFS scheduler is getting a bit unwieldy in its complexity (though the goal is to make it scale to very large numbers of cores, not all machines have that).
So I grabbed a spare machine that doesn’t have a ton of horsepower on purpose, and tested using the same benchmark (NetApp’s venerable postmark) on the same part of the disk, with Windows, Ubuntu 10.04LTS beta, and PCLOS 2010 beta2.
Here’s some of the averaged data (did multiple runs):
|time||IOPS||Creation||Reads||Appends||Deletes||Read MB/s||Write MB/s|
|Ubuntu 10.04b deadline||136||173||500||86||87||10184||4.03||8.51|
|PCLOS ext4 CFQ||105||239||1011||118||120||4478||5.25||11.09|
|PCLOS ext4 deadline||97||240||1158||119||121||7828||5.73||12.21|
|PCLOS XFS tuned deadline||187||136||476||68||68||509||2.93||6.19|
Some explanation on the fields:
- Time: Total time to run
- IOPS: mixed transactions per second (maybe the most useful number)
- Creation: files created per second (not mixed with transactions)
- Reads: reads per second
- Appends: appends per second
- Deletes: pure file deletions per second (not mixed with transactions)
- Read and write MB/s: the effective MB/s
- Windows was just vanilla XP with all service packs, tuned as a server (box was too weak to take Win7)
- Supercache was using a portion of memory for the Supercache filter driver
- Uptempo is a similar filter driver that provides caching (see previous posts here and here)
- Ubuntu was the latest available beta of 10.04
- Whenever you see “deadline”, the deadline scheduler was used instead of CFQ
- PCLOS is the latest available beta of PCLOS 2010.
- I mounted ext4 with barrier=0,noatime
- I mounted XFS with nobarrier,noatime,nodiratime,logbufs=8,logbsize=256k and made it with mkfs.xfs -f -d agcount=4 -l lazy-count=1 -l size=128m
Some pretty pictures for the ADD among us:
- Some operations that are metadata-heavy like deletion, can get heavily cached in Linux, so that skews the MB/s and total time results when 10,000 files can be deleted in an instant. I wanted to show the numbers because, depending on what you do, that may or may not be important.
- Intelligent caching still helps tremendously, note the results for Uptempo on Windows and the crazy file creation times on Linux (also skews the MB/s but is a useful number to know if you’re creating a lot of files constantly)
- Depending on what you’re doing, Windows can be Just Fine
- The BFS seems to have been an excellent scheduler choice for PCLOS and something other Linux vendors should start looking at seriously â€“ unless there are serious other I/O tweaks in the kernel that I donâ€™t know about, the difference in performance is staggering between Ubuntu and PCLOS (Phoronix figured that out as well here – they have tons of additional benchmarks showing things other than I/O).
- And, last but by no means least, the deadline scheduler is ahead of CFQ again, both for Ubuntu and PCLOS. Not by a huge margin but it’s a safe choice especially if you’re running Linux on enterprise storage that has its own decent I/O scheduler built-in. There have been cases of CFQ dramatically lowering performance with external arrays in some cases. Remember – most of the guys writing code for Linux don’t have access to enterprise storage, using the defaults could be harming your Linux performance!