Itâ€™s been a while since I checked the status of Linux-land regarding filesystems and CPU and I/O schedulers, so I thought Iâ€™d post some results.
A bunch of new distributions are coming out with Linux kernel 2.6.32 as standard (Ubuntu 10.04LTS one of them), and one distribution (PC Linux OS) will have Con Kolivasâ€™ BFS as default process scheduler.
Since Iâ€™m a scheduler and filesystem aficionado, I was intrigued by the simplicity of the BFS and wanted to see how it might affect I/O performance. I donâ€™t believe that thereâ€™s ever a single scheduling algorithm that fits all possible use cases, and, furthermore, I think the default Linux CFS scheduler is getting a bit unwieldy in its complexity (though the goal is to make it scale to very large numbers of cores, not all machines have that).
So I grabbed a spare machine that doesnâ€™t have a ton of horsepower on purpose, and tested using the same benchmark (NetAppâ€™s venerable postmark) on the same part of the disk, with Windows, Ubuntu 10.04LTS beta, and PCLOS 2010 beta2.
Hereâ€™s some of the averaged data (did multiple runs):
|time||IOPS||Creation||Reads||Appends||Deletes||Read MB/s||Write MB/s|
|Ubuntu 10.04b deadline||136||173||500||86||87||10184||4.03||8.51|
|PCLOS ext4 CFQ||105||239||1011||118||120||4478||5.25||11.09|
|PCLOS ext4 deadline||97||240||1158||119||121||7828||5.73||12.21|
|PCLOS XFS tuned deadline||187||136||476||68||68||509||2.93||6.19|
Some explanation on the fields:
- Time: Total time to run
- IOPS: mixed transactions per second (maybe the most useful number)
- Creation: files created per second (not mixed with transactions)
- Reads: reads per second
- Appends: appends per second
- Deletes: pure file deletions per second (not mixed with transactions)
- Read and write MB/s: the effective MB/s
- Windows was just vanilla XP with all service packs, tuned as a server (box was too weak to take Win7)
- Supercache was using a portion of memory for the Supercache filter driver
- Uptempo is a similar filter driver that provides caching (see previous posts here and here)
- Ubuntu was the latest available beta of 10.04
- Whenever you see â€œdeadlineâ€ the deadline scheduler was used instead of CFQ
- PCLOS is the latest available beta of PCLOS 2010.
- I mounted ext4 with barrier=0,noatime
- I mounted XFS with nobarrier,noatime,nodiratime,logbufs=8,logbsize=256k and made it with mkfs.xfs -f -d agcount=4 -l lazy-count=1 -l size=128m
Some pretty pictures for the ADD among us:
- Some operations that are metadata-heavy like deletion, can get heavily cached in Linux, so that skews the MB/s and total time results when 10,000 files can be deleted in an instant. I wanted to show the numbers because, depending on what you do, that may or may not be important.
- Intelligent caching still helps tremendously, note the results for Uptempo on Windows and the crazy file creation times on Linux (also skews the MB/s but is a useful number to know if youâ€™re creating a lot of files constantly)
- Depending on what youâ€™re doing, Windows can be Just Fineâ€¦
- The BFS seems to have been an excellent scheduler choice for PCLOS and something other Linux vendors should start looking at seriously â€“ unless there are serious other I/O tweaks in the kernel that I donâ€™t know about, the difference in performance is staggering between Ubuntu and PCLOS (Phoronix figured that out as well here â€“ they have tons of additional benchmarks showing things other than I/O).
- And, last but by no means least â€“ the deadline scheduler is ahead of CFQ again, both for Ubuntu and PCLOS. Not by a huge margin but itâ€™s a safe choice especially if youâ€™re running Linux on enterprise storage that has its own decent I/O scheduler built-in. There have been cases of CFQ dramatically lowering performance with external arrays in some cases. Remember â€“ most of the guys writing code for Linux donâ€™t have access to enterprise storageâ€¦ using the defaults could be harming your Linux performance!