File Systems Benchmarks
Benchmark of various file systems
Benchmark command
I'm using bonnie++ for benchmarking.# create a file used as the test partition dd if=/dev/zero of=/dev/testdrv bs=1024k count=1024 mkfs.??? /testdrv mount -oloop -t??? /dev/testdrv /mnt/test # Test with 256M file, limiting memory to 128M bonnie++ -s256 -r128 -uroot -d/mnt/test # Updated - specify -n 1 to test the creation of 1024 files bonnie++ -d /tmp -r 512 -u root -s 1g -n 1 # This produce cleaner output. Output=write, Input=read bonnie++ -d /storage -r 512 -u root -s 2g -n0 -q -f -b
Show iostat
iostat -Nx
Testbed
Fedora core 7 x86_64 Kernel 2.6.23.1-21.fc7 (SMP)Intel C2D6850 / 4G ddr2 / WD2000JS-00MHB0 sata 7200rpm 8M
Results
EXT3Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP ism.comme.ca 256M 67760 99 105487 18 +++++ +++ 62100 99 +++++ +++ +++++ +++ ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
EXT4 - 4 LVM stripes over SAN storage
bonnie++ -d /OS -r 512 -u root -s 2g -n0 -q -f -b Version 1.03e ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP mongodb2.thumbpl 2G 87118 99 205443 57 1873668 99 3210 6 mongodb2.thumbplay.com,2G,,,87118,99,205443,57,,,1873668,99,3209.6,6,,,,,,,,,,,,,
JFS - supported by Fedora by default. Get jfsutils from http://jfs.sourceforge.net/
Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP ism.comme.ca 256M 71175 99 +++++ +++ +++++ +++ 104159 99 +++++ +++ +++++ +++ ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 +++++ +++ +++++ +++ +++++ +++ 10525 34 +++++ +++ +++++ +++
XFS - Get xfsprogs from ftp://oss.sgi.com/projects/xfs/cmd_tars or yum install xfsprogs
Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP ism.comme.ca 256M 94347 96 +++++ +++ +++++ +++ 71158 99 +++++ +++ +++++ +++ ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 13829 43 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
ReiserFS3 - yum install reiserfs-utils
Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP ism.comme.ca 256M 76964 96 30437 5 16506 1 103556 99 +++++ +++ +++++ +++ ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 +++++ +++ +++++ +++ +++++ +++ 32444 99 +++++ +++ +++++ +++
Conclusion
In this test, JFS & ReiserFS offer the best read performance, while XFS offers the best write performance. But that's just sequential read/write of a not-so-large file. I've had some good experience with Oracle on JFS and some bad ones with ReiserFS. ReiserFS4 and ext4 have promising performance but I'm too lazy to get my kernel to support them, and they may not be as matured as the others.One also have to take into consideration about how often random access is used. Certainly not to under-estimate the effect that ext3 is officially supported and is widely used for obvious reasons. Think about when your fs crashes, you certainly don't want to learn that the fs is not supported by your favorate live-cd/rescue-cd.
Another test (IOZone)
http://www.iozone.org/"The tests performed by IOzone measure many things, starting with the simpler metrics such as read, write, reread, and rewrite performance. Read and write are obvious. Reread is measures how well a system caches a file that was recently read. There are two rewrite tests; the one reported as simply rewrite overwrites an existing file. The record rewrite writes to a specific location in a file over and over again. " http://www.linux.com/archive/feature/139744
./iozone -M -l2 -u2 -r1m -s512m -i0 -i1 -i2 -F /storage/t1 /storage/t2 # 3 threads, 1m record size, 512M, read/write/random tests only.
Output
Command line used: iozone -l3 -u3 -r16k -s512m -i0 -i1 -i2 -F /opt/t1 /opt/t2 /opt/t3 Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. Min process = 3 Max process = 3 Throughput test with 3 processes Each process writes a 524288 Kbyte file in 16 Kbyte records Children see throughput for 3 initial writers = 190551.92 KB/sec Parent sees throughput for 3 initial writers = 95358.14 KB/sec Min throughput per process = 59399.04 KB/sec Max throughput per process = 67102.94 KB/sec Avg throughput per process = 63517.31 KB/sec Min xfer = 464848.00 KB Children see throughput for 3 rewriters = 192879.43 KB/sec Parent sees throughput for 3 rewriters = 97202.22 KB/sec Min throughput per process = 61424.02 KB/sec Max throughput per process = 67892.16 KB/sec Avg throughput per process = 64293.14 KB/sec Min xfer = 475440.00 KB Children see throughput for 3 readers = 5169973.62 KB/sec Parent sees throughput for 3 readers = 5043432.88 KB/sec Min throughput per process = 1274313.12 KB/sec Max throughput per process = 2559450.75 KB/sec Avg throughput per process = 1723324.54 KB/sec Min xfer = 254640.00 KB Children see throughput for 3 re-readers = 5141331.25 KB/sec Parent sees throughput for 3 re-readers = 5082621.95 KB/sec Min throughput per process = 1280551.88 KB/sec Max throughput per process = 2565787.75 KB/sec Avg throughput per process = 1713777.08 KB/sec Min xfer = 259664.00 KB Children see throughput for 3 random readers = 4574113.12 KB/sec Parent sees throughput for 3 random readers = 4539378.00 KB/sec Min throughput per process = 1218295.25 KB/sec Max throughput per process = 2109693.00 KB/sec Avg throughput per process = 1524704.38 KB/sec Min xfer = 303872.00 KB Children see throughput for 3 random writers = 49672.07 KB/sec Parent sees throughput for 3 random writers = 30601.92 KB/sec Min throughput per process = 15888.48 KB/sec Max throughput per process = 17488.86 KB/sec Avg throughput per process = 16557.36 KB/sec Min xfer = 476272.00 KB
sysbench
sysbench --test=fileio --num-threads=4 --file-num=4 --file-total-size=512M --file-test-mode=rndrw prepare sysbench --test=fileio --num-threads=4 --file-num=4 --file-total-size=512M --file-test-mode=rndrw run sysbench --test=fileio --num-threads=4 --file-num=4 --file-total-size=512M --file-test-mode=rndrw cleanup
Result on Seagate hybrid:
Operations performed: 6001 Read, 4004 Write, 396 Other = 10401 Total Read 93.766Mb Written 62.562Mb Total transferred 156.33Mb (3.8785Mb/sec) 248.22 Requests/sec executed Test execution summary: total time: 40.3067s total number of events: 10005 total time taken by event execution: 6.2396 per-request statistics: min: 0.01ms avg: 0.62ms max: 350.00ms approx. 95 percentile: 0.11ms Threads fairness: events (avg/stddev): 2501.2500/34.74 execution time (avg/stddev): 1.5599/0.18
Compared with a LinuxVM sitting on san
Operations performed: 6002 Read, 3999 Write, 401 Other = 10402 Total Read 93.781Mb Written 62.484Mb Total transferred 156.27Mb (89.85Mb/sec) 5750.43 Requests/sec executed Test execution summary: total time: 1.7392s total number of events: 10001 total time taken by event execution: 6.1917 per-request statistics: min: 0.00ms avg: 0.62ms max: 369.01ms approx. 95 percentile: 1.55ms Threads fairness: events (avg/stddev): 2500.2500/370.49 execution time (avg/stddev): 1.5479/0.12
