{{parent page="LinuxFileSystem"}}

===File Systems Benchmarks===

~===[[FileSystemsBenchMarkSDD]]===
===Benchmark of various file systems===

==Benchmark command==
I'm using [[http://www.coker.com.au/bonnie++ bonnie++]] for benchmarking.

%%
# create a file used as the test partition
dd if=/dev/zero of=/dev/testdrv bs=1024k count=1024
mkfs.??? /testdrv
mount -oloop -t??? /dev/testdrv /mnt/test

# Test with 256M file, limiting memory to 128M
bonnie++ -s256 -r128 -uroot -d/mnt/test

# Updated - specify -n 1 to test the creation of 1024 files
bonnie++ -d /tmp -r 512 -u root -s 1g -n 1

# This produce cleaner output. Output=write, Input=read
bonnie++ -d /storage -r 512 -u root -s 2g -n0 -q -f -b
%%

Show iostat
%%
iostat -Nx
%%

==Testbed==
Fedora core 7 x86_64 Kernel 2.6.23.1-21.fc7 (SMP)
Intel C2D6850 / 4G ddr2 / WD2000JS-00MHB0 sata 7200rpm 8M

==Results==
EXT3
%%
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
ism.comme.ca 256M 67760 99 105487 18 +++++ +++ 62100 99 +++++ +++ +++++ +++
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
%%

EXT4 - 4 LVM stripes over SAN storage
%%
bonnie++ -d /OS -r 512 -u root -s 2g -n0 -q -f -b
Version 1.03e ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
mongodb2.thumbpl 2G 87118 99 205443 57 1873668 99 3210 6
mongodb2.thumbplay.com,2G,,,87118,99,205443,57,,,1873668,99,3209.6,6,,,,,,,,,,,,,
%%

JFS - supported by Fedora by default. Get jfsutils from http://jfs.sourceforge.net/
%%
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
ism.comme.ca 256M 71175 99 +++++ +++ +++++ +++ 104159 99 +++++ +++ +++++ +++
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ 10525 34 +++++ +++ +++++ +++
%%

XFS - Get xfsprogs from ftp://oss.sgi.com/projects/xfs/cmd_tars or yum install xfsprogs
%%
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
ism.comme.ca 256M 94347 96 +++++ +++ +++++ +++ 71158 99 +++++ +++ +++++ +++
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 13829 43 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
%%

ReiserFS3 - yum install reiserfs-utils
%%
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
ism.comme.ca 256M 76964 96 30437 5 16506 1 103556 99 +++++ +++ +++++ +++
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ 32444 99 +++++ +++ +++++ +++
%%

==Conclusion==
**In this test**, JFS & ReiserFS offer the best read performance, while XFS offers the best write performance. But that's just sequential read/write of a not-so-large file. I've had some good experience with Oracle on JFS and some bad ones with ReiserFS. ReiserFS4 and ext4 have promising performance but I'm too lazy to get my kernel to support them, and they may not be as matured as the others.

One also have to take into consideration about how often random access is used. Certainly not to under-estimate the effect that ext3 is officially supported and is widely used for obvious reasons. Think about when your fs crashes, you certainly don't want to learn that the fs is not supported by your favorate live-cd/rescue-cd.

===Another test (IOZone)===
http://www.iozone.org/
"The tests performed by IOzone measure many things, starting with the simpler metrics such as read, write, reread, and rewrite performance. Read and write are obvious. Reread is measures how well a system caches a file that was recently read. There are two rewrite tests; the one reported as simply rewrite overwrites an existing file. The record rewrite writes to a specific location in a file over and over again. " http://www.linux.com/archive/feature/139744

%%
./iozone -M -l2 -u2 -r1m -s512m -i0 -i1 -i2 -F /storage/t1 /storage/t2
# 3 threads, 1m record size, 512M, read/write/random tests only.
%%

Output
%%
Command line used: iozone -l3 -u3 -r16k -s512m -i0 -i1 -i2 -F /opt/t1 /opt/t2 /opt/t3
Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Min process = 3
Max process = 3
Throughput test with 3 processes
Each process writes a 524288 Kbyte file in 16 Kbyte records

Children see throughput for 3 initial writers = 190551.92 KB/sec
Parent sees throughput for 3 initial writers = 95358.14 KB/sec
Min throughput per process = 59399.04 KB/sec
Max throughput per process = 67102.94 KB/sec
Avg throughput per process = 63517.31 KB/sec
Min xfer = 464848.00 KB

Children see throughput for 3 rewriters = 192879.43 KB/sec
Parent sees throughput for 3 rewriters = 97202.22 KB/sec
Min throughput per process = 61424.02 KB/sec
Max throughput per process = 67892.16 KB/sec
Avg throughput per process = 64293.14 KB/sec
Min xfer = 475440.00 KB

Children see throughput for 3 readers = 5169973.62 KB/sec
Parent sees throughput for 3 readers = 5043432.88 KB/sec
Min throughput per process = 1274313.12 KB/sec
Max throughput per process = 2559450.75 KB/sec
Avg throughput per process = 1723324.54 KB/sec
Min xfer = 254640.00 KB

Children see throughput for 3 re-readers = 5141331.25 KB/sec
Parent sees throughput for 3 re-readers = 5082621.95 KB/sec
Min throughput per process = 1280551.88 KB/sec
Max throughput per process = 2565787.75 KB/sec
Avg throughput per process = 1713777.08 KB/sec
Min xfer = 259664.00 KB

Children see throughput for 3 random readers = 4574113.12 KB/sec
Parent sees throughput for 3 random readers = 4539378.00 KB/sec
Min throughput per process = 1218295.25 KB/sec
Max throughput per process = 2109693.00 KB/sec
Avg throughput per process = 1524704.38 KB/sec
Min xfer = 303872.00 KB

Children see throughput for 3 random writers = 49672.07 KB/sec
Parent sees throughput for 3 random writers = 30601.92 KB/sec
Min throughput per process = 15888.48 KB/sec
Max throughput per process = 17488.86 KB/sec
Avg throughput per process = 16557.36 KB/sec
Min xfer = 476272.00 KB

%%


==sysbench==

%%
sysbench --test=fileio --num-threads=4 --file-num=4 --file-total-size=512M --file-test-mode=rndrw prepare
sysbench --test=fileio --num-threads=4 --file-num=4 --file-total-size=512M --file-test-mode=rndrw run
sysbench --test=fileio --num-threads=4 --file-num=4 --file-total-size=512M --file-test-mode=rndrw cleanup
%%

Result on Seagate hybrid:
%%
Operations performed: 6001 Read, 4004 Write, 396 Other = 10401 Total
Read 93.766Mb Written 62.562Mb Total transferred 156.33Mb (3.8785Mb/sec)
248.22 Requests/sec executed

Test execution summary:
total time: 40.3067s
total number of events: 10005
total time taken by event execution: 6.2396
per-request statistics:
min: 0.01ms
avg: 0.62ms
max: 350.00ms
approx. 95 percentile: 0.11ms

Threads fairness:
events (avg/stddev): 2501.2500/34.74
execution time (avg/stddev): 1.5599/0.18

%%

Compared with a LinuxVM sitting on san
%%
Operations performed: 6002 Read, 3999 Write, 401 Other = 10402 Total
Read 93.781Mb Written 62.484Mb Total transferred 156.27Mb (89.85Mb/sec)
5750.43 Requests/sec executed

Test execution summary:
total time: 1.7392s
total number of events: 10001
total time taken by event execution: 6.1917
per-request statistics:
min: 0.00ms
avg: 0.62ms
max: 369.01ms
approx. 95 percentile: 1.55ms

Threads fairness:
events (avg/stddev): 2500.2500/370.49
execution time (avg/stddev): 1.5479/0.12
%%