Test Disk Write Speed Without the Page Cache Using dd

A dd write test reports speeds far above what the disk can do, because the data only reached the page cache.

dd if=/dev/zero of=plain.bin bs=1M count=256 2>&1 | tail -n 1
dd if=/dev/zero of=direct.bin bs=1M count=256 oflag=direct 2>&1 | tail -n 1
dd if=/dev/zero of=sync.bin bs=1M count=256 conv=fdatasync 2>&1 | tail -n 1
dd if=/dev/zero of=odd.bin bs=1000 count=1000 oflag=direct 2>&1 | head -n 1

268435456 bytes (268 MB, 256 MiB) copied, 0.179601 s, 1.5 GB/s
268435456 bytes (268 MB, 256 MiB) copied, 0.0956704 s, 2.8 GB/s
268435456 bytes (268 MB, 256 MiB) copied, 0.686185 s, 391 MB/s
dd: error writing 'odd.bin': Invalid argument

A plain dd write finishes when the kernel has accepted the data, and the figure it prints mostly measures memory. oflag=direct opens the file with O_DIRECT, so each block goes to the device. conv=fdatasync keeps the cache but flushes at the end and counts the flush in the time. The run shows 1.5 GB/s plain, 2.8 GB/s direct and 391 MB/s with the flush; on this virtual disk direct writes were not the slowest, so these numbers describe the VM and not a real disk. See dd invocation.

Direct IO requires the block size to be a multiple of the device's sector size, and bs=1000 failed with Invalid argument. Some file systems, tmpfs among them, refuse O_DIRECT. Results vary from run to run, so repeat the test and use fio for anything you will make decisions on.