Skip to main content

FIO Commands for Storage Performance Tests on Linux

This document describes sample FIO commands you can use to run performance tests on Linux-based systems.

Installing FIO​

Install FIO using your distribution's package manager. For example:

# Debian/Ubuntu
sudo apt-get update && sudo apt-get install fio -y
# RHEL/CentOS/Oracle Linux
sudo dnf install fio -y

FIO Commands​

IOPS Performance Tests​

Use the following FIO commands to test IOPS performance. You can run the commands directly or create a job file and run that instead. In some cases, job files produce more consistent results.

Caution: Do not run FIO tests with a write workload (readwrite, randrw, write, trimwrite) directly against a device that is in use.

Test Random Reads​

Run the following command directly:

sudo fio --filename=<device name> --direct=1 --rw=randread --bs=4k \
  --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 \
  --time_based --group_reporting --name=iops-test-job \
  --eta-newline=1 --readonly

Alternatively, create a job file fiorandomread.fio with the following content:

[global]
bs=4K
iodepth=256
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=4
name=raw-randread
rw=randread

[job1]
filename=<device name>

Then run it with:

fio fiorandomread.fio

Test Random Read/Writes​

Run the following command directly:

sudo fio --filename=<device name> --direct=1 --rw=randrw --bs=4k \
  --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 \
  --time_based --group_reporting --name=iops-test-job --eta-newline=1

Add both the read IOPS and the write IOPS returned.

Alternatively, create a job file fiorandomreadwrite.fio with the following content:

[global]
bs=4K
iodepth=256
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=4
name=raw-randreadwrite
rw=randrw

[job1]
filename=<device name>

Then run it with:

fio fiorandomreadwrite.fio

Test Sequential Reads​

Run the following command directly:

sudo fio --filename=<device name> --direct=1 --rw=read --bs=4k \
  --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 \
  --time_based --group_reporting --name=iops-test-job \
  --eta-newline=1 --readonly

Alternatively, create a job file fioread.fio with the following content:

[global]
bs=4K
iodepth=256
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=4
name=raw-read
rw=read

[job1]
filename=<device name>

Then run it with:

fio fioread.fio

Throughput Performance Tests​

Use the following FIO commands to test throughput performance.

Caution: Do not run FIO tests with a write workload (readwrite, randrw, write, trimwrite) directly against a device that is in use.

Test Random Reads​

Run the following command directly:

sudo fio --filename=<device name> --direct=1 --rw=randread --bs=256k \
  --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 \
  --time_based --group_reporting --name=throughput-test-job \
  --eta-newline=1 --readonly

Alternatively, create a job file fiorandomread.fio with the following content:

[global]
bs=256K
iodepth=64
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=4
name=raw-randread
rw=randread

[job1]
filename=<device name>

Then run it with:

fio fiorandomread.fio

Test Random Read/Writes​

Run the following command directly:

sudo fio --filename=<device name> --direct=1 --rw=randrw --bs=256k \
  --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 \
  --time_based --group_reporting --name=throughput-test-job \
  --eta-newline=1

Add both the read MB/s and the write MB/s returned.

Alternatively, create a job file fiorandomreadwrite.fio with the following content:

[global]
bs=256K
iodepth=64
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=4
name=raw-randreadwrite
rw=randrw

[job1]
filename=<device name>

Then run it with:

fio fiorandomreadwrite.fio

Test Sequential Reads​

Run the following command directly:

sudo fio --filename=<device name> --direct=1 --rw=read --bs=256k \
  --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 \
  --time_based --group_reporting --name=throughput-test-job \
  --eta-newline=1 --readonly

Alternatively, create a job file fioread.fio with the following content:

[global]
bs=256K
iodepth=64
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=4
name=raw-read
rw=read

[job1]
filename=<device name>

Then run it with:

fio fioread.fio

Latency Performance Tests​

Use the following FIO commands to test latency performance.

Caution: Do not run FIO tests with a write workload (readwrite, randrw, write, trimwrite) directly against a device that is in use.

There are two approaches to latency testing, depending on what you want to measure:

  • RBD engine (--ioengine=rbd): Talks directly to Ceph via librbd, bypassing the Linux kernel, filesystem, and page cache entirely. This gives the most accurate picture of raw Ceph storage latency, and is useful for isolating storage-layer performance from the rest of the stack.
  • libaio with a mounted filesystem (--ioengine=libaio): Tests latency as seen by an application using a mounted filesystem. This includes overhead from the kernel block layer and the filesystem, and is more representative of real-world application latency.

Use the RBD engine approach when you want to benchmark the storage cluster itself. Use the mounted filesystem approach when you want to understand the latency experienced by applications running on top of the storage.

Test Write Latency via RBD Engine​

This test uses the rbd ioengine to write directly to a Ceph RBD image, bypassing the kernel and filesystem:

fio --ioengine=rbd --direct=1 --invalidate=1 --bs=4K --iodepth=1 \
  --rw=write --runtime=60 --clientname=admin --pool=<pool name> \
  --rbdname=<rbd image name> --numjobs=1 --name=latency

Test Write Latency via Mounted Filesystem​

First, mount the RBD image at a suitable mount point with a filesystem such as ext4:

# Mount the RBD image
sudo mount /dev/<rbd image name> /mnt/<mount point>

Then run the latency test against the mounted filesystem:

fio /mnt/<mount point> --ioengine=libaio --direct=1 --sync=1 \
  --invalidate=1 --bs=4K --iodepth=1 --rw=write --runtime=60 \
  --name=latency

Test Random Reads for Latency​

Run the following command directly:

sudo fio --filename=<device name> --direct=1 --rw=randread --bs=4k \
  --ioengine=libaio --iodepth=1 --numjobs=1 --time_based \
  --group_reporting --name=readlatency-test-job --runtime=120 \
  --eta-newline=1 --readonly

Alternatively, create a job file fiorandomreadlatency.fio with the following content:

[global]
bs=4K
iodepth=1
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=1
name=readlatency-test-job
rw=randread

[job1]
filename=<device name>

Then run it with:

fio fiorandomreadlatency.fio

Test Random Read/Writes for Latency​

Run the following command directly:

sudo fio --filename=<device name> --direct=1 --rw=randrw --bs=4k \
  --ioengine=libaio --iodepth=1 --numjobs=1 --time_based \
  --group_reporting --name=rwlatency-test-job --runtime=120 \
  --eta-newline=1

Alternatively, create a job file fiorandomrwlatency.fio with the following content:

[global]
bs=4K
iodepth=1
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=1
name=rwlatency-test-job
rw=randrw

[job1]
filename=<device name>

Then run it with:

fio fiorandomrwlatency.fio