Measure Storage Latency Before Buying Faster Disks

A dashboard reporting 600 MB/s does not tell you whether an order confirmation is fast. Large sequential transfers can produce impressive throughput while small synchronous writes wait behind a busy queue. Begin with the operation users notice, then find out how much of its time is actually spent waiting for storage.

Keep the workload attached to the number

Record request size, read/write mix, concurrency and whether a successful write must reach durable storage. A benchmark that writes to memory-backed cache answers a different question from a database commit that waits for persistence. Likewise, a test with dozens of outstanding requests may hide latency that dominates a single-threaded workload.

Compare the same time window across the application, filesystem and block device. Linux block statistics describe requests reaching that layer. They do not directly measure application response time, and aggregation across devices or time can hide short stalls.

Work through a plausible example

Consider a hypothetical service whose slowest one percent of commits take 180 ms while its median stays near 4 ms. If the spikes line up with a backup, reduce backup concurrency in a controlled test. If commit latency improves without changing the disk, you have evidence of contention. If device waits stay low during the spikes, inspect database locks, CPU scheduling or networked storage before recommending new hardware.

Treat the average as a comparison, not a promise to an individual user. Preserve both the latency distribution and the amount of work completed; a lower latency achieved by serving far fewer requests is a different outcome.

Test the purchase decision

Use a disposable volume, representative data and the same durability settings for both candidates. Never run a destructive write benchmark on production data. Decide in advance what improvement would justify the upgrade, then compare cost, sustained behavior and performance under the workload that caused the complaint.

Further reading

Linux kernel: Block layer statistics

Related guide: Read a PostgreSQL Query Plan Before Adding an Index.