Storage Performance Troubleshooting Playbook

There is no universal "good latency" number

NetApp explicitly notes that it does not define one universal latency or IOPS threshold for all workloads. Use the stable normal state of the real application as a baseline and investigate changes together with user symptoms.

Measure the workload shape

• Read vs write.
• Random vs sequential.
• I/O size.
• IOPS.
• Throughput.
• Latency.
• Queue depth/concurrency.

Low IOPS can still show odd average latency

With very few operations, one slow request can skew the average strongly. NetApp documents this statistical effect. Do not declare a storage system unhealthy from one average-latency value without checking operation count and application symptoms.

Find the delay layer

1. Application: is the app waiting on storage or something else?
2. Host: filesystem, queue, CPU, memory pressure?
3. Network: packet loss, errors, congestion, iSCSI path?
4. Array: controller/cache/data layer?
5. Media: SSD/HDD response and rebuild/background work?

Background operations

RAID rebuild, scrubbing, snapshots, replication, deduplication and backup can consume resources. Correlate performance changes with maintenance activity before replacing hardware.

Baseline

Keep normal busy-hour graphs for IOPS, latency and throughput. A baseline is more useful than a generic Internet threshold because it shows what the application normally needs on this exact system.

Latency breakdown

When the platform exposes component latency, separate network, protocol, controller/data and disk/media delay. This prevents a high total latency number from being blamed on drives when the delay is actually in the network or application layer.

Read/write asymmetry

A workload can read well and write poorly because parity, cache policy, replication or media characteristics affect writes differently. Always split read and write metrics when the platform supports it.

Benchmark carefully

Do not run destructive or unrealistic benchmarks on production data. A synthetic sequential test may prove interface throughput but say little about a database workload with small random writes.

Rebuild impact

Measure performance before, during and after a rebuild if the system is business critical. The client should know whether a second failure or heavy production load during rebuild changes the operational risk.

Throughput math

Approximate throughput can be related to IOPS and average I/O size: throughput is roughly IOPS multiplied by bytes per operation. This is arithmetic, not a performance guarantee. Protocol overhead, caching and mixed I/O patterns affect what the application sees.

Queue depth

A deeper queue can keep fast storage busy, but it can also increase latency because more work waits in line. Compare queue depth with application response time rather than tuning it in isolation.

Cache hit effects

Reads served from cache can be dramatically faster than reads from media. A benchmark that fits entirely in cache may not represent a large production dataset. Record test duration and working-set size.

HDD vs SSD behaviour

Rotating disks pay mechanical seek/rotation cost, so random I/O behaves differently from sequential I/O. SSDs remove mechanical seek but still have controller, flash-management and endurance characteristics. Pick media from workload and resilience requirements, not only headline throughput.

Network ceiling

A storage array capable of high throughput can still be limited by a 1 GbE client link, an oversubscribed switch uplink or a single TCP/session path. Confirm every layer's negotiated speed and utilisation before concluding the array is slow.