Break down the data wait time from the dataset to the GPU
Data path: the actual route that delivers stored data to the reader, to CPU processing, to memory, and into the GPU.
Training data does not jump straight from storage into GPU memory. It passes through the object or filesystem client, the page cache, CPU memory, PCIe, and the data loader queue, and waits and conversions at each boundary create GPU idle time.
Large sequential shards are sensitive to bandwidth, while small files and random samples are sensitive to metadata, IOPS, and latency. Compression, decode, and augmentation can create a CPU bottleneck that looks like a storage bottleneck.
Before adding storage just because GPU utilization is low, look at queue depth, CPU saturation, read throughput, and data loader wait on the same time axis.
In the training example GPU utilization is 30% while one CPU preprocessing thread stays busy. Increasing storage bandwidth leaves the segment that thread processes serially unchanged. Separate read waiting, decode and transform time, and GPU waiting, then change one variable at a time and compare. Record the cold and warm conditions as well, so you do not mistake a second run that was faster from cache hits for a storage improvement.
- Why does this happen?
- Large sequential shards are sensitive to bandwidth, while small files and random samples are sensitive to metadata, IOPS, and latency. Compression, decode, and augmentation can create a CPU bottleneck that looks like a storage bottleneck.
- When is it a problem?
- If GPU idle time and the causal signal do not line up in time, there are insufficient grounds to proceed.
- Common beginner misconceptions
- Do not mix a single warm-cache result with cold-start results. Record the cache conditions and the dataset revision.
- How to verify it yourself
- Record the sample size, the file count, the access pattern, and the batch conditions. Collect the client read bandwidth, IOPS, latency, and cache hit rate.