Skip to content

06 — Shared storage: capacity, bandwidth, and failure domains

Disaggregated storage lets compute and storage capacity grow independently, but it also introduces shared resources and network dependencies. Design it from workload measurements rather than a promise of linear scaling.

Prerequisites: data path and measurement. Artifact: a storage plan with capacity, bandwidth, redundancy, and failure assumptions. Allow 1–2 hours for analysis; network deployment is an optional hardware exercise.

What remote storage changes

application -> filesystem/block layer -> transport -> storage target -> media
                                      -> network and target failure domains

NVMe over Fabrics extends NVMe over a fabric; its transports and implementation support depend on the selected stack. The official NVM Express specifications define the protocol families. A file-reading application can often retain its logical API while the underlying path changes. That does not guarantee unchanged latency, supported GDS mode, error behavior, or code/configuration requirements.

Local caching and shared storage can coexist. The useful comparison is among actual workload designs: dataset copies, cache reuse, access concurrency, checkpoint write bursts, recovery, and operational cost.

A capacity and bandwidth plan you can calculate

The following hypothetical inputs are teaching values, not OpenFlex or another product's specifications:

  • 32 GPU workers need 10 decimal GB/s each during an I/O-heavy phase;
  • each storage unit supplies a measured 75 GB/s useful throughput for that pattern;
  • each unit has 300 TB of usable capacity after its internal protection;
  • the dataset is 500 TB and an additional replication policy requires 1.5× storage.

Bandwidth requires ceil(320 / 75) = 5 units. Capacity requires ceil(500 × 1.5 / 300) = 3. Under these assumptions the initial count is 5. It is only feasible if the network, client paths, placement, and software can carry the demand. Avoid applying a redundancy multiplier twice if it is already included in the usable-capacity input.

from math import ceil

workers = 32
worker_GB_s = 10.0
unit_GB_s = 75.0
usable_unit_TB = 300.0
stored_TB = 500.0 * 1.5
units = max(ceil(workers * worker_GB_s / unit_GB_s),
            ceil(stored_TB / usable_unit_TB))
print(units)  # 5, under the stated assumptions

Use ceil, not int(x) + 1: the latter overallocates when the requirement is an exact integer. Check that capacities and throughput inputs are positive before turning this calculation into a reusable tool.

Follow-up: if one unit must fail without missing the same peak bandwidth target, six units are an initial N+1 arithmetic candidate. That still requires suitable data placement, failover behavior, network headroom, and measured degraded-mode performance. Simply buying six independent units does not establish resilience.

Why linear scale-out can stop

Storage devices, controller paths, switch uplinks, clients, metadata services, CPU processing, and the application can all constrain scaling. Adding two equal storage units doubles their nominal aggregate capacity; it does not guarantee that each training job sees twice the throughput.

Compare these cases with the same dataset and correctness check:

Experiment What it separates
One client, one target Baseline path and integrity
Several clients, one target Target and shared-link contention
One client, several targets Client-side aggregation limits
Several clients, several targets Placement, fairness, and scale-out behavior
One controlled unavailable path Recovery behavior and remaining capacity

Report the slowest worker and completion-time distribution when synchronized training makes stragglers important. Aggregate bandwidth alone can hide one worker that repeatedly stalls an entire step.

Network configuration belongs to the selected stack

RoCE deployments require deliberate congestion, queue, priority, and loss-recovery configuration. Do not treat “lossless network” as an unconditional guarantee or copy a switch CLI for a different network operating system. Use the selected NIC, firmware, switch, and driver documentation, and validate the agreed traffic class end to end.

For a controlled lab, begin with read-only inventory and the documented diagnostic commands. Then establish transport connectivity, a mounted test path, payload integrity, and finally the intended GDS mode. Keep network throughput and application/file-I/O throughput as distinct measurements.

Use the official nvme-cli project for command semantics and the GDS configuration guide for supported storage-path testing. Discovery, connection, formatting, and multi-host mounting are different operations; this lesson does not require formatting a device or changing a production network.

Apply it to robot learning

A robot-learning dataset may contain many small compressed images, metadata, actions, and episode boundaries. A sequential tensor-read benchmark omits file lookup, decoding, sample grouping, and random access. Measure the loader that preserves episode identity and timestamp alignment. A faster loader that pairs an image with the wrong action is incorrect.

Independent task: design a local-cache plus shared-storage loader for a recorded-data experiment. Specify the cache key, dataset revision, eviction and invalidation rules, behavior after a partial download, and checksum verification. Measure repeated-epoch and first-epoch behavior separately if hardware is available.

Self-check with answer: does shared storage eliminate failure? No. It changes which failures are coupled. Storage and network failure can affect multiple compute nodes; replication, failover, backups, and recovery tests address different risks.

Exit criteria: calculate the two sizing constraints, identify at least one shared bottleneck, and distinguish analytical sizing from measured support and performance. Return to the course index.