Interconnects and AI fabrics

Carrier (physical signal detection): the state in which an interface detects the link signal from the other end; it is not a sufficient condition for healthy data delivery.

Whether a cable is plugged in, the optical signal, PCS alignment, carrier, and the interface admin state are facts at different layers. A single "link up" line guarantees neither error-free delivery nor the correct speed.

Transceivers and cables carry the signal on each lane, and the NIC recovers frames from it. If the speed, FEC, or lane negotiation does not match, or if the optical power is marginal, corrected error counts can accumulate while carrier stays up.

An intermittent fault can disappear right after reseating, so save the counters and the module identity before touching anything. Put the port names at both ends and the cable labels in the same record so you can identify what to replace.

In the training example the carrier stayed up but CRC errors grew by 12 over 60 seconds. Looking only at the cumulative counter of 12 tells you nothing about whether the errors are old or current. Record the start value, the end value, the measurement window and the traffic sent during it. Do not immediately declare a bad cable because FEC corrections increased; first compare the settings at both ends and the supported module combinations.

Why does this happen?

Transceivers and cables carry the signal on each lane, and the NIC recovers frames from it. If speed, FEC, or lane negotiation does not match, or optical power is marginal, the carrier can stay up while corrected errors accumulate.

When is it a problem?

If you see degraded speed, rising CRC errors, or a module mismatch, there are insufficient grounds to proceed.

Common beginner misconceptions

Clearing counters to 0 or reseating the cable first destroys the original failure evidence. Take a snapshot first, then change only one approved variable.

How to verify it yourself

Compare the switch and NIC ports at both ends and the cable labels against the cabling sheet. Check the driver, firmware, and module identities and the supported combinations.

VLAN (virtual LAN): a configuration that logically divides Ethernet broadcast domains even on the same physical switch.

Ethernet forwarding is a chain of MAC learning, VLAN, IP subnet, route, ARP/ND, and queues. The mere fact that two devices are plugged into the same switch guarantees neither the same broadcast domain nor the ability to communicate.

The host decides whether the destination is on the local subnet, and if it is remote it sends the frame to the gateway MAC. The switch forwards by MAC within the VLAN, and the router crosses subnets. Congestion builds up in the egress queue and produces drops and tail latency.

Before concluding that a communication failure is the firewall's fault, narrow the boundary in the order of local address, route, neighbor, VLAN, and path. Use packet capture only within the minimum scope and the rules for handling sensitive information.

In the training example the sending port transmits tagged frames on VLAN 120 while the receiving port is configured as untagged VLAN 130. Even with both links up and correct IP addresses, this is not the same connection path. Treating a small ping failure as a routing problem right away misses the VLAN boundary underneath. Check port mode, allowed VLANs and MAC learning first, then move up to IP and MTU.

Why does this happen?

The host decides whether the destination is on the local subnet and, if it is remote, sends the frame to the gateway MAC. A switch forwards frames by MAC within a VLAN, and a router carries traffic across subnets. Congestion builds up in egress queues and causes drops and tail latency.

When is it a problem?

If you see a VLAN, MTU, or gateway mismatch, there are insufficient grounds to proceed.

Common beginner misconceptions

Setting jumbo frames on only the two end hosts is not enough. Check that each segment of the path accepts the required frame and packet sizes, and account for encapsulation overhead. Devices may display MTU on different bases, so do not pass a path just because the numbers are the same.

How to verify it yourself

Compare the IP, prefix, and MTU of the source and destination. Check on which interface, and to which MAC, each neighbor entry is resolved.

RDMA (Remote Direct Memory Access): a method of delivering data into remote memory through a supported NIC and transport.

Remote Direct Memory Access (RDMA) reduces the involvement of the CPU and the kernel data path in transfers to remote memory, but it does not automatically make the network fast and lossless.

The RDMA device, GID, queue pair, and memory registration must all match, and RoCE is operated together with the loss and congestion design of the Ethernet network. Overusing Priority Flow Control (PFC) creates new failure modes: head-of-line blocking and pause storms.

Split the functional check into three stages: the device is visible, small messages get through, and bandwidth, latency and retries are acceptable at the target message size and concurrency.

In the training example, the ordinary TCP test passed, but the RDMA test could not establish a connection. Working TCP communication does not prove the RDMA device, GID, or queue pair configuration. For RoCE, investigate the IP, GID, MTU, and congestion control path; for InfiniBand, investigate the state of that fabric's management separately. Do not treat the configuration of the two transports as a single common command.

Why does this happen?

The RDMA device, GID, queue pairs, and memory registration must all be correct, and RoCE must be operated together with Ethernet loss and congestion design. Overusing Priority Flow Control (PFC) introduces new failure modes: head-of-line blocking and pause storms.

When is it a problem?

If retries, pause frames, or tail latency surge, there are insufficient grounds to proceed.

Common beginner misconceptions

Running an unauthorized bandwidth test on the production network affects real workloads and switch queues. Get approval for an isolated window and rate limits.

How to verify it yourself

Check that the rdma link and GID point to the designed netdev and VLAN. Make sure the server and client use the same device, port, and GID index.

Collective (collective communication): a data exchange operation that multiple ranks perform together in a fixed order.

Fabric acceptance is not a single top-speed run; it is the process of approving inventory completeness, path symmetry, error-free sustained load, per-node variation, and recoverability.

A collective workload is limited by its slowest participant and by barriers. Even if the average exceeds the target, tail latency on one node, incorrect NUMA placement, flaps, or corrected errors will delay the entire job in production.

The test matrix must include cable/port identity, firmware, topology, message size, the number of concurrent jobs, the measurement duration, and the allowed variation. Record the revision and hash of each result file.

In the training example, 7 of the 8 ranks are healthy, but one rank's communication time has increased. Even if the average bandwidth exceeds the target, the whole job can slow down at the point where every rank waits. Tie the per-rank transport and path, along with the per-message-size results, to the same run ID. Checking whether the problem follows the slow rank when you move it lets you distinguish the node hypothesis from the path hypothesis.

Why does this happen?

A collective workload is limited by its slowest participant and by barriers. Even if the average exceeds the target, tail latency on one node, incorrect NUMA placement, flaps, or corrected errors will delay the entire job in production.

When is it a problem?

If you see node-to-node variation, link flaps, or rising errors, there are insufficient grounds to proceed.

Common beginner misconceptions

If you approve a slow node by burying it in the average, the collective's barrier slows down the entire real job. An outlier needs its own cause analysis and retest.

How to verify it yourself

Check that the cabling sheet and the actual port, module, and firmware lists are complete. Compare each link with a point-to-point test under identical conditions.