Skip to content
Netlume
NewInfrastructure intelligence for the networks powering AI

AI-native operations for modern network infrastructure.

Netlume connects to your infrastructure, understands topology, device state, telemetry, logs, routing, and configuration, and helps engineers investigate and resolve incidents across complex multi-vendor environments.

  • Multi-vendor by design
  • Read-only by default
  • Operator approval for every change
  • On-premise or private cloud
  1. Incidents
  2. /INC-4127
OC
OverviewTopologyDevicesIncidentsInvestigationsAgentsChangesConfigurationsAI Fabric
INC-4127HighRoot cause candidateopened 20:31 UTC

Intermittent packet loss between GPU worker group and leaf fabric

Request approval Run verification
Affected service
GPU Training Cluster
Scope
dc-east-1 · AI Fabric A
Devices analyzed
23
Affected paths
18

Incident path · AI Fabric A

4 hops · ECMP
SPECTRUM-01ECMP set changedSPECTRUM-02nominalLEAF-05nominalLEAF-06CNP ↑LEAF-07Et12 q3 saturatedLEAF-08nominalGPU-041nominalGPU-042step +23%GPU-043nominal
  • Healthy
  • Warning
  • Critical
  • Incident path

Correlated signals

5
  • BGP path change detected

    NVIDIA-SPECTRUM-01 · bgp

    20:27:14
  • ECMP distribution changed 4 minutes before incident

    fabric · ecmp-groups

    20:27:15
  • ECN queue saturation detected

    ARISTA-LEAF-07 · Et12 · TC3

    20:33:02
  • PFC pause duration increased

    ARISTA-LEAF-07 · Et12 · prio 3

    20:33:08
  • Packet drops detected on leaf-07

    ARISTA-LEAF-07 · counters

    20:33:41

LEAF-07 · Ethernet12

Critical
ECN-marked packets · TC392%
PFC pause rx · prio 341 ms/s
Queue depth
97% buf
Drops (5m)
18,214
Rx power
−2.1 dBm
Utilization
94.6%

Root cause candidate

evidence-linked

East-west congestion following ECMP path redistribution resulted in queue saturation on leaf-07.

At 20:27 a BGP path change on NVIDIA-SPECTRUM-01 removed one next-hop from the ECMP group toward the GPU leaf tier. Flows rehashed onto the remaining members and concentrated on ARISTA-LEAF-07 Ethernet12. The lossless RoCE queue (TC3) saturated, ECN marking reached 92% and PFC pause duration increased, producing intermittent drops for GPU-WORKER-042.

Confidence

92%

Ruled out

  • Optic / physical layer fault (Rx power nominal)
  • Host NIC firmware regression
  • PFC storm / deadlock

Investigation timeline

UTC
  1. 20:31

    Packet loss anomaly detected

    18 GPU↔leaf paths above loss threshold

  2. 20:31

    Agent began topology-aware investigation

    Scope: AI Fabric A, 2 spines · 4 leaves · 17 hosts

  3. 20:32

    Telemetry collected from 23 devices

    gNMI counters, queue stats, BGP state, syslog

  4. 20:32

    BGP path change correlated

    SPECTRUM-01 next-hop withdrawn at 20:27:14

  5. 20:33

    Interface queue anomalies detected

    LEAF-07 Et12 TC3 depth at 97% of buffer

  6. 20:34

    ECN saturation detected on leaf-07 Ethernet12

    ECN-marked 92% of TC3 packets

  7. 20:34

    Potential root cause identified

    Confidence 92% · 3 alternatives ruled out

  8. 20:35

    Validation steps generated

    4 read-only commands ready to run

Verification commands

read-only
ARISTA-LEAF-07#show interfaces counters errors
Port        FCS  Align  Symbol  Rx   Runts  Giants  Tx
Et11          0      0       0   0      0       0   0
Et12          0      0       0   0      0       0   0
Et13          0      0       0   0      0       0   0

No physical errors — rules out optics/cabling.

Illustrative product UI with simulated demo data. Not real customer data or statistics.

Multi-vendor

One investigation layer across your infrastructure.

Real networks are heterogeneous. Netlume is built for multi-vendor environments — connect your existing routers, switches, firewalls, clouds and clusters without replacing anything.

Designed for heterogeneous infrastructure. Names indicate target platforms and environments, not partnerships or certifications. See coverage by category

  • Juniper
  • Cisco
  • Arista
  • NVIDIA
  • Nokia
  • Huawei
  • HPE Aruba
  • Dell
  • Extreme
  • Fortinet
  • Palo Alto
  • F5
  • SONiC
  • Linux
  • Kubernetes
  • AWS
  • Azure
  • Google Cloud

The problem

Troubleshooting is a correlation problem spread across a dozen tools.

Modern infrastructure teams operate thousands of devices from multiple vendors, with massive telemetry streams, routing systems, logs, configuration, topology dependencies, cloud environments and AI fabrics.

When something breaks, the evidence exists — but it is scattered. Engineers open SSH sessions to one device after another, compare counters by hand, scroll syslog, check what changed, and rebuild the topology in their head.

Netlume acts as the investigation layer across the environment. It knows how everything connects, gathers the right evidence from the right devices, and explains what it found — so engineers spend their time deciding, not collecting.

  • CLI sessions

    show commands, device by device

  • Streaming telemetry

    counters, queues, optics

  • Syslog & alarms

    millions of lines a day

  • Routing state

    BGP · EVPN · IS-IS · MPLS

  • Configuration

    running vs. intended

  • Change records

    what changed, and when

  • Cloud consoles

    VPCs, gateways, interconnects

  • Topology diagrams

    often out of date

Netlume investigation layer

topology-aware
CollectCorrelateExplain

One answer

Root cause candidate, the evidence behind it, what was ruled out, and the safest next step.

Capabilities

Everything an investigation needs, in one place.

Netlume combines infrastructure context with AI reasoning so that answers are grounded in your topology, your telemetry and your configuration.

AI Troubleshooting

Ask a question in operational terms — a prefix, a service, a symptom. Netlume plans the investigation, runs read-only checks on the right devices and shows its work.

› why are BFD sessions flapping on PE-04?

Topology Intelligence

Physical, underlay, overlay and service relationships in one live model, so every investigation starts from how traffic actually flows.

LLDP + IS-IS + BGP + EVPN → one graph

Incident Correlation

Alarms, routing events, counters, logs and changes aligned on a single timeline. Related symptoms collapse into one incident with a blast radius.

5 signals · 23 devices → 1 incident

Configuration Intelligence

Understand running configuration across vendors, detect drift from intended state and connect recent changes to the incidents that follow them.

CHG-2291 → best-path change in 89 s

Network Path Analysis

Trace the forwarding path for any flow across leaf, spine, border, WAN and cloud — including ECMP members — and inspect every hop on it.

10.20.4.15 → 10.44.18.0/24 · 6 hops

AI Fabric Operations

Operate GPU fabrics with RoCEv2 context: ECN, PFC, DCQCN, buffers, NIC health and rail topology linked to the training jobs they affect.

leaf-07 Et12 · ECN 92% · PFC 41 ms/s

How it works

From signal to verified resolution.

A consistent operating model for every incident — with engineers in control of every change.

  1. 01

    Connect

    Read-only connectors reach devices, controllers and clouds over the protocols they already speak.

    SSH · NETCONF · gNMI · SNMP · APIs

  2. 02

    Observe

    State, telemetry, logs and configuration are normalized and attached to a live topology model.

    3,842 devices · 26 networks

  3. 03

    Investigate

    Agents follow the topology from symptom to suspect, collecting only the evidence that matters.

    11 devices traced on path

  4. 04

    Explain

    Findings are written up with linked evidence, a timeline and what was ruled out.

    Root cause · 88% confidence

  5. 05

    Recommend

    Remediation is proposed with blast radius, verification plan and rollback — never applied silently.

    Approval required

  6. 06

    Verify

    After a change, the same checks run again to prove the symptom is gone.

    Probe loss 3.4% → 0.0%

Topology

Understands relationships before it analyzes incidents.

Netlume continuously builds a model of your environment — physical links, underlay and overlay routing, ECMP groups, VRFs and service dependencies — so an investigation can follow traffic instead of guessing.

  • LLDP / physical adjacency
  • BGP · OSPF · IS-IS
  • EVPN / VXLAN overlay
  • MPLS · SR-MPLS · SRv6
  • ECMP · MLAG · LACP
  • VRFs and tenants
Topology/dc-east-1 · all layers
21 devices in view3,842 managedINC-4127 path
INTERNET / CLOUDEDGE / SECURITYSPINELEAFGPU / HOSTSINTERNETAS64500 transit ×2CLOUDAWS · Azure · GCPJUN-MX-EDGE-01edge router · JunosJUN-MX-EDGE-02edge router · JunosPA-FW-01firewall · PAN-OSNOKIA-CORE-01DCI / WAN · SR OSCISCO-N9K-SPINE-01spine · NX-OSCISCO-N9K-SPINE-02spine · NX-OSNVIDIA-SPECTRUM-01AI spine · CumulusNVIDIA-SPECTRUM-02AI spine · CumulusARISTA-LEAF-05leaf · EOSARISTA-LEAF-06leaf · EOSARISTA-LEAF-07leaf · EOSARISTA-LEAF-08leaf · EOSSONIC-LEAF-09leaf · SONiCSTORAGE-NVME-01NVMe-oF targetGPU-WORKER-0418× GPU · 8× 400GGPU-WORKER-0428× GPU · 8× 400GGPU-WORKER-0438× GPU · 8× 400GCOMPUTE-POOL-Bk8s workers ×64
  • Healthy
  • Warning
  • Critical
  • Incident path

Scroll sideways to see the full fabric · tap a device

Illustrative product UI with simulated demo data. Not real customer data or statistics.

Investigation agent

Ask about the network. Get an investigation, not a chat reply.

Netlume turns a question into a plan, traces the path, runs read-only commands on the devices involved and returns evidence, a timeline and a probable root cause — with the exact commands so you can check its work.

Plans
which devices and data to inspect
Traces
the forwarding path hop by hop
Correlates
routing, counters, logs, changes
Shows
every command and its output
  1. Investigations
  2. /INV-20931
OC
OverviewTopologyDevicesIncidentsInvestigationsAgentsChangesConfigurationsAI Fabric
INV-20931·asked by netops-oncallVRF PROD · dc-east-1 → dc-west-2read-only

Why is traffic to 10.44.18.0/24 experiencing packet loss?

Investigation complete · 11 devices · 3m 42s

Agent plan

6/6
  1. Resolving 10.44.18.0/24 in VRF PROD
  2. Tracing forwarding path across 11 network devices
  3. Collecting interface counters, optics and queues
  4. Correlating routing events and BGP best-path history
  5. Checking change history for devices on path
  6. Ranking hypotheses and generating validation steps
Devices inspected
11
Routing events correlated
2
Interface anomaly detected
1
Root cause candidate
Identified

Timeline

UTC
  1. 13:57:12

    CHG-2291 committed on JUN-MX-EDGE-01

  2. 13:58:41

    BGP best path for 10.44.18.0/24 changed

  3. 13:59:05

    CRC errors begin increasing on et-0/0/2

  4. 14:01:30

    Probe loss crosses 1% threshold

  5. 14:02:10

    Investigation opened by netops-oncall

  6. 14:03:52

    Root cause candidate identified (88%)

Forwarding path · VRF PROD

ECMP 1 of 2
  1. Source

    app-​web-​17

    10.20.4.15

  2. Leaf

    ARISTA-​LEAF-​03

    EOS · VTEP 10.0.250.3

  3. Spine

    CISCO-​N9K-​SPINE-​02

    NX-OS

  4. Border Leaf

    ARISTA-​BL-​01

    EOS · VRF PROD

  5. Router

    JUN-​MX-​EDGE-​01

    Junos · et-0/0/2

    Suspect hop
  6. Destination

    10.44.18.0/24

    dc-west-2 · payments-db

Evidence

5 findings
  • Recent config change

    CHG-2291 raised local-preference to 200 on DCI-EXPORT

    13:57:12

    JUN-MX-EDGE-01 · commit by netops-automation · policy-statement DCI-EXPORT term PREFER-EDGE-01

  • BGP route change

    Best path for 10.44.18.0/24 moved to JUN-MX-EDGE-01

    13:58:41

    ARISTA-BL-01 · next-hop 172.16.0.9 (EDGE-02) → 172.16.0.5 (EDGE-01) · 2 events

  • Interface errors

    Input CRC errors rising on et-0/0/2

    13:59 →

    JUN-MX-EDGE-01 · +18,442 CRC in 15 min · Rx power −11.9 dBm (low warn −10.0 dBm)

  • Packet drops

    Path loss 0.1% → 3.4%

    13:59 →

    Synthetic probes app-web-17 → 10.44.18.10 · 600 probes / min

  • Latency increase

    p95 RTT 1.2 ms → 9.8 ms

    13:59 →

    Retransmissions on payments-db sessions ↑ 7.1×

Probable root cause

CHG-2291 shifted traffic for 10.44.18.0/24 onto JUN-MX-EDGE-01 et-0/0/2, a DCI link with a degrading optic. The link previously carried little traffic, so the fault was latent until the policy change.

88%

Recommended actions

  • Confirm optic degradation on et-0/0/2Read only
  • Revert CHG-2291 local-preference on DCI-EXPORTApproval required
  • Schedule optic replacement for et-0/0/2Recommend

Commands executed (read-only)

read-only
JUN-MX-EDGE-01#show interfaces et-0/0/2 extensive | match "CRC|errors"
  Input errors:
    Errors: 18442, Drops: 0, Framing errors: 18442, Runts: 0,
    Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0,
  CRC/Align errors            18442            0

CRC errors increasing at ~20/s since 13:59.

Illustrative product UI with simulated demo data. Not real customer data or statistics.

AI infrastructure

Built for the networks behind AI.

GPU clusters are only as fast as the fabric between them. Netlume understands rail-optimized leaf-spine designs, RoCEv2 congestion control and the link between network symptoms and training-job impact.

  • GPU clusters
  • Leaf-spine
  • RoCEv2
  • ECN
  • PFC
  • DCQCN
  • Buffers
  • Microbursts
  • NIC health
  • InfiniBand
AI Fabric A· rail-optimized · RoCEv2 · 400G
Fabric healthy1 leaf under watch
spine-01spine-02spine-03spine-04leaf-00rail 0leaf-01rail 1leaf-02rail 2leaf-03rail 3leaf-04rail 4leaf-05rail 5leaf-06rail 6leaf-07rail 7GPU-WORKER-041GPU-WORKER-042GPU-WORKER-043GPU-WORKER-044GPU n → rail leaf n · 1 NIC per GPU

Congestion signalled to RoCEv2 senders (DCQCN)

leaf-00
leaf-01
leaf-02
leaf-03
leaf-04
leaf-05
leaf-06
leaf-07
Et1Et12Et24
lowhigh · % of TC3 packets marked

Illustrative product UI with simulated demo data. Not real customer data or statistics.

Security & control

Safe by default. Engineers stay in control.

Netlume is designed for production networks. It reads before it recommends, recommends before anything changes, and never takes destructive action on its own.

  • Read-only mode

    Every connector starts read-only. Write access is a separate, explicit decision.

  • RBAC

    Roles scope who can view which networks, run which checks and approve which changes.

  • Approval workflows

    Proposed changes show the exact diff, blast radius and rollback before anyone approves.

  • Audit trail

    Every command, recommendation, approval and verification is recorded and exportable.

  • Least privilege

    Collectors use narrowly scoped device accounts and command allow-lists.

  • Encrypted transport

    Connector and platform traffic is encrypted in transit with TLS and SSH.

  • Customer-controlled credentials

    Credentials stay in your vault or deployment boundary — you can rotate or revoke at any time.

  • Private deployment

    Run on-premise or in your private cloud, including environments with no inbound access.

Approval required

rb-2291 · INC-4126

Revert CHG-2291 on JUN-MX-EDGE-01

Restore local-preference 100 for DCI-EXPORT so 10.44.18.0/24 prefers JUN-MX-EDGE-02 while the et-0/0/2 optic is replaced.

[edit policy-options policy-statement DCI-EXPORT term PREFER-EDGE-01 then]
-    local-preference 200;
+    local-preference 100;
Blast radius
1 device · 1 term
Prefixes moved
1 (/24)
Method
commit confirmed 5
Rollback
automatic if unconfirmed

Verification plan

  • Best path for 10.44.18.0/24 via 172.16.0.9
  • Probe loss < 0.1% for 5 minutes
  • No new BGP flaps on ARISTA-BL-01

Requires role: network-lead · 1 of 1 approvals

Reject Approve change

Illustrative approval request. Changes apply only after an authorized operator approves.

Turn infrastructure complexity into clear answers.

See Netlume investigate a real-world failure scenario on a multi-vendor fabric, and talk with the engineers building it about your environment.