DetectionScattered across three tools
XIDs live in kernel logs, ECC counters live in DCGM, and node death shows up somewhere else entirely. Teams end up correlating them by hand, across tools.
Open source — for Kubernetes GPU fleets
DetectsProtectsRecoversMeasures
Vendor-neutral by construction: KubeNeuron detects degradation, protects the workloads running on the device, automates safe recovery through a policy ladder with approval gates and a complete audit trail — and measures recovered capacity, so what the fleet got back is a number rather than a feeling.
curl -sfL https://github.com/kubeneuron/kubeneuron/releases/latest/download/install.sh | bash -s -- --version latest
Dry-run by default: the workflow runs end to end and records exactly what it would do, so you see the decision before anything touches a node.
Runs alongside the stack you already have
The 3am problem
One uncorrectable GPU memory error can kill hours or days of training work. Today that means a human paging through kernel logs and DCGM dashboards, then guessing whether to reset the device, drain the node, or file an RMA — under time pressure, without a record.
XIDs live in kernel logs, ECC counters live in DCGM, and node death shows up somewhere else entirely. Teams end up correlating them by hand, across tools.
The right answer for XID 64 is not the right answer for XID 94. That knowledge sits in a few engineers' heads and a stale runbook.
Scripts that reboot nodes on a metric threshold are exactly how one bad signal takes out a fleet. So most teams never trust them enough to turn them on.
The platform
The product definition is a sentence with four claims in it. Each one has a page that shows the work behind it, including where the work stops.
Two detection paths converging on one incident, the escalation ladder, and the signal catalog that decides which failure gets which response.
ProtectsDry-run by default, a blast radius you declare, human authority on the rungs that deserve it, and unknown state stopping the action rather than guessing.
RecoversEvery decision visible and every actor named — the approval gate, the audit trail, the fleet map, and the dashboard that ships with it.
MeasuresThe whole ladder against a synthetic XID, no cluster and no hardware, in about fifteen minutes — and the recovered-capacity report at the end of it.
What recovery gave back
An incident closing tells you the workflow finished. It does not tell you how much accelerator capacity came back, how long it took, or how often the fleet healed without waking anybody. KubeNeuron measures all three from its own incident store — exact, not sampled, and reproducible from a database snapshot.
366.5GPU-hours recovered, of 412.8 degraded — 88.8%
27 of 31recoveries that finished without a human — 87.1%
12m22smedian time to resolution (p50, n=31; p90 52m0s)
kubeneuronctl report --since 30d
degraded GPU-hours 412.8
recovered GPU-hours 366.5 88.8% of degraded
incidents recovered 31 of 37 83.8%
without a human 27 of 31 87.1% of recovered
MTTR (resolved, n=31) p50 12m22s p90 52m0s
Why not just…
How most GPU fleets run today
but…
The cron job nobody dares to enable
but…
Detection is automatic. Anything destructive runs a ladder you wrote, behind gates you configured, with a named approver on the record — and dry-run shows you the decision before you let it act.
No shell in config. No surprise reboots. No unexplained decisions.
Get started
CRDs, operator, TLS bootstrap, and a starter config. Dry-run by default.
Install KubeNeuronThe real controller against a simulated fleet, on your laptop, in a minute.
Start the demoClick through incidents, the fleet map, and playbooks captured from a real EKS run.
Open the consoleThree slots for teams running production GPU fleets. Direct roadmap influence, direct access.
Read the offerLatest release
Every release carries signed multi-arch images, an SBOM, checksums, a single-file install manifest, and a changelog entry that says what was proven and what was not.
Release notes, checksums, signed images and the install manifest.
Read the release notesBuild the safe path first
Read the source, run the dry-run ladder, and tell us where the policy model breaks.