What eBPF Host Monitoring Actually Costs: 33 Days on Two Live Servers
I ran a read-only eBPF recorder on two servers I run for a month, and here is what it cost, including the places it failed.
Why I measured this
A recent review of 54 eBPF security studies found that 47 tested only on a testbed, with a median reported overhead of 2.4% CPU (Kostopoulos et al. 2026, preprint). I wanted numbers from servers I actually run. So I ran a read-only eBPF recorder I built on two small cloud servers I run, 4 vCPU and about 8 GB each, for 33 days. They host low-traffic sites and an automated management agent, so they are quiet. Two quiet servers I run are not a population. Read that first.
One more caveat up front: the collector ran live, but the part that scores events ran later, on replay, on another machine. The full pipeline never ran live on these hosts.
What it cost
The collector used 0.119% and 0.318% of one core on the two hosts. Time spent inside the eBPF programs themselves was 0.004% and 0.0018% of a core. I ran a different collector build on each host on purpose, so these are not a clean A/B.
Where I missed my own budgets
Memory: median 43 MB, high-water 69 MB, against a 30 MB budget. Exec probe latency: mean 13.4 and 5.1 microseconds, against a 3 microsecond p99 target. Disk: 932 and 954 bytes per fact. That passes, but only because I raised the budget from 300 to 1,000 bytes during the project. On the original number it fails.
The lane that was not supposed to drop
I built a lane I called “never-drop.” It dropped 25,115 of 10,485,812 events on one server, 0.24%. All 25,115 losses happened in 40 windows of about one second each, while the server averaged under 5 events per second. The collector’s records name npm and Node for about 15,000 of them and rm for 8,302, and the worst window lost 5,973 events in about a second. The ring holds 487 records and by my estimate overflows above roughly 49,000 file events per second. I had written and tested a larger ring but never deployed it to that host. A never-drop lane that drops is a design failure, not a tuning note.
The claim I had to retract
I had claimed no learned model could run on this stream at a useful size. The measured rates: mean 3.33 and 4.56 events per second, peak hour 128 and 268 per second, peak second about 2,700 facts. My budgets were set at a 5,000 events per second stress point, about a thousand times the mean. I never thought that rate was typical.
So I measured. On one core of a Xeon, forward passes only, with untrained weights: at these servers’ mean rate of reduced facts, the largest model I tested (2.08M parameters) needs 1.31% of a core. At the mean raw event rate, all models need under 3% except the two largest transformers, 4.42% and 7.9%. At the stress point or the peak second, all of them need more than a core without buffering.
So my claim does not hold at these mean rates. Others had already shown learned models keeping up on lab and benchmark data, so the fit is not news. The measured live rates are what I add. And nothing was trained, so this says a model fits, not that it detects anything.
Ranking: close to a tie, on weak evidence
I spliced five synthetic attacks I wrote into 10 held-out days and compared my ranker to two classic count-based methods, with thresholds tuned the same way. My first-seen counter ranker found 300 of 300 and showed a median of 0.5 and 1 group a day. A Kneser-Ney bigram found 297 of 300 and showed 2. That is about a tie. It is weak evidence, because every injected attack uses paths new to the host by construction. A STIDE-style method found 177 of 300, missing short attacks at its tuned threshold.
My first draft compared my tuned ranker against untuned baselines, which made mine look much better. A separate AI agent reviewing the draft flagged it and I fixed it. The tuning rule I fixed in advance picked a score floor of 1.0, not the 0.5 I had shipped. At floor 1.0, 41 groups that scored exactly 1.0 are hidden, against 45 above it.
And a small attack that runs daily is reported for three runs. From the fourth it drops out of the digest in three of five scenarios, because the system learns that periodic things are routine. A burst of 5,000 new paths stays visible.
How this was made
I built this with substantial help from Claude, an AI assistant from Anthropic, and drafted the paper text with it. I am responsible for all of it. I have a commercial interest in the recorder, so weigh that. Its source is not public; aggregate data and scripts come with the paper. Raw recordings are not released, because they hold command lines from live servers.