Auditing · misleading signals

Five green lights that lied: anatomy of a routine Linux checkup

2026-07-20 · HardStacked · 7 min read

A routine checkup on one of our hardened Linux machines. Final verdict: nothing to fix, nothing suspicious, hardening score stable. An uneventful hour, on the surface. Except that on the way to that all-green report, five perfectly reassuring signals tried to lie to us, and two very nearly got through.

Auditing is not collecting OKs. It is asking, in front of every number: what could be hiding the truth here? Here are the five traps from that round, in the order they showed up, each with the habit that defuses it.

1. A grep that returns nothing is not a zero

First reflex of the checkup: re-read the score from the last audit in the report file. The command runs, returns nothing. The tempting read: no line, no score, maybe this machine was never scanned. The true read: the file is readable by root only, and the grep died on a permission error, not on an absence of matches.

Empty output has three possible causes: the pattern is absent, the file is unreadable, or the command itself failed. They look identical on screen and mean completely different things.

2. "Free" memory lies, "available" memory tells the truth

Second stop: memory. On an 8 GB machine, free shows 351 MiB free. The tempting read: the box is suffocating, kill something. The true read: the kernel deliberately uses idle RAM as disk cache, and that cache is handed back instantly the moment a program needs it.

The only number that measures real pressure is available: the kernel's own estimate of what can be served without touching swap. Here, 2.4 GiB, 30 % of the machine. No pressure at all. A monitor keyed on free would have raised an alarm for nothing; worse, someone could have "freed" memory by killing a healthy service.

3. "inactive" also means "that name does not exist"

This one is our own mistake, and it stays in the article because it teaches. When it was time to check the machine's HIDS, we queried the service under its agent name: answer, inactive. For several minutes the HIDS was considered down. It was running perfectly: it runs in manager mode, under another unit name, and the one we were querying has never existed on this machine.

The trap is structural: systemctl is-active answers inactive for a unit that does not exist, with the same exit code as a genuinely stopped service (unit names genericized below):

In an audit report, "the HIDS is down" and "I misspelled the HIDS" produce exactly the same output line. Only a second look tells them apart.

4. Yesterday's file can carry a two-week-old number

Back to the score. Unable to read the official report without privileges, we hunted for a readable copy in our working archives, sorted by modification time, newest first. It announced a score of 70. The machine sits at 80. Ten points gone? No: the file had been copied the day before, but its content was two weeks old. Sorting by mtime had surfaced an old photograph in a brand-new frame.

Every copy, every backup, every misconfigured rsync refreshes an mtime without refreshing a single byte of meaning. On an audit number, the gap between the two can make you conclude a regression that never happened, or hide one that did.

5. The score says the door is locked, not whether anyone rattled the handle

The hardening score was good, stable, with two warnings long documented as distribution false positives. The temptation at this point: close the case. But a score is a photograph of the configuration at rest. It says nothing about what actually happened on the machine. The second half of the checkup lives in three places the score never looks at.

First, the real surface. SSH keys found on disk raised an eyebrow: exposed surface? No. Keys make the machine a client that reaches out. The attack surface is a daemon that listens, and what it attracts once it listens, we have measured elsewhere:

Second, the failure counters. An empty faillock table and an auth journal where every privilege elevation traces back to a known action are living proof that nobody even tried. No score contains that information.

Third, the allowlists. The machine carries an exclusion list for its file integrity monitoring. Every line of an allowlist is a chosen blind spot: "stop warning me when this file changes". We re-read them one by one. Three justified themselves; the fourth required finding out who had changed what, and why. It was legitimate. But the only way to know was to ask the question.

What the round came to in the end

Nothing. And that is the best possible result: score stable at 80 (how that number is earned and lost, we measured fix by fix), two warnings long identified as false positives, an empty failure table, clean journals, allowlists revalidated line by line. Zero fixes applied.

But the five greens we ended up trusting are greens we first challenged. The same checkup taken at face value would have declared a running HIDS dead, a breathing machine suffocated, and a ten-point regression that never existed. Three false alarms, three possible "corrections" on a healthy system. Methodical doubt did not just prevent wrong conclusions: it prevented wrong actions.