Attack surface · earlyoom

Something killed your process and the kernel log is empty

Published 2026-08-16 · HardStacked · 11 min

A process that mattered is gone. It did not crash: there is no core dump, no exception in the application log, and dmesg has no Out of memory line anywhere. You have a dead process and, apparently, no killer.

The killer usually exists, and it usually kept a diary. It is just not the diary you have been reading.

Everything quoted below was measured on one Debian 13 (trixie) arm64 machine running earlyoom 1.8.2-1+b1 from the Debian repository: forty-five kills counted over seven days, 2026-07-25 to 2026-07-31, with the unit file and configuration read back on 2026-08-16.

01. The two line verdict

02. How to find out who did it

Three possible killers, three different journals. Ask them in this order; each answer is a line or an empty screen, and both tell you something.

The kernel first, so you can rule it out:

-k reads the kernel's messages out of the journal — the same source as dmesg, without the ring buffer's habit of overwriting itself. It covers the kernel's own OOM killer and every cgroup memory limit, because both are kernel decisions. Note that -k only shows the current boot; if the machine rebooted since the kill, add -b -1. If your process is named here, you are done — it was the kernel, and section 06 tells the two kernel cases apart.

Then the userspace daemons, each in its own unit journal:

A daemon that kills from userspace logs where every other service logs: in its own unit. Not in dmesg, not in your application's log — in a journal you have to name to see.

Last, the process itself:

That is everything the journal holds from the process's own mouth. Often nothing. Sometimes a last complaint that changes the picture.

An empty first screen and a full second one is this page's case, and section 04 shows what you will find there. Four empty screens and a process that is somehow running again: hold that thought for section 06 — a supervisor may be restarting it faster than you notice the death.

03. Why the kernel log is empty

The kernel's OOM killer is a last resort by design. It wakes when an allocation cannot be satisfied any other way — when the machine is already at the wall. Everything it does at that point lands in the kernel log, loudly.

earlyoom exists so that moment never comes. It is a small daemon that keeps reading MemAvailable and free swap out of /proc/meminfo at short intervals, and once its thresholds are crossed it picks the process with the highest oom_score and sends it a plain SIGTERM — SIGKILL if things keep sliding. Memory frees while the kernel can still satisfy allocations, so the kernel's own trigger is never reached. Nothing in that sequence is a kernel decision: reading a proc file and calling kill() is what any process with the right capability may do.

The unit file says all of it in a dozen lines. This is the [Service] section as shipped on Debian 13, read on 2026-08-16:

CAP_KILL and OOMScoreAdjust=-100 tell the whole story between them: a daemon whose only power is to kill, shielded from the very shortage it manages, capped at 50 MB so the watcher can never become the problem. The rest of the unit, cut above, is a hardening block that takes away everything else — no network, no /home, no devices. A process built like that leaves exactly one trace when it acts, and that trace is in its own journal.

04. What it looks like when you measure it

Strip the comments and the machine's whole policy is one line in /etc/default/earlyoom:

In order: -r 3600 prints a memory report once an hour. -m 12,6 sets the thresholds — SIGTERM when available memory drops under 12% of RAM, SIGKILL under 6%. -s 100 sets the swap threshold at 100%, crossed as soon as a single page sits in swap — in practice, the decision rides on RAM alone. -p gives earlyoom priority and shields it from the OOM it polices, doubling what the unit already grants. And --prefer is the interesting one: a regex of process names whose oom_score gets a large flat bonus, pushing them to the front of the kill queue. Here it names a browser and its child processes.

This is what one kill looks like in journalctl -u earlyoom. The command line is trimmed — the [...] is ours — everything else is verbatim:

One line, in one unit's journal, for a process holding 4,640 MiB. Nothing in dmesg, before or after.

Counted over seven days, 2026-07-25 to 2026-07-31: forty-five kills. Among the named victims, sixteen Isolated Web Co, six WebExtensions, four Web Content, one Privileged Cont — browser children, all of them — and one working process. The browser filed zero crash reports over the same window: a clean SIGTERM is not a crash. Tabs quietly reloaded, and nobody complained.

Now put the configuration next to the journal line, because together they show something no documentation gives you. In the cascade measured on 2026-07-31, earlyoom sent around ten SIGTERMs to browser processes first — logged at oom_score 1119, 1107, 1081, 1068, 1066, 1062, 1061 and so on — each holding between 90 and 326 MB. The process actually holding the memory, the 4,640 MiB one, was logged at 1023 and fell only after them. That is --prefer doing exactly what it says: the flat bonus lifts a 90 MB tab above a 4.6 GB worker, and the natural logic of oom_score — biggest holder first — runs backwards. Ten processes died to free less than a third of what one process held.

Resist reading that as a misconfiguration. It is a trade. A killed tab reloads itself and loses nothing; a killed worker loses its work. --prefer encodes exactly that: burn the processes that are cheap to restart, even ten of them, before touching the one whose death costs something — at the price of freeing memory in 90 MB sips while 4.6 GB sits in one place. Which side is right depends on what the big process is to you. On this machine it was the work being protected; if on yours it is a leak, the same line is protecting the leak. This article's job is to show you the mechanism executing. Which side of the trade to stand on is your call.

The trade also has a lever, and it is worth knowing by name. The same binary ships --sort-by-rss: earlyoom then picks its victim by largest resident memory instead of highest oom_score — the criterion that would have pointed at the 4,640 MiB process first. It is not the fix; it is the other side of the same trade, with the mirror-image cost: the largest holder is often the process doing the work. Choose the side that matches what the machine is for — and choose it in the configuration, not by default.

05. The test that will lie to you

Here is the test that produces the wrong conclusion with complete confidence:

Silence. And silence reads as an answer: the machine did not run out of memory, so the process must have died of something else — a segfault, a bug, someone's stray kill. You are now reading application code, hunting for core dumps, maybe blaming a deploy, while the actual cause was memory pressure, handled, logged, and filed under a unit name you never queried.

The flaw is not in the command. It is in the scope we grant the answer. dmesg is not the system's diary; it is the kernel's. It records what the kernel did — OOM kills, cgroup limit breaches, oopses — and nothing that a userspace process did with its ordinary powers. To the kernel, earlyoom killing your process is one signal among the thousands it delivers every minute. There was nothing to record, so nothing was recorded.

An empty kernel log is not the absence of an event. It is the absence of that log's author from the event.

The machine above pushes the lesson one layer further: forty-five kills, and the browser's crash reporter counted zero, because a clean SIGTERM is not a crash. Two independent monitors, both reporting that nothing happened, both perfectly correct about their own patch of ground. The event fell in nobody's jurisdiction except the killer's own journal — which is why section 02 starts by naming suspects instead of searching one log harder.

06. The neighbours

Same symptom, different killers. We have only measured earlyoom on this machine, so for the neighbours you get the mechanism and the shape of the message to search for, not a painted terminal.

systemd-oomd. Ships enabled on several desktop distributions. It watches cgroup v2 pressure information (PSI) — how long tasks stall waiting for memory — rather than an absolute threshold, and it kills a whole cgroup at once, not a single process. Yours can die for a sibling's appetite. It logs in its own unit journal, journalctl -u systemd-oomd, and the message names the cgroup it killed and the pressure that triggered it. Kernel log: empty, for the same reason as earlyoom.

A MemoryMax= on the unit. If your service's unit sets MemoryMax=, the cgroup hits its ceiling and the kernel — this time it really is the kernel — kills inside it. dmesg gets a line beginning Memory cgroup out of memory: Killed process, with names. So an empty kernel log rules this one out too, which is worth something.

The kernel's own OOM killer. The loud one. Out of memory: Killed process followed by a task dump you cannot miss. If that is absent from journalctl -k, the kernel did not do it — which is precisely the fact this article started from.

The supervisor that hides the body. Restart=always — the same directive that keeps earlyoom itself immortal — restarts a killed service in seconds. You never see a dead process at all: same name in the process table, new PID, uptime reset to now. The unit's own journal shows the death and the restart back to back; the process table shows only the young replacement. If your monitoring asks “is the process there” rather than “is it the same process”, this one can run for months.

07. Check which one you have

Before touching any threshold, establish which of the four you actually have. Three commands, each answering with a fact rather than an interpretation.

One word per daemon. active means a userspace killer is on duty and its unit journal is worth reading; inactive on both removes half the suspects at once.

MemoryMax=infinity means no cgroup ceiling on that unit — the kernel was never entitled to kill it over a limit. A number means it was, and section 06 told you where that gets logged.

The thresholds and, above all, the --prefer list. Read it before you blame your application: it is the list of processes elected to die first, and section 04 showed how far that election overrides size.

Identify the killer before you change anything. Raising earlyoom's thresholds when the killer is a cgroup limit, or lifting MemoryMax= when the killer is earlyoom, treats the wrong patient on a machine that is already short of memory.

And if you know exactly who kills, and why, and it keeps happening — the question is no longer the killer. It is the platform underneath: how much memory the machine really has, what shares it, and what deserves protection from what. That is a longer conversation than a threshold.

A tool answering a narrower question than the one you asked is also how a firewall reports green while blocking nothing.