Config drift is a process failure

September 28, 2026

During a quarterly audit we found a production host whose nginx configuration had eleven directives that existed nowhere else in the fleet. All added by hand, none documented, two of them demonstrably from an emergency at least a year old. Nothing was broken. That is exactly the problem.

Drift is not a config problem

The config did not drift. The process did. Someone had an incident at 23:40, fixed it the only way available at 23:40 — directly on the box — and then went back to bed. The commit that says "NGINX: raise client_max_body_size for the intake endpoint (incident #4281)" never happened, so the next operator sees a value with no story attached and a fleet baseline that disagrees with production.

The weekly stroll

One rule has survived since the seventies of machine administration: walk the config once a week, by hand, with a diff against the source of truth. It sounds démodé until the first time it catches something. Ours took four minutes and found:

--- expected/httpd.conf
+++ reality/httpd.conf
@@ -14,5 +14,6 @@
  KeepAlive on
  KeepAliveTimeout 5
+ RequestReadTimeout header=20-40,MinRate=500 body=20,MinRate=500
  IncludeOptional conf.d/*.conf

One line, added under pressure, matching nothing, breaking nothing yet. The fix is not a rollback; the fix is a commit plus a link to the incident.

Make the honest path cheapest

Every operator takes the shortest path from "it hurts" to "it doesn't". Drift prevention means making the short path concrete:

• a department of one: every change lands in the template first;
• the weekly diff is a cron that people actually read;
• emergency edits are always followed, next morning, by the commit — no exceptions, and holidays are not exceptions either.

The box is boring again. Next quarter the audit will find nothing, because there will be nothing to find.