Notes on replication without drama

October 12, 2026

Last month we moved a service that had been sitting on the same machine for three years. The requirement was boring and absolute: no one should notice. The interesting part turned out to be not the move itself but the assumptions quietly attached to every row of data.

Timestamps are contract, not metadata

Every record carried updated_at, and every consumer assumed it meant the same thing. It did not — two writers used local time, one used UTC, and a nightly job had been normalising everything into UTC-3 for reasons nobody could recall. The diff at cutover came out EMPTY only after we made the contract explicit:

-- contract, version 2, written down this time
-- updated_at: always UTC, always the writer's wall clock
ALTER TABLE events
  MODIFY updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
                             ON UPDATE CURRENT_TIMESTAMP;

Checksum the moves, not just the data

We shipped the state machine-by-machine with rsync -a --checksum and a per-directory manifest. The manifest is what mattered: at cutover we compared three lists and they agreed to a byte. Without it the "success" is a confident shrug.

rsync -a --checksum /srv/state/ newnode:/srv/state/
find /srv/state -type f -exec sha256sum {} \; > manifest.old
ssh newnode 'find /srv/state -type f -exec sha256sum {} \;' \
    > manifest.new
diff manifest.old manifest.new   # empty, we were done

What the checklist missed

The first pass scheduled cutover at a "quiet hour". Quiet for batch jobs, apparently, not for the reporting script that starts at exactly 01:00 and holds a read lock for a minute. The second pass added one line: list every cron on both sides, by hand. Rehabilitation for cron is optional; enumerating it is not.

Aftermath

The old machine stayed up for a week, serving nothing but the health check, in case the rollback path needed to be an actual path and not a story we told ourselves. Cost: one idle box for seven days. Worth it.