🔄 Upgrade FreeLLMAPI without losing your data — and keep a way back

Tutorials & Methods
, ,
1

:counterclockwise_arrows_button: Upgrade it, verify it, keep a way back

The whole upgrade, in five glowing stages: before old v0.13.9 still running, copy the database (a filename is not a backup), one override moves the traffic, livez 200 readyz 200, old version kept in place, and a dashed amber line labelled the way back
The whole upgrade, in five glowing stages: before old v0.13.9 still running, copy the database (a filename is not a backup), one override moves the traffic, livez 200 readyz 200, old version kept in place, and a dashed amber line labelled the way back3072×1714 760 KB

Downloading new code is not an upgrade. Building it is not an upgrade. Restarting the service is not proof that it worked.

I upgraded a live FreeLLMAPI box from v0.13.9-ish to v0.13.10 — a Proxmox VE 9 container, Debian, plain Node.js and systemd, SQLite, no Docker. It finished clean, and the way it finished clean is the whole point of this post.

:bullseye: Why it matters: an upgrade can look finished while the old code is still running, or while the new service quietly builds itself an empty database. Those two failures are invisible until you need your data.
:sparkles: What makes it unique: nothing gets overwritten. The old version stays installed, and the entire switch is one systemd override — flip it forward, or flip it back.
:heart: Why you should care: you keep a way back. That turns an upgrade from a gamble into a procedure.

:shuffle_tracks_button: The road, in one picture

        users
          |
   [ systemd switch ]
      /            \
   NEW              OLD
 v0.13.10         kept, ready

One lever. The new version takes the traffic, and the old version stays installed, whole, ready to take it back.

:compass: Six stages, and the trap hiding in each

1 · Know what is running. Which directory, which tag, which systemd unit, where the database lives, where the keys live. systemctl cat, systemctl status, git describe --tags --always.
:mouse_trap: The trap: the database is not always inside the app folder, and the version the health check reports is not always the version Git checked out. A clean checkout proves nothing about what systemd actually launched.

2 · Back up before you touch anything. The application directory, the systemd config, a separate SQLite rollback copy, and a full container backup.
sqlite3 /path/to/freeapi.db ".backup '/path/to/rollback.db'"
:mouse_trap: The trap: a filename is not a backup. Restore it somewhere and look at it before you trust it.

3 · Keep the original, install beside it. /opt/freellmapi-original/ and /opt/freellmapi-v0.13.10/. Nothing is deleted, nothing is overwritten.

4 · Fetch the exact release. The tag you chose — not the moving branch.

5 · Install dependencies and build. A successful build is a step, not a result.

6 · Mind the migrations. A release that reshapes the database can leave the old binary unable to read it. App and database are one unit — that is why step 2 exists.

:white_check_mark: Then verify twice, because once is not enough

First, the service: the unit is active, and /livez and /readyz both answer HTTP 200.
Then, the real path: check it the way your users reach it, over the network — not just from inside the box. SSH access to the upgraded instance answered 200 in my run.

:mouse_trap: The trap that catches monitoring setups: treating anything that is not 200 as “down”. They are different conditions.

  • 401 — the server answered, authentication is required
  • 403 — the server answered, access is forbidden
  • 404 — the server answered, that path is not there
  • 500 — the server answered, and it has an internal error
  • no response at all — that is the only real “cannot reach it”

A 401 proves that authentication is required, not that the service is down. A 200 on a health endpoint proves the service answers, not that every API call works.

:receipt: What was actually proven, and what was not

Verified: new tag checked out · dependencies built · original source kept · existing database retained · migrations completed · integrity check passed · systemd pointing at the new version · service active after cutover · /livez 200 · /readyz 200 · network path 200 · old instance still reachable · rollback materials preserved.

Not yet proven: every provider integration · every model answering · compatibility with every existing API client · recovery after a host reboot. Those are separate tests, and saying so is part of doing the job properly.

:prohibited: What I would not do

  • Overwrite a working directory without a backup
  • Treat a successful build as a finished upgrade
  • Treat 200 as proof that everything works, or 401 as proof that nothing does
  • Run migrations without a rollback copy
  • Swap a systemd unit without preserving its original
  • Delete the old installation right away
  • Run package cleanup just before a critical reboot
  • Expose admin interfaces to the internet to make testing easier
  • Paste environment files, API keys or setup codes into troubleshooting logs
  • Call the upgrade done before the running service and its database were both checked

:fast_reverse_button: And the way back, told honestly

Rollback means: the old directory, the old systemd config, the database copy, the container backup, and a note of the new override. Two truths to hold:

1 · An older app against a migrated database can fail. They move together.
2 · Restoring an old database discards everything written since that copy. Rollback is a controlled recovery, not a git checkout.

:link: The pieces

:bullseye: Do it once. Document it. Verify it. Keep a way back.

2 Likes