Docs / Architecture
Reliability and recovery
How ChannelForge keeps running when a machine, a cloud or a platform fails: retries, self-healing, backups and standbys.
Things fail every day at this scale: a GPU machine is taken back by its cloud, a zone runs out of GPUs, a platform times out, a model is overloaded, a deploy restarts a service. ChannelForge is designed so that none of these loses work, charges you for our failure, or posts anything twice — and so that a person hears about what the system could not fix by itself.
What fails, and what happens#
| When this fails | What happens |
|---|---|
| a worker process stops mid-job | the job's lease runs out; within a few minutes it goes back to the queue, or fails after its last attempt — never silently lost |
| a GPU machine is taken back or crashes | its jobs go back to the queue and are retried more patiently than an ordinary failure, without charge; a film resumes from its last checkpoint; the fleet starts another machine |
| a zone or a cloud has no GPUs left | an incident opens and the fleet tries the next machine, in another zone or cloud |
| a service restarts for a deploy | GPU jobs keep running in their own runner processes; workers finish what they are doing first, and anything cut off returns through its lease |
| the model provider is overloaded | the gateway retries and falls back to another model; the job retries with back-off; if it still fails, it is not charged |
| the model credential is refused | a one-token check every 30 minutes notices it, and a critical incident is emailed to the operators; posting goes on from the episodes already ready |
| the narration service stops answering | it is checked every two minutes and an incident opens; narrated jobs wait and retry while it restarts |
| a platform refuses or times out | the post is retried or explains what to fix; the slot's other platforms go on; an incident opens when several posts fail within two hours |
| a platform's daily allowance is used up | its posts wait for the next day without using a try; the slot says why |
| the service is down at a posting time | the slot is recorded as missed, never posted late; the episode goes out on the next slot |
| the cache is lost | it holds nothing that matters: sessions, jobs and money are in the database; limits that protect money refuse rather than open while it is back |
| the application server is lost | a standby in a second cloud can take over within minutes (below) |
| the database is damaged | it is restored to the moment before the damage, from point-in-time recovery or an hourly copy |
The sentinel#
One leader of the sentinel watches the whole system once a minute, acts where it can, and opens an incident where it cannot.
- Jobs. Puts back jobs whose worker stopped renewing their lease; settles credit holds a crash left behind.
- Credits. Expires credits past their date and rolls unused plan credits over, hourly; sends the spending caps' alerts at 80 % and 100 %; tells workspaces that are running low while they generate.
- Capacity. Opens an incident when paid GPU jobs wait more than half an hour.
- Health. Checks the model credential and the gateway, the narration service and how much of the day's narration is used; opens incidents for jobs failing on our side (five or more in an hour, with their reasons) and for posts failing (three or more in two hours).
- Retention. Removes what has passed its time and deletes deleted workspaces' tokens and platform data on schedule (see Data, storage and retention).
- Telling people. Emails new serious incidents to the operators, once per incident and at most hourly, with what the system already did; writes the health snapshot behind the public status page every 30 seconds.
The fleet heals its own side too: a machine that fails to start waits out a growing back-off, a machine stuck starting is given up on, a machine left idle is stopped, and a job past its time limit is stopped.
Retries in one place#
| What | How it is retried |
|---|---|
| a job that failed on our side | after 30 seconds, 2 minutes, 8 minutes …, up to its attempts; then failed, not charged |
| a job whose GPU machine was lost | up to three more times (six for a film), not charged |
| a model call | the gateway retries after a second, then a few seconds, then falls back to another model |
| a webhook delivery | after 1 minute, 5 minutes, 30 minutes, 2, 6, 12 and 24 hours — 8 tries over about two days |
| a post | up to its attempts; one whose answer was lost is checked for, not sent again blindly |
| a machine that would not start | skipped for a while, longer after each failure |
Every retry is safe to make: jobs carry their attempt, money is settled once per job, posts and webhooks carry identifiers that are never sent twice as new.
Deploys without downtime#
- The web app runs as more than one instance and is restarted one at a time, each answering its health check before the next goes; open browser tabs keep working, because the previous build's files are kept for a week.
- A new web build is made before anything is switched: a build that fails leaves production as it was.
- After every deploy, every signed-in page is opened and a set of real forms is submitted the way a browser submits them; a form that fails stops the deploy being called done.
- Workers finish what they are running before they stop; GPU jobs are not interrupted, and each keeps the pipeline code it started with.
- Changes that need the database to change first are deployed in the order that keeps the old and new code working side by side.
Backups and recovery#
- The database can be restored to any moment of the last 7 days, to the second. An encrypted copy is also taken every hour and kept with a second cloud provider.
- Media is copied, encrypted, to a second cloud provider every hour.
- Code and configuration live in version control: what runs is compared with the repository every night, and on 1 November 2026 a fresh copy of the repository passed its tests and matched what runs, byte for byte.
- A standby of the application server in a second cloud can take over within minutes; the switch-over was rehearsed on 1 November 2026. When it takes over, the old server stands down by itself, so two copies never run at once. While it runs, a few features that depend on the main cloud — such as email and very large uploads — wait.
- GPU machines need no backup: they hold nothing between jobs and are rebuilt from their recipe.
- After any restore, the recovery procedure holds posting until someone has checked what the platforms already have, so a backup that is a few hours old cannot post those hours' episodes a second time.
What you can watch#
- The status page shows whether the web app and API answer, the job queue (queued and running), the GPU fleet (machines running and on standby — GPUs start on demand), the jobs done and failed in the last 24 hours, and open incidents.
- In the app, every job shows its live progress and, if it failed, why in plain words and whether it was charged.
- Webhooks tell your systems about failures as they happen:
job.failed,post.failed,film.failed,credits.low.
Related#
- The GPU fleet and isolation — lost machines and time limits.
- The publishing engine — why a post is never sent twice.
- Security — monitoring and how incidents are reported.
- Troubleshooting — what to do when something of yours fails.
Something missing or unclear? Tell us and we will improve this page.