The GPU fleet and isolation

Episodes, kits, brand suites, translations, clips and films are rendered on GPU machines that ChannelForge starts when there is work and stops when there is none. Jobs of different workspaces can share a machine. This page explains how machines are chosen, how their time is shared fairly, and the walls that keep each job apart from every other.

The machines#

• NVIDIA L4 GPUs, in Google Cloud and AWS, in regions in Europe, the United States and Asia. Having several clouds and regions keeps work moving when one runs out of GPUs, which happens often.
• Slots. Each machine runs a few jobs side by side, one per slot. A long-form film takes a whole machine of its own, never shared.
• Interchangeable. Every machine is built from the same recipe and keeps nothing between jobs: a job's files are laid out for it and removed after it. The pipeline's code is installed on a machine, per version, the first time a job needs it; a job always runs the version it started with.
• Given as little as possible. A machine holds no cloud, model or platform credential, has no cloud service account, and accepts remote sessions only from ChannelForge's control plane.

Starting and stopping#

One leader of the fleet decides, every few seconds:

1. Demand — the GPU jobs waiting that may run now: their workspace is active and under its plan's number of GPU jobs at once.
2. Scale — when there are more of them than free or arriving slots, the fleet starts the stopped machine with the lowest expected cost an hour: its price, plus the risk that the cloud takes it back (a spot machine's pre-emptions, measured machine by machine) times what that would lose, since a pre-emption loses every job on the machine. A quiet spot machine goes before an on-demand one; an on-demand one before a machine that keeps being taken back.
3. Assign — free slots on healthy machines go to the next jobs by fair share (below).
4. Idle — a machine with nothing to do for a few minutes is stopped. GPUs run only while there is work.
5. Limits — a job running past its kind's time limit is stopped.

A machine that fails to start waits out a back-off before it is tried again; when a zone has no GPUs left, the next machine is tried and an incident is opened. There is a fixed ceiling on how many machines run at once.

Sharing fairly#

Free slots are given out in this order:

1. Plan tier first — a higher plan's jobs before a lower one's.
2. Then fair share — the workspace that has used the least GPU time over the last 24 hours, for the weight of its plan, goes next. Time is counted in slot-seconds, so a film's hour on a whole machine counts as every slot's hour.
3. Then age — the oldest job.

No workspace can starve another, and none runs more GPU jobs at once than its plan allows — the number for each plan is on Pricing. Paid jobs that wait more than half an hour for a GPU open an incident for the operators.

Films need a whole machine. A film that finds none earmarks the cheapest suitable machine and starts it — or, when no more machines may start, lets the running machine whose jobs end soonest drain; a running job is never cut off to make room. While short episodes are waiting, films may hold only part of the film-capable machines: shorts feed posting slots; a film is not time-critical.

Keeping tenants apart#

Several workspaces' jobs can run on one machine at once. These walls stand between them, for every workspace:

• Its own user. Each slot runs as its own unprivileged user. A job's folder can be entered only by that user; no other slot can list the jobs on the machine, let alone read one. The pipeline's code below the folder is shared read-only and is the same for every job.
• Its own processes and its own temporary files. Each job runs in its own process and mount namespaces: it sees and can signal only its own processes, nothing of it outlives the job, and its temporary folders and shared memory are private to it.
• No network while it runs code a writer produced. A slot running an episode, a kit, a brand suite or anything else an agent writes code for has no internet access and no name lookups — only its own encrypted tunnels to the model gateway and the voice service, each of which still requires the job's own token. A film, which has a machine to itself, runs under a user whose outbound traffic is closed in the same way. The few jobs that run only ChannelForge's own code (joining a compilation, rendering a clip of your footage) keep outbound access to read the member's media from short-lived signed links. No job can reach the cloud's metadata service or a private network. A slot that cannot be closed does not run the job, and that failure is not charged.
• Credentials that are useless elsewhere. A job's tokens name its workspace, the job and the attempt, cap what it may spend, and expire. The gateway and the voice service refuse a token as soon as its job is no longer running on that attempt. A narration token is not a model token, nor the reverse.
• No tenant value reaches a shell. Every value a tenant can influence (a colour, a topic, a name) is quoted at every level of the job's command line, so it can never be run as a command on the machine — tested with exactly such a payload.
• Clean before and after. The slot user's home, temporary files and processes are wiped before a job starts and after it ends, and the job's folder is removed whatever happened.
• No secret in the logs. A token a machine prints is redacted before its job's log reaches the job's page.

These walls were proved in a throwaway container that tries to cross each one, then switched on for one real job of every common kind, and only then for every workspace.

Lost machines and time limits#

• A machine taken back or crashed. The job's lease runs out, the job goes back to the queue and is retried more patiently than an ordinary failure, and it is not charged. A film resumes from its last checkpoint.
• A fleet restart. Each GPU job runs in its own runner process, so restarting the fleet (a deploy) does not stop running jobs; the new fleet adopts every job whose lease is fresh.
• A job that runs too long. Each kind has a limit, measured from the start of the current attempt — a short episode's is three and a half hours, a film's making nine — after which the job is stopped.

What GPU time costs you#

A job's GPU time is its slot-seconds at the machine's hourly price divided by its slots, then at your plan's rate. It is a small part of an episode's price: most of a standard episode's cost is the writer's and the critic's model time and the narration. A failure on our side, a lost machine and a held episode cost nothing.

Related#

• The episode pipeline — what a job does on its slot.
• Security — the same walls among the rest of the protections.
• Reliability and recovery — what happens when a machine or a cloud fails.
• Credits and plans — GPU jobs at once, and how charges are worked out.