Distributed rendering

1Preview-then-handoff

Spectral path tracing through a high-index gemstone with dozens of internal bounces is expensive, and streaming every interactive frame to a remote machine would add visible input latency. Indicatrix instead renders locally while the camera moves, and hands off to its one configured remote endpoint once the camera settles: the client sends a compact SceneState (serialized with postcard) and the remote streams traced radiance back, which is folded into the client's own accumulation buffer. That endpoint is always a coordinator, indicatrix-worker serve, which renders the request itself, splits it across render workers that have joined it, or both (section 3); the desktop app never talks to a worker directly.

The settings dialog's Live Compute choice decides what happens at the settle: Local only, Remote only, or Local + Remote. With Local + Remote both sides trace the same image against one shared sample budget — the live view's single target sample count, there is no separate remote setting — through one sample cursor per settle (LiveEpoch, apps/indicatrix-cut/src/bridge/sample_cursor/live.rs), so their samples are combined, never duplicated. The remote side claims short chunks sized to about 1.5 s of its measured rate (8 samples for the first chunk, before a rate is known; LIVE_CHUNK_TARGET_SECS, bridge/remote/live_lane), so a camera drag wastes at most a second or two of remote time. A chunk that fails keeps the prefix the remote already traced and returns the untraced remainder to the local tracer; after two consecutive failures the remote sits out the rest of that settle. Each settle is stamped with the scene it was dispatched for, so a scene change during the handoff can never merge samples of the old scene into the new image.

2Sample-range additivity

Splitting a frame into screen-space tiles load-balances poorly for a gemstone render: background tiles finish almost instantly while tiles covering the stone itself carry the full bounce budget. Because each sample's RNG seed is a deterministic function of pixel index and sample number, sample contributions are additive and order-independent, so Indicatrix instead partitions work by sample-index range across the whole frame — one node traces samples [0, 64), another [64, 128), and so on — with no tile-seam stitching and linear load balancing across heterogeneous machines.

Disjointness is the whole correctness condition, so it is enforced in one place: a SampleCursor in the GUI-free crates/indicatrix-dispatch hands out disjoint absolute sample ranges to every contributor of one image — the export's local and remote lanes, the live view's local tracer and remote chunks, and the coordinator's own lane and joined workers alike. Every backend also applies the same non-finite rule (add_finite_sample, crates/indicatrix/src/optics/raytracer/sampling.rs, with its twin in the GPU reduction): a sample with a NaN or infinite component is dropped but still counted, so one bad sample can never poison a pixel's sum and the divisor of a merge is always the number of samples actually traced. A statistical test renders one range on the CPU and the next on the GPU, merges them the production way, and checks the result against an independent CPU reference (renderer/gpu/merge_tests.rs).

3The coordinator and joining workers

indicatrix-worker serve is the coordinator (apps/indicatrix-worker/src/serve, src/coordinator). It always serves the read-only design library to viewers. On a build with the worker feature it also opens a separate worker port that render workers dial into, and it traces samples with its own CPU/GPU only when started with --render (--only-gpu/--only-cpu imply it); without that flag it acquires no GPU and traces nothing itself. Its WELCOME advertises exactly that: no render capability for a bare coordinator with no joined workers (the desktop app then sees a library-only remote and renders locally), the plain CPU or GPU backend for --render alone, and a Coordinator { workers, threads, gpus } backend as soon as a worker has joined. A capability change between requests is announced to the viewer with CAPABILITY_CHANGED.

A worker is indicatrix-worker join <coordinator-host:port> (apps/indicatrix-worker/src/join): it connects out to the coordinator's worker port over mutual TLS with a worker certificate, reports its render capability in its HELLO, and then serves the coordinator's requests on that connection. No inbound port is needed on the worker, so a machine behind NAT or a cloud VM can join. --slots K opens K parallel connections (default 1, at most 64), each serving one request stream at a time; every slot reconnects forever with jittered backoff, 1 s doubling to 60 s.

The worker port sits two ports above --bind and each enrollment listener one above its own port, on --bind's host unless overridden (WORKER_PORT_OFFSET, apps/indicatrix-worker/src/cli/mod.rs). Every listener binds loopback only unless --allow-remote is passed. --no-workers skips the worker port and its enrollment listener; --no-enroll skips both enrollment listeners.

How a request runs depends on its kind (src/coordinator/job). An export-type request — a still export, a tilt-video frame — becomes a job: a lane pool (indicatrix-dispatch's LanePool) over the request's sample range, with one lane for every idle joined worker whose pixel cap accepts the image plus the coordinator's own lane, each claiming chunks from the shared sample cursor sized to its measured rate. A live-view request stays on the own lane alone, the lowest-latency path; a coordinator without --render gives it the single fastest idle worker instead. If a worker dies mid-job, the samples it had not finished go back to the cursor and the remaining lanes trace them, so the viewer still receives every sample of the request exactly once. Only when no lane at all is left does the viewer get ALL_WORKERS_LOST, and it then renders the unfinished part locally. The coordinator pings every idle joined connection after 10 s without traffic and drops it after 30 s of silence (LivenessConfig, src/coordinator/registry.rs); a joined worker that hears nothing for 45 s reconnects (JOIN_IDLE_TIMEOUT).

Two advanced flags change the live-view rule: --interactive-workers N also gives each live-view request up to N of the fastest idle joined workers (default 0), and --pin-interactive-worker <label> prefers one named worker while it is connected, idle and accepts the image. Throughput jobs — an export streamed final-only, or any final-picture request — wait in a per-viewer queue, one active job per viewer certificate, and all in-flight jobs share one memory budget, --max-job-memory-mib (default 2048; each job is charged width × height × 48 bytes, and a job past the cap is refused, not queued).

4What travels on the wire

• Lossless result compression. A radiance delta (FRAME/PREVIEW payload) can travel compressed, negotiated per connection: the client lists the encodings it can decode in its HELLO and the server picks the first entry of its own preference list the client accepts (negotiate, crates/indicatrix-net/src/messages/encoding.rs). The default preference is a byte shuffle (four planes: every value's first byte, then every second byte, …) followed by Zstandard level 1, then shuffle plus LZ4, then raw; a loopback peer gets raw only, and raw is also the fallback whenever the two sides share nothing. Each frame header names its own encoding, so a payload that would not shrink is simply sent raw. The decoded bytes are bit-identical to the raw f32 buffer, NaN payloads and signed zeros included. On captured real frames the default came out 1.54–1.99× smaller — 35–50 % fewer bytes on the link — encoding at 465–621 MB/s and decoding above 1 GB/s on one laptop core. Decoding is bounded against decompression bombs: the declared raw length must equal width × height × 12 before anything is decoded, the output buffer is exactly that size, and output that is short or would overflow it is rejected (radiance/payload.rs). Both hops negotiate it: viewer to coordinator, and coordinator to each joined worker.
• Final picture only. The export dialog and the tilt-video export have a Transfer choice (its default is set in the Remote Coordinator form): Full data sends float radiance that is merged with the local lanes; Final picture only has the coordinator render, tone-map with the desktop export's own function (tonemap_accumulation) and send one PNG per image, the same PNG the desktop would have written from the same samples, while the local lanes stay idle. The Live Transfer setting offers the same trade for the viewport: Final picture streams finished, denoised 8-bit display frames (denoised on the coordinator with the desktop's own À-Trous denoiser, its guides from the coordinator's own primary-ray prepass) instead of float deltas. A finished picture cannot be summed with anything, so this mode is remote-only by construction: the local GPU/CPU pauses after the handoff. A remote that does not implement it answers UNSUPPORTED_REQUEST, and the app falls back to full data with one note.
• HDR environment maps by content hash. A scene lit by a loaded HDR panorama names the map only by the SHA-256 of its file bytes (crates/indicatrix-net/src/messages/asset.rs). A node that lacks it answers NEED_ASSET; the requester sends the bytes once (ASSET, at most 256 MiB, checked before anything is allocated); the receiver verifies the hash and keeps the bytes in a bounded on-disk cache (asset-cache next to --db or --cert-dir, or $INDICATRIX_ASSET_CACHE_DIR; size cap $INDICATRIX_ASSET_CACHE_MIB, default 2048 MiB), and a later request naming the same hash is never asked again. The coordinator asks the viewer once and forwards the map to each joined worker that asks. A remote advertises HDR support in its WELCOME only when some lane can render it, and HDR jobs only go to workers whose own cache works. For a remote without HDR support, or a map that cannot be sent (not loaded from a file, or over the size limit), HDR scenes render locally, with one status note.

5Mutual TLS, roles and token enrollment

The wire protocol lives in crates/indicatrix-net, typed message framing with no networking of its own. indicatrix-worker's serve and join commands carry the actual sockets and TLS.

• TLS 1.3 only, via rustls with the ring crypto provider; TLS 1.2 is excluded from the feature set at compile time, not merely disabled at runtime (tls::server_config/tls::client_config).
• Mutual authentication: both sides present a certificate signed by the same private CA (indicatrix-worker cert subcommands manage that CA). The coordinator's own certificate is used on both of its ports.
• Every client certificate carries a role: a worker certificate's Common Name is worker:<label>, a viewer's is the plain label (which may not start with worker:). Each listener checks the role and its own fingerprint allowlist — allowlist-viewers.txt and allowlist-workers.txt next to the CA, with a pre-role allowlist.txt still honoured as the viewers list while it is the only one — so a viewer certificate is never accepted on the worker port, nor a worker certificate on the viewer port. The HELLO carries the peer's role too, and a mismatch is refused with ROLE_REFUSED. --trust-any-client-cert skips both allowlists, never the role check.
• One-time token enrollment (the wire protocol and claim-side verification in crates/indicatrix-net/src/enroll; the server-side registry that mints and expires tokens in apps/indicatrix-worker/src/enroll): indicatrix-worker cert issue-token asks a running serve process's enrollment listener, from a loopback connection only, to mint a 180-second, single-use token for a viewer, or with --role worker for a worker. The desktop app's Redeem button, cert claim, or join --token redeems it — verifying the listener's certificate chain against the CA fingerprint carried inside the token itself, before sending the token's secret — and writes the same three-file certificate bundle cert issue-client would, without a manual copy. A listener refuses a token request for the other role.
• Plaintext serving (--insecure-no-tls) is available for local debugging, refused on any non-loopback bind address, and disables the worker port and both enrollment listeners.
• Every connection negotiates a protocol version during its HELLO/WELCOME handshake (PROTOCOL_VERSION, currently 14, crates/indicatrix-net/src/messages/mod.rs), together with a build hash and, where both sides can establish it, a content hash of the indicatrix source tree; a mismatched peer is refused outright (BUILD_MISMATCH) rather than risking a misread frame or a merge of samples from different physics.
• Every message is length-prefixed and bounded before a payload is even allocated (framing::MAX_FRAME_LEN, 512 MiB; the enrollment listener caps its own request further, at 4 KiB), and a just-accepted socket carries a 20-second read/write timeout through its TLS handshake until its first HELLO arrives (HANDSHAKE_TIMEOUT, serve/socket.rs) — an idle or slow-loris connection cannot pin a thread indefinitely. --max-connections (default 64) bounds authenticated connections, counted separately for viewers and joined workers, and a second limiter at four times that cap bounds connections still in their handshake.

6The indicatrix-worker CLI

indicatrix-worker has four subcommands: render (one-shot scene-to-PNG, no networking), serve (the coordinator: the design-library protocol over TLS, the worker port, and with --render its own render lane), join (a render worker dialling a coordinator), and cert (manages the private CA: init, issue-server, issue-client, issue-token, claim). render, join and serve --render need a build with the worker feature (gpu implies it). Flags below are copied from the CLI's own --help text in apps/indicatrix-worker/src/cli/mod.rs.

indicatrix-worker render \
  --scene scene.json \
  --out render.png \
  --width 3840 \
  --height 2160 \
  --samples 8192 \
  --threads 16

--only-gpu / --only-cpu force a backend; the default is a hybrid split across both when it measures as worthwhile.

indicatrix-worker serve \
  --bind 0.0.0.0:7878 --allow-remote \
  --ca pki/ca.pem --cert pki/server.pem --key pki/server.key \
  --render

--bind defaults to loopback only (127.0.0.1:7878); --allow-remote is required for any other address, on any of the four listeners. --db points at the design-library SQLite file (default facet_diagrams.sqlite in the working directory). Leave out --render for a coordinator that only distributes work to joined workers. --threads and --only-gpu/--only-cpu govern the own lane; --max-connections defaults to 64.

indicatrix-worker join coordinator.local:7880 --slots 2

The address is the coordinator's worker port (also accepted as --coordinator). --cert-dir names the worker certificate bundle (ca.pem, client.pem, client.key; default worker-cert in the working directory); --threads and --only-gpu/--only-cpu work as for render. With one GPU, more than one slot mainly helps CPU-only machines.

7Setting up a coordinator, its workers and the desktop app

# 1. create a CA
indicatrix-worker cert init --dir pki/

# 2. issue the coordinator's own certificate (used on the viewer and worker ports)
indicatrix-worker cert issue-server --dir pki/ --host coordinator.local --ip 192.0.2.10

# 3. start the coordinator; drop --render if this machine should not trace samples itself
indicatrix-worker serve --bind 0.0.0.0:7878 --allow-remote \
  --ca pki/ca.pem --cert pki/server.pem --key pki/server.key --render

# 4. mint one-time, 180-second enrollment tokens (loopback only)
indicatrix-worker cert issue-token --ca pki/ca.pem --admin-addr 127.0.0.1:7879 --name my-laptop
indicatrix-worker cert issue-token --ca pki/ca.pem --admin-addr 127.0.0.1:7881 --name gpu-box --role worker

# first run: claim the worker token into worker-cert/, then join
indicatrix-worker join --coordinator coordinator.local:7880 --token GW1-...

# every later run reuses the claimed bundle
indicatrix-worker join coordinator.local:7880

serve's viewer enrollment listener defaults to the same host as --bind, one port up, and the worker enrollment listener to one port above the worker port (--enroll-bind and --worker-enroll-bind override them), but cert issue-token only ever succeeds from a loopback peer regardless of that address — the operator runs it on the coordinator itself, not remotely. join --token claims one port above the worker port it was given (7881) unless --enroll-addr says otherwise; a token is single-use, so later runs leave it out. For an air-gapped machine, cert issue-client --role worker writes the same bundle for copying by hand.

In indicatrix-cut, open the Remote panel (Remote Coordinator) and choose Set up remote coordinator: enter the coordinator's viewer address (coordinator.local:7878), paste the enrollment address (coordinator.local:7879) and the printed token, and press Redeem. The client redeems it (cert claim's wire equivalent), stores the returned certificate bundle, fills in the certificate folder, and completes the mutual-TLS handshake on the next Test connection or render. The same form holds the live stream preferences and the default export Transfer; the app keeps exactly one such endpoint, and a settings file from an older build that listed several workers keeps only the first.