5. Building block view

Living document. The system is described through two complementary views: the actor view (§5.1 — what the moving parts are and how they collaborate to perform an operation) and the layer view (§5.2 — how those parts map onto scope tiers, and why bulk data never crosses a shared component). They answer different questions and are both load-bearing: the actor view is how you learn the system; the layer view is how you understand its scalability. See diagrams/c4-container.mermaid for the container diagram (the actor view) and diagrams/layer-stack.mermaid for the layer stack.

Why two views

The actor view and the layer view are two decompositions of the same system, cut along different axes:

• The actor view answers "what are the components, and how do they collaborate to do a write?" It is the operational and learning frame — the one you reach for to explain the system or reason about a request. Its centre is the commit point.
• The layer view answers "what scope does each part have, and why doesn't bulk data bottleneck on a shared component?" It is the scalability frame — the one that justifies the whole architecture. Its decisive property is that the globally-scoped tier holds only tiny, high-value truth.

Neither subsumes the other. A component's identity and collaborators come from the actor view; its scope and the scaling consequences come from the layer view. The sections below give each in turn.

5.1 The actor view — the components and how they collaborate

The system is, at its core, five components orbiting a single commit point. Everything else (gateways, coordination, replication) is in service of these.

The metadata store sits at the centre because the atomicity guarantee lives in one mutation there; the other four components collaborate around it.

Holding these together, off the data path: coordination (L5) — service discovery, leader election, locks — where D servers register and the custodian leader-elects, so the actors can find one another.

How they collaborate — the write path as actor interaction

The single most important collaboration, and the one that defines the system:

1. The client registers chunk ids in the metadata store's pending-chunk ledger (leased; the file exists nowhere yet).
2. The client erasure-codes the data and writes fragments directly to the D servers — bulk data flows client→D-server, crossing no shared component. The D servers verify checksums. A failure here is harmless, unreferenced garbage.
3. The client issues one atomic mutation to the metadata store: write the chunk map, set COMMITTED, bump the version (conditional on the prior). This is the commit point — the file now exists. Concurrent writers conflict here; exactly one wins.
4. The client releases the ledger entries.

Later and asynchronously, the custodian scrubs the D servers' fragments against the map, rebuilds any that are lost (committing the new locations via the same atomic mutation pattern), and emits durability telemetry; the replication worker (multi-zone) copies committed chunks to other zones. Readers never see a hybrid — only the old version or the new.

The shape to carry away: dumb storage at the bottom, a thick client and background custodians holding the intelligence, and a metadata commit point at the centre as the locus of atomicity — with cross-zone replication off to the side. The layer view (§5.2) now shows how these actors map onto scope tiers.

5.2 The layer view — scope tiers and the scalability property

The system is organized into five layers. L4 and L5 operate within a single zone (one datacenter) and are unaware that other zones exist. L1, L2, and L3 are the layers that span zones. The decisive property throughout: the layer with global scope (L2) holds only small, slow-changing, high-value truth, so it can afford global consensus without becoming a bottleneck, while bulk data flows directly between client and storage servers, crossing no shared component.

L1  Access layer            S3 / FUSE / WebDAV / SDK gateways      (stateless)
L2  Global control plane     namespace, placement, ACLs, registry   (global consensus)
L3  Cross-zone replication    async chunk copy, global custodians    (off the data path)
L4  Zonal file system         metadata store, D servers, custodians, client library
L5  Bootstrap & coordination  service discovery, leader election, locks, config

Where the §5.1 actors live: the client is defined in L4 but embedded in the L1 gateways; the metadata store, D servers, and custodians are L4 (zonal); the replication worker is L3 (cross-zone); coordination is L5. The actors are unchanged — this view re-cuts them by scope.

L1 — Access layer

Stateless protocol translators. They hold no durable state, scale horizontally, authenticate the caller, and embed the client library (L4) to do the actual storage work. All consistency logic lives below them.

L2 — Global control plane

The brain of the multi-zone system: the one component with a global, authoritative view of what exists and where it should live. It controls by deciding and recording; it distributes by routing actors to the right zones, never by carrying the payload itself.

Holds metadata about metadata — kilobytes per file. Consulted on namespace and placement operations (create, rename, share, delete, "where do I read this"), which are rare relative to data operations and ideally cached by clients.

The default L2 backend is the Apache-2.0, CNCF TiKV stack — TiDB for the SQL surface over it — with YugabyteDB or PostgreSQL as alternatives behind the NamespaceStore trait (ADR-0020). CockroachDB is Spanner-class and works as a backend, but it is not a default: it moved to a source-available (non-OSI) licence in 2024, which is at odds with the project's fully-open, sovereignty-friendly substrate goal.

L3 — Cross-zone replication

Turns independent single-datacenter systems into one geographically distributed system, without ever sitting on the foreground write path.

Default replication is async (home-zone commit acknowledges immediately). Sync N-zone replication is a per-tenant opt-in that eliminates the data-loss window at the cost of cross-region write latency.

L4 — Zonal file system (the Colossus analog, one per datacenter)

Where atomicity lives and where the bulk of correctness risk concentrates.

A zone is the unit of atomicity and the unit of (intra-provider) federation.

The metadata model

Hierarchical: inode + dirent, not path-as-key.

• inode:<id> → attributes, chunk map (or inline data for small files), state, version, and the object metadata the S3 surface serves (ADR-0047): the content digest (etag, an opaque change-token — the lowercase-hex SHA-256, not MD5), the writing client's declared content_type, and the content-publication time (modified). Each is optional with a serde default and skipped when absent, so a record written before the fields decodes, degrades on the wire to the pre-metadata behaviour, and re-encodes byte-identically — the latter load-bearing because every commit is a full-value CAS on the record's stored bytes.
• dirent:<parent_id>/<name> → child inode id.

Object metadata is stamped only at content publication (create / overwrite, in the same atomic commit as the chunk map). The repair paths — reconstruction, backfill, rebalance — re-commit the same content, so they preserve the stored trio: a placement-maintenance commit never moves Last-Modified or drops the content type.

This makes rename a single dirent mutation (atomic under the same mechanism as a write) instead of a mass key rewrite, and makes cross-zone sharing expressible (a dirent pointing at an inode owned elsewhere). It is the strongest concrete driver of the requirement that the metadata store offer an atomic multi-key transaction, because file creation must atomically write both the inode and its dirent. That requirement is what disqualified HBase-class stores and what every backend choice since has had to satisfy; the backend satisfying it in production is now FoundationDB (ADR-0042, superseding ADR-0008's TiKV). See section 6.

The store's read surface is two operations, and the split is a durability requirement rather than an optimization. scan(prefix) returns a whole namespace as one consistent cut and fails loud past its per-listing cap instead of truncating — a silently short inode: listing would shrink GC's never-reclaim safety set, which is data loss (#262, ADR-0011). scan_page(prefix, after, limit) walks a namespace larger than that cap, in cursor-keyed pages ordered by raw byte-lexicographic key, with an exclusive cursor and a continuation that may not skip a key present throughout the walk (#634). The second exists because two multipart-era populations outgrow the cap — the deliberately unbounded retirement drain, and GC's orphan ledger, where one maximum segmented-object retirement installs ~1.78 M marks against a cap of 1,048,576 — and a namespace that cannot be enumerated cannot be reclaimed. Both contracts are normative for every backend and asserted on all of them by the shared conformance suite (crates/metadata-conformance), which is what keeps redb, FoundationDB and TiKV from disagreeing about how a listing behaves.

The write protocol (the commit point)

The same collaboration described in §5.1 as actor interaction, here in terms of the L4 mechanics it touches:

1. Intent — client registers chunk IDs in the pending-chunk ledger with a lease. The chunks exist nowhere in the namespace.
2. Data path — client erasure-codes and writes fragments directly to D servers, which verify checksums. Failures here are harmless garbage.
3. Commit — one atomic metadata mutation writes the chunk map + sets state COMMITTED + bumps the version, conditional on the prior version. This is the atomicity. Concurrent writers conflict here; exactly one wins.
4. Release — delete the ledger entries. Crash between 3 and 4 leaves the ledger to a custodian sweep; crash before 3 leaves leased garbage for GC.

Readers never consult the ledger; they see the old version or the new one, never a hybrid.

L5 — Bootstrap and coordination

The smallest layer by data volume and the one everything else depends on. It answers the questions a distributed system must answer before it can do any real work. Stores kilobytes; never on the data path.

The Colossus-style bootstrap recursion bottoms out here: etcd depends on nothing above it. Losing L5 loses no data; established connections keep working from cached state. What is lost is the ability to react — failovers, elections, topology changes — until coordination returns.

openraft is reserved as a future embedded coordination backend (a no-external-dependency production mode) behind the same trait, validated by the existing DST harness once the trait semantics are pinned by two implementations. Not built for v1 (ADR-0006).

5.3 Cross-cutting components

The dependency rule (ADR-0010): implementations and consumers depend on traits, never on each other's concretes. Only server knows the concrete backends, which is what makes "swap redb for FoundationDB" or "in-memory for etcd" a composition change rather than a refactor. The seam earned its keep in M4: swapping the production backend from TiKV to FoundationDB (ADR-0042) touched the server wiring and added a crate, and changed no consumer.