Wyrd

Prior art & foundations

Wyrd did not appear from nowhere. Its shape descends from a known lineage of distributed storage systems, it was designed in deliberate contrast to its closest open-source sibling, and it consumes well-solved commodity primitives rather than reinventing them. The sources below are grouped by the role each played.

Architectural lineage

The systems whose architecture Wyrd follows: metadata and bulk data kept separate, dumb storage servers, intelligence in a thick client and background maintenance services, a single metadata commit as the point of atomicity.

  • Colossus — Google's file system — The canonical public description of the architecture Wyrd descends from: a horizontally scalable metadata service (Curators), dumb “D” file servers, client-side erasure coding, and a direct client-to-server data path.
  • Colossus — data placement — The companion post on how Colossus places and tiers data across storage media for performance and cost.
  • Tectonic — Meta's exabyte-scale filesystem (FAST '21) — The strongest single influence on Wyrd's metadata model. Its layered metadata schema and the authors' account of what they would do differently directly informed the inode/dirent design and the choice of a multi-key-transactional metadata store (see ADR-0008).
  • Tectonic — paper (PDF) — The full paper.

Prior art we studied

Systems we engaged with seriously — the closest sibling, and the incumbents the “why not just harden the existing thing?” question must be answered against.

  • Garage / Deuxfleurs — The most mission-aligned open-source sibling: Rust, sovereign, self-hostable, geo-distributed, publicly funded. Garage makes the opposite core trade-offs — eventual consistency with CRDTs and whole-copy replication, by deliberate design — where Wyrd chooses strong consistency, a linearizable commit point, and erasure coding. Kindred in values, complementary in design.
  • Garage — design goals — Their goals page, which states the no-erasure-coding, no-POSIX, eventual-consistency non-goals explicitly — the clearest evidence of where Wyrd's design center diverges.
  • Garage — source
  • Garage — NLnet/NGI grant — A model for the European sovereign-infrastructure funding path.

Conceptual influences

Ideas, not systems, that shaped specific decisions.

  • Delta Lake — The commit-point idea in a different domain: a single linearizable metadata operation makes a write visible, with data staged invisibly beforehand — the conceptual ancestor of Wyrd's atomic commit.

Foundations we build on

The hard, well-solved commodity primitives Wyrd consumes rather than rebuilds. The novelty budget is spent on the commit protocol, not on re-solving consensus or transactional storage.

  • Rust — The implementation language for all v1 components: memory safety and no GC tail latency in the paths where bugs become silent corruption, plus access to the Rust deterministic-simulation-testing ecosystem (see ADR-0004).
  • Protocol Buffers + gRPC — The versioned wire-contract foundation for internal services. Wyrd defines protobuf contracts and uses the Rust tonic / prost stack so components can be replaced behind stable RPC boundaries (see ADR-0010 and architecture §8.6).
  • etcd — Coordination: service discovery, leader election, locks with fencing (see ADR-0006).
  • TiKV — The default L2 metadata backend (Apache-2.0, CNCF), behind the MetadataStore trait; native multi-key transactions are why it was chosen over an HBase-style store. TiDB adds a SQL surface on the same engine; YugabyteDB or PostgreSQL are alternatives behind the trait (see ADR-0008).
  • redb — The embedded metadata backend for the single-binary and local profiles: small, Rust-native, and deliberately replaceable behind the same MetadataStore seam as TiKV (see ADR-0008).
  • step-ca / Smallstep — The production internal CA chosen for the mTLS fabric now: self-hosted, short-lived, auto-rotated certificates without paying SPIRE's operational cost before fleet scale (see ADR-0036).
  • SPIFFE / SPIRE — The reserved workload-identity upgrade path. Wyrd shapes service identities like SPIFFE IDs from day one, but keeps the full SPIRE server/agent topology deferred until secret-less workload attestation is worth the operational weight (see ADR-0025 and ADR-0036).
  • OpenTelemetry — The single instrumentation commitment; storage and visualization backends stay the operator's choice (see ADR-0012).
  • Prometheus — The scrape-compatible observability edge Wyrd exposes alongside OTLP, and part of the recommended reference stack; the binary still hardcodes no telemetry storage backend (see ADR-0012).
  • madsim — Deterministic simulation testing for async Rust (see ADR-0009).
  • turmoil — Deterministic network simulation for testing (see ADR-0009).
  • reed-solomon-simd — The erasure-coding library used in the client data path (see ADR-0004).
  • CRC32C — The default fragment and header checksum for the v1 chunk format: fast, hardware-accelerated integrity checking for reads and scrub, with cryptographic integrity reserved separately (see ADR-0019).