0045. Metadata validation boundaries: parse-don't-validate at decode, liberal read / strict maintenance
Context
Several operational paths today trust a metadata record the moment JSON decoding
succeeds. That is safe only while every record is produced by current, correct,
in-process Wyrd code. It stops being safe once we account for old on-disk
versions, bugs, manual repair, partial corruption, and — imminently — a
distributed backend: TiKV (ADR-0008) and now FoundationDB (ADR-0042) end the
store's process-privacy, so records become network-visible and writable by
non-Wyrd tooling, and proposal 0014's M7.3 restores metadata from backup into a
custodian verify pass, driving possibly-inconsistent restored records through
every one of these boundaries. Two concrete failures already surfaced and were
fixed piecemeal: #285 (invalid EC metadata such as k = 0 panics lower layers)
and #290 (the read path preallocated directly from an untrusted inode size).
The systematic posture, however, already exists for exactly one field. ADR-0040
built it for placement: classify-before-use (placement_is_valid /
checked_fragments), a typed error (MalformedPlacement), a liberal-read
/ strict-maintenance asymmetry (serve what you safely can; never let a
maintenance loop act on or rewrite a record it cannot trust), NEEDS-HUMAN
emission, and GC treating an unparsable reference as referenced (fail-safe).
The open question is not what posture to invent — it is to generalize
ADR-0040's posture across all metadata records, and to pin the boundary so it
cannot silently regress.
Decision
We adopt a single validation posture for all metadata, generalizing ADR-0040 (which it refines, not supersedes), answering the four questions #291 poses:
- Parse-don't-validate at decode, for structural invariants. A validated
metadata::decodepath returns a typedMetadataValidationErrorfor any record that violates a structural invariant; a value that decodes is structurally trustworthy thereafter. Contextual checks that need external state (placement-vs-fleet, domain membership) — and, explicitly, placement length, which stays under ADR-0040's liberal-read rule — remain at the operation boundary, never at decode, so a still-readable record is never turned into a read fault (ADR-0040 decision 4). - Post-validation assumptions are documented per type. After a successful
validated decode, the erasure math MAY assume the scheme is
erasure::supported(sok ≥ 1andm ≥ 1); indexing MAY trustfragment_count(); each type records which invariants downstream code is entitled to assume, so the assumption and its guarantee live together. - Fail-closed surfacing is the ADR-0040 asymmetry, generalized. The read path MUST surface a typed read fault, never panic (the #285 class). Maintenance loops (custodian, repair, GC sweeps) MUST classify, skip, and emit NEEDS-HUMAN — never act on or rewrite a malformed record. GC MUST fail safe (treat an unparsable reference as referenced), never reclaim on doubt.
- The boundary is pinned by an adversarial decode corpus. A malformed-record
table —
k = 0,m = 0(and any!erasure::supported(k, m)pair),k + moverflow, size↔chunk-map mismatch, wrong-length placement, absurd size,u64::MAXversion/lease — is run at Tier-0 againstdecodeand each boundary, applying theinvalid/conformance-vector pattern to metadata.
The structural invariants to pin, by record:
| Record | Invariants |
|---|---|
EcScheme::ReedSolomon |
erasure::supported(k, m) — the exact predicate the read path already applies (read.rs), so k ≥ 1 and m ≥ 1 (a stored rs(k, 0) is as invalid as rs(0, m)); k, m additionally bounded to the format header widths (ec_k/ec_m are u8, ec_fragment_index u16) so metadata can never describe a chunk the format cannot encode |
ChunkRef |
scheme validity; logical length consistent with the scheme. Placement is not a decode-time invariant — it stays under ADR-0040 decision 4's liberal-read / strict-maintenance split (reads keep the per-index identity fallback, so a still-readable record is never faulted over a length quirk; only maintenance gates on a malformed placement). Placement length is a boundary check, never a decode failure |
InodeRecord |
size ↔ chunk-map consistency; state validity; checked arithmetic on version increments |
PendingEntry / repair / desired / orphan |
lease-timestamp checked arithmetic; parseable keys (the #283 strict-parsing sibling) |
| read reassembly | allocation caps before buffering (the #290 pattern, generalized) |
Consequences
Metadata is hardened against every untrusted-input path the distributed backend, restore-verify, old versions, and corruption open — the read path degrades to a typed fault instead of a panic, and maintenance loops quarantine rather than propagate bad records. The posture is one rule, generalized from ADR-0040, so there is a single mental model rather than per-field improvisation. The cost is validation work on the decode path, kept bounded by validating only structural invariants there and leaving contextual checks at operation boundaries.
This review produces the implementation issues #291's contract calls for; two are
already visible and filed against this ADR: (i) the validated-decode layer with
its typed MetadataValidationError and the adversarial decode corpus; (ii)
checked arithmetic for version increments and lease timestamps. Any further gaps
the implementation surfaces are filed the same way.
This should land before records go network-visible on the production path (the FoundationDB backend, #438) and before M7.3's restore-verify pass exists, since both make the untrusted-input assumption real. It commits the project to a typed metadata-validation error surface and the liberal-read / strict-maintenance asymmetry as standing rules; reversing toward trust-on-decode would reopen the #285 panic class on any non-current record. Refs ADR-0040 (the posture template), #285/#290 (closed inputs), #283 (strict-parsing sibling), #438 (FDB consumer).