Provider / S3-compatible gateway

Experimental

Apache Ozone

Apache Ozone is exercised through its S3 gateway rather than a new mount-rs core dependency. It is a useful object-store target, but the current all-in-one harness is a loopback test deployment, not a production-authenticated replicated cluster.

Metadata

Ozone supplies the object block plane only. Namespace metadata, revisions, leases, and chunk references remain in a separate provider such as SQLite, PGlite, TiDB, or FoundationDB.

Immutable blocks

Blocks are immutable S3 objects below a caller-owned bucket/prefix. Use the gateway's List/Head/Get/Range operations to inspect the object keys; the bytes are not stored in the Ozone metadata tables by mount-rs.

Storage anatomy

Follow one file through both planes.

A mount-rs namespace stores names, attributes, directory entries, layouts, and block references. The bytes stay in the independent block plane. The example below is illustrative; the provider-specific schema, keys, and prefixes are the inspectable contract.

Logical file/docs/readme.mdinode 42 · size 8,192NodeMetadata → NodeData::File(FileLayout)
blocksOzone bucket objects under an owned prefixBlockId → raw bytes
Illustrative Namespace / FileLayout record (shape, not serialized bytes)
Namespace {
  nodes[42] = NodeMetadata {
    stats: { ino: 42, size: 8192, mode: regular },
    data: File(FileLayout {
      chunker: { algorithm: "fixed-size", version: 1, chunk_size: 4096 },
      extents: [
        { file_offset: 0,    block: "b012...", block_offset: 0, length: 4096 },
        { file_offset: 4096, block: "bfed...", block_offset: 0, length: 4096 }
      ]
    })
  }
}

BlockId "b012..." -> [4 KiB raw bytes]
BlockId "bfed..."  -> [4 KiB raw bytes]

Object storage

Apache Ozone S3 gateway prefix map

block-only bucket

Ozone is observed through its S3 gateway. The current all-in-one harness uses loopback resources and keeps namespace metadata outside the bucket.

Bucketmount-rs
mount-rs/
└─mount-rs-tests/owned-run/
├─b0123456789abcdef0123456789abcdef
└─bfedcba9876543210fedcba987654321
selected relational/KV provider

Stores the namespace and the references that make each object part of a file.

blocksOzone bucket objects under an owned prefix

Stores only raw bytes at exact keys. LIST shows a prefix; it does not reveal a directory tree.

mount-rs-tests/harness scope

logical S3 prefix; no filesystem directory

mount-rs-tests/owned-run/owned prefix

LIST/cleanup boundary

mount-rs-tests/owned-run/b0123456789abcdef0123456789abcdefgateway object

raw immutable block bytes

mount-rs-tests/owned-run/bfedcba9876543210fedcba987654321gateway object

raw immutable block bytes

What the key does not tell you

The object key shape is real for the S3 gateway path, while hosted topology, authentication, and mixed-provider acceptance remain experimental. The file name, inode, offsets, and chunk layout are not encoded in the object key; inspect the metadata plane to map them.

Experimental — scoped evidence

Pinned 2.2.1 gateway and arm64 block/restart/CAS/range evidence exist; the latest exact-tip hosted packet is diagnostic after hard 1,000-IOPS misses, while customer topology, backup/DR, secure tenancy, and release gates remain external. The current maturity is Experimental: the real arm64 gateway harness has the block, CAS, range, failure, restart/reopen, and cleanup checkpoint above. Its SQLite/PGlite compositions passed through the real Ozone gateway, and a separate single-node TiDB/Ozone run passed the direct TiDB contract, ChunkedFs partial/truncate/CAS/stale-fencing/ reopen, ambiguous-commit, and cleanup checks. The arm64 Node provider and Node/Rust CLI run recorded pass=7 skip=1 fail=0. A durable FoundationDB/Ozone run also passed with three fixed-address FoundationDB 7.4.7 servers, persistent volumes, double/SSD configuration, transaction readiness across a replicated-node restart, fresh-client reopen, and owned cleanup. A dedicatedozone-tidb hosted job is wired for durable v8.5.7 TiDB with Node 24 against the real Ozone gateway, but the current replacement job is queued and has no result yet. A dedicated hostedozone-compositions job now installs PGlite, builds the public Node addon, and runs the real SQLite/PGlite mixed-metadata gate. Its opt-in consumer phase covers the Node provider matrix, Node CLI, and Rust CLI against live Ozone/PGlite/R2-compatible services, but the hosted result is still pending and revision-specific. Durable multi-node TiDB, secure/replicated Ozone deployment, and the broader backend matrix are not yet accepted. The retained W26 packet on9c098e5 / hosted run 35585066458 remains the last accepted scoped result. The newer hosted packet35635486040 completed all four configured provider lifecycles but failed the 1,000-IOPS gate; aggregate job106458415293 rejected the missing pass marker, so its artifacts are diagnostic only. W26's credential-free rollout contract fixes the Tier 1 99.99% availability objective, 1,000 IOPS per drive, five-minute RPO/RTO, TLS/SigV4, tenant-scoped prefixes, and three-node/three-replica/three-domain topology; these are customer/Ozone requirements, not proof of a deployed environment. Replacement run 35641941218 is terminal for W26 even though unrelated native jobs kept the parent workflow active. Its producer jobs 106473112920, 106473112915, and106473112409, plus aggregate106477164582, failed closed on the hard IOPS gate. The local overlap/read/shutdown regression suite is green, but this first remediation is insufficient; secure customer topology, backup/DR, measured SLO/capacity, and release/canary/rollback remain open. The current publication-barrier implementation adds a conservative provider capability that skips only the redundant post-publish flush probe while retaining explicit syncfs; the focused ChunkedFs suite passes 20/20 and the full locked workspace/Clippy checks are green. Fresh hosted run 35672928117 targets exact SHA fbc8346d, but its Ozone producers were queued or in progress and no aggregate existed at the recorded snapshot, so it is not acceptance evidence. The later terminal diagnostic packet35674425514 completed 1,200/1,200 lifecycle operations with zero timeouts and cleanup failures for every provider row, but measured only 637.01 IOPS for SQLite/R2, 457.65 for PGlite/R2, 147.31 for TiDB/R2, and 126.50 for FoundationDB/R2 against the hard 1,000-IOPS target; its aggregate failed closed without pass markers. The newer manual run 35683158821 targeted exact SHA14dbf2c6 and is now terminal but diagnostic: SQLite/R2 measured 754.59 IOPS, PGlite/R2 817.10, TiDB/R2 143.04, and FoundationDB/R2 345.17against the unchanged 1,000-IOPS target. Each provider completed 1,200/1,200 lifecycle operations with zero timeouts and cleanup failures, but the provider jobs failed the hard threshold and the aggregate job 106606577835 failed closed on missingOZONE_IOPS_PASS markers. No acceptance or production claim is made from this run. The newer terminal packet 35686340751 selected exact SHA1e64bc25: base Ozone passed, SQLite/R2 measured997.06 IOPS, PGlite/R2 592.43, and TiDB/R2277.59; FoundationDB produced no benchmark after its locked-preflight failure, and aggregate job106616062969 failed closed. A fresh manual dispatch35691451007 selected exact SHA dccd8351 and remains diagnostic: SQLite/R2 measured 213.947686 IOPS, PGlite/R2 2065.669446, TiDB/R2 333.356025, and FoundationDB/R2 363.254472. Every row completed 1,200/1,200 lifecycle operations with zero timeouts and cleanup failures. PGlite cleared the row target, but SQLite, TiDB, and FoundationDB missed the hard 1,000 IOPS threshold, so the aggregate failed closed without OZONE_IOPS_PASS. No acceptance or production claim is made from this run; the next work is correctness-preserving variance diagnosis and requalification. The latest terminal packet 35698854392 at exact source116e9ed4 passed the Ozone base checks and recorded SQLite/R2 at 1051.976655 IOPS, but PGlite/R2 measured837.779225, TiDB/R2 463.080116, and FoundationDB/R2 450.696578 against the unchanged hard1000-IOPS target. Each row completed 1,200/1,200 operations with zero timeouts and cleanup failures; the aggregate failed closed without OZONE_IOPS_PASS. This is diagnostic provider evidence, not an Ozone acceptance or production claim.

Layout and chunking

Fixed-size chunks are chosen by ChunkedFs and referenced by the independent metadata store. Conditional object creation, range reads, and delete ownership must remain visible in the test evidence.

Durability and consistency

The current arm64 gateway run passed create-only publication and duplicate rejection, ETag stale-read/stale-write rejection and CAS, concurrent writers, full/range reads, missing objects, binary payloads, a bounded stopped-gateway failure, service restart/reopen, and owned cleanup. The actual cluster's replication, Ratis, auth, TLS, and failure policy are not inferred from that single-node or all-in-one run.

Across every split-store composition, new blocks are flushed before metadata references are published. A provider acknowledgement is not automatically a power-loss or distributed-locking guarantee.

Use S3-compatible tools against the Ozone gateway

Read-only inspection example
export OZONE_ENDPOINT='http://127.0.0.1:<s3-port>'
export OZONE_BUCKET='mount-rs'
export OZONE_PREFIX='mount-rs-tests/owned-run'

aws s3api list-objects-v2 --endpoint-url "$OZONE_ENDPOINT"   --bucket "$OZONE_BUCKET" --prefix "$OZONE_PREFIX/"
aws s3api head-object --endpoint-url "$OZONE_ENDPOINT"   --bucket "$OZONE_BUCKET" --key "$OZONE_PREFIX/<block-id>"
aws s3api get-object --endpoint-url "$OZONE_ENDPOINT"   --bucket "$OZONE_BUCKET" --key "$OZONE_PREFIX/<block-id>"   --range bytes=0-63 /tmp/block-prefix.bin

Credentials, cleanup, and safe operations

Keep the Ozone cluster and gateway isolated, use an owned bucket/prefix, and remove only resources created by the harness. Production cleanup must follow Ozone's retention and replication policy; do not use a broad bucket delete in examples.

Current limits

The latest revision-bound hosted packet35635486040 completed its base Ozone job, but all four configured provider rows missed the hard 1,000-IOPS target: SQLite/R261.97, PGlite/R2 63.56, TiDB/R214.14, and FoundationDB/R2 23.07. Each row still completed 1,200/1,200 lifecycle operations with zero timeouts and cleanup failures; the aggregate correctly failed closed because no provider emitted OZONE_IOPS_PASS. The failed packet is diagnostic only, not a retained acceptance result. The non-secure all-in-one service is loopback-only and is not production authentication or durability evidence; durable FoundationDB remains test-deployment evidence and does not imply production auth, TLS, or power-loss durability. W26.15 remains open after the first safe concurrency/publication remediation. Replacement run 35641941218 at exact W26 code head 88b707ba completed all four provider rows with 1,200/1,200 successful lifecycle operations and zero timeouts or cleanup failures, but missed the hard 1,000-IOPS target: SQLite/R2 measured85.70, PGlite/R2 94.04, TiDB/R214.50, and FoundationDB/R2 30.46. The producer jobs and aggregate failed closed without OZONE_IOPS_PASS; this is diagnostic evidence, not a new acceptance result.

← Back to providers