Pool Migration Integrity Review: #261, #263, and #265
This record reviews #261, #263, and #265, together with their fixes #262, #264, and #266. It also records the maintainer follow-up #276. It is a design and review record, not a release note.
Status: Shipped in SILO Server 20261109 (d998292387cc). PRs #267 (superseding #262), #264, #266, and #276 (with lifecycle-delete replication removed before release) all shipped together in RELEASE.2026-11-09T00-00-00Z.
Decision at a glance
| Report | Is the problem real? | PR assessment | Recommended action |
|---|---|---|---|
| #261 | Yes. A readable mixed-EC object can be omitted from migration and lost when its old pool is removed. | The narrow fix was merged through maintainer PR #267; #262 was superseded. | Merged in #267; shipped in SILO Server 20261109. |
| #263 | Yes. An internal tier cleanup entry is treated as a real version by lifecycle, listing, and migration paths. | #264 fixes the whole boundary while keeping healing visibility. | Merged in #264; shipped in SILO Server 20261109. |
| #265 | Yes. ILM and migration disagree about whether a version still exists and who removes it. | #266 fixes the contributor-reported local ordering bug. Maintainer follow-up #276 preserves tier cleanup, lifecycle-generation, and whole-object completion contracts. The replication of lifecycle deletes that #276 also added was removed before SILO Server 20261109 shipped; see the 2026-11-06 update. |
Merged #266 and #276; shipped in SILO Server 20261109. |
The practical order was: merge the mixed-EC fix, merge #264, merge #266 to preserve the contributor’s change, and immediately follow it with #276. The first three changes are now in main, and #276 supplies the maintainer-owned safety boundaries that the original PR could not safely implement in isolation.
What was verified
The three reports reproduce against the pre-fix main with focused tests and the supplied native-server scenarios. The merged #264 and #267 code was checked before #266. The merged #266 plus #276 tree passed the focused migration and replication tests, go vet ./cmd, make build, the repository CI lint and generated-file checks, and the full go test ./cmd -count=1 run. The companion site checks pass after this document update.
The important behavioral evidence is:
- On main, the #261 mixed-EC migration tests return no entry for cases where latest-only GET and ListObjectsV2 already find a readable object. The same tests pass with #262.
- On main, the #263 tests show a free entry as a latest delete marker, hide a readable key from ListObjectsV2, queue the wrong version for lifecycle expiry, and move a plain delete marker during decommission. The full #264 suite fails on main and passes on the PR.
- On main, the #265 tests either leave the current version in the source pool or leave duplicate moved copies. The #266 suite passes on the PR, including decommission and rebalance cases with a delayed expiry queue.
The test evidence proves the reported failures and the local migration fixes. It does not by itself prove that the direct source purge in #266 preserves bucket replication, remote-tier ownership, lifecycle-configuration changes, or a safe completion fence for whole-object expiry. Those are separate acceptance conditions.
Why these reports arrived together
All three problems are variations of the same rule: a storage-pool worker must use the same logical object view as GET, and it must preserve the object until a safe destination or a deliberate deletion is known.
The failures were hidden because different workers read the same xl.meta through different projections:
- A latest-only read selected a readable majority across mixed EC layouts, while an all-versions walk used a stricter path.
- GET already ignored tier free versions, while list, lifecycle, and migration code still saw them.
- ILM knew that some actions would delete data eventually, while decommission and rebalance treated the queued action as if the deletion had already completed.
The right repairs are local boundary repairs. They do not require a new metadata format or a broad rewrite of pool migration.
Affected release scope and advisory
The three issue fixes are treated as a storage-integrity advisory for releases through RELEASE.2026-10-26, which predate the repairs. All four PRs (#264, #267, #266, and #276 with lifecycle-delete replication removed) shipped together in SILO Server 20261109 (RELEASE.2026-11-09T00-00-00Z); operators on 20261026 or earlier should upgrade to SILO Server 20261109 before running pool decommission or rebalance.
The code-level windows are different:
- The free-version listing and migration behavior entered through upstream 29a25a538 on 2024-06-07. Releases containing that change can hold the latent problem. Actual user-visible damage still requires a tiered version to leave a free record pending while lifecycle, listing, or migration observes it.
- The mixed-EC discrepancy became especially visible after e90cb2067 on 2026-10-16: latest-only reads recovered a readable majority while all-version migration walks could still omit it. The released 20261018, 20261025, and 20261026 tags contain that behavior; #267 fixes it.
This is a compound-condition problem, not a statement that every installation or every object is damaged. A free version by itself is normal pending-cleanup state. Mixed EC metadata by itself is a recoverable result of an availability-optimized degraded write. The damaging outcomes require a second operation while that state exists:
| State and operation | Result | Recovery position |
|---|---|---|
| A free version exists briefly and the scanner completes | No user-visible corruption; the record and remote object are cleaned. | Normal after upgrade; do not edit metadata by hand. |
| A tiered free version is seen by lifecycle or ordinary listing before cleanup | Wrong noncurrent decision, hidden key, or misleading delete marker. | Future scans are corrected by #264; already deleted real versions need backup or another authoritative copy. |
| A free version is moved by old decommission/rebalance | A plain delete marker can be created and the remote reference can be lost. | The marker cannot be identified automatically; compare source copies, replication targets, backups, and audit history. |
| Mixed EC metadata is scanned by old decommission/rebalance | The readable object can be omitted; removing the source pool can lose it. | If the source remains, run the fixed migration path and heal; if it was removed, recover from a copy or backup. |
Operators still running SILO Server 20261026 or earlier should pause pool decommission and rebalance—especially while drives are degraded, tier cleanup is pending, or lifecycle rules are active—preserve the old pool and backups, upgrade to SILO Server 20261109, and verify GET, ListObjectsV2, ListObjectVersions, and object counts after migration. The advisory is conditional on the compound states above, and rebalance still skips transitioned versions by design. A CVE-style security label is unnecessary unless a project classifies storage-integrity failures that way; the SILO Server 20261109 release notes and operational guidance cover the upgrade boundary.
SILO Server 20261109 delivers the complete migration repair set (#264, #267, #266, and #276, paired with the multi-pool and exact-key lifecycle fixes); a release containing only #264 and #267 would have been insufficient for lifecycle-aware migration. Installations that already completed pool migrations on 20261026 or earlier should audit source copies, replication targets, backups, and audit logs before assuming historical data was unaffected.
Merge and delivery summary
- Shipped #264, #267, #266, and #276 together in SILO Server 20261109 (
d998292387cc), establishing the logical object view, mixed-EC migration listing, lifecycle action semantics, and external cleanup contracts. - Preserved the distinction between a versioned current Expiration, a version-only purge, and a whole-object deletion rather than replacing it with a generic “skip every ILM action” rule.
- Retained regression coverage for lifecycle deletes with replication configured (which must queue no replication), transitioned-version cleanup, lifecycle configuration changes, whole-object expiry, and resumable decommission.
- Published the migration-integrity guidance in SILO Server 20261109; installations that migrated pools on earlier builds should still audit historical state against backups or replicas.
- Track rebalance support for transitioned versions separately; #276 deliberately does not broaden that scope.
The detailed records are:
- Mixed EC Layouts and Migration Listing
- Tier Free Versions Are Internal Cleanup Records
- ILM Expiry and Pool Migration Ordering
Residual limits
These repairs do not heal a disk that is still unavailable, recreate historical data already lost by an earlier completed decommission, or distinguish a plain delete marker created by an older buggy build from a client-created marker. #264 can make decommission fail when a tier is unavailable, because that is safer than dropping a cleanup record whose remote object may still exist. Rebalance still skips transitioned versions by design; that is a separate scope decision from the completed decommission fix. Lifecycle configuration changes are fail-closed during migration, so a stale walk must be retried under the new configuration.