Repair And Anti-Entropy
DSM uses steady-state replication for normal traffic and repair loops for missed, delayed, or divergent state. The important production idea is simple: a dropped delta should not require a manual rebuild.
Runtime Repair Model
The repair path is collection-addressed. It uses the same locator identity that application code uses: tenantId/applicationId/collectionId.
Proactive Anti-Entropy Sweep
RuntimeDataPlaneReplicationService starts background sweeps by default.
| Setting | Default | Meaning |
|---|---|---|
dsm.sync.anti-entropy.sweep-enabled | true | Enables background repair sweeps. |
dsm.sync.anti-entropy.sweep-interval | 30s | Interval between sweep rounds. |
dsm.sync.anti-entropy.max-concurrent-repairs | 2 | Maximum repair attempts initiated in one sweep. |
Collections are visited stale-first, and peers are selected by least-recent repair observation. Recent in-flight attempts enter cooldown so a stuck repair does not monopolize every sweep.
Repair Modes
| Mode | Use when | Risk controlled |
|---|---|---|
| None | Digests already match, or requester is clearly ahead of source. | Avoids rolling back healthier peers. |
| Replay | Replay window is available and cheaper than snapshot. | Sends only missed deltas. |
| Snapshot | Replay is unavailable, unsafe, or estimated to be more expensive. | Restores full visible state for that collection. |
The planner refuses backward snapshots when the requester advertises a newer observed sequence, or the same sequence with a larger entry set.
Adaptive Planning
AdaptiveRepairPlanner extends the fixed planner with per-peer cost profiles.
PeerRepairProfile.recordReplaySample(...)updates replay latency, throughput, and density estimates.PeerRepairProfile.recordSnapshotSample(...)updates snapshot estimates.- When replay is technically available, the planner compares estimated replay duration with estimated snapshot duration.
- Until a peer profile has real samples, the planner falls back to fixed replay-window behavior.
There is no Spring property for this feature. It is automatic inside the data-plane replication service.
Relay Health Gating
When a RelayHealthMonitor is supplied, data-plane repair skips failed relay peers.
| Relay health | Repair behavior |
|---|---|
HEALTHY | Peer is eligible for targeted repair and proactive anti-entropy. |
DEGRADED | Peer remains visible but should be watched operationally. |
FAILED | repairFromPeer rejects the peer and proactive sweeps exclude it. |
| recovered | Recovery triggers an immediate catch-up sweep. |
Schema-Incompatible Upserts
RecordCodec prefixes serialized records with an 8-byte schema fingerprint. If a remote register upsert carries a fingerprint that does not match the local record schema, RegisterDeltaSyncHandler catches DSM-6003 and ignores the remote upsert while preserving local state.
This is a compatibility safety behavior, not convergence magic. Sustained recordSchemaFingerprintMismatch(recordType) metrics mean the cluster is running incompatible entity schemas and must be fixed.
Metrics To Watch
| Metric hook | Meaning |
|---|---|
recordAntiEntropySweepOutcome(locator, outcome) | Per-collection sweep result: success, failed, or skipped. |
recordSchemaFingerprintMismatch(recordType) | Remote register upsert was discarded due to schema fingerprint mismatch. |
recordBackpressureDecision(...) | Scheduler accepted, delayed, or rejected traffic under pressure. |
reportQueueDepth(...) | Inbound/outbound traffic queue depth for a collection scope. |
Test Evidence
AdaptiveRepairPlannerTestcovers adaptive replay-vs-snapshot decisions and fixed fallback behavior.RuntimeDataPlaneReplicationServiceTestcovers failed relay skipping, proactive peer exclusion, relay recovery catch-up, and adaptive planner integration.ChaosIntegrationTestcovers partition healing and anti-entropy catch-up behavior.