Troubleshooting

This page is the operator-facing runbook for common DSM integration failures. Start with the symptom you see, then use diagnostics and observability together.

Universal Triage Flow

Symptom: Lease Thrashing

Typical signs:

• ownership changes too often
• renewRejectCount grows
• acquireRejectCount and uncertainAcquireCount grow
• downstream sees fencing rejections

Failure shape: worker-a acquires the shard, renewal fails, worker-b acquires it, and worker-a keeps retrying with stale ownership assumptions.

What to check:

1. leaseCollections() for renew and acquire rejection counters
2. whether renew-skew is too small relative to scheduling jitter
3. whether term is too short for the runtime environment
4. whether downstream processing ignores fencing and keeps acting as a stale owner

Likely causes:

• renewals scheduled too late
• overloaded node misses renewal window
• duplicate workers competing for the same keys
• environment pauses causing expiry churn

Fix direction:

• increase lease term if operationally appropriate
• keep renewals comfortably inside the renew-skew window
• verify only one worker should target the key at a time
• inspect recordLeaseRenew, recordLeaseAcquire, and recordFencingReject metrics

Symptom: Wrong Or Missing Locator

Typical signs:

• one node has entries but peers do not
• expected collection does not appear in collections()
• data appears isolated even though the runtime is running

Failure shape: node-a writes shared/gateway/route-hints while node-b reads shared/worker/route-hints. The entry key may match, but the locator does not, so the two nodes are not sharing the same collection state.

What to check:

1. collections() on every affected node
2. locator triplet: tenantId/applicationId/collectionId
3. schemaId
4. collection consistency tier

Likely causes:

• typo in one locator segment
• one service registered under a different application domain
• Spring Boot property mismatch between environments
• one side changed schemaId or consistency tier incompatibly

Fix direction:

• make the locator identical on all intended peers
• keep schemaId aligned for compatible payloads
• centralize registration to avoid drift between modules

Symptom: Spring Bean Validation Failure At Startup

Typical signs:

• application fails during Spring Boot startup
• error mentions missing codec bean, missing entity factory, duplicate locator, or wrong consistency tier

Failure shape: Spring binds dsm.* properties, validates collection definitions, resolves supporting beans, and then builds DsmRuntime. Validation failures happen before the runtime is usable.

What to check:

1. required common fields are present: tenant-id, application-id, collection-id, and schema-id
2. either codec-bean is set or entity-type can derive a record codec
3. consistency-tier matches type
4. lease collections set lease.entity-factory-bean
5. CRDT collections set state-codec-bean, initial-state-bean, and merger-bean
6. explicit bean-name values are unique

Likely causes:

• bean name typo
• missing supporting bean definition
• copied register config reused for lease or CRDT without required nested fields
• duplicate locator definitions in one application

Fix direction:

• compare the failing config to Spring Properties
• compare the target workload to the matching cookbook page
• keep one centralized configuration source for all collection definitions

Symptom: clusterId Or serviceId Mismatch

Typical signs:

• node sees no peers or fewer peers than expected
• recordServiceIdMismatch rises
• peers appear present on the network but do not join the same DSM fabric

Failure shape: node-a and node-b may share the same network and clusterId, but if one uses gateway-service and the other uses worker-service, membership or replication traffic is rejected or ignored.

What to check:

1. clusterView().clusterId() and clusterView().serviceId() on healthy and unhealthy nodes
2. deployment-time environment variables or property overrides
3. metrics for recordServiceIdMismatch and cluster admission denial

Likely causes:

• wrong environment profile
• copied service config from another service family
• one node deployed with stale configuration

Fix direction:

• keep clusterId stable per environment and cluster
• keep serviceId stable per service family
• validate both values during deployment rollout, not after startup

Symptom: Security Or Replay Rejections

Typical signs:

• recordAuthFailure or recordReplayRejection grows
• replication traffic is dropped or rejected

What to check:

1. cluster secret alignment
2. nonce window and clock drift settings
3. min-acceptable-key-version
4. payload encryption mode across all peers
5. whether one deployment rolled with a different signing configuration
6. sender ban metrics for repeated verification failures

Fix direction:

• align security settings across the cluster
• verify time synchronization assumptions
• roll encryption and key-version changes intentionally
• inspect the security integration test pattern before changing runtime behavior

Symptom: Schema Fingerprint Mismatch

Typical signs:

• recordSchemaFingerprintMismatch rises
• a node accepts local writes but ignores remote register upserts
• data appears stuck after a rolling deployment

What to check:

1. entity record class deployed on each node
2. schema-id values
3. whether the record fields or generic signatures changed
4. deployment order and rollback state

Fix direction:

• align entity binaries across peers
• change schema-id when payload semantics are incompatible
• use a planned compatibility migration rather than relying on ignored upserts

Symptom: QUORUM Lease Rejections

Typical signs:

• lease acquire, renew, or transfer fails with membership-unstable
• lease operations fail with quorum-unavailable
• cluster membership is changing during ownership handoff

What to check:

1. visible cluster size and recent membership events
2. lease.quorum-stability-rounds
3. whether suspicion or failure transitions are flapping
4. whether the workload truly requires QUORUM mode

Fix direction:

• wait for membership to settle before ownership changes
• tune membership stability rounds conservatively
• inspect cluster discovery and network churn

Symptom: Relay Or Federation Repair Failure

Typical signs:

• federation binding status is DEGRADED or DISCONNECTED
• recordFederationRepair reports failed or repeated bootstrap_fallback
• anti-entropy repair skips a relay peer marked FAILED

What to check:

1. relay health status
2. federation target (clusterId, serviceId, locator)
3. whether the collection tier supports the expected federation behavior
4. whether register-only repair expectations are being applied to lease or CRDT bindings

Fix direction:

• restore relay health before expecting catch-up
• verify custom FederationRelay transport semantics
• remember that federated leases are read-only remote observations

Symptom: Queue Growth Or Backpressure

Typical signs:

• reportQueueDepth grows steadily
• recordBackpressureDecision increases
• updates appear delayed under load

What to check:

1. which locator is showing queue growth
2. whether the workload matches the selected collection type and QoS profile
3. whether traffic volume is much higher than expected for control-plane usage

Fix direction:

• reduce unexpected write amplification
• confirm you are not using DSM as a bulk data plane
• inspect transport and downstream processing lag

Read Next

• Diagnostics
• Observability
• Spring Properties
• Error Codes