DSM

Appearance

Observability ​

DSM uses DsmMetrics as its metrics SPI instead of coupling the runtime to one telemetry framework. The practical model is:

Observability flow

Core Signals ​

At minimum the SPI exposes:

  • recordSyncLatency(long latencyMs)
  • reportClusterSize(int count)

Additional callbacks cover:

  • authentication failures
  • replay rejection
  • cluster admission denial
  • service ID mismatch
  • LWW discards and dropped messages
  • suspected partitions
  • lease acquisition, renewal, transfer, release, and verification
  • fencing rejection
  • active lease holder counts
  • backpressure decisions and queue depth
  • change-stream overflow
  • encryption downgrade rejection and sender bans
  • anti-entropy sweep outcomes
  • federation repair outcomes
  • schema fingerprint mismatches
  • nonce sender evictions
  • trace context injection

Signal Groups ​

Cluster Health ​

  • reportClusterSize
  • recordServiceIdMismatch
  • recordClusterAdmissionDenied
  • recordPartitionSuspected

Interpretation:

Metric patternLikely interpretation
Cluster size drops and service ID mismatch rises.Isolation or discovery mismatch.
Cluster size drops and partition suspicion rises.Membership or network instability.

Security And Transport Integrity ​

  • recordAuthFailure
  • recordReplayRejection
  • recordEncryptionDowngradeRejected
  • recordSenderBanned
  • recordNonceSenderEviction
  • recordMessageDropped

Interpretation:

Metric patternLikely interpretation
Auth failures spike.Cluster secret or signing mismatch.
Replay rejection spikes.Nonce-window issue, replay-protection issue, or duplicate traffic pattern.
Sender bans rise.Repeated verification failures from one sender or spoofed sender identity.

Lease Health ​

  • recordLeaseAcquire
  • recordLeaseRenew
  • recordLeaseTransfer
  • recordLeaseRelease
  • recordLeaseVerify
  • recordFencingReject
  • reportActiveLeaseHolders

Interpretation:

Metric patternLikely interpretation
Renew failures and fencing rejects rise together.Investigate lease thrash and stale holders.
Active holders are unstable and acquire rejections are high.Investigate ownership churn or timing windows.

Backpressure And Queueing ​

  • recordBackpressureDecision
  • reportQueueDepth

Interpretation:

If queue depth grows steadily while backpressure decisions increase, treat the runtime as under sustained pressure or downstream transport lag.

Repair And Compatibility ​

  • recordAntiEntropySweepOutcome
  • recordSchemaFingerprintMismatch
  • recordFederationRepair
  • recordTraceContextInjected

Interpretation:

Metric patternLikely interpretation
Schema fingerprint mismatches rise.Incompatible record entity rollout.
Anti-entropy failures rise.Repair path, relay health, or peer availability issue.
Federation repair fallback rises.Dropped remote mutation or stale remote register state.

How To Correlate Metrics With Diagnostics ​

When an alert fires, first check the relevant DsmMetrics series for trend direction, then query runtime.diagnostics() on an affected node. Use the combination to decide whether the issue is membership, collection registration, lease churn, transport rejection, or security rejection.

No-Op Default ​

DsmMetrics.noop() is available when you need a runtime that operates without external metrics wiring.

Map the SPI into your existing metrics system with these shapes:

  • timers for sync latency
  • gauges for cluster size, active holders, and queue depth
  • counters for failures, mismatches, rejections, and drops

First Alerts To Add ​

  • cluster size below expected baseline
  • sustained replay rejection or auth failure
  • rising lease renew rejection or fencing rejection
  • queue depth above normal operating range
  • sustained backpressure decisions on one collection locator
  • sustained schema fingerprint mismatch
  • sustained anti-entropy or federation repair failures
  • sender bans or encryption downgrade rejections