Skip to content

Disconnected Spool Replay Testing

Disconnected backend testing proves the operational pattern used when a final destination is temporarily unreachable and local custody is intentional. The test does not change delivery semantics. It uses the existing Edge Spool Sink as the durable boundary during the outage phase, then replays those committed spool records to the recovered backend.

What The Test Proves

The certification flow uses synthetic, non-sensitive messages and runs these phases:

  1. Write 1001 messages directly to a reachable backend.
  2. Prove that the backend write path is unavailable.
  3. Write another 1001 messages to encrypted local spool custody.
  4. Replay the local spool to the recovered backend.
  5. Write a final 1001 messages directly to the backend.
  6. Verify that the backend contains 3003 unique messages and the local spool is empty.

The important safety point is the success boundary. In the direct phases, the backend commit is the success boundary. In the disconnected phase, the encrypted local spool commit is the success boundary. Replay is a second at-least-once workflow, so every tested backend must treat duplicate delivery as an update, no-op, or safe duplicate result rather than creating duplicate mission records.

Backend Coverage

Backend Test mode Command
Deterministic fake backend Unit test, no network python -m pytest tests/unit/test_disconnected_spool_replay.py -q
Oracle Database verification guard Unit test, no network python -m pytest tests/unit/test_oracle_disconnected_replay_verification.py -q
Oracle MySQL Database Local container python scripts/run-mysql-sink-e2e.py
Oracle NoSQL Database Local KVLite container python scripts/run-oracle-nosql-sink-e2e.py
Oracle Coherence Community Edition Local container python scripts/run-coherence-sink-e2e.py
Oracle Database Live integration gate NATS_SINKS_ORACLE_DISCONNECTED_REPLAY=1 python -m pytest -m integration tests/integration/test_oracle_sink.py -q

The Oracle MySQL Database, Oracle NoSQL Database, and Oracle Coherence Community Edition runners enable their disconnected replay checks themselves. They start or use short-lived local backends, bind only to loopback, use fake payloads, and clean up by default.

Oracle Database certification stays behind the existing live integration environment because this repository does not maintain a local Oracle Database test container. Source the ignored local Oracle environment files before running that live check. Do not paste connection strings, usernames, passwords, wallet locations, certificate material, or private endpoints into test reports or issue comments.

Full Container Gate

Run the container-backed Oracle-family suite with:

python -m pip install -e ".[mysql,oracle-nosql,coherence]"
NATS_SINKS_RUN_CONTAINER_E2E=1 scripts/check-sinks.sh

Expected successful tail output:

Oracle MySQL sink container e2e test passed.
Oracle NoSQL sink container e2e test passed.
Oracle Coherence sink e2e test passed.
Full container-backed sink e2e suite passed.

This gate is local and explicit. It is not enabled by default and is not a normal GitHub Actions trigger. It is intended for release preparation and developer certification when Docker and the optional backend clients are available.

How The Harness Works

The reusable helper lives in nats_sinks.testing.disconnected_spool_replay. Backend-specific tests provide a small adapter with three responsibilities:

  • create a fresh reachable sink;
  • create a fresh unreachable sink that must fail closed;
  • verify the expected synthetic records in the backend after replay.

The helper then creates deterministic NatsEnvelope objects for the three message phases. The local spool sink writes encrypted .spool.json records under the test-managed temporary directory. The replay helper deletes each spool file only after the target sink returns success.

The test report generated by the helper is intentionally payload-free. It contains counts such as:

{
  "backend": "Oracle MySQL Database",
  "direct_after_records": 1001,
  "direct_before_records": 1001,
  "expected_backend_records": 3003,
  "messages_per_phase": 1001,
  "outage_proved": true,
  "replayed_records": 1001,
  "schema_version": 1,
  "spool_remaining_records": 0,
  "spooled_records": 1001,
  "stream": "MYSQL_DISC_EXAMPLE",
  "unique_idempotency_keys": 3003
}

Running Individual Backends

Run Oracle MySQL Database:

python -m pip install -e ".[mysql]"
python scripts/run-mysql-sink-e2e.py

Run Oracle NoSQL Database:

python -m pip install -e ".[oracle-nosql]"
python scripts/run-oracle-nosql-sink-e2e.py

Run Oracle Coherence Community Edition:

python -m pip install -e ".[coherence]"
python scripts/run-coherence-sink-e2e.py

Run live Oracle Database when the ignored local environment is ready:

set -a
. .local/oracle-adb/integration.env
set +a
export NATS_SINKS_ORACLE_DISCONNECTED_REPLAY=1
python -m pytest -m integration tests/integration/test_oracle_sink.py -q

Failure Handling

If a backend is still unreachable during replay, replay stops and the spool file remains committed on disk. This is the expected fail-closed behavior: the operator can retry after the backend is repaired, and the original local custody record has not been discarded.

If the test finds a project defect, raise a managed GitHub bug before fixing it. Keep the bug report and evidence sanitized. Do not include payloads, connection strings, private network locators, credentials, wallet material, certificate material, or generated secret values.