Architecture

This page explains how the project is divided so that delivery safety is easy to reason about. The most important concept is the difference between the core runtime and a sink. The core runtime talks to NATS JetStream and owns message delivery decisions. A sink talks to a destination system such as Oracle Database, Oracle Autonomous Database on Oracle Cloud Infrastructure (OCI), a file store, an object store, or another approved durable backend and owns only destination writes.

The main architectural rule is:

Core owns delivery semantics. Sinks own destination writes.

This separation keeps JetStream ACK behavior consistent across destinations. A sink implementation should be able to focus on writing to its destination and committing durable state. It should not need to know how to ACK, NAK, publish to DLQ, or manage a JetStream consumer.

This boundary matters especially in mission and defence-style deployments, where multiple systems may depend on the same operational event stream. A database outage, file-system delay, or schema issue should result in visible redelivery or DLQ handling, not a quiet ACK that removes the event from the processing path before it has crossed a durable boundary.

Component Model

The diagram below shows the framework boundary. Current destination modules fit into the same shape whether they write to Oracle-family databases, filesystem storage, object storage, HTTP endpoints, edge spool storage, or experimental platform connectors: the runner manages JetStream, and sinks receive normalized envelopes instead of raw NATS messages.

flowchart TB
    subgraph NATS
        Stream[JetStream stream]
        Consumer[Durable pull consumer]
    end

    subgraph Core[nats_sinks.core]
        Runner[JetStreamSinkRunner]
        Envelope[NatsEnvelope]
        Authenticity[Optional message authenticity gate]
        Encrypt[Optional payload encryption]
        Policy[Optional pre-sink policy gate]
        Custody[Optional custody hash evidence]
        Priority[Optional priority lanes]
        DLQ[DLQ publisher]
        Metrics[Metrics hooks]
    end

    subgraph Sinks[nats_sinks.sinks]
        Protocol[Sink protocol]
        Registry[Safe registry]
    end

    subgraph Destinations[Destination modules]
        OracleDb[nats_sinks.oracle]
        OracleMySQL[nats_sinks.mysql]
        OracleNoSQL[nats_sinks.oracle_nosql]
        Coherence[nats_sinks.coherence]
        File[nats_sinks.file]
        ReviewPackage[nats_sinks.review_package]
        Spool[nats_sinks.spool]
        HTTP[nats_sinks.http]
        S3[nats_sinks.s3]
        Foundry[nats_sinks.foundry]
        Gotham[nats_sinks.gotham]
        Future[future sink modules]
    end

    Stream --> Consumer --> Runner
    Runner --> Envelope --> Authenticity --> Encrypt --> Policy --> Custody --> Priority --> Protocol
    Protocol --> OracleDb
    Protocol --> OracleMySQL
    Protocol --> OracleNoSQL
    Protocol --> Coherence
    Protocol --> File
    Protocol --> ReviewPackage
    Protocol --> Spool
    Protocol --> HTTP
    Protocol --> S3
    Protocol --> Foundry
    Protocol --> Gotham
    Protocol --> Future
    Runner --> DLQ
    Runner --> Metrics
    Registry --> OracleDb
    Registry --> OracleMySQL
    Registry --> OracleNoSQL
    Registry --> Coherence
    Registry --> File
    Registry --> ReviewPackage
    Registry --> Spool
    Registry --> HTTP
    Registry --> S3
    Registry --> Foundry
    Registry --> Gotham
    Registry --> Future

Diagram Components

• JetStream stream is the NATS persistence boundary that stores events before the sink worker consumes them. See Advanced JetStream Topology.
• Durable pull consumer tracks delivery state and redelivery for the worker. See JetStream Stream Management Planning and Commit Then ACK.
• nats_sinks.core is the delivery-control runtime. It owns fetch, retry, DLQ, ACK, NAK, shutdown, and envelope construction. See Configuration and Commit Then ACK.
• JetStreamSinkRunner is the core runner that connects the durable consumer to one configured sink while preserving commit-then-ACK behavior. See Python Usage.
• NatsEnvelope is the safe message object passed to sinks instead of raw NATS messages, so destination code cannot ACK early. See Public API Compatibility.
• Optional message authenticity gate verifies configured message signatures before sink delivery. See Message Authenticity.
• Optional payload encryption encrypts payload bytes before sink delivery when configured. See Payload Encryption.
• Optional pre-sink policy gate rejects messages that fail configured metadata or policy checks before any destination write. See Security, Configuration, and Mission Metadata.
• Optional custody hash evidence computes tamper-evident custody metadata before destination writes. See Tamper-Evident Custody.
• Optional priority lanes reorder already-fetched bounded batches without changing delivery guarantees. See Priority Lanes.
• DLQ publisher publishes permanent failures only according to the delivery policy and ACKs only after DLQ publication succeeds. See Dead Letter Queues.
• Metrics hooks record bounded operational counters without deciding delivery behavior. See Metrics Snapshot And CLI and Observability.
• nats_sinks.sinks contains the shared sink connector interfaces and certification expectations. See Sink Connector Framework and Sink Certification.
• Sink protocol is the destination-facing contract implemented by each sink. It accepts normalized envelopes and reports durable success or failure. See Sink Connector Framework.
• Safe registry resolves only approved built-in or allow-listed external connectors. It does not allow arbitrary module imports from configuration. See Sink Connector Framework.
• nats_sinks.oracle writes to Oracle Database and Oracle Autonomous Database on Oracle Cloud Infrastructure (OCI). See Oracle Sink.
• nats_sinks.mysql writes to Oracle MySQL Database. See Oracle MySQL Sink.
• nats_sinks.oracle_nosql writes to Oracle NoSQL Database. See Oracle NoSQL Database Sink.
• nats_sinks.coherence writes JSON payloads to Oracle Coherence Community Edition key/value storage. See Oracle Coherence Community Edition Sink.
• nats_sinks.file writes durable local filesystem records. See File Sink.
• nats_sinks.review_package writes disabled-by-default deterministic review packages with manifest and custody evidence. See Review Package Writer Sink.
• nats_sinks.spool provides durable local spool storage for disconnected or staged delivery patterns. See Edge Spool Sink.
• nats_sinks.http writes events to configured HTTP endpoints. See HTTP Sink.
• nats_sinks.s3 writes objects to S3-compatible object storage. See S3-Compatible Object Sink.
• nats_sinks.foundry is an experimental Palantir Foundry connector. See Palantir Foundry Sink.
• nats_sinks.gotham is an experimental Palantir Gotham connector. See Palantir Gotham Sink.
• Future sink modules should follow the same connector boundary, certification expectations, and ACK-after-durable-success invariant. See Sink Connector Framework and Sink Certification.

Processing Path

The processing path is intentionally linear. A batch moves from JetStream to a destination write, then to a durable commit, and only then to a JetStream ACK.

JetStream stream
  -> durable consumer
  -> nats-sinks core runner
  -> optional message authenticity verification
  -> optional payload encryption
  -> optional pre-sink policy gate
  -> optional tamper-evident custody metadata
  -> optional in-batch priority lane ordering
  -> sink.write_batch(...)
  -> durable destination commit
  -> JetStream ACK

Runtime Responsibilities

The core runtime handles:

• NATS and JetStream connectivity.
• Pull-based consumption.
• Bounded batch fetching.
• Conversion from raw NATS messages to NatsEnvelope.
• Optional message authenticity verification before any sink delivery.
• Optional payload encryption before sink delivery.
• Optional fail-closed policy enforcement after normalization and core payload transformation, but before any destination write.
• Optional tamper-evident custody metadata computation after policy acceptance, but before any destination write.
• Optional priority-lane ordering for already-fetched bounded batches.
• Sink lifecycle.
• Temporary versus permanent failure handling.
• DLQ publication.
• ACK and NAK behavior.
• Metrics hooks.
• Graceful shutdown.

Destination sinks handle:

• connection management for the destination,
• batch writes,
• durable commit,
• destination-specific error translation,
• destination-specific idempotency behavior.

Mission-oriented deployments should treat this split as an accountability boundary. The core provides a consistent delivery contract, while each sink documents exactly what counts as durable success for its destination.

JetStream Topology Boundary

The runner binds to the stream, durable consumer, and subject declared in configuration. Advanced JetStream topology that exists before that point, such as mirrors, sources, subject transforms, republish rules, stream compression, placement, and stream metadata, remains a NATS platform concern.

Those choices can still affect the envelope the runner receives. A transform may change the subject used by sink routing. A source or mirror may change how operators interpret stream and sequence identity. Placement can affect latency and reconnect behavior. The core does not hide these concerns; it keeps them outside the delivery-control boundary so they can be reviewed as platform architecture.

See Advanced JetStream Topology for the detailed operator guidance.

Stream Administration Boundary

The core runner is not a stream administration engine. It may bind to or prepare its configured durable pull consumer according to consumer_management.mode, but broad stream creation, stream update, stream retention policy, discard policy, storage type, replica count, and duplicate window choices remain NATS control-plane decisions.

The optional nats-sink stream-plan command lives on the safe side of this boundary. It reads local JSON configuration and generates an offline plan that operators can review, but it does not connect to NATS and does not mutate stream state.

flowchart LR
    Config[Runtime JSON config] --> Plan[nats-sink stream-plan]
    Plan --> Review[Operator review]
    Review --> Admin[Separate NATS admin identity]
    Admin --> Stream[Create or update JetStream stream]
    Stream --> Consumer[Durable pull consumer]
    Consumer --> Runner[JetStreamSinkRunner]
    Runner --> Sink[Destination sink]

This split keeps elevated stream-management permissions out of the ordinary sink worker. It also prevents stream planning from becoming a hidden runtime side effect that could change retention or replay behavior during message processing. See JetStream Stream Management Planning.

Durable Consumer Management

Before a runner fetches messages, it can now explicitly check the configured durable pull consumer. The default consumer_management.mode is create_if_missing, which preserves the previous developer-friendly behavior while making the action visible in configuration. Production environments that provision streams and consumers through infrastructure-as-code can switch to bind_only so the worker fails if the durable consumer is missing. Controlled platform deployments can use reconcile to submit the configured durable pull-consumer settings when the existing consumer is already compatible.

Consumer management is part of the startup boundary, not the message-processing boundary. The runner validates the durable name, single or plural filter subjects, explicit ACK policy, pull-consumer shape, deliver and replay policy, and configured delivery-sensitive fields before calling pull_subscribe. Managed fields can include AckWait, server-side BackOff, MaxDeliver, MaxAckPending, MaxWaiting, HeadersOnly, consumer replicas, memory-storage state, and bounded consumer metadata. If drift is unsafe, startup fails closed before any message can be fetched or ACKed.

flowchart LR
    Config[JSON configuration] --> Desired[Desired durable pull consumer]
    Existing[Existing JetStream consumer] --> Check{Compatible?}
    Desired --> Check
    Check -->|yes| Pull[Bind pull subscription]
    Check -->|missing and allowed| Create[Create consumer]
    Create --> Pull
    Check -->|unsafe drift| Stop[Fail closed before fetch]
    Pull --> Sink[Sink write and commit]
    Sink --> Ack[ACK last]

Why Raw NATS Messages Are Not Passed To Sinks

Raw nats-py messages expose ack, nak, and related methods. Passing raw messages into destination code would make it easy for a sink to ACK before durable success. NatsEnvelope prevents this by carrying payload and metadata without delivery-control methods.

Headers-only JetStream consumers keep the same sink boundary. NatsEnvelope still carries data: bytes for compatibility, but it also records payload-presence metadata so sinks and DLQ records can distinguish a producer-empty message from a body that the NATS server intentionally omitted. See Headers-Only Delivery.

Ordered consumers are also intentionally separate from the production sink runner. They are useful for inspection and analysis, but they are not a replacement for durable pull consumers when writing to Oracle, files, or future sinks. See Ordered Consumer Evaluation for the evaluation and follow-up tooling split.

Push consumers are available as an explicit opt-in runner mode for deployments that already standardize on server-initiated delivery. They add callback scheduling, client pending buffers, deliver subjects, flow-control messages, and more complex shutdown behavior, so pull consumers remain the production default. Push mode is manual-ACK only, bounded, and still routes messages through the same ACK-after-durable-success pipeline. The certification tests exercise callback error containment, temporary failure, permanent DLQ, flow-control, heartbeat, overflow, and shutdown paths. See Push Consumer Evaluation.

Extension Model

Future sinks should implement:

class Sink(Protocol):
    async def start(self) -> None: ...
    async def write_batch(self, messages: Sequence[NatsEnvelope]) -> None: ...
    async def stop(self) -> None: ...

A future sink is production-ready only when it can demonstrate:

• no ACK ownership,
• durable success before returning from write_batch,
• idempotent duplicate handling,
• clear temporary/permanent error classification,
• deterministic unit tests,
• documentation for failure behavior.

Adding a new sink should be an additive release: a new module, optional dependency extra, registry entry, tests, and destination-specific documentation. The core NatsEnvelope, Sink protocol, commit-then-acknowledge ordering, and existing Oracle configuration should remain compatible.

Generic route matching and fan-out orchestration are also part of the core. The routing configuration evaluates subject, priority, classification, labels, and approved header hints against a normalized NatsEnvelope and returns logical target names plus ACK-gating policy for those targets. The active fanout sink binds those target names to named child sinks, writes the selected message subsets, waits for every required child sink, and bounds optional side copies without moving JetStream acknowledgement ownership into destination modules.

The route selector, ACK-gate helper, and production fan-out sink are covered by reusable fan-out certification tests. Those tests use synthetic route policies and operation plans to prove that a required target failure blocks ACK, optional targets are bounded, no-route behavior is explicit, and the documented NATO SECRET and NATO UNCLASS examples select the expected logical sink names. Destination-specific sink certification still applies to every child sink.

Payload encryption is also part of the core, not a sink-specific responsibility. When enabled, the runner encrypts NatsEnvelope.data and passes a copied envelope to the sink. Metadata remains clear. This lets all sinks store the same encrypted payload envelope without duplicating cryptographic code.

Message authenticity verification is also part of the core. When enabled, the runner checks producer-supplied signature headers against an allow-listed subject rule before payload encryption, policy enforcement, custody hashing, priority lanes, or sink delivery. A verification rejection is handled as a permanent validation failure: the rejected message does not reach the sink, and the core follows DLQ-before-ACK behavior when a DLQ is configured. This complements NATS authentication and TLS; it does not replace them. See Message Authenticity.

Pre-sink policy enforcement is also part of the core. The policy gate is configured with explicit allow-listed checks such as required priority, classification, labels, mission metadata, encrypted payloads, and bounded payload size. It does not run dynamic code or destination-specific SQL. A policy rejection is handled as a permanent validation failure: the rejected message does not reach the sink, and the core follows DLQ-before-ACK behavior when a DLQ is configured.

Tamper-evident custody metadata is another core transformation. When enabled, the runner computes deterministic hashes over the normalized payload and stable metadata, attaches the custody object to NatsEnvelope, and then calls the sink. Sinks persist the custody object but do not decide whether it is required, valid, or sufficient for ACK. A custody computation failure is a pre-sink permanent validation failure, so the runner does not ACK the message unless configured DLQ publication succeeds first. See Tamper-Evident Custody Metadata.