Skip to content

NATS Feature Gap Analysis

This page compares capabilities documented by NATS with the current scope of nats-sinks. It is not a criticism of the project. nats-sinks is deliberately focused on one job: at-least-once delivery from JetStream to external destinations with commit-then-acknowledge processing and idempotent sink support.

NATS is a broad platform. Many NATS features are server-side topology, administration, publishing, request/reply, or storage features that are adjacent to, but not required for, the current JetStream sink runner.

For new readers, this page is a roadmap aid. It helps explain why a feature may exist in NATS but not yet appear in nats-sinks. The table entries distinguish between features that are intentionally out of scope, features that should be managed by NATS operators, and features that are good candidates for future certified support in the sink framework.

Current Scope

nats-sinks currently supports:

  • pull-based JetStream consumption by default and explicit opt-in push consumption when configured,
  • one stream, one consumer, and one configured subject filter per runner,
  • durable or ephemeral subscription selection,
  • bounded batch fetches,
  • normalized NatsEnvelope objects,
  • commit-then-acknowledge processing,
  • NAK or leave-unacked temporary failure handling,
  • DLQ publication for permanent failures,
  • Oracle as the first production database sink,
  • local files as the first production filesystem sink,
  • NATS token and username/password authentication,
  • TLS server verification with a local CA file,
  • multiple NATS seed URLs,
  • reconnect tuning and NATS connection event metrics,
  • offline JetStream stream-management planning for retention, discard, storage, replicas, duplicate-window, and permission review,
  • client-side credentials-file, NKEY seed-file, and TLS client certificate option construction with validation, redaction, documentation, and gated integration-test hooks.

High-Level Gap Map

mindmap
  root((NATS Capabilities))
    Currently supported
      Pull consumer
      Explicit ACK after sink commit
      Token auth
      User password auth
      TLS local CA
      Credentials file auth
      NKEY seed auth
      TLS client certificates
      WebSocket connection guardrails
      DLQ publishing
      Stream planning helper
    Not yet supported
      Rich consumer config
      Stream mutation
      Push and ordered consumers
      AckSync and InProgress
      Advanced WebSocket proxy runbooks
      KV and object stores
      Request reply and services
      Advisories and system events

Connection And Authentication Gaps

NATS supports multiple connection URLs for clustered deployments, WebSocket connections, reconnection tuning, connection event callbacks, NKEY challenge authentication, decentralized JWT authentication/authorization, TLS certificate identity, subject-level authorization, accounts, exports, and imports.

Current gap details:

NATS capability NATS support Current nats-sinks status Suggested priority
Multiple seed URLs for clusters NATS clients can connect with multiple seed URLs. Supported through nats.urls, which is passed to nats-py as servers. Implemented
Reconnect tuning NATS clients expose reconnect wait, max reconnects, buffers, ping behavior, and event callbacks. Supported through JSON fields for reconnect enablement, connect timeout, reconnect wait, maximum reconnect attempts, ping settings, pending bytes, and drain timeout. Implemented
Connection event metrics NATS clients can report disconnect, reconnect, closed, discovered server, and async error events. Runner wraps nats-py callbacks and records connection event metrics while preserving user-supplied callbacks. Implemented
No-echo connection option NATS clients can request server-side suppression of messages published on the same connection being echoed back to matching subscriptions on that connection. Supported through nats.no_echo, passed to nats-py as no_echo. The default is false because normal JetStream pull-consumer sink workers do not depend on same-connection echo suppression. Implemented
WebSocket connections NATS URLs can use ws:// for WebSocket connections, and NATS server WebSocket support can run alongside traditional TCP connections. Supported with project guardrails for mixed transport rejection, credential-free URLs, wss:// local CA handling, validated optional headers, and an optional local certification harness. See WebSocket Connection Evaluation. Implemented; continue hardening proxy-specific runbooks.
TLS certificate identity auth NATS can use client certificate/key material and server-side TLS verification. Client cert/key option construction, redaction, validation, and documentation are certified on the client side. Server-side certificate identity policy remains an operator responsibility. Implemented for client configuration; keep expanding live certification runbooks.
NKEY challenge auth NATS supports challenge-response auth using Ed25519 NKEYs. nkey_seed_file is validated, redacted, passed to nats-py as nkeys_seed, and covered by unit plus gated integration tests. Implemented for client configuration; keep expanding live certification runbooks.
Decentralized JWT auth NATS supports operator/account/user JWT auth with credentials files and resolvers. creds_file is validated, redacted, passed to nats-py as user_credentials, and documented as the client-side credentials-file workflow. Account resolver setup remains server-side policy. Implemented for client configuration; keep expanding live certification runbooks.
Accounts, exports, imports, permissions NATS supports account isolation and subject-level permissions. Least-privilege runtime, DLQ, management, and advisory permission templates are documented. Account export/import designs remain server-side operator policy. Implemented for runtime templates; deeper account design remains Phase 2
Auth callouts NATS supports auth callout extensions. Server-side feature; not supported or documented for sink deployments. Phase 3

JetStream Consumer Gaps

NATS JetStream consumers support durable and ephemeral consumers, pull and push delivery, ordered consumers, multiple deliver policies, ACK policies, ACK wait, maximum delivery attempts, backoff, maximum ACK pending, replay policy, server-side filtering, headers-only delivery, metadata, replicas, and memory storage for consumer state.

Current gap details:

NATS capability NATS support Current nats-sinks status Suggested priority
Explicit consumer creation/update Consumers have a rich server-side configuration model. consumer_management.mode supports bind_only, create_if_missing, and reconcile for durable pull consumers. Existing unsafe drift fails closed before messages are fetched. Implemented for durable pull consumers
AckWait Controls when unacked messages redeliver. consumer_management.ack_wait_seconds can create, reconcile, or validate the server setting for durable pull consumers. It is rejected when backoff_seconds is configured because JetStream BackOff overrides AckWait. Implemented for durable pull consumers
MaxDeliver Controls maximum redelivery attempts before advisories. consumer_management.max_deliver can create, reconcile, or validate the server setting. delivery.max_retries still controls active delayed NAK attempts before leaving messages redeliverable. Implemented for durable pull consumers
BackOff Server-side redelivery backoff sequence. consumer_management.backoff_seconds can create, reconcile, or validate bounded server-side BackOff sequences when max_deliver is also configured. Local delayed NAK backoff remains under delivery.retry_backoff_*. Implemented for durable pull consumers
MaxAckPending Server-side flow control for outstanding unacked messages. consumer_management.max_ack_pending can create, reconcile, or validate the server setting. delivery.batch_size remains the client-side fetch bound. Implemented for durable pull consumers
DeliverPolicy Start at all, new, last, sequence, time, or last-per-subject. consumer_management.deliver_policy supports all, last, new, and last_per_subject. Sequence and time starts remain future work. Partially implemented
Multiple FilterSubjects Consumers can filter on multiple subjects. consumer_management.filter_subjects supports bounded plural filters that must remain within nats.subject; Oracle table routing still happens after delivery. Implemented for durable pull consumers
HeadersOnly delivery Consumers can deliver only headers and expose the omitted body size through a NATS header. consumer_management.headers_only can create, reconcile, or validate the server setting. The runtime records payload-presence metadata, rejects unsafe payload-hash fallback when bodies are omitted, and certifies sink/DLQ metadata-only behavior. Implemented for durable pull consumers
Consumer metadata Consumers support user metadata. consumer_management.metadata supports bounded low-sensitivity string metadata and rejects secret-looking keys. Implemented for durable pull consumers
Push consumers NATS supports push delivery to a subject, optional queue-style deliver groups, MaxAckPending, FlowControl, and IdleHeartbeat. push_consumer provides an explicit opt-in bounded manual-ACK runner mode with fail-closed configuration guardrails and focused delivery-contract certification. Pull remains the default. Implemented as opt-in
Ordered consumers NATS supports ordered consumers for inspection and analysis workflows. nats-sink inspect-ordered provides bounded, read-only ordered inspection when the NATS Python client exposes ordered-consumer support. It fails closed when unsupported and never replaces durable pull-consumer sink writes. Durable replay-to-sinks remains future work. Implemented for inspection
Queue-style push subscriptions Push delivery can use queue groups. Supported only through explicit push_consumer.deliver_group configuration in opt-in push mode. Pull consumers remain preferred for sink work. Partially implemented
Consumer replicas and memory storage Consumer state can have replica and memory options. consumer_management.num_replicas and consumer_management.memory_storage can create, reconcile, or validate these settings when explicitly configured. Implemented for durable pull consumers

JetStream ACK And Redelivery Gaps

NATS supports several acknowledgement variants: normal ACK, double ACK (AckSync in client APIs), NAK, in-progress, term, and next-message ACK behavior for pull consumers.

Current gap details:

NATS capability NATS support Current nats-sinks status Suggested priority
Double ACK / AckSync Client can wait for the server to confirm receipt of the ACK. delivery.ack_confirmation is disabled by default and can use bounded ack_sync only after durable sink success or successful DLQ publication. Metrics distinguish attempts, successes, timeouts, failures, unsupported paths, and elapsed confirmation time. Implemented as opt-in
In-progress ACK Extends AckWait while long processing continues. Optional disabled-by-default runtime heartbeat is available with effective AckWait-only guardrails, bind-only consumer-policy inspection, BackOff rejection, and stable metrics. BackOff-aware heartbeat timing remains future work. See InProgress Evaluation. Implemented with BackOff support deferred
Term ACK Stops redelivery without marking successful processing. Supported as explicit dead_letter.ack_term_after_publish policy only after DLQ publication succeeds. Disabled by default. Implemented
AckAll ACK one message and implicitly ACK earlier messages. Intentionally unsupported because commit-then-ack requires explicit per-message safety. Not planned
AckNone Server treats delivery as success without client ACK. Intentionally unsupported because it violates commit-then-ack. Not planned
AckNext ACK and request more messages in one protocol operation. Not planned unless scope changes. The runner fetches batches explicitly so ACK, backpressure, timeout, and retry behavior stay separately testable. Not planned

JetStream Stream Management Gaps

NATS streams support retention policies, discard policies, storage type, replicas, maximum message limits, maximum age, duplicate windows, subject transforms, republish, mirrors, sources, rollups, compression, metadata, placement, direct get, and newer options such as per-message TTL or atomic publish.

Current gap details:

NATS capability NATS support Current nats-sinks status Suggested priority
Stream creation and reconciliation Streams have rich configuration. nats-sink stream-plan provides offline planning guidance and permission separation. It does not mutate NATS; users still create or update streams externally. Guidance helper implemented; mutation remains outside runtime scope
Retention and discard policies Limits, interest, and work-queue retention are server-side stream options. nats-sink stream-plan validates allow-listed planning values and documents operational tradeoffs. It does not enforce server state. Guidance helper implemented
Duplicate window Streams can deduplicate publisher writes by Nats-Msg-Id. nats-sink stream-plan includes duplicate-window guidance. Consumed Nats-Msg-Id can still be used for sink idempotency; publisher dedupe remains complementary, not a replacement. Guidance helper implemented
Mirrors and sources Streams can replicate from other streams. Not managed by the runner; topology considerations and idempotency impacts are documented. Guidance implemented; management remains Phase 3
Subject transforms NATS can transform subjects at stream ingress, source, mirror, or republish boundaries. Not managed by the runner; documentation explains that sink routing sees the delivered subject. Guidance implemented; management remains Phase 3
RePublish Streams can republish stored messages to another subject. Not managed. Documentation separates server-side RePublish from sink DLQ publishing. Guidance implemented; management remains Phase 3
Stream compression File streams can use compression. Not managed. Documentation explains that server-side stream compression is transparent to the runner. Guidance implemented; management remains Phase 3
Stream metadata and placement Streams can carry metadata and placement preferences. Not managed. Documentation explains operational, latency, and per-message metadata boundaries. Guidance implemented; management remains Phase 3
Per-message TTL, schedules, counters, atomic publish Newer stream capabilities exist in recent NATS versions. Out of scope for the sink runner. Phase 3

Core NATS Feature Gaps

nats-sinks is not intended to be a general-purpose Core NATS framework. NATS supports capabilities that are important but outside the sink-runner contract.

NATS capability NATS support Current nats-sinks status Suggested priority
Core NATS pub/sub runtime Core NATS supports at-most-once pub/sub. Only a manual live probe script uses core subscribe/publish. The sink runner is JetStream-focused. Not planned
Queue groups Core NATS can load-balance subscribers with queue groups. Not supported in runtime. Pull consumers are the sink scaling mechanism. Not planned
Request/reply NATS supports request/reply and no-responder behavior. Out of scope for sink writes. Not planned
NATS services framework NATS supports service-style request/reply abstractions. Out of scope. Not planned
No-echo subscriptions Clients can disable echo of their own messages. Not exposed in JSON. Phase 3

JetStream Data Abstraction Gaps

JetStream also enables data abstractions beyond streams and consumers.

NATS capability NATS support Current nats-sinks status Suggested priority
Key/Value store JetStream supports buckets with put/get/delete/watch/history and atomic update operations. Not supported. Not planned unless a sink needs it
Object store JetStream supports object buckets and chunked object transfer. Not supported. Not planned unless a sink needs it

Observability And Operations Gaps

NATS exposes server monitoring endpoints, system events, and JetStream advisories. These can help operators understand redeliveries, maximum-delivery events, API activity, stream changes, and consumer changes.

NATS capability NATS support Current nats-sinks status Suggested priority
JetStream advisories $JS.EVENT.ADVISORY.> publishes operational events such as stream and consumer actions. Optional observation is supported through the disabled-by-default advisories config. The observer emits aggregate counters only and remains separate from sink ACK decisions. Implemented for selected advisory counters
MaxDeliver advisory handling NATS emits advisories when messages hit maximum delivery attempts. Optional observation records jetstream_advisory_max_deliver_total. DLQ is still driven by sink exceptions and pre-sink policy, not by advisories. Implemented as observation only
Server monitoring endpoints NATS exposes monitoring such as /jsz and /healthz. Implemented as a separate disabled-by-default nats-sink-observe connector with explicit endpoint and field allow lists. The delivery worker still does not poll server monitoring endpoints. Implemented for selected fields
Reconnect/disconnect metrics Client libraries expose connection event callbacks. Runner records disconnect, reconnect, close, discovered-server, and async-error callback metrics. Implemented
Prometheus/OpenTelemetry export NATS and application metrics can be exported externally. Basic counters, gauges, timing observations, a local JSON snapshot, nats-sink-metrics, policy-controlled Prometheus textfile export, and an optional native Prometheus HTTP endpoint exist. OpenTelemetry is not shipped yet. Phase 2 for OpenTelemetry

Design Notes

Some gaps should remain intentional:

  • AckNone and early ACK behavior conflict with commit-then-acknowledge.
  • Core NATS queue groups and request/reply are not destination sink semantics.
  • Stream and server topology management may belong in infrastructure-as-code rather than inside the sink process. The current documentation gives deployment-design guidance without making the sink worker a stream management tool.
  • Ordered consumers are useful for inspection and replay, but they do not match the durable destination-write model.
  • Push consumers are available only as an explicit manual-ACK runner mode with bounded in-flight work, flow-control handling, shutdown tests, and optional live NATS certification.

Other gaps are good candidates for future work:

  • certified credentials-file, NKEY, and JWT workflows,
  • explicit consumer configuration and reconciliation,
  • explicit BackOff-aware InProgress heartbeat timing,
  • optional live push-consumer certification on disposable local NATS servers,
  • durable replay-to-sinks guidance that keeps destination writes on durable pull consumers,
  • multi-subject filters,
  • JetStream advisory consumption beyond the documented read-only advisory permission template.
  • broader NATS server monitoring field recipes, if operators need more documented field selections in addition to the current explicit allow-list connector.

Source References