Skip to content

Observability Connector Roadmap

This page records the evaluation behind observability connector work. It turns the broad connector roadmap into separate feature requests while keeping every connector aligned to the same security, configuration, testing, and documentation contract.

nats-sinks already has:

  • a local JSON metrics snapshot,
  • nats-sink-metrics for local inspection,
  • nats-sink-observe for disabled-by-default observability policy work,
  • a Prometheus textfile connector,
  • an optional native Prometheus HTTP endpoint,
  • an OpenTelemetry OTLP/HTTP JSON metrics connector,
  • an Elastic Observability profile,
  • a Grafana Alloy profile,
  • a Splunk HEC observability connector,
  • an OCI Monitoring observability connector,
  • a StatsD observability connector,
  • a Datadog observability connector,
  • an Amazon CloudWatch observability connector,
  • an Azure Monitor observability connector,
  • a syslog observability bridge,
  • a NATS server monitoring snapshot connector.

Future connectors must build on that model. Existing connectors must continue to follow it. No connector may become an independent way to leak metrics, subjects, classifications, labels, file paths, table names, payloads, credentials, or operational tempo without an explicit policy review.

Shared Connector Contract

All future connectors must follow the same functional contract:

Requirement Standard For Every Connector
Runtime isolation The connector runs outside the message delivery path. It must not participate in ACK, NAK, DLQ, retry, sink success, or idempotency decisions.
Default state Disabled by default. A connector exports nothing until the global observability policy and the connector-specific section are explicitly enabled.
Input source Read from the existing local metrics snapshot or a shared read-only metrics provider abstraction. Do not connect to NATS, Oracle, file-sink output directories, or future sinks directly.
Metric selection Use the existing observability allow-list and deny-list model. No connector may bypass allowed_metrics, denied_metrics, stale-snapshot checks, or observation controls.
Naming Use stable metric suffixes from MetricNames. Platform-specific names may add a safe namespace or prefix, but must map predictably back to the canonical nats-sinks metric names.
Labels and dimensions Export no high-cardinality labels by default. Subjects, labels, classification values, usernames, table names, file paths, message IDs, stream sequences, hostnames, and mission metadata values are excluded unless a future reviewed policy explicitly allows bounded fields.
Configuration Use the same common options where practical: enabled, snapshot_file, policy_file, namespace, allowed_metrics, denied_metrics, include_observations, stale_after_seconds, timeout_seconds, retry_attempts, retry_backoff_seconds, max_payload_bytes, dry_run, and connector-specific authentication references.
Authentication Secrets must come from environment variables, service identity, workload identity, instance principals, managed identity, or protected configuration files. Do not use command-line secrets or print resolved credentials.
Transport TLS verification stays enabled by default for network connectors. Local CA support should be documented when the target platform supports it.
Failure behavior Connector failures are logged safely and counted where practical. They must not stop the core sink worker or change delivery outcomes.
Bounds Every network call needs timeouts, bounded retries, bounded payload size, and safe behavior for stale snapshots.
Tests Unit tests cover disabled-by-default behavior, policy filtering, metric name mapping, authentication configuration redaction, timeout handling, retry limits, malformed snapshot handling, and sanitized logging. Optional integration tests stay behind markers.
Documentation Each connector gets its own sub-page under the Observability documentation section with configuration, security notes, examples, sample output, test guidance, and limitations. 
flowchart LR
    Worker[nats-sink worker] --> Snapshot[Local metrics snapshot]
    Policy[Observability policy] --> Connector[Connector service]
    Snapshot --> Connector
    Connector --> Platform[External observability platform]
    Connector -. never controls .-> Worker

Evaluation Matrix

Connector Priority Recommended Shape Why It Is Valuable Main Security Concern Feature Request
OpenTelemetry OTLP Implemented Native connector that exports approved metrics to an OpenTelemetry Collector through OTLP/HTTP JSON. OTLP is a stable OpenTelemetry exporter path and is a common neutral bridge to many platforms. Collector endpoints and resource attributes must not leak sensitive deployment details. See OpenTelemetry OTLP Integration.
StatsD Implemented Lightweight UDP or Unix-datagram connector for approved counters, gauges, and observation summaries. Useful in older or constrained environments with existing StatsD-compatible aggregation. UDP and datagram transports are best-effort; avoid pretending they provide durable telemetry custody. See StatsD Integration.
Datadog Implemented DogStatsD through a local or explicitly approved Datadog Agent listener. Datadog is widely used for hosted operational dashboards and alerting. Tags and custom metrics can create cost, cardinality, and confidentiality risk. See Datadog Integration.
Splunk HEC Implemented HEC metric event containing only policy-approved aggregate metric fields. Valuable for security operations, incident response, and SIEM-adjacent environments. HEC tokens are sensitive and payload shaping must avoid event-style leakage of operational metadata. See Splunk HEC Integration.
Elastic Observability Implemented Elastic profile over the shared OTLP connector, intended for a local or gateway OpenTelemetry Collector that forwards to Elastic. Useful for organizations that standardize on Elasticsearch-backed observability while keeping nats-sinks on the shared policy model. Index fields and labels can expose sensitive operational detail or create high cardinality. See Elastic Observability Profile.
Grafana Alloy Implemented Alloy profile over the shared OTLP connector, intended for a local or gateway Alloy otelcol.receiver.otlp component. Alloy bridges approved metrics into Grafana Cloud, Mimir, and LGTM-style observability stacks while preserving the nats-sinks policy model. Avoid leaking sensitive metric dimensions and keep Alloy credentials out of the delivery worker. See Grafana Alloy Profile.
OCI Monitoring Implemented OCI-native connector using Monitoring custom metrics with instance principals, resource principals, or configured OCI identity. Natural fit for Oracle Cloud deployments and Oracle-heavy nats-sinks users. Compartments, dimensions, tenancy metadata, and signer configuration require least-privilege review. See OCI Monitoring Integration.
Amazon CloudWatch Implemented AWS SDK based connector using CloudWatch custom metrics and bounded PutMetricData requests. Useful for AWS deployments that use CloudWatch as the operational source of truth. IAM permissions, namespace design, dimensions, and API cost controls need careful bounds. See Amazon CloudWatch Integration.
Azure Monitor Implemented REST custom metrics connector using a Microsoft Entra bearer token supplied from an environment variable. Useful for Microsoft cloud deployments and existing Azure operational teams. Resource identifiers, locations, dimensions, and bearer-token handling need strict redaction. See Azure Monitor Integration.
Syslog Implemented RFC 5424-style structured-data bridge for restricted networks over UDP or Unix datagrams. Useful where pull-based scraping and cloud APIs are not available. Syslog has transport and format pitfalls; messages must be bounded and sanitized. See Syslog Bridge.

Source Notes

The evaluation used public vendor and standards documentation:

Certification Checklist

Before any connector ships, the implementation issue must prove:

  1. The connector is optional and disabled by default.
  2. No runtime dependency is added to the base package unless it is already required by the core.
  3. The connector reads approved metrics from the shared snapshot/provider path.
  4. The connector applies the same observability policy model as existing Prometheus connectors.
  5. The connector exports no sensitive labels or dimensions by default.
  6. Network calls have explicit timeouts, bounded retries, and bounded payloads.
  7. Authentication is least-privilege and never passed through command-line arguments.
  8. Dry-run and validation modes exist for operator review.
  9. Tests cover disabled policy, malformed policy, stale snapshot, allowed and denied metrics, timeout behavior, retry exhaustion, redaction, and safe logging.
  10. The connector documentation is a sub-page under Observability and includes configuration examples, sample output, limitations, security guidance, and operational tests.

Implementation Order Guidance

The recommended order after the implemented Prometheus, OTLP, Elastic, Grafana Alloy, Splunk HEC, OCI Monitoring, StatsD, Datadog, Amazon CloudWatch, Azure Monitor, syslog, and NATS monitoring connectors is:

  1. Additional connector candidates only after they can reuse the shared policy, bounded export, dry-run, and documentation model proven by the implemented connectors.