Skip to content

Subject-Aware Observability Evaluation

This page records the evaluation for bounded subject-aware observability policies in nats-sinks. It is paired with the Subject-Aware Observability Runbook and is written for operators who want insight into subject-family behavior without accidentally publishing sensitive operational metadata or creating unbounded metric cardinality.

The conclusion is intentionally conservative:

  • subject-aware metric export must remain disabled by default,
  • current low-cardinality aggregate metrics should remain the normal production posture,
  • subject-aware export should be controlled by a reviewed, opt-in policy model,
  • raw subjects should not become labels by default,
  • any subject-aware feature must be bounded, fail closed, and unable to influence ACK behavior, sink writes, retries, or DLQ handling.

The current release adds the policy model, a bounded prepared metric-series format, and certification tests with runbook guidance. Subject-labeled export remains disabled unless an operator enables a reviewed policy and a safe aggregation path emits prepared labeled_metrics rows.

Background

Prometheus metrics are stored as time series identified by a metric name and labels. The Prometheus data model states that changing a label value creates a new time series. The Prometheus instrumentation guidance also warns that every label set has RAM, CPU, disk, and network cost, recommends that most metrics have no labels, and advises moving analysis away from monitoring when cardinality can grow too large. See Prometheus Data Model, Prometheus Metric And Label Naming, and Prometheus Instrumentation.

OpenTelemetry also treats attribute cardinality as an explicit design concern. The metrics SDK specification describes cardinality limits, overflow handling, and the need to apply filtering before cardinality-limit enforcement. See OpenTelemetry Metrics SDK.

Those principles matter for nats-sinks because NATS subjects can encode mission, customer, environment, platform, tenant, routing, or operational tempo information. In defence and mission-support environments, a subject name can be sensitive even when the payload is encrypted and no credentials are present.

Current Safe Baseline

The current observability design is intentionally low-cardinality:

  • the core metrics snapshot contains aggregate counters, gauges, and timing summaries;
  • the Prometheus connector exports no subjects, table names, file paths, message IDs, classification values, labels, or payload fields;
  • generated observability policies are disabled by default;
  • generated policy files may include known subject patterns as disabled review hints, but those hints are not exported as metric labels.
flowchart LR
    Runner[Sink runner] --> Snapshot[Local metrics snapshot]
    Snapshot --> Policy[Disabled observability policy]
    Policy --> Exporter[Observability connector]
    Exporter -->|default| None[No external subject sharing]

This remains the recommended production default.

Threat Model

Subject-aware observability has two risk families.

First, subjects can disclose operational context:

  • business domain or mission thread,
  • platform or system family,
  • location, unit, tenant, or environment naming conventions,
  • incident tempo through per-subject rates,
  • classification workflow hints,
  • routing design and system boundaries.

Second, subjects can create metric cardinality pressure:

  • unbounded subjects can create many time series,
  • wildcard subject families can hide high variation,
  • dynamic subject tokens can resemble user IDs or message IDs,
  • per-subject observations can multiply timing series,
  • exporters can become slower or more expensive even when delivery remains safe.

The safe default is therefore no subject sharing.

Policy Shape

The subject-aware policy is separate from the current aggregate metric allow list. It makes operators state exactly what they intend to share and how it should be represented. Exporters render subject-family labels only from prepared labeled_metrics rows that were built from this reviewed policy.

Example policy shape:

{
  "subject_metrics": {
    "enabled": true,
    "default_action": "deny",
    "max_subject_families": 20,
    "overflow_action": "aggregate_other",
    "overflow_label": "other",
    "allow_raw_subjects": false,
    "rules": [
      {
        "subject": "orders.*",
        "label": "orders",
        "display_mode": "label",
        "allowed_metrics": [
          "messages_fetched_total",
          "messages_written_total",
          "messages_failed_total"
        ]
      }
    ]
  }
}

The important properties are:

  • disabled by default,
  • default deny,
  • explicit subject-family allow list,
  • stable low-cardinality labels rather than raw subjects where possible,
  • maximum subject-family count,
  • deterministic overflow behavior,
  • no payload, header, message ID, table, file path, classification, or free-form label values in metric labels.

Hashing And Redaction

Hashing raw subjects may help compare repeat occurrences without exposing the literal subject. It is not a confidentiality boundary by itself. Subject spaces are often small enough that an attacker with context can guess values and compare hashes.

If hashing is added later, it should be treated as one display mode, not as a replacement for allow lists and cardinality caps. A safer default is to map approved subject patterns to operator-chosen labels such as orders, telemetry, or dlq.

Delivery Boundary

Subject-aware observability must never affect delivery semantics.

sequenceDiagram
    participant R as Runner
    participant S as Sink
    participant M as Metrics policy
    participant JS as JetStream

    R->>S: write_batch(envelopes)
    S-->>R: durable success
    R->>M: observe aggregate or subject-family metric
    M-->>R: allowed, denied, capped, or failed closed
    R->>JS: ACK after durable success

If metrics export is disabled, denied, capped, stale, malformed, or unable to write, message delivery behavior must remain unchanged. Metrics failure is an observability event, not a delivery decision.

The evaluation recommends three separate implementation boundaries:

  1. Add a subject-aware observability policy model with threat-model defaults.
  2. Add bounded subject-family metric aggregation.
  3. Add subject-aware observability certification tests and runbook guidance.

This split keeps policy design, runtime aggregation, and certification evidence reviewable as separate changes.

Current Status

This release adds the disabled-by-default subject-aware policy model, the bounded labeled_metrics snapshot extension, and a focused certification suite with operator runbook guidance. Subject-family rows are prepared from approved policy decisions and stable family labels rather than from raw subject strings. The certification tests prove disabled defaults, allow and deny handling, malformed policy rejection, cardinality caps, sanitized connector output, and delivery non-interference. Existing aggregate counters remain unchanged.

Run the focused release gate with:

python -m pytest tests/unit/test_subject_observability_certification.py -q