OpenTelemetry Operator Guide

Status: Available feature with comprehensive operator guidance.

pg_trickle supports W3C Trace Context propagation through the refresh pipeline. When enabled, distributed traces initiated in application sessions are linked to CDC capture events and refresh spans, allowing full end-to-end latency attribution from user write to materialized result.

How It Works

1. Application session sets pg_trickle.trace_id to a W3C traceparent header.
2. CDC trigger captures pg_trickle.trace_id from the session GUC and stores it in the __pgt_trace_context column of the change buffer table.
3. Scheduler reads the trace context when consuming the change buffer.
4. Refresh pipeline propagates the trace context through the DIFF/FULL refresh cycle and exports a child span to the configured OTLP endpoint.

Application session                pg_trickle background worker
      │                                       │
      │ SET pg_trickle.trace_id = '...'       │
      │ INSERT INTO source_table ...          │
      │  └─► CDC trigger captures trace ──────┤
      │                                       │ Reads change buffer
      │                                       │ Opens child span
      │                                       │ Exports span to OTLP
      │                                       └─► Refresh complete

Configuration

Minimal setup

-- Enable trace propagation
ALTER SYSTEM SET pg_trickle.enable_trace_propagation = true;

-- Set the OTLP/HTTP endpoint
ALTER SYSTEM SET pg_trickle.otel_endpoint = 'http://localhost:4318';

SELECT pg_reload_conf();

Per-session usage

-- Set traceparent before DML (propagates through CDC to refresh span)
SET pg_trickle.trace_id = '00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01';

-- Your DML here
INSERT INTO orders (id, total) VALUES (42, 99.99);
-- pg_trickle CDC trigger stores the trace context in the change buffer

GUC Reference

Collector Configuration Examples

Jaeger (all-in-one)

docker run -d \
  -p 4318:4318 \
  -p 16686:16686 \
  jaegertracing/all-in-one:latest

ALTER SYSTEM SET pg_trickle.otel_endpoint = 'http://localhost:4318';
ALTER SYSTEM SET pg_trickle.enable_trace_propagation = true;
SELECT pg_reload_conf();

Access traces at http://localhost:16686. Look for service name pg_trickle.

When the endpoint is configured, the existing monitoring cadence also sends a bounded OTLP/HTTP metrics batch to {endpoint}/v1/metrics. It contains the target freshness, exact p95, and breach-duration gauges in seconds. Metrics use bounded db_oid, db_name, schema, and name attributes plus the controller status; missing percentile evidence is omitted.

Grafana Tempo

# docker-compose.yaml excerpt
tempo:
  image: grafana/tempo:latest
  ports:
    - "4318:4318"   # OTLP HTTP
    - "3200:3200"   # Tempo HTTP API

ALTER SYSTEM SET pg_trickle.otel_endpoint = 'http://tempo:4318';

OpenTelemetry Collector (recommended for production)

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      http:
        endpoint: "0.0.0.0:4318"

exporters:
  otlp:
    endpoint: "your-backend:4317" # Collector export remains gRPC
    tls:
      insecure: false

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      exporters: [otlp]

ALTER SYSTEM SET pg_trickle.otel_endpoint = 'http://otel-collector:4318';

Failure Behavior

pg_trickle's trace export is best-effort:

Important: Trace export failures never block or delay refresh cycles. Monitoring refresh latency with pgtrickle.sla_summary() is unaffected by OTLP endpoint health.

Verifying Trace Export

Check that trace context is captured

-- After enabling trace propagation and making a write:
SET pg_trickle.trace_id = '00-aaaabbbbccccdddd0000111122223333-0102030405060708-01';
INSERT INTO my_source_table VALUES (...);

-- Check the change buffer (replace <oid> with the source table OID):
SELECT __pgt_trace_context
FROM pgtrickle_changes.changes_<oid>
ORDER BY ctid DESC
LIMIT 5;
-- Should return the traceparent you set

Check the OTLP/HTTP endpoint

# Quick check: verify the collector accepts the metrics route
curl -i -X POST http://localhost:4318/v1/metrics \
  -H 'Content-Type: application/json' \
  --data '{"resourceMetrics":[]}'

Span Attributes

Refresh spans exported by pg_trickle include:

Stable Span Names

These names are part of the v0.97 compatibility surface. Dashboards and collector routing can match them without depending on implementation details:

The OTLP scope version is the packaged pg_trickle version. This lets a collector or trace backend identify the exact extension build that emitted a span.

Troubleshooting

Spans not appearing in the collector

1. Verify enable_trace_propagation = true:

   SHOW pg_trickle.enable_trace_propagation;
   

2. Verify otel_endpoint is set and reachable:

   SHOW pg_trickle.otel_endpoint;
   

3. Check PostgreSQL logs for OTLP export warnings:

   grep -i "otel\|otlp\|trace" /var/log/postgresql/postgresql.log
   

4. Ensure the collector is listening for OTLP/HTTP on the configured port and accepts /v1/traces and /v1/metrics.

__pgt_trace_context column missing

The column is added automatically when upgrading. If it is missing from a change buffer table:

-- Re-run the migration (idempotent):
ALTER TABLE pgtrickle_changes.changes_<oid>
  ADD COLUMN IF NOT EXISTS __pgt_trace_context TEXT;

Integration Test

A dockerized integration test against a local OTLP collector is available under tests/e2e_otel_tests.rs. It verifies:

• Span export success to a live collector
• Timeout and endpoint-failure handling (export is skipped, refresh completes)
• Trace context round-trip from session GUC through change buffer to span

Run with:

just test-e2e -- --test e2e_otel_tests

See also: CONFIGURATION.md · TROUBLESHOOTING.md