Monitoring¶

A well-monitored RockLake deployment tells you three things at a glance: Is it healthy? Is it performing well? Is anything trending toward a problem? RockLake exposes Prometheus-compatible metrics that give you visibility into catalog operations, resource usage, storage interactions, and session state. Combined with proper alerting, these metrics let you catch issues before they affect users.

This page covers the metrics endpoint configuration, the complete metrics catalog with explanations, alerting rules for common failure modes, Grafana dashboard setup, and integration with cloud-native monitoring services.

Enabling Metrics¶

RockLake exposes metrics in Prometheus exposition format on a configurable HTTP endpoint:

rocklake \
    --catalog s3://bucket/catalog/ \
    --bind 0.0.0.0:5432 \
    --metrics-bind 0.0.0.0:9090 \
    --metrics-path /metrics

Or via environment variables:

export ROCKLAKE_METRICS_BIND=0.0.0.0:9090
export ROCKLAKE_METRICS_PATH=/metrics

The metrics endpoint is a plain HTTP server (separate from the PG-wire listener) that responds to GET requests with the current metric values in Prometheus text format.

Prometheus Scrape Configuration¶

scrape_configs:
  - job_name: 'rocklake'
    scrape_interval: 15s
    static_configs:
      - targets: ['rocklake:9090']
    metrics_path: /metrics

For Kubernetes with Prometheus Operator:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: rocklake
  namespace: rocklake
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: rocklake
  endpoints:
    - port: metrics
      interval: 15s
      path: /metrics

Complete Metrics Catalog¶

The following metrics are emitted by CatalogMetrics::render_prometheus() in crates/rocklake-catalog/src/metrics.rs. All are exposed in Prometheus text format on the configured --metrics-path endpoint.

Snapshot / Catalog Metrics¶

Object Storage Metrics¶

These track interactions with the underlying object store (S3/GCS/Azure/local):

Session Metrics¶

Writer Metrics¶

CDC Data-Quality Metrics¶

Alerting Rules¶

Critical Alerts (Page Immediately)¶

groups:
  - name: rocklake-critical
    rules:
      - alert: RockLakeDown
        expr: up{job="rocklake"} == 0
        for: 30s
        labels:
          severity: critical
        annotations:
          summary: "RockLake is down"
          description: "No metrics received from RockLake for 30 seconds"

      - alert: RockLakeSessionsExhausted
        expr: rocklake_active_sessions / rocklake_max_sessions > 0.95
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "RockLake session capacity >95% — new connections will be rejected"

Warning Alerts (Investigate Within Hours)¶

      - alert: RockLakeStorageThrottling
        expr: rate(rocklake_object_store_throttles_total[5m]) > 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Object storage is throttling RockLake requests"

      - alert: RockLakeHighRetryRate
        expr: rate(rocklake_object_store_retries_total[5m]) > 5
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Elevated object-store retry rate — transient failures"

      - alert: RockLakeWriterEpochStale
        expr: rocklake_writer_epoch_age_ms > 300000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Writer epoch is more than 5 minutes old — check for stuck writer"

      - alert: RockLakeCDCMismatch
        expr: increase(rocklake_cdc_record_count_mismatch_total[1h]) > 0
        labels:
          severity: warning
        annotations:
          summary: "CDC record-count mismatch detected — Parquet file row counts differ from catalog metadata"

Grafana Dashboard¶

Recommended Panels¶

A comprehensive RockLake dashboard includes these panels:

Row 1: Overview

• Current sessions (gauge) — rocklake_active_sessions
• Session capacity (gauge) — rocklake_active_sessions / rocklake_max_sessions
• Snapshots/min (graph) — rate(rocklake_snapshots_created_total[1m])

Row 2: Object Storage

• Storage requests/sec (graph) — rate(rocklake_object_store_requests_total[1m])
• Bytes read/written (graph) — rate(rocklake_object_store_bytes_read_total[1m]) / rate(rocklake_object_store_bytes_written_total[1m])
• Throttle rate (graph) — rate(rocklake_object_store_throttles_total[1m])
• Retry rate (graph) — rate(rocklake_object_store_retries_total[1m])

Row 3: Writer Health

• Writer epoch age (graph) — rocklake_writer_epoch_age_ms
• Files per snapshot (graph) — rocklake_files_per_snapshot

Row 4: Data Quality

• CDC mismatch total (stat) — rocklake_cdc_record_count_mismatch_total
• Keys scanned per query (graph) — rocklake_last_query_keys_scanned

Cloud-Native Monitoring Integration¶

AWS CloudWatch¶

Use the CloudWatch Agent's Prometheus scraping to forward metrics:

{
  "metrics": {
    "metrics_collected": {
      "prometheus": {
        "prometheus_config_path": "/etc/cwagent/prometheus.yaml",
        "emf_processor": {
          "metric_namespace": "RockLake",
          "metric_unit": {
            "rocklake_writer_epoch_age_ms": "Milliseconds",
            "rocklake_active_sessions": "Count"
          }
        }
      }
    }
  }
}

Google Cloud Managed Prometheus¶

On GKE with Managed Prometheus, the ServiceMonitor configuration works automatically — Google scrapes Prometheus endpoints and stores metrics in Cloud Monitoring.

Datadog¶

Use the Datadog Agent's OpenMetrics integration:

# datadog-agent/conf.d/openmetrics.d/conf.yaml
instances:
  - prometheus_url: http://rocklake:9090/metrics
    namespace: rocklake
    metrics:
      - rocklake_*

What "Normal" Looks Like¶

Understanding baseline behavior helps identify anomalies:

OpenTelemetry Tracing (v0.39.0)¶

RockLake can export distributed traces to any OpenTelemetry-compatible backend (Jaeger, Tempo, OTLP/HTTP collectors).

Enabling OTLP Export¶

rocklake serve \
    --catalog s3://bucket/catalog/ \
    --otlp-endpoint http://jaeger:4318

Or via environment variable:

export ROCKLAKE_OTLP_ENDPOINT=http://jaeger:4318
rocklake serve --catalog s3://bucket/catalog/

When --otlp-endpoint is not set (the default), no spans are exported and there is zero overhead.

Instrumented Operations¶

The following catalog write paths are instrumented with OTLP spans:

Jaeger Quick-Start¶

# docker-compose.yml
services:
  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "4318:4318"   # OTLP HTTP
      - "16686:16686" # Jaeger UI

Then set --otlp-endpoint http://localhost:4318 and open http://localhost:16686 to view traces.

Further Reading¶

• Diagnostics — rocklake diagnose health report and orphan file sweep
• Health Checks — Probing operational readiness
• Logging — Complementary diagnostic information
• Troubleshooting — Investigating alerts
• Configuration — Metrics endpoint configuration