Skip to content

Credential Isolation

One of the most important security practices for a RockLake deployment is keeping the credentials for the catalog plane (RockLake's access to SlateDB's SST files and WAL segments) completely separate from the credentials for the data plane (DuckDB's access to Parquet data files). When credentials are isolated, a compromise of any single component can only access what that component legitimately needs — it cannot pivot to the other plane. This page explains why isolation matters, how to implement it on each major cloud provider, and how to verify that your isolation is effective.

Why Credential Isolation Matters

Without isolation, a single set of credentials provides access to both your catalog metadata and your analytical data. This creates unnecessary risk:

Compromised catalog → data exfiltration. If an attacker gains the credentials used by RockLake (perhaps through a configuration file in a compromised container or a logged environment variable), they can use those credentials to read all your Parquet data files — even though RockLake itself never reads those files. The credentials did not need to grant data access, but they did.

Compromised data writer → catalog corruption. If an attacker gains the write credentials used by a data ingestion pipeline, without isolation they could also write to the catalog prefix — corrupting catalog metadata or injecting malicious file registrations that point DuckDB at attacker-controlled files.

Broad blast radius. Without isolation, the failure mode for any credential compromise is "full lakehouse access." With isolation, the failure mode is either "catalog metadata access" (limited impact on confidentiality) or "data access" (limited impact on integrity), never both simultaneously.

The principle of least privilege — giving each component only the permissions it needs to function — is the right design for any production system, and the catalog/data plane separation makes it unusually clean to implement for RockLake.

What Each Component Needs

Before configuring credentials, map out what each component actually accesses:

RockLake (catalog plane): - Needs: Read/write/delete on the catalog prefix (e.g., s3://bucket/catalog/) - Needs: List on the catalog prefix (for SST file enumeration during manifest reconciliation) - Does NOT need: Any access to the data prefix (e.g., s3://bucket/data/) - Does NOT need: Access to other buckets

DuckDB data writer (ingestion pipelines): - Needs: Write and delete on the data prefix (e.g., s3://bucket/data/) - Needs: List on the data prefix (for pipeline coordination) - Needs: Write to RockLake via PG wire (to register files in the catalog) - Does NOT need: Direct access to the catalog prefix - Does NOT need: Read access to catalog SST files

DuckDB data reader (query execution): - Needs: Read on the data prefix (e.g., s3://bucket/data/) - Needs: Read from RockLake via PG wire (to look up catalog metadata) - Does NOT need: Any direct bucket access beyond the data prefix - Does NOT need: Write access anywhere

AWS S3: IAM Policy Design

Create three separate IAM identities: one for RockLake, one for data writers, one for data readers.

RockLake Catalog Policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CatalogObjectOperations",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:HeadObject"
      ],
      "Resource": "arn:aws:s3:::my-lakehouse/catalog/*"
    },
    {
      "Sid": "CatalogListOperations",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::my-lakehouse",
      "Condition": {
        "StringLike": {
          "s3:prefix": ["catalog/*", "catalog/"]
        }
      }
    }
  ]
}

Data Writer Policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DataWriteOperations",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:HeadObject"
      ],
      "Resource": "arn:aws:s3:::my-lakehouse/data/*"
    },
    {
      "Sid": "DataListOperations",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::my-lakehouse",
      "Condition": {
        "StringLike": {
          "s3:prefix": ["data/*", "data/"]
        }
      }
    }
  ]
}

Data Reader Policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DataReadOperations",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:HeadObject"
      ],
      "Resource": "arn:aws:s3:::my-lakehouse/data/*"
    },
    {
      "Sid": "DataListForPlanning",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::my-lakehouse",
      "Condition": {
        "StringLike": {
          "s3:prefix": ["data/*", "data/"]
        }
      }
    }
  ]
}

Creating IAM Roles

For EC2/ECS/EKS workloads, create IAM roles instead of static users:

# Create roles
aws iam create-role --role-name RockLakeCatalogRole \
  --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"ec2.amazonaws.com"},"Action":"sts:AssumeRole"}]}'

aws iam create-role --role-name DuckDBWriterRole \
  --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"ec2.amazonaws.com"},"Action":"sts:AssumeRole"}]}'

aws iam create-role --role-name DuckDBReaderRole \
  --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"ec2.amazonaws.com"},"Action":"sts:AssumeRole"}]}'

# Create and attach policies
aws iam create-policy --policy-name RockLakeCatalogPolicy \
  --policy-document file://rocklake-catalog-policy.json
aws iam attach-role-policy --role-name RockLakeCatalogRole \
  --policy-arn arn:aws:iam::123456789012:policy/RockLakeCatalogPolicy

# Repeat for writer and reader roles...

Verifying the Isolation

Test that the RockLake role cannot access data files:

# Assume the RockLake role
aws sts assume-role \
  --role-arn arn:aws:iam::123456789012:role/RockLakeCatalogRole \
  --role-session-name test-isolation

# Export the temporary credentials
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_SESSION_TOKEN=...

# This should SUCCEED (catalog access)
aws s3 ls s3://my-lakehouse/catalog/
# Expected: list of SST files

# This should FAIL (data access denied)
aws s3 ls s3://my-lakehouse/data/
# Expected: AccessDenied error

If the data access check succeeds when it should fail, your policy conditions are not working correctly. Review the StringLike condition syntax in the ListBucket statement.

Google Cloud Storage: Service Account Isolation

Three Service Accounts

# Create the service accounts
gcloud iam service-accounts create rocklake-catalog \
  --display-name="RockLake Catalog" --project=my-project

gcloud iam service-accounts create duckdb-writer \
  --display-name="DuckDB Data Writer" --project=my-project

gcloud iam service-accounts create duckdb-reader \
  --display-name="DuckDB Data Reader" --project=my-project

Scoped IAM Bindings

Grant each service account access only to its respective prefix using IAM conditions:

STORAGE_ID=$(gcloud storage buckets describe gs://my-lakehouse \
  --format="value(name)")

# RockLake: catalog prefix only
gcloud storage buckets add-iam-policy-binding gs://my-lakehouse \
  --member="serviceAccount:rocklake-catalog@my-project.iam.gserviceaccount.com" \
  --role="roles/storage.objectAdmin" \
  --condition='title=CatalogOnly,expression=resource.name.startsWith("projects/_/buckets/my-lakehouse/objects/catalog/")'

# DuckDB writer: data prefix, write access
gcloud storage buckets add-iam-policy-binding gs://my-lakehouse \
  --member="serviceAccount:duckdb-writer@my-project.iam.gserviceaccount.com" \
  --role="roles/storage.objectAdmin" \
  --condition='title=DataWrite,expression=resource.name.startsWith("projects/_/buckets/my-lakehouse/objects/data/")'

# DuckDB reader: data prefix, read access
gcloud storage buckets add-iam-policy-binding gs://my-lakehouse \
  --member="serviceAccount:duckdb-reader@my-project.iam.gserviceaccount.com" \
  --role="roles/storage.objectViewer" \
  --condition='title=DataRead,expression=resource.name.startsWith("projects/_/buckets/my-lakehouse/objects/data/")'

Azure Blob Storage: Managed Identity Isolation

Three Managed Identities

az identity create --name rocklake-catalog --resource-group rocklake-rg
az identity create --name duckdb-writer --resource-group rocklake-rg
az identity create --name duckdb-reader --resource-group rocklake-rg

STORAGE_ID=$(az storage account show --name mylakehouse \
  --resource-group rocklake-rg --query id --output tsv)

Scoped RBAC Assignments

# RockLake: catalog container access
CATALOG_SCOPE="${STORAGE_ID}/blobServices/default/containers/catalog"

ROCKLAKE_PRINCIPAL=$(az identity show --name rocklake-catalog \
  --resource-group rocklake-rg --query principalId --output tsv)

az role assignment create \
  --assignee-object-id $ROCKLAKE_PRINCIPAL \
  --assignee-principal-type ServicePrincipal \
  --role "Storage Blob Data Contributor" \
  --scope $CATALOG_SCOPE

# DuckDB writer: data container write access
DATA_SCOPE="${STORAGE_ID}/blobServices/default/containers/data"

WRITER_PRINCIPAL=$(az identity show --name duckdb-writer \
  --resource-group rocklake-rg --query principalId --output tsv)

az role assignment create \
  --assignee-object-id $WRITER_PRINCIPAL \
  --assignee-principal-type ServicePrincipal \
  --role "Storage Blob Data Contributor" \
  --scope $DATA_SCOPE

# DuckDB reader: data container read access
READER_PRINCIPAL=$(az identity show --name duckdb-reader \
  --resource-group rocklake-rg --query principalId --output tsv)

az role assignment create \
  --assignee-object-id $READER_PRINCIPAL \
  --assignee-principal-type ServicePrincipal \
  --role "Storage Blob Data Reader" \
  --scope $DATA_SCOPE

Deploying with Isolated Credentials

Kubernetes Example

In Kubernetes, create separate pods/deployments with different service accounts:

# RockLake deployment with catalog credentials
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rocklake
spec:
  template:
    spec:
      serviceAccountName: rocklake-sa  # Has catalog access
      containers:
        - name: rocklake
          image: ghcr.io/rocklake/rocklake:latest
          args: [serve, --catalog, s3://my-lakehouse/catalog/, --bind, 0.0.0.0:5432]
---
# DuckDB ingestion job with data writer credentials
apiVersion: batch/v1
kind: Job
metadata:
  name: data-ingestion
spec:
  template:
    spec:
      serviceAccountName: duckdb-writer-sa  # Has data write access
      containers:
        - name: ingestion
          image: duckdb/duckdb:latest
          # DuckDB connects to RockLake via PG wire for catalog,
          # and uses AWS credentials for direct S3 data writes

Docker Compose Example

services:
  rocklake:
    image: ghcr.io/rocklake/rocklake:latest
    command: serve --catalog s3://bucket/catalog/ --bind 0.0.0.0:5432
    environment:
      # Catalog-only credentials
      AWS_ACCESS_KEY_ID: ${ROCKLAKE_AWS_KEY}
      AWS_SECRET_ACCESS_KEY: ${ROCKLAKE_AWS_SECRET}
      AWS_REGION: us-east-1

  ingestion:
    image: my-pipeline:latest
    environment:
      # Data-write credentials
      AWS_ACCESS_KEY_ID: ${WRITER_AWS_KEY}
      AWS_SECRET_ACCESS_KEY: ${WRITER_AWS_SECRET}
      AWS_REGION: us-east-1
      # PG wire endpoint for catalog registration
      ROCKLAKE_ENDPOINT: rocklake:5432

Network-Level Isolation

Beyond IAM/RBAC isolation, consider network-level controls:

VPC/VNet placement. Run RockLake and DuckDB within the same VPC/VNet, with the RockLake port (5432) accessible only within the VPC. This prevents external access to the catalog even if credentials are leaked — an attacker would also need network access.

Security groups/firewall rules. Restrict inbound connections to RockLake's port to only the IP ranges of known DuckDB clients:

# AWS Security Group: allow port 5432 only from the ingestion subnet
aws ec2 authorize-security-group-ingress \
  --group-id sg-rocklake \
  --protocol tcp \
  --port 5432 \
  --cidr 10.0.0.0/8

# Deny everything else
aws ec2 revoke-security-group-ingress \
  --group-id sg-rocklake \
  --protocol tcp \
  --port 5432 \
  --cidr 0.0.0.0/0

Object storage VPC endpoints. Use VPC endpoints for S3 (AWS) or Private Service Connect (GCS) to keep all catalog I/O within the private network. This prevents catalog SST files from transiting the public internet even though S3 is technically a public service.

Auditing Credential Usage

Enable access logging to verify that the isolation is working and detect any unexpected access patterns:

# AWS: Enable S3 server access logging
aws s3api put-bucket-logging \
  --bucket my-lakehouse \
  --bucket-logging-status '{
    "LoggingEnabled": {
      "TargetBucket": "my-lakehouse-logs",
      "TargetPrefix": "s3-access-"
    }
  }'

# Alert if the catalog credential accesses the data prefix
# (this should never happen if isolation is configured correctly)

Set up monitoring rules that alert when: - The RockLake IAM role accesses any path outside catalog/ - The DuckDB writer role accesses any path outside data/ - Any identity accesses the catalog/ prefix directly (bypassing RockLake)

Unexpected cross-prefix access is a signal that either the IAM configuration is wrong or credentials have been misappropriated.

Further Reading