Deploying with MinIO¶

MinIO is a high-performance, S3-compatible object storage server that you can run on your own infrastructure. It is the ideal backend for RockLake when you cannot or do not want to use cloud storage: on-premises data centers, air-gapped environments, development workstations without cloud access, CI/CD pipelines, and edge deployments where data sovereignty prevents sending catalog metadata to a public cloud. Because MinIO implements the S3 API, RockLake treats it identically to Amazon S3 — the only difference is the endpoint URL.

This guide covers running MinIO alongside RockLake for development, deploying MinIO in single-node and distributed configurations for production-like environments, configuring RockLake to connect to MinIO, and the common pitfalls that appear when using S3-compatible APIs.

Why MinIO?¶

MinIO occupies a specific niche: it is the best choice when you need S3 API compatibility without S3's cloud costs or data egress restrictions. The most common use cases are:

Local development. Running a full lakehouse stack on a developer laptop with no cloud credentials, no network dependency, and no cost. MinIO starts in seconds and provides the full S3 API for testing RockLake, DuckDB, and data pipelines against a realistic object-store backend.

CI/CD pipelines. Integration tests that exercise the full storage stack need a real object store, not a mock. MinIO as a Docker service in a CI pipeline gives tests realistic latency and error modes without requiring cloud credentials in CI secrets.

On-premises production. Organizations that require data to remain within their own data centers can deploy MinIO on bare metal or on-premises Kubernetes. MinIO's distributed mode provides erasure coding and cross-node redundancy comparable to cloud storage.

Edge and air-gapped environments. Deployments where the RockLake sidecar runs alongside processing systems in environments without internet connectivity.

Quick Start: MinIO with Docker¶

The fastest path to a working local environment combines MinIO and RockLake using Docker Compose:

# docker-compose.yml
services:
  minio:
    image: minio/minio:latest
    command: server /data --console-address ":9001"
    ports:
      - "9000:9000"   # S3 API endpoint
      - "9001:9001"   # MinIO web console
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: minioadmin
    volumes:
      - minio-data:/data
    healthcheck:
      test: ["CMD", "mc", "ready", "local"]
      interval: 5s
      timeout: 3s
      retries: 5

  minio-init:
    image: minio/mc:latest
    depends_on:
      minio:
        condition: service_healthy
    entrypoint: >
      /bin/sh -c "
        mc alias set local http://minio:9000 minioadmin minioadmin;
        mc mb local/my-lakehouse;
        mc mb local/my-lakehouse-data;
        echo 'Buckets created';
      "

  rocklake:
    image: ghcr.io/rocklake/rocklake:latest
    depends_on:
      minio:
        condition: service_healthy
      minio-init:
        condition: service_completed_successfully
    command: >
      serve
      --catalog s3://my-lakehouse/catalog/
      --bind 0.0.0.0:5432
    ports:
      - "5432:5432"
    environment:
      AWS_ENDPOINT_URL: http://minio:9000
      AWS_ACCESS_KEY_ID: minioadmin
      AWS_SECRET_ACCESS_KEY: minioadmin
      AWS_REGION: us-east-1
      ROCKLAKE_S3_PATH_STYLE: "true"

volumes:
  minio-data:

Start the stack:

docker compose up -d

# Verify everything is running
docker compose ps

# Watch RockLake's logs
docker compose logs -f rocklake

Expected RockLake startup output:

INFO  RockLake v0.8.0 starting
INFO  Storage backend: aws-s3 (endpoint: http://minio:9000)
INFO  Path-style addressing: enabled
INFO  Catalog path: s3://my-lakehouse/catalog/
INFO  Opening SlateDB...
INFO  Catalog initialized (new catalog)
INFO  Listening on 0.0.0.0:5432
INFO  Ready to accept connections

Connect DuckDB:

INSTALL ducklake;
LOAD ducklake;

ATTACH 'ducklake:host=localhost;port=5432' AS lake;

CREATE SCHEMA lake.analytics;
CREATE TABLE lake.analytics.events (
  id BIGINT,
  name VARCHAR,
  ts TIMESTAMP
);
INSERT INTO lake.analytics.events VALUES (1, 'hello', NOW());
SELECT * FROM lake.analytics.events;

Without Docker: Running MinIO Directly¶

If you prefer to run MinIO directly as a binary (useful for development when Docker is not available):

# Download the MinIO binary for macOS
curl -O https://dl.min.io/server/minio/release/darwin-amd64/minio
chmod +x minio

# For Linux:
# curl -O https://dl.min.io/server/minio/release/linux-amd64/minio

# Start MinIO
export MINIO_ROOT_USER=minioadmin
export MINIO_ROOT_PASSWORD=minioadmin
mkdir -p ~/minio-data
./minio server ~/minio-data --console-address ":9001"

Create a bucket using the MinIO CLI (mc):

# Install mc
curl -O https://dl.min.io/client/mc/release/darwin-amd64/mc
chmod +x mc

# Configure the alias
./mc alias set local http://localhost:9000 minioadmin minioadmin

# Create the catalog bucket
./mc mb local/my-lakehouse

Start RockLake:

export AWS_ENDPOINT_URL=http://localhost:9000
export AWS_ACCESS_KEY_ID=minioadmin
export AWS_SECRET_ACCESS_KEY=minioadmin
export AWS_REGION=us-east-1

rocklake serve \
  --catalog s3://my-lakehouse/catalog/ \
  --bind 127.0.0.1:5432 \
  --s3-path-style

Critical Configuration: Path-Style Addressing¶

Without path-style addressing, you will see errors like:

Error: NoSuchBucket: The specified bucket does not exist

or connection refused errors as RockLake tries to connect to a hostname like my-lakehouse.localhost that does not exist.

Endpoint Configuration¶

RockLake needs to know MinIO's endpoint URL. There are two ways to configure this:

Environment variable (recommended):

export AWS_ENDPOINT_URL=http://localhost:9000

Command-line flag:

rocklake serve \
  --catalog s3://my-lakehouse/catalog/ \
  --s3-endpoint http://localhost:9000 \
  --s3-path-style \
  --bind 127.0.0.1:5432

For Docker deployments where RockLake and MinIO are on the same Docker network, use the service name as the hostname:

AWS_ENDPOINT_URL=http://minio:9000  # Inside Docker network

MinIO Authentication¶

MinIO uses the same access key / secret key model as AWS S3. The credentials you set with MINIO_ROOT_USER and MINIO_ROOT_PASSWORD when starting MinIO are the root credentials. For production MinIO deployments, create dedicated users and policies (equivalent to IAM policies) through MinIO's identity management:

# Using mc
mc admin user add local rocklake-user rocklake-password

# Create a policy
cat > rocklake-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket"],
      "Resource": ["arn:aws:s3:::my-lakehouse/*", "arn:aws:s3:::my-lakehouse"]
    }
  ]
}
EOF

mc admin policy create local rocklake-policy rocklake-policy.json
mc admin policy attach local rocklake-policy --user rocklake-user

Then use the dedicated credentials for RockLake:

export AWS_ACCESS_KEY_ID=rocklake-user
export AWS_SECRET_ACCESS_KEY=rocklake-password

Distributed MinIO for Production-Like Environments¶

For environments that need production-comparable durability (CI integration environments, staging clusters, on-premises deployments), run MinIO in distributed mode with erasure coding:

# docker-compose.yml for distributed MinIO (4-node setup)
services:
  minio1:
    image: minio/minio:latest
    command: server --console-address ":9001"
      http://minio{1...4}/data
    ports:
      - "9001:9001"
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: minioadmin
    volumes:
      - minio1-data:/data
    networks:
      - minio-net

  minio2:
    image: minio/minio:latest
    command: server --console-address ":9001"
      http://minio{1...4}/data
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: minioadmin
    volumes:
      - minio2-data:/data
    networks:
      - minio-net

  minio3:
    image: minio/minio:latest
    command: server --console-address ":9001"
      http://minio{1...4}/data
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: minioadmin
    volumes:
      - minio3-data:/data
    networks:
      - minio-net

  minio4:
    image: minio/minio:latest
    command: server --console-address ":9001"
      http://minio{1...4}/data
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: minioadmin
    volumes:
      - minio4-data:/data
    networks:
      - minio-net

  haproxy:
    image: haproxy:latest
    ports:
      - "9000:9000"
    volumes:
      - ./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg
    depends_on:
      - minio1
      - minio2
      - minio3
      - minio4
    networks:
      - minio-net

  rocklake:
    image: ghcr.io/rocklake/rocklake:latest
    depends_on:
      - haproxy
    command: >
      serve
      --catalog s3://my-lakehouse/catalog/
      --bind 0.0.0.0:5432
    ports:
      - "5432:5432"
    environment:
      AWS_ENDPOINT_URL: http://haproxy:9000
      AWS_ACCESS_KEY_ID: minioadmin
      AWS_SECRET_ACCESS_KEY: minioadmin
      AWS_REGION: us-east-1
      ROCKLAKE_S3_PATH_STYLE: "true"
    networks:
      - minio-net

volumes:
  minio1-data:
  minio2-data:
  minio3-data:
  minio4-data:

networks:
  minio-net:
    driver: bridge

A 4-node MinIO deployment with erasure coding provides N/2 redundancy: the cluster can lose any 2 nodes and continue operating. For CI, a 2-node setup is typically sufficient.

TLS with MinIO¶

For any environment beyond local development, enable TLS on MinIO. This requires generating or providing certificates:

# For development: generate self-signed certificates
mkdir -p ~/minio-certs
openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
  -keyout ~/minio-certs/private.key \
  -out ~/minio-certs/public.crt \
  -subj "/CN=minio.local"

# Start MinIO with TLS
./minio server ~/minio-data \
  --certs-dir ~/minio-certs

Tell RockLake to use HTTPS and where to find the CA certificate:

export AWS_ENDPOINT_URL=https://localhost:9000
# For self-signed certificates, add the CA to the system trust store
# or use the --tls-ca-bundle flag:
rocklake serve \
  --catalog s3://my-lakehouse/catalog/ \
  --s3-endpoint https://localhost:9000 \
  --s3-path-style \
  --tls-ca-bundle ~/minio-certs/public.crt \
  --bind 127.0.0.1:5432

Performance Characteristics¶

MinIO on local hardware typically provides lower latency than cloud object storage:

Local MinIO is 5–20× faster than S3 Standard for catalog operations. This makes it attractive for development workflows and integration testing where fast iteration matters. It also means performance tests run against MinIO may underestimate the latency you will see in production against S3.

Troubleshooting¶

NoSuchBucket despite bucket existing: Forgot --s3-path-style. Enable it with the flag or environment variable.

SignatureDoesNotMatch: The access key or secret key is wrong. Verify against the MinIO credentials you configured.

Connection refused or timeout: MinIO is not running or the endpoint URL is wrong. Check docker ps or ./minio server output to confirm MinIO is listening on the expected port.

Bucket names with uppercase letters or special characters: MinIO follows S3's bucket naming rules: lowercase letters, numbers, and hyphens only. Bucket names with uppercase letters or dots cause signature calculation mismatches.

RockLake connects but reads/writes fail: The bucket exists but the user does not have the required permissions. Check MinIO's access policies with mc admin policy list local.

Further Reading¶

• Docker Deployment — Running RockLake and MinIO together with Docker Compose
• AWS S3 Deployment — When you are ready to move from MinIO to AWS S3 in production
• Credential Isolation — Applying the least-privilege principle to MinIO deployments
• Object Store Durability — The durability model and its implications for MinIO vs. cloud storage