Development Setup¶

This page walks you through setting up a complete local development environment for RockLake. By the end, you will be able to build all crates, run the full test suite, start a local RockLake instance, connect to it with DuckDB, and make changes with confidence that the test suite will catch regressions. The process is straightforward — if you can compile Rust code and have basic command-line skills, you can be contributing within 30 minutes.

Prerequisites¶

Required¶

Rust installation:

# Install rustup (manages Rust versions and components)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Ensure you have the latest stable toolchain
rustup update stable

# Verify installation
rustc --version    # Should show 1.80.0 or higher
cargo --version    # Should show corresponding cargo version

C compiler (needed for some native dependencies like ring, openssl-sys):

xcode-select --install

sudo apt install build-essential pkg-config libssl-dev

sudo dnf install gcc openssl-devel

Recommended¶

DuckDB installation:

# macOS (Homebrew)
brew install duckdb

# Or download from https://duckdb.org/docs/installation/

Docker (for MinIO):

# macOS
brew install --cask docker

# Linux
curl -fsSL https://get.docker.com | sh

Clone and Build¶

# Clone the repository
git clone https://github.com/rocklake/rocklake.git
cd rocklake

# Build all crates in debug mode (faster compile, slower runtime)
cargo build

# Build in release mode (slower compile, faster runtime — for benchmarks)
cargo build --release

The first build downloads and compiles all dependencies. This takes 2–5 minutes depending on your machine. Subsequent builds are incremental and take seconds.

Workspace Structure¶

After cloning, the workspace contains:

rocklake/
├── Cargo.toml            # Workspace manifest (lists all member crates)
├── crates/
│   ├── rocklake-core/       # Foundation: types, keys, values, MVCC
│   ├── rocklake-catalog/    # Persistence: read/write operations, GC
│   ├── rocklake-sql/        # SQL classifier (pattern matching)
│   ├── rocklake-pgwire/     # PG wire protocol server (binary lives here)
│   ├── rocklake-ffi/        # C FFI for native DuckDB extension
│   ├── rocklake-datafusion/ # DataFusion catalog provider
├── tests/
│   ├── fixtures/             # Wire corpus and test data
│   └── golden/               # Golden test runner
├── docs/                     # Documentation source (MkDocs)
├── benchmarks/               # Benchmark results (JSON)
└── extension/                # DuckDB native extension (C++)

Dependency Graph¶

Crates depend on each other in a strict hierarchy:

graph TD
    pgwire[rocklake-pgwire<br/>Binary + protocol] --> sql[rocklake-sql<br/>SQL classifier]
    sql --> catalog[rocklake-catalog<br/>Persistence]
    catalog --> core[rocklake-core<br/>Types + encoding]
    ffi[rocklake-ffi<br/>C FFI] --> catalog
    datafusion[rocklake-datafusion<br/>DataFusion] --> catalog

You only need to understand the crates relevant to your change. Most contributions touch one or two crates.

Running Tests¶

# Run the complete test suite (all crates)
cargo test

# Run tests for a specific crate
cargo test -p rocklake-core
cargo test -p rocklake-catalog
cargo test -p rocklake-sql
cargo test -p rocklake-pgwire

# Run a specific test by name
cargo test test_schema_key_roundtrip

# Run tests with output visible (for print debugging)
cargo test -- --nocapture

# Run only integration tests
cargo test -p rocklake-catalog --test integration_tests

# Run property-based tests
cargo test -p rocklake-core --test property_tests

The complete test suite should pass on a fresh clone without any additional configuration. Tests use local filesystem storage by default (no S3 access required). If a test requires external services (MinIO, DuckDB), it is gated behind a feature flag or environment variable.

Test Performance¶

Running RockLake Locally¶

For end-to-end testing, run a local RockLake instance:

# Start RockLake with local filesystem storage
cargo run -- serve --catalog ./dev-catalog --bind 127.0.0.1:5432

# In another terminal, connect with psql
psql -h 127.0.0.1 -p 5432 -U rocklake

# Or connect with DuckDB
duckdb -c "ATTACH 'ducklake:postgresql://localhost:5432/rocklake' AS lake;"

The local filesystem storage is perfect for development — zero latency, no cloud credentials needed, and you can inspect the catalog files directly.

Using MinIO for S3-Compatible Testing¶

For testing with S3-compatible storage (more realistic than local filesystem):

# Start MinIO
docker run -p 9000:9000 -p 9001:9001 \
  -e MINIO_ROOT_USER=minioadmin \
  -e MINIO_ROOT_PASSWORD=minioadmin \
  minio/minio server /data --console-address :9001

# Create a bucket
mc alias set local http://localhost:9000 minioadmin minioadmin
mc mb local/rocklake-dev

# Start RockLake pointing to MinIO
AWS_ACCESS_KEY_ID=minioadmin \
AWS_SECRET_ACCESS_KEY=minioadmin \
AWS_ENDPOINT_URL=http://localhost:9000 \
cargo run -- serve --catalog s3://rocklake-dev/catalog/ --bind 127.0.0.1:5432

Editor Setup¶

VS Code¶

Recommended extensions:

• rust-analyzer — Full IDE features (completion, navigation, diagnostics)
• Even Better TOML — Syntax highlighting for Cargo.toml
• CodeLLDB — Debugging support

Settings (.vscode/settings.json):

{
  "rust-analyzer.check.command": "clippy",
  "rust-analyzer.cargo.features": "all",
  "editor.formatOnSave": true,
  "[rust]": {
    "editor.defaultFormatter": "rust-lang.rust-analyzer"
  }
}

IntelliJ/CLion¶

Install the Rust plugin. The workspace is automatically detected from Cargo.toml.

Neovim¶

Use rust-analyzer via the native LSP client or nvim-lspconfig. Configuration varies by setup.

Development Workflow¶

The standard workflow for contributing:

1. Create a Feature Branch¶

git checkout main
git pull origin main
git checkout -b fix/description-of-change

Branch naming conventions: - fix/ — Bug fixes - feat/ — New features - docs/ — Documentation changes - test/ — Test additions - refactor/ — Code restructuring

2. Make Your Changes¶

Edit files, add tests, update documentation as needed.

3. Validate Locally¶

# Format code (required — CI rejects unformatted code)
cargo fmt

# Lint (required — CI rejects clippy warnings)
cargo clippy --all-targets --all-features

# Run tests (required — CI rejects failing tests)
cargo test

4. Commit¶

git add .
git commit -m "fix: correct MVCC visibility at boundary snapshots"

Use Conventional Commits format: - feat: — New feature - fix: — Bug fix - docs: — Documentation only - test: — Adding or fixing tests - refactor: — Code change that neither fixes a bug nor adds a feature - perf: — Performance improvement

5. Push and Open PR¶

git push origin fix/description-of-change

Open a Pull Request on GitHub. The PR template asks for: - Description of the change - Related issue (if any) - Test coverage explanation - Breaking change notice (if applicable)

Documentation Development¶

RockLake's documentation is built with MkDocs Material:

# Install Python dependencies
pip install -r requirements-docs.txt

# Serve locally with hot reload (auto-refreshes on file changes)
mkdocs serve
# → Open http://127.0.0.1:8000

# Build the static site (for verifying production build)
mkdocs build --strict

The --strict flag makes MkDocs treat warnings as errors (broken links, missing pages). This is what CI uses — fix warnings before pushing.

Troubleshooting¶

Build Fails with Missing System Dependencies¶

# macOS: ensure Xcode CLI tools are installed
xcode-select --install

# Linux: install OpenSSL development headers
sudo apt install libssl-dev    # Debian/Ubuntu
sudo dnf install openssl-devel # Fedora/RHEL

Tests Fail on First Run¶

If tests fail immediately after cloning, verify: - Rust version: rustc --version (must be 1.80+) - Clean build: cargo clean && cargo build - No conflicting environment variables (AWS_*, ROCKLAKE_*)

Slow Compilation¶

• Use cargo build (not cargo build --release) for development
• The sccache tool can speed up repeated builds: cargo install sccache
• Incremental compilation is enabled by default in debug mode

Linker Errors on Linux¶

If you see linker errors related to OpenSSL or other system libraries:

# Install all development dependencies at once (Ubuntu/Debian)
sudo apt install build-essential pkg-config libssl-dev cmake

# Verify pkg-config can find OpenSSL
pkg-config --modversion openssl

Port Already in Use¶

If RockLake fails to start with "address already in use":

# Check what is using port 5432
lsof -i :5432

# Use a different port
cargo run -- serve --catalog ./dev-catalog --bind 127.0.0.1:5433

Common Development Tasks¶

Adding a Wire Corpus Entry¶

When you need to capture new SQL patterns from DuckDB:

# Start RockLake with verbose protocol logging
RUST_LOG=rocklake_pgwire=trace cargo run -- serve --catalog ./dev-catalog --bind 127.0.0.1:5433

# In another terminal, run DuckDB with the new operation
duckdb -c "
ATTACH 'ducklake:postgresql://localhost:5433/rocklake' AS lake;
USE lake;
CREATE SCHEMA test_schema;
"

# Copy the SQL from the log output to the corpus file
# tests/fixtures/wire-corpus/duckdb-X.Y.Z/new-operation.sql

Debugging a Specific Test¶

# Run a single test with full output and backtrace
RUST_BACKTRACE=1 cargo test test_name_here -- --nocapture

# Run with debug logging
RUST_LOG=debug cargo test test_name_here -- --nocapture

Profiling Performance¶

# Install flamegraph tool
cargo install flamegraph

# Generate a flamegraph of the benchmark suite
cargo flamegraph --bench catalog_bench -- --bench prefix_scan

# Or profile the running server
cargo build --release
flamegraph -- ./target/release/rocklake serve --catalog ./bench-catalog

Inspecting Storage Contents¶

During development, it can be useful to inspect what RockLake has written to storage:

# For local filesystem storage
ls -la ./dev-catalog/
find ./dev-catalog/ -type f | head -20

# For MinIO
mc ls local/rocklake-dev/ --recursive

Further Reading¶

• Architecture Guide — Understanding the codebase structure
• Testing — Writing and running tests
• Code Style — Formatting and conventions