# 🏰 EVM Cloud
> Deploy, manage, and scale EVM blockchain data infrastructure — Nodes, RPC proxies, indexers, databases, and networking — on any cloud or bare metal
import { ArchitectureDiagram } from '../../components/ArchitectureDiagram'
import { DataFlowDiagram } from '../../components/DataFlowDiagram'
## Architecture
How evm-cloud's Terraform modules fit together to deploy a complete EVM indexing stack.
### High-Level Overview
### Data Flow at Runtime
### Layer Model
evm-cloud separates concerns into two layers. The `workload_mode` variable controls how much Terraform manages.
#### Layer 1: Infrastructure
Everything needed before workloads can run.
| Component | What Terraform Creates |
| -------------- | ------------------------------------------------------------------------ |
| **Networking** | VPC, public/private subnets, security groups, NAT gateway, VPC endpoints |
| **Compute** | EC2 instances, EKS cluster + node groups, or k3s host |
| **Database** | RDS PostgreSQL instance (or outputs for BYODB) |
| **IAM** | Instance profiles, roles, policies for Secrets Manager access |
| **Secrets** | AWS Secrets Manager entries for DB credentials, RPC keys |
#### Layer 2: Workloads
The actual containers and configuration that run on top of infrastructure.
| Component | What Gets Deployed |
| -------------------- | -------------------------------------------------------------------------------------- |
| **eRPC** | RPC proxy container with your `erpc.yaml` config |
| **rindexer** | Indexer container with your `rindexer.yaml` + ABI files |
| **Config injection** | YAML/ABI files delivered to containers via bind mounts, ConfigMaps, or Secrets Manager |
#### Workload Modes
```hcl
variable "workload_mode" {
type = string
default = "terraform" # or "external"
}
```
| Mode | Behavior | Use Case |
| ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| `terraform` | Terraform manages both Layer 1 and Layer 2. Containers are deployed via cloud-init (EC2), Helm (k3s/EKS), or SSH provisioner (bare metal). | Simple deployments, dev/staging, teams that want one `terraform apply` for everything. |
| `external` | Terraform manages Layer 1 only. Outputs a `workload_handoff` object with connection details, endpoints, and credentials for external tools to deploy workloads. | Teams with existing CI/CD, GitOps (ArgoCD/Flux), or Helm-based deploy pipelines. Separates infra and app lifecycles. |
**`workload_handoff` output** (available when `workload_mode = "external"`):
```hcl
output "workload_handoff" {
value = {
runtime = {
ec2 = { public_ip, private_ip, ssh_user, ssh_key_name }
eks = { cluster_name, cluster_endpoint, kubeconfig_cmd }
k3s = { kubeconfig, node_ip }
}
database = {
host, port, username, password_secret_arn, database_name
}
networking = {
vpc_id, private_subnet_ids, security_group_ids
}
config = {
erpc_config_path, rindexer_config_path, abi_dir_path
}
}
}
```
### Module Map
| Module | Path | Purpose |
| -------------------------- | ----------------------------------- | --------------------------------------------------------------------------------------------------- |
| **core/capabilities** | `modules/core/capabilities/` | Provider-neutral capability contracts. Defines the interface every provider adapter must implement. |
| **core/k8s/k3s-bootstrap** | `modules/core/k8s/k3s-bootstrap/` | Installs k3s on any host via SSH. Used by both `aws/k3s-host` and `bare_metal` providers. |
| **aws/networking** | `modules/providers/aws/networking/` | VPC, subnets, security groups, NAT gateway, VPC endpoints. |
| **aws/ec2** | `modules/providers/aws/ec2/` | EC2 instance with Docker Compose workloads. Cloud-init for setup, bind mounts for config. |
| **aws/eks\_cluster** | `modules/providers/aws/eks/` | EKS cluster, managed node groups, OIDC provider, CoreDNS/kube-proxy addons. |
| **aws/k3s-host** | `modules/providers/aws/k3s-host/` | EC2 instance bootstrapped with k3s via the `core/k8s/k3s-bootstrap` module. |
| **aws/postgres** | `modules/providers/aws/postgres/` | RDS PostgreSQL with automated backups, security group rules, and Secrets Manager integration. |
| **bare\_metal** | `modules/providers/bare_metal/` | SSH + Docker Compose on any VPS. No cloud provider required. |
#### Module Dependency Graph
```
root module (main.tf)
│
├── modules/core/capabilities/ # Capability contracts
│
├── modules/providers/aws/ # AWS adapter
│ ├── networking/ # VPC, subnets, SGs
│ ├── ec2/ # EC2 + Docker Compose
│ ├── eks/ # EKS cluster
│ ├── k3s-host/ # EC2 + k3s
│ │ └── modules/core/k8s/k3s-bootstrap/
│ └── postgres/ # RDS PostgreSQL
│
└── modules/providers/bare_metal/ # Bare metal adapter
└── modules/core/k8s/k3s-bootstrap/ (optional)
```
### Compute Engine Comparison
| | EC2 + Docker Compose | EKS | k3s | Bare Metal |
| ----------------------- | -------------------------- | -------------------- | ------------------------------------ | ---------------------------- |
| **Provider** | AWS | AWS | AWS or bare\_metal | bare\_metal |
| **Kubernetes** | No | Yes (managed) | Yes (lightweight) | No |
| **Control plane cost** | $0 | \~$73/mo | $0 | $0 |
| **Workload management** | Docker Compose | K8s deployments | K8s deployments | Docker Compose |
| **Config delivery** | cloud-init + bind mounts | ConfigMap/Secret | Helm values -> ConfigMap | SSH + file provisioner |
| **Scaling** | Vertical only | HPA, node autoscaler | Manual / limited HPA | Vertical only |
| **Best for** | Dev/staging, simple setups | Production K8s teams | Cost-conscious K8s, single-node prod | Self-hosted, no cloud vendor |
**Decision guide:**
* **Just want it to work** -- EC2 + Docker Compose. Simplest path, lowest cost.
* **Need Kubernetes but not the EKS bill** -- k3s. Same K8s API, $0 control plane.
* **Production K8s with managed upgrades** -- EKS. AWS handles the control plane.
* **No AWS account / own hardware** -- Bare Metal. SSH to any box with Docker installed.
### Data Flow
See [Data Flow at Runtime](#data-flow-at-runtime) above for the full pipeline diagram.
### Provider Abstraction
The root module is provider-neutral. The `infrastructure_provider` variable routes to the correct adapter.
```
variables.tf (root)
│
│ infrastructure_provider = "aws" ──→ modules/providers/aws/
│ infrastructure_provider = "bare_metal" ──→ modules/providers/bare_metal/
│
└── Every adapter implements the same output contract:
- networking: { vpc_id, subnet_ids, security_group_ids }
- compute: { host_ip, ssh_user, connection_type }
- database: { host, port, credentials_arn }
- workload: { deploy_method, config_paths }
```
#### How It Works
1. **Root module** declares provider-neutral variables (`instance_type`, `database_engine`, `compute_engine`, etc.)
2. **`infrastructure_provider`** selects which adapter directory to use
3. **Adapter module** translates generic variables into provider-specific resources (e.g., `instance_type = "medium"` becomes `t3.medium` on AWS)
4. **Adapter outputs** conform to the capability contract defined in `modules/core/capabilities/`
5. **Workload deployment** reads adapter outputs to place containers on the provisioned infrastructure
#### Adding a New Provider
Adding support for a new cloud provider (GCP, Azure, Hetzner, etc.) requires:
1. Create `modules/providers//` with networking, compute, and database sub-modules
2. Implement the output contract from `modules/core/capabilities/`
3. Add the provider option to `infrastructure_provider` validation in `variables.tf`
4. No changes to root module logic, workload deployers, or existing providers
### Related Pages
* **[Getting Started](./getting-started.mdx)** -- Deploy your first indexer
* **[Core Concepts](./concepts.mdx)** -- Providers, compute engines, workload modes in depth
* **[Variable Reference](./variable-reference.mdx)** -- All configuration options
* **[Examples](./examples/index.mdx)** -- 7 runnable deployment patterns
## Core Concepts
evm-cloud is built around three orthogonal dimensions: **where** your infrastructure runs (provider), **how** compute is managed (engine), and **who** deploys the workloads (mode). Understanding these three choices — and how they compose — is the key to using evm-cloud effectively.
### Infrastructure Providers
The `infrastructure_provider` variable determines what cloud platform (or lack thereof) provisions your resources.
#### `aws` (default)
Full AWS stack. Terraform creates and manages:
* VPC with public/private subnets
* Compute (EC2, EKS, or k3s on EC2)
* Database (RDS PostgreSQL or BYODB ClickHouse)
* Secrets Manager for credentials
* IAM roles with least-privilege policies
* Security groups with tight ingress rules
```hcl
module "evm_cloud" {
source = "github.com/evm-cloud/evm-cloud"
infrastructure_provider = "aws" # default, can omit
compute_engine = "ec2"
# ...
}
```
#### `bare_metal`
Any VPS with SSH access and Docker installed. No cloud services. Terraform connects over SSH to provision containers directly.
* You manage networking (firewall rules, DNS, TLS)
* You manage secrets (no Secrets Manager — credentials go in `.env` files)
* You provide the host IP and SSH key
```hcl
module "evm_cloud" {
source = "github.com/evm-cloud/evm-cloud"
infrastructure_provider = "bare_metal"
compute_engine = "docker_compose"
bare_metal_host = "203.0.113.10"
bare_metal_ssh_key = file("~/.ssh/id_ed25519")
bare_metal_ssh_user = "deploy"
# ...
}
```
#### How to Choose
| Factor | AWS | Bare Metal |
| ------------------------ | ---------------------------------- | ------------------------------ |
| Cost | \~$50-140/mo minimum | \~$5-20/mo (Hetzner, OVH) |
| Managed services | RDS, Secrets Manager, IAM | None — you manage everything |
| Networking | VPC with private subnets | Public IP, your firewall rules |
| Security baseline | IAM + SGs + encryption at rest | SSH hardening + manual TLS |
| Compliance / sovereignty | AWS regions, shared responsibility | Full control, any jurisdiction |
Use **AWS** when you want managed services, security defaults, and operational simplicity. Use **bare\_metal** when cost matters most, when you need geographic sovereignty, or when you already have VPS infrastructure.
***
### Compute Engines
The `compute_engine` variable determines how containers run on the provisioned infrastructure.
#### `ec2`
Docker Compose on a single EC2 instance. The simplest deployment path.
* Terraform provisions the instance, installs Docker, writes config files via cloud-init
* All services (eRPC, rindexer, database client) run as Docker Compose services
* Good for dev, staging, and small production workloads
* No orchestrator overhead
```hcl
compute_engine = "ec2"
```
#### `eks`
AWS-managed Kubernetes (Elastic Kubernetes Service).
* Terraform creates the EKS cluster, node groups, and OIDC provider
* Workloads deploy as Kubernetes pods via Helm charts or Terraform K8s resources
* Best for teams already running K8s who want evm-cloud to fit into their existing cluster operations
* Adds \~$73/mo for the EKS control plane
```hcl
compute_engine = "eks"
```
#### `k3s`
Lightweight Kubernetes on a single EC2 instance (or bare metal VPS). Real Kubernetes API at zero control plane cost.
* Phase 1: Terraform provisions the instance and installs k3s
* Phase 2: A deployer script applies Helm charts to the running cluster
* Runs the same Helm charts as EKS but on a single node
* Two-phase deployment is required — see [Two-Phase Deployment](#two-phase-deployment) below
```hcl
compute_engine = "k3s"
```
#### `docker_compose`
Same model as `ec2` but on a `bare_metal` provider. SSH + Docker Compose on your own VPS.
* Terraform connects over SSH, writes configs, and manages Docker Compose services
* Functionally identical to `ec2` mode but without AWS networking or cloud-init
```hcl
infrastructure_provider = "bare_metal"
compute_engine = "docker_compose"
```
#### Valid Combinations
Not every engine works with every provider:
| Provider | `ec2` | `eks` | `k3s` | `docker_compose` |
| ------------ | ----- | ----- | ----- | ---------------- |
| `aws` | Yes | Yes | Yes | No |
| `bare_metal` | No | No | Yes | Yes |
Terraform validates this at plan time — invalid combinations produce a clear error message.
***
### Workload Modes
The `workload_mode` variable controls **who** deploys the application workloads (eRPC, rindexer) onto the provisioned compute.
#### `terraform` (default)
Terraform deploys and manages both infrastructure AND workloads. One `terraform apply` does everything: provisions compute, creates the database, writes configs, and starts containers.
```hcl
workload_mode = "terraform" # default, can omit
```
This is the simplest path. Use it when:
* You want a single command to deploy everything
* You do not need separate infra/app deploy cadences
* Your team does not already have a CI/CD pipeline for container deployments
#### `external`
Terraform provisions infrastructure only (VPC, compute, database) and outputs a structured [workload handoff](#the-workload-handoff) containing everything an external deployer needs. You deploy workloads yourself.
```hcl
workload_mode = "external"
```
Use `external` when:
* You have CI/CD pipelines (GitHub Actions, GitLab CI) that deploy containers
* You use GitOps (ArgoCD, Flux) to manage Kubernetes workloads
* Infrastructure and application teams operate independently
* You need different deploy cadences — infra changes monthly, app deploys daily
**Note:** `k3s` always operates in external mode regardless of the `workload_mode` setting. The two-phase architecture requires it — see [Two-Phase Deployment](#two-phase-deployment).
***
### The Workload Handoff
When `workload_mode = "external"` (or when using `k3s`), Terraform outputs a `workload_handoff` JSON object containing everything an external deployer needs to put workloads onto the provisioned infrastructure.
#### Accessing the Handoff
```bash
terraform output -json workload_handoff
```
The output is marked `sensitive = true` because it may contain kubeconfig data or database passwords.
#### Structure
```json
{
"version": "v1",
"mode": "external",
"compute_engine": "k3s",
"runtime": {
"ec2": {
"ssh_command": "ssh -i key.pem ubuntu@203.0.113.10",
"instance_ip": "203.0.113.10",
"config_path": "/opt/evm-cloud/config/"
},
"eks": {
"cluster_name": "evm-cloud-prod",
"cluster_endpoint": "https://ABCDEF.gr7.us-east-1.eks.amazonaws.com",
"oidc_provider_arn": "arn:aws:iam::123456789012:oidc-provider/..."
},
"k3s": {
"kubeconfig_base64": "YXBpVmVyc2lvbjogdjEK...",
"cluster_endpoint": "https://203.0.113.10:6443",
"host_ip": "203.0.113.10"
}
},
"services": {
"erpc": { "name": "erpc", "port": 4000, "internal_url": "http://erpc:4000" },
"rindexer": { "name": "rindexer", "port": 3001, "internal_url": "http://rindexer:3001" }
},
"data": {
"database_type": "postgresql",
"host": "evm-cloud-db.abc123.us-east-1.rds.amazonaws.com",
"port": 5432,
"username": "rindexer",
"password": "generated-password",
"database": "rindexer"
},
"artifacts": {
"config_channel": "helm"
}
}
```
Only the `runtime` block matching the active `compute_engine` is populated. The others are `null`.
#### Using the Handoff in CI/CD
```bash
# Extract values for a Helm deploy
KUBECONFIG_B64=$(terraform output -json workload_handoff | jq -r '.runtime.k3s.kubeconfig_base64')
DB_HOST=$(terraform output -json workload_handoff | jq -r '.data.host')
DB_PASSWORD=$(terraform output -json workload_handoff | jq -r '.data.password')
echo "$KUBECONFIG_B64" | base64 -d > /tmp/kubeconfig
export KUBECONFIG=/tmp/kubeconfig
helm upgrade --install rindexer ./deployers/charts/indexer \
--set database.host="$DB_HOST" \
--set database.password="$DB_PASSWORD"
```
***
### Config Injection
eRPC and rindexer each require YAML configuration files. How those configs reach the running containers depends on the compute engine and workload mode.
#### EC2 (Docker Compose)
cloud-init writes config files to `/opt/evm-cloud/config/` during instance provisioning. Docker Compose bind-mounts this directory into the containers.
```yaml
# docker-compose.yml (generated)
services:
erpc:
volumes:
- /opt/evm-cloud/config/erpc.yaml:/etc/erpc/erpc.yaml:ro
rindexer:
volumes:
- /opt/evm-cloud/config/rindexer.yaml:/etc/rindexer/rindexer.yaml:ro
```
#### EKS (terraform mode)
Terraform creates Kubernetes ConfigMaps and Secrets directly. Pod specs mount them as volumes.
#### EKS / k3s (external mode)
Helm values embed the config content. The chart templates create ConfigMaps from those values, and pods mount the ConfigMaps.
```bash
helm upgrade --install rindexer ./deployers/charts/indexer \
--set-file rindexerConfig=./config/rindexer.yaml \
--set-file erpcConfig=./config/erpc.yaml
```
#### Bare Metal (Docker Compose)
Terraform's SSH file provisioner writes configs to the remote host. Docker Compose bind-mounts them, same as EC2.
***
### Secrets Delivery
Database passwords, API keys, and other credentials need to reach the workloads securely. The delivery mechanism varies by provider and engine.
#### EC2 on AWS
1. Terraform stores secrets in AWS Secrets Manager
2. A `pull-secrets.sh` script on the instance fetches them at boot
3. Secrets are written to a `.env` file with `chmod 0600`
4. Docker Compose reads the `.env` file
#### EKS (terraform mode)
Terraform creates Kubernetes Secrets directly from the generated credentials. Pod specs reference them as environment variables or mounted files.
#### k3s (external mode)
The database password is included in the [workload handoff](#the-workload-handoff). The deployer script passes it to Helm, which creates a Kubernetes Secret.
Future improvement: support for External Secrets Operator to pull from Secrets Manager or Vault.
#### Bare Metal
Terraform's SSH provisioner writes a `.env` file to the remote host with `chmod 0600`. Docker Compose loads it.
See the [Secrets Management guide](./guides/secrets-management.mdx) for the full security model, rotation procedures, and production hardening steps.
***
### Two-Phase Deployment
k3s and EKS external mode use a two-phase deployment pattern. This is an architectural requirement, not a preference.
#### The Problem
The Terraform `kubernetes` and `helm` providers need a kubeconfig to connect to the cluster. But the kubeconfig does not exist until Terraform finishes creating the cluster. This is a chicken-and-egg problem that cannot be solved within a single `terraform apply`.
#### The Solution
**Phase 1: Infrastructure** — `terraform apply`
Provisions compute (EC2 with k3s installed, or EKS cluster), networking, database, and outputs the `workload_handoff`.
```bash
cd examples/k3s_rds
terraform init
terraform apply
```
**Phase 2: Workloads** — deployer script
Uses the kubeconfig from Phase 1 to deploy Helm charts onto the cluster.
```bash
cd deployers/k3s
./deploy.sh ../../examples/k3s_rds
```
The deployer script reads `workload_handoff` from the Terraform state, extracts the kubeconfig, and runs `helm upgrade --install` for each service.
#### Teardown
Reverse order. Remove workloads before destroying infrastructure:
```bash
# Phase 2 teardown
cd deployers/k3s
./teardown.sh ../../examples/k3s_rds
# Phase 1 teardown
cd examples/k3s_rds
terraform destroy
```
If you skip Phase 2 and go straight to `terraform destroy`, the EC2 instance (and its containers) will be terminated. No resources leak, but Helm release state is lost.
See the [Two-Phase Workflow guide](./guides/two-phase-workflow.mdx) for a complete walkthrough with all commands.
***
### Lifecycle Behavior
Understanding what Terraform does (and does not do) when you change configuration after initial deployment.
#### Changing `compute_engine`
Changing the compute engine on an existing deployment (e.g., `ec2` to `k3s`) **destroys and recreates all compute resources**. The database is preserved — RDS instances and BYODB connections are independent of compute.
```bash
# This will destroy the EC2 instance and create a new one with k3s
compute_engine = "k3s" # was "ec2"
terraform apply # plan will show destroy + create
```
Always review the plan carefully before applying compute engine changes.
#### Changing Config Content
Updating `erpc_config_yaml` or `rindexer_config_yaml` triggers a redeployment on the next `terraform apply`:
* **EC2/Docker Compose**: Terraform detects the config file change and re-provisions via SSH
* **EKS (terraform mode)**: ConfigMap update triggers a pod rollout
* **k3s/EKS (external mode)**: Re-run the deployer script with updated configs
#### EC2 User Data Behavior
EC2 instances use `lifecycle { ignore_changes = [user_data] }`. This means:
* The **initial** deploy uses cloud-init to configure the instance
* **Subsequent** config changes do NOT recreate the instance
* Config updates after initial deploy are applied via SSH provisioner or the external deployer
This prevents accidental instance replacement (and downtime) when config files change. If you need a fresh instance, taint it explicitly:
```bash
terraform taint 'module.evm_cloud.module.compute.aws_instance.this[0]'
terraform apply
```
## Cost Estimates
Estimated monthly costs for each deployment pattern. All prices are US East (N. Virginia) as of 2025. Actual costs vary by usage, data transfer, and reserved instance pricing.
### By Deployment Pattern
| Pattern | Components | Est. Monthly Cost |
| ------------------------------- | ---------------------------------------- | ------------------------------ |
| **EC2 + Managed RDS** | t3.medium EC2 + db.t4g.micro RDS + VPC | \~$45-60 |
| **EC2 + BYO ClickHouse** | t3.medium EC2 + VPC | \~$35-50 (+ your ClickHouse) |
| **EC2 + BYO ClickHouse (dev)** | t3.micro EC2 + VPC | \~$10-15 (+ your ClickHouse) |
| **EKS + BYO ClickHouse** | EKS control plane + t3.medium node + VPC | \~$110-140 (+ your ClickHouse) |
| **k3s + BYO ClickHouse** | t3.medium EC2 (no EKS fee) + VPC | \~$35-50 (+ your ClickHouse) |
| **Bare Metal + BYO ClickHouse** | Your VPS cost only | \~$5-20 (Hetzner/OVH) |
### What Drives Cost
#### Biggest Cost Drivers
| Component | Cost | How to Reduce |
| --------------------- | ----------------------- | ------------------------------------------------------- |
| **EC2 instance type** | $8-133/mo | Use `t3.micro` for dev, `t3.medium` for production |
| **EKS control plane** | $73/mo (fixed) | Use k3s instead ($0 control plane) |
| **NAT gateway** | \~$35/mo + data | Set `network_enable_nat_gateway = false` for dev |
| **RDS instance** | $13-175/mo | Use `db.t4g.micro` for dev, or switch to BYO ClickHouse |
| **EBS storage** | \~$2.40/mo per 30GB gp3 | Default is fine for most workloads |
| **Data transfer** | $0.09/GB outbound | Minimal for indexers (inbound RPC data is free) |
#### Free or Near-Free
| Component | Cost | Notes |
| -------------------------- | ----------------- | -------------------------------------- |
| VPC, subnets, SGs | $0 | No charge for VPC resources themselves |
| Secrets Manager | \~$0.40/secret/mo | Negligible |
| S3 VPC endpoint (gateway) | $0 | Free gateway endpoint |
| k3s control plane | $0 | Runs on your EC2 instance |
| ClickHouse Cloud free tier | $0 | 10M rows, good for dev/testing |
### Cost Optimization Tips
#### Dev Environment
Save \~80% on dev costs:
```hcl
# Use smallest instances
ec2_instance_type = "t3.micro" # $8/mo vs $33/mo
postgres_instance_class = "db.t4g.micro" # $13/mo (if using RDS)
# Skip expensive networking
network_enable_nat_gateway = false # saves ~$35/mo
network_enable_vpc_endpoints = false # saves ~$7-22/mo
# Fast secret cleanup
ec2_secret_recovery_window_in_days = 0 # immediate deletion
```
#### k3s Instead of EKS
If you want Kubernetes but not the EKS bill:
| | EKS | k3s | Savings |
| ------------- | --------------- | ---------------------- | ---------- |
| Control plane | $73/mo | $0 | $73/mo |
| Worker node | t3.medium ($33) | Same instance runs k3s | $0 |
| **Total** | **\~$106/mo** | **\~$33/mo** | **$73/mo** |
Both run the same Helm charts and K8s workloads. Trade-off: k3s is single-node (no HA), uses static kubeconfig (not IAM-based).
#### BYO ClickHouse vs Managed RDS
| | RDS PostgreSQL | ClickHouse Cloud (free) | Self-hosted ClickHouse |
| ------------ | -------------------- | ----------------------- | ---------------------- |
| Monthly cost | $13-175 | $0 (10M rows) | Your infra cost |
| Setup | Automatic | Sign up, get URL | You manage |
| Best for | Small datasets, ACID | Analytics, time-series | Large-scale analytics |
### Example: Cheapest Production Setup
Bare metal VPS (Hetzner CX22) + ClickHouse Cloud free tier:
| Component | Monthly Cost |
| ------------------------------ | ------------ |
| Hetzner CX22 (2 vCPU, 4GB RAM) | \~$4 |
| ClickHouse Cloud free tier | $0 |
| Domain + DNS | \~$1 |
| **Total** | **\~$5/mo** |
```hcl
infrastructure_provider = "bare_metal"
compute_engine = "docker_compose"
bare_metal_host = "your-hetzner-ip"
indexer_storage_backend = "clickhouse"
```
### Example: Production AWS Setup
| Component | Monthly Cost |
| ----------------- | ------------ |
| t3.medium EC2 | $33 |
| VPC + NAT gateway | $35 |
| db.t4g.small RDS | $26 |
| Secrets Manager | $0.40 |
| EBS (30GB gp3) | $2.40 |
| **Total** | **\~$97/mo** |
import { DeployDiagram } from '../../components/DeployDiagram'
import { WhatJustHappened } from '../../components/WhatJustHappened'
## Getting Started
Deploy an EVM blockchain indexer on AWS in under 10 minutes. This guide uses the simplest example: EC2 + Docker Compose + external ClickHouse.
\~5 min to deploy
$0/mo (AWS free tier)
EC2 (free tier) + Docker + ClickHouse
#### What You're About to Build
### Prerequisites
| Tool | Version | Install |
| ---------------- | -------------- | -------------------------------------------------------------------------------------------------------- |
| **Terraform** | >= 1.5.0 | [terraform.io/downloads](https://developer.hashicorp.com/terraform/downloads) |
| **AWS CLI** | v2 | [docs.aws.amazon.com/cli](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) |
| **jq** | any | `brew install jq` or `apt install jq` |
| **SSH key pair** | Ed25519 or RSA | `ssh-keygen -t ed25519` |
Optional (for development/QA):
| Tool | Purpose | Install |
| ----------- | ------------------ | ------------------------------------------------- |
| **tflint** | Terraform linting | `brew install tflint` |
| **checkov** | Security scanning | `pip install checkov` |
| **Docker** | LocalStack testing | [docker.com](https://docs.docker.com/get-docker/) |
#### AWS Credentials
Configure AWS CLI with credentials that have EC2, VPC, IAM, and Secrets Manager permissions:
```bash
aws configure
# Or export environment variables:
export AWS_ACCESS_KEY_ID=your-key
export AWS_SECRET_ACCESS_KEY=your-secret
export AWS_DEFAULT_REGION=us-east-1
```
#### SSH Key
If you don't have one:
```bash
ssh-keygen -t ed25519 -C "evm-cloud" -f ~/.ssh/evm-cloud
# Public key: ~/.ssh/evm-cloud.pub
# Private key: ~/.ssh/evm-cloud
```
### Your First Deployment
We'll use the `minimal_aws_byo_clickhouse` example — EC2 instance running eRPC + rindexer with an external ClickHouse database.
#### 1. Clone and Navigate
```bash
git clone https://github.com/ExoMonk/evm-cloud.git
cd evm-cloud/examples/minimal_aws_byo_clickhouse
```
#### 2. Configure Secrets
```bash
cp secrets.auto.tfvars.example secrets.auto.tfvars
```
Edit `secrets.auto.tfvars` with your values:
```hcl
# SSH key (the public key content, not the path)
ssh_public_key = "ssh-ed25519 AAAA... your-email"
# ClickHouse connection (from ClickHouse Cloud or your own instance)
indexer_clickhouse_url = "https://your-instance.clickhouse.cloud:8443"
indexer_clickhouse_password = "your-password"
```
> **Don't have ClickHouse?** Sign up for [ClickHouse Cloud](https://clickhouse.cloud/) free tier, or use the `minimal_aws_rds` example instead (uses managed PostgreSQL, no external DB needed).
#### 3. Initialize and Plan
```bash
terraform init
terraform plan -var-file=minimal_clickhouse.tfvars
```
Review the plan. You should see resources for:
* VPC, subnets, security groups
* EC2 instance (t3.micro — AWS free tier eligible)
* IAM role + Secrets Manager secret
* Docker Compose services (eRPC + rindexer)
#### 4. Apply
```bash
terraform apply -var-file=minimal_clickhouse.tfvars
```
This takes 3-5 minutes. Terraform will:
1. Create a VPC with public subnets
2. Launch an EC2 instance with Docker pre-installed
3. Deploy eRPC (RPC proxy) and rindexer (indexer) as Docker containers
4. Wire rindexer to your ClickHouse database via Secrets Manager
#### 5. Verify
```bash
# Get the instance IP
terraform output -json workload_handoff | jq -r '.runtime.ec2.public_ip'
# SSH into the instance
ssh -i ~/.ssh/evm-cloud ubuntu@
# Check containers are running
sudo docker compose -f /opt/evm-cloud/docker-compose.yml ps
```
You should see `erpc` and `rindexer` containers in `running` state.
#### 6. Teardown
```bash
terraform destroy -var-file=minimal_clickhouse.tfvars
```
### What Just Happened
Here's what Terraform created and how data flows:
**Key components:**
* **eRPC** aggregates multiple RPC endpoints with automatic failover, caching, and load balancing
* **rindexer** reads your `rindexer.yaml` config to index specific contracts/events into your database
* **Secrets Manager** securely stores database credentials and injects them at runtime
### Next Steps
* **Try a different example**: See [Examples](./examples/index.mdx) for all 7 deployment patterns
* **Understand the architecture**: Read [Architecture](./architecture.mdx) and [Core Concepts](./concepts.mdx)
* **Tune performance**: See [Variable Reference](./variable-reference.mdx) for instance types, memory limits, and other knobs
* **Go to production**: Follow the [Production Checklist](./guides/production-checklist.mdx)
### Other Starting Points
| If you want... | Use this example |
| ---------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| Managed PostgreSQL (no external DB) | [`minimal_aws_rds`](./examples/ec2-docker-compose-rds.mdx) |
| Kubernetes (EKS) | [`aws_eks_BYO_clickhouse`](./examples/eks-clickhouse.mdx) |
| Lightweight Kubernetes (k3s, no EKS fee) | [`minimal_aws_k3s_byo_clickhouse`](./examples/k3s-clickhouse.mdx) |
| Any VPS, no AWS | [`bare_metal_byo_clickhouse`](./examples/bare-metal-clickhouse.mdx) |
| Infra only (deploy workloads yourself) | [`minimal_aws_external_ec2_byo`](./examples/external-ec2.mdx) or [`external_eks`](./examples/external-eks.mdx) |
import { DeployDiagram } from '../../components/DeployDiagram'
## EVM Cloud
Open-source infrastructure platform for EVM blockchain data. Deploy, manage, and scale a complete data stack — RPC proxies, event indexers, databases, networking, and monitoring — on AWS, GCP, or bare metal with a single command.
### What It Deploys
### Supported Configurations
#### Compute Engines
| Engine | Description | Best For | Monthly Cost |
| ------------------------ | ------------------------------------------- | --------------------------------------- | ------------ |
| **EC2 + Docker Compose** | Single instance, all services in containers | Dev, small projects | \~$15-50 |
| **EKS** | AWS-managed Kubernetes | Production, teams needing K8s | \~$110-140 |
| **k3s** | Lightweight Kubernetes on a single EC2 | K8s without EKS cost ($0 control plane) | \~$35-50 |
| **Bare Metal** | Docker Compose on any VPS via SSH | Self-hosted, no cloud vendor | \~$5-20 |
#### Databases
| Backend | Mode | Notes |
| -------------- | ---------------------- | ------------------------------------------------------------------- |
| **PostgreSQL** | Managed (RDS) | Default, zero-ops. Terraform provisions and wires it automatically. |
| **ClickHouse** | BYODB (bring-your-own) | You provide the endpoint. Great for high-throughput analytics. |
#### Workload Modes
| Mode | Who Deploys Workloads | Use Case |
| --------------------- | ------------------------------- | --------------------------------- |
| `terraform` (default) | Terraform manages everything | Simple, all-in-one deployment |
| `external` | Your CI/CD, GitOps, or Helm CLI | Separate infra and app lifecycles |
### Quick Links
* **[Getting Started](./getting-started.mdx)** — Deploy your first indexer in 5 minutes
* **[Architecture](./architecture.mdx)** — How the modules fit together
* **[Core Concepts](./concepts.mdx)** — Providers, compute engines, workload modes
* **[Variable Reference](./variable-reference.mdx)** — All configuration options
* **[Examples](./examples/index.mdx)** — 7 ready-to-use deployment examples
* **[Guides](./guides/secrets-management.mdx)** — Secrets, config updates, production checklist
* **[Cost Estimates](./cost-estimates.mdx)** — What each deployment pattern costs
* **[Troubleshooting](./troubleshooting.mdx)** — Common issues and fixes
* **[Roadmap](./roadmap.mdx)** — What's coming next
### Repository Layout
```
evm-cloud/
main.tf, variables.tf, outputs.tf # Root module (consumers use this)
modules/
core/ # Provider-neutral logic (capabilities, K8s)
providers/aws/ # AWS adapter (VPC, EC2, EKS, RDS, k3s)
providers/bare_metal/ # Bare metal adapter (SSH + Docker Compose)
deployers/
charts/ # Shared Helm charts (rpc-proxy, indexer)
k3s/ # k3s Helm deployer scripts
eks/ # EKS deployer + ArgoCD manifests
examples/ # 7 runnable examples
tests/ # LocalStack harness + kind K8s tests
runbooks/ # Operational guides
documentation/ # This documentation (Vocs)
```
## Outputs Reference
Terraform outputs from evm-cloud. Access any output with `terraform output -json `.
### Output Summary
| Output | Description | Always Present |
| --------------------- | ------------------------------------------------ | ------------------------------- |
| `provider_selection` | Active provider, deployment target, architecture | Yes |
| `capability_contract` | Provider-neutral capability flags | Yes |
| `adapter_context` | Provider-specific adapter metadata | Yes |
| `networking` | VPC, subnet, and security group IDs | AWS only |
| `postgres` | RDS endpoint, port, database name, secret ARN | When `postgres_enabled = true` |
| `rpc_proxy` | eRPC service name and port | When `rpc_proxy_enabled = true` |
| `indexer` | rindexer service name and log group | When `indexer_enabled = true` |
| `workload_handoff` | Full deployment contract for external tools | Yes (sensitive) |
### Accessing Outputs
```bash
# All outputs (redacts sensitive values)
terraform output
# Specific output as JSON
terraform output -json workload_handoff
# Extract a nested field
terraform output -json workload_handoff | jq '.runtime.ec2.public_ip'
```
> **Note:** `workload_handoff` is marked `sensitive = true` because it may contain kubeconfig credentials (k3s) or database passwords. Use `terraform output -json` to access it.
### workload\_handoff v1 Schema
The `workload_handoff` is the primary integration point for external deployers. It contains everything needed to deploy workloads outside of Terraform.
#### Top-Level Fields
| Field | Type | Description |
| ---------------- | -------- | ------------------------------ |
| `version` | `string` | Contract version: `"v1"` |
| `mode` | `string` | `"terraform"` or `"external"` |
| `compute_engine` | `string` | `"ec2"`, `"eks"`, or `"k3s"` |
| `project_name` | `string` | Project identifier |
| `aws_region` | `string` | AWS region (when provider=aws) |
#### identity
IAM identity for workloads.
| Field | Present When | Contents |
| ------------------------------- | ------------------------ | ------------------------ |
| `identity.ec2_instance_profile` | `compute_engine = "ec2"` | `{ name, role_arn }` |
| `identity.eks_irsa_role_arns` | `compute_engine = "eks"` | `{ rpc_proxy, indexer }` |
#### network
VPC and security group information.
| Field | Type | Description |
| ----------------------------------- | -------- | ------------------ |
| `network.vpc_id` | `string` | VPC ID |
| `network.public_subnet_ids` | `list` | Public subnet IDs |
| `network.private_subnet_ids` | `list` | Private subnet IDs |
| `network.security_groups.rpc_proxy` | `string` | SG ID for eRPC |
| `network.security_groups.indexer` | `string` | SG ID for rindexer |
#### runtime
Compute-engine-specific connection details. Only one of `ec2`, `eks`, or `k3s` is populated.
##### runtime.ec2
| Field | Type | Description |
| ---------------------- | -------- | ------------------------------------------------- |
| `instance_id` | `string` | EC2 instance ID |
| `public_ip` | `string` | Instance public IP |
| `ssh_command` | `string` | Ready-to-use SSH command |
| `config_dir` | `string` | Config path on instance (`/opt/evm-cloud/config`) |
| `compose_file` | `string` | Docker Compose file path |
| `secret_arn` | `string` | Secrets Manager secret ARN |
| `cloudwatch_log_group` | `string` | CloudWatch log group name |
##### runtime.eks
| Field | Type | Description |
| ------------------- | -------- | ---------------------- |
| `cluster_name` | `string` | EKS cluster name |
| `cluster_endpoint` | `string` | EKS API endpoint |
| `oidc_provider_arn` | `string` | OIDC provider for IRSA |
##### runtime.k3s
| Field | Type | Description |
| ------------------- | -------- | ------------------------------------- |
| `host_ip` | `string` | EC2 instance public IP |
| `instance_id` | `string` | EC2 instance ID |
| `cluster_endpoint` | `string` | k3s API endpoint (`https://IP:6443`) |
| `kubeconfig_base64` | `string` | Base64-encoded kubeconfig (sensitive) |
| `node_name` | `string` | k3s node name |
#### services
Workload service metadata.
| Field | Present When | Contents |
| -------------------- | ------------------- | ----------------------------------------------------------- |
| `services.rpc_proxy` | `rpc_proxy_enabled` | `{ service_name, port, internal_url }` |
| `services.indexer` | `indexer_enabled` | `{ service_name, single_writer_required, storage_backend }` |
#### data
Database connection information.
| Field | Present When | Contents |
| ----------------- | ------------------------------------ | ------------------------------------- |
| `data.backend` | `indexer_enabled` | `"postgres"` or `"clickhouse"` |
| `data.postgres` | backend=postgres + postgres\_enabled | `{ host, port, db_name, secret_arn }` |
| `data.clickhouse` | backend=clickhouse | `{ url, user, db, password }` |
#### artifacts
| Field | Type | Values |
| -------------------------- | -------- | --------------------------------------------------- |
| `artifacts.config_channel` | `string` | `"ssh"` (EC2), `"helm"` (k3s), `"k8s_config"` (EKS) |
### Example: EC2 Handoff
```json
{
"version": "v1",
"mode": "external",
"compute_engine": "ec2",
"project_name": "my-indexer",
"aws_region": "us-east-1",
"runtime": {
"ec2": {
"instance_id": "i-0abc123def456",
"public_ip": "54.123.45.67",
"ssh_command": "ssh -i ~/.ssh/key ubuntu@54.123.45.67",
"config_dir": "/opt/evm-cloud/config",
"compose_file": "/opt/evm-cloud/docker-compose.yml",
"secret_arn": "arn:aws:secretsmanager:us-east-1:123456:secret:my-indexer-abc",
"cloudwatch_log_group": "/evm-cloud/my-indexer"
},
"eks": null,
"k3s": null
},
"services": {
"rpc_proxy": { "service_name": "erpc", "port": 4000, "internal_url": "http://erpc:4000" },
"indexer": { "service_name": "rindexer", "single_writer_required": true, "storage_backend": "clickhouse" }
},
"data": {
"backend": "clickhouse",
"postgres": null,
"clickhouse": { "url": "https://ch.example.com:8443", "user": "default", "db": "default", "password": null }
},
"artifacts": { "config_channel": "ssh" }
}
```
### Example: k3s Handoff
```json
{
"version": "v1",
"mode": "external",
"compute_engine": "k3s",
"project_name": "evm-cloud-k3s",
"runtime": {
"ec2": null,
"eks": null,
"k3s": {
"host_ip": "44.213.127.224",
"instance_id": "i-0def789abc012",
"cluster_endpoint": "https://44.213.127.224:6443",
"kubeconfig_base64": "YXBpVmVyc2lvbjogdjEK...",
"node_name": "evm-cloud-k3s-server-0"
}
},
"data": {
"backend": "clickhouse",
"clickhouse": { "url": "https://ch.clickhouse.cloud:8443", "user": "default", "db": "default", "password": "secret" }
},
"artifacts": { "config_channel": "helm" }
}
```
import { ArchitectureDiagram } from '../../components/ArchitectureDiagram'
## Roadmap
EVM Cloud is evolving from Terraform modules into a full-stack platform for EVM blockchain data infrastructure.
### Architecture Vision
Where we're headed — every layer pluggable, every component optional:
### 🔨 Building Now
#### Managed ClickHouse
Self-hosted ClickHouse — no more BYODB-only.
```
terraform apply
→ EC2 or K8s pod with ClickHouse
→ Schema auto-provisioned from rindexer config
→ Automated backups to S3
→ Wired to rindexer + eRPC automatically
```
### Status Overview
| Component | Status | Description |
| ------------------ | ----------- | -------------------------------------------------- |
| AWS Provider | ✅ Shipped | VPC, subnets, security groups, IAM |
| EC2 + Docker | ✅ Shipped | Single-instance Docker Compose deployments |
| EKS (K8s) | ✅ Shipped | AWS-managed Kubernetes clusters |
| k3s (K8s) | ✅ Shipped | Lightweight Kubernetes on EC2 |
| Bare Metal | ✅ Shipped | Docker Compose on any VPS via SSH |
| eRPC Proxy | ✅ Shipped | Multi-upstream RPC with failover and caching |
| rindexer | ✅ Shipped | No-code EVM event indexer |
| PostgreSQL (RDS) | ✅ Shipped | AWS-managed database |
| ClickHouse BYODB | ✅ Shipped | Bring-your-own ClickHouse endpoint |
| Secrets Management | ✅ Shipped | AWS Secrets Manager, K8s secrets, env-file |
| External Mode | ✅ Shipped | Separate infra and workload lifecycles |
| Managed ClickHouse | 🔨 Building | Self-hosted ClickHouse with auto-provisioning |
| Monitoring | 🔨 Building | Grafana + Prometheus dashboards and alerting |
| Ingress / TLS | 🔨 Building | Caddy or ALB with automatic TLS |
| K8s Test Suite | 🔨 Building | kind-based chart validation and connectivity tests |
| GCP Provider | 📋 Planned | GCE, GKE, Cloud SQL — same interface as AWS |
| EVM Cloud CLI | 📋 Planned | `init`, `deploy`, `status`, `logs`, `scale` |
| Deployment Presets | 📋 Planned | One-variable sizing profiles (dev/prod/analytics) |
| Bare Metal v2 | 📋 Planned | Docker Swarm, Ansible, automated backups |
| Event Streaming | 📋 Planned | Kafka, SNS/SQS, CDC fan-out |
| Self-Hosted Nodes | 📋 Planned | Reth / Erigon on NVMe with eRPC integration |
#### Monitoring & Observability
```
┌─────────────────────────────────────────────┐
│ Grafana Dashboard │
├─────────────┬──────────────┬────────────────┤
│ rindexer │ eRPC │ Database │
│ │ │ │
│ Block lag │ Upstream │ Query p99 │
│ Events/sec │ latency │ Disk usage │
│ Error rate │ Cache hit % │ Connections │
│ Chain sync │ Failovers │ Replication │
└─────────────┴──────────────┴────────────────┘
▲ ▲ ▲
└───── Prometheus Scraping ────┘
```
* Pre-built dashboards per service
* Alerting rules: stalled indexer, RPC errors, disk pressure
* Deployed as part of the stack — one variable to enable
#### Ingress & TLS
```
Internet
│
▼
┌──────────────────────────┐
│ Caddy / ALB │
│ *.your-domain.com │
│ Auto-renewing TLS │
├──────────────────────────┤
│ /graphql → rindexer │
│ /grafana → monitoring │
│ /rpc → eRPC proxy │
└──────────────────────────┘
```
#### K8s Test Suite
Automated validation using [kind](https://kind.sigs.k8s.io/):
```bash
make test-k8s
# → Creates throwaway kind cluster
# → Renders Helm charts
# → Validates K8s resources
# → Tests service connectivity
# → Tears down — zero cleanup needed
```
***
### 📋 Planned
#### Event Streaming
Fan out indexed events to downstream consumers:
```
rindexer
│
├──▶ Kafka (MSK or self-hosted)
│ └──▶ Your consumers
│
├──▶ SNS/SQS (AWS-native)
│ └──▶ Lambda / Fargate
│
└──▶ CDC (change data capture)
└──▶ Real-time DB replication
```
#### Self-Hosted EVM Node
```hcl
# Add to your terraform config:
node_enabled = true
node_client = "reth" # or "erigon"
node_instance = "i3en.xlarge" # NVMe storage
node_chain = "ethereum"
# eRPC auto-discovers the local node as primary upstream
# Third-party RPCs become fallback only
```
* NVMe-optimized instances with large EBS
* Automatic peer discovery and sync monitoring
* Local node as eRPC primary, third-party as fallback
#### GCP Provider
Same interface, different cloud:
```hcl
# AWS deployment
infrastructure_provider = "aws"
compute_engine = "ec2_docker"
# Becomes GCP deployment — same variables, same stack
infrastructure_provider = "gcp"
compute_engine = "gce_docker"
```
* GCE instances, GKE clusters, Cloud SQL
* VPC networking, firewall rules, IAM
* Feature parity with AWS provider
#### EVM Cloud CLI
```bash
# Scaffold a new project
evm-cloud init --chain ethereum --db clickhouse
# Deploy with smart defaults
evm-cloud deploy --preset dev
# Live status
evm-cloud status
# ┌─────────────────────────────────────────────┐
# │ Chain: Ethereum mainnet │
# │ Blocks: 19,847,231 / 19,847,235 (-4) │
# │ Events: 1,247/sec │
# │ eRPC: 3/3 upstreams healthy │
# │ DB: ClickHouse — 42GB indexed │
# │ Cost: ~$38/mo (EC2 t3.medium) │
# └─────────────────────────────────────────────┘
# Stream logs from all services
evm-cloud logs --follow
# Scale without editing HCL
evm-cloud scale --instance t3.xlarge
```
#### Deployment Presets
```hcl
# One variable to configure the full sizing profile
preset = "dev" # ~$35-50/mo — single instance, minimal resources
preset = "production" # ~$500-2k/mo — multi-AZ, HA, proper sizing
preset = "analytics" # ~$1-5k/mo — ClickHouse-heavy, read replicas
```
| Preset | Compute | Database | Monitoring | Cost |
| ------------ | -------------------- | --------------------- | ---------- | ------------ |
| `dev` | EC2 `t3.medium` | RDS `db.t3.micro` | Off | \~$35-50/mo |
| `production` | EKS or k3s, multi-AZ | ClickHouse cluster | Full stack | \~$500-2k/mo |
| `analytics` | EKS, large instances | ClickHouse + replicas | Full stack | \~$1-5k/mo |
#### Bare Metal v2
* Multi-node Docker Swarm deployments
* Ansible integration for fleet management
* Automated backups and restore scripts
***
### Contributing
Pick any item and open a PR. Check [GitHub Issues](https://github.com/ExoMonk/evm-cloud/issues) for detailed specs on each roadmap item, or start a [Discussion](https://github.com/ExoMonk/evm-cloud/discussions) to propose something new.
## Troubleshooting
Common issues and solutions, organized by compute engine.
### General
#### Terraform plan shows instance recreation
**Symptom:** `terraform plan` wants to destroy and recreate the EC2 instance after you changed config files.
**Cause:** EC2 user\_data changed. Terraform sees this as a replacement trigger.
**Fix:** EC2 uses `lifecycle { ignore_changes = [user_data] }` so config changes after initial deploy should NOT trigger recreation. If you see recreation, something else changed (instance type, AMI, subnet). Check the plan output carefully.
For config updates after deploy, see [Updating Configuration](./guides/config-updates.mdx).
#### Terraform destroy hangs on internet gateway
**Symptom:** `aws_internet_gateway.this: Still destroying... [10m elapsed]`
**Cause:** Orphaned network interfaces (ENIs) from k3s/EKS pods are still attached to the VPC. The IGW can't be deleted until all ENIs are removed.
**Fix:**
```bash
# Find orphaned ENIs
VPC_ID=$(aws ec2 describe-internet-gateways --internet-gateway-ids \
--query 'InternetGateways[0].Attachments[0].VpcId' --output text)
aws ec2 describe-network-interfaces --filters "Name=vpc-id,Values=$VPC_ID" \
--query 'NetworkInterfaces[?Status==`available`].[NetworkInterfaceId]' --output text \
| while read eni; do
aws ec2 delete-network-interface --network-interface-id "$eni"
done
# Re-run destroy
terraform destroy -var-file=your.tfvars
```
**Prevention:** Always run the teardown script before `terraform destroy` for K8s deployments:
```bash
./deployers/k3s/teardown.sh handoff.json # or Helm uninstall for EKS
terraform destroy -var-file=your.tfvars
```
#### Secrets Manager: "already scheduled for deletion"
**Symptom:** `terraform apply` fails with "secret is scheduled for deletion".
**Cause:** You destroyed and re-created with the same project name, but the secret is in its recovery window.
**Fix:**
```bash
# Force delete the secret
aws secretsmanager delete-secret --secret-id --force-delete-without-recovery
# Re-run apply
terraform apply -var-file=your.tfvars
```
**Prevention:** Set `ec2_secret_recovery_window_in_days = 0` for dev environments.
***
### EC2 + Docker Compose
#### Containers not running after apply
**Symptom:** SSH into instance, `docker compose ps` shows no containers or exited status.
**Check:**
```bash
# Check cloud-init completed
cloud-init status
# Check cloud-init logs for errors
sudo cat /var/log/cloud-init-output.log | tail -50
# Check Docker Compose logs
sudo docker compose -f /opt/evm-cloud/docker-compose.yml logs
```
**Common causes:**
* Cloud-init still running (takes 2-3 min after instance launch)
* Docker image pull failed (check egress security group)
* Config YAML syntax error (check rindexer/eRPC logs)
#### rindexer can't connect to database
**Symptom:** rindexer logs show "could not connect to database" or "connection refused".
**Check:**
```bash
# Verify secrets were pulled
sudo cat /opt/evm-cloud/.env
# Test database connectivity from instance
curl -s https://your-clickhouse:8443/ping
# Or for Postgres:
psql -h -U rindexer -d rindexer -c "SELECT 1"
```
**Common causes:**
* Security group doesn't allow outbound to database port (8443 for ClickHouse, 5432 for Postgres)
* ClickHouse URL is wrong (must include protocol and port: `https://host:8443`)
* RDS is in a private subnet but EC2 is in a different VPC
***
### k3s
#### deploy.sh fails: "handoff mode must be 'external'"
**Symptom:** `ERROR: handoff mode must be 'external', got ''`
**Cause:** The handoff JSON was consumed by a previous pipe read (stdin exhaustion) or is empty.
**Fix:** Write the handoff to a file instead of piping:
```bash
terraform output -json workload_handoff > handoff.json
chmod 0600 handoff.json
./deployers/k3s/deploy.sh handoff.json --config-dir ./config
```
#### Pods can't resolve external DNS names
**Symptom:** `Could not resolve host: your-database.clickhouse.cloud` from inside a pod.
**Cause:** Two possible issues:
1. **systemd-resolved stub:** Ubuntu's `/etc/resolv.conf` points to `127.0.0.53`, unreachable from pods.
2. **CIDR collision:** k3s default pod CIDR (`10.42.0.0/16`) overlaps with your VPC CIDR.
**Check:**
```bash
# Check CoreDNS logs
sudo k3s kubectl -n kube-system logs -l k8s-app=kube-dns --tail=20
# If you see "connection refused" to 10.42.0.2 — it's a CIDR collision
# If you see "connection refused" to 127.0.0.53 — it's systemd-resolved
```
**Fix:** evm-cloud v12+ handles both automatically:
* Sets `resolv-conf: /run/systemd/resolve/resolv.conf` in k3s config
* Uses non-conflicting CIDRs: `10.244.0.0/16` (pods) and `10.245.0.0/16` (services)
If you hit this on an older deployment, SSH in and apply:
```bash
sudo tee /etc/rancher/k3s/config.yaml > /dev/null <<'EOF'
resolv-conf: "/run/systemd/resolve/resolv.conf"
cluster-cidr: "10.244.0.0/16"
service-cidr: "10.245.0.0/16"
EOF
sudo systemctl stop k3s
sudo rm -rf /var/lib/rancher/k3s/server/db # reset cluster state
sudo systemctl start k3s
```
#### k3s provisioner fails: "Illegal option -o pipefail"
**Symptom:** `/tmp/terraform_XXX.sh: 2: set: Illegal option -o pipefail`
**Cause:** Ubuntu's `/bin/sh` is `dash`, which doesn't support `set -o pipefail`.
**Fix:** This is fixed in evm-cloud v12+. The provisioner uses `set -eu` (POSIX-compatible). If you're on an older version, update the k3s-bootstrap module.
#### kubectl: "connection to server was refused"
**Symptom:** `The connection to the server 127.0.0.1:6443 was refused`
**Cause:** k3s service isn't running (crashed after config change, or instance rebooted).
**Fix:**
```bash
# Check k3s service
sudo systemctl status k3s
sudo journalctl -u k3s --no-pager -n 30
# Restart k3s
sudo systemctl restart k3s
```
***
### EKS
#### Pods stuck in Pending
**Symptom:** `kubectl get pods` shows pods in `Pending` state.
**Check:**
```bash
kubectl describe pod
# Look for Events section — common causes:
# - "0/N nodes are available: insufficient memory" → instance type too small
# - "no nodes match pod topology spread constraints" → node group scaling
```
**Fix:** Increase node instance type or adjust node group scaling.
#### Helm chart deployment fails
**Symptom:** `helm upgrade --install` fails with template errors.
**Check:** Dry-run the chart:
```bash
helm template my-release deployers/charts/indexer/ -f values.yaml
```
**Common causes:**
* Missing required values (rindexerYaml, databaseUrl)
* Chart path wrong (should be `deployers/charts/`, not `deployers/eks/charts/`)
***
### Bare Metal
#### SSH provisioner fails
**Symptom:** `Error: timeout - last error: dial tcp: connection refused`
**Check:**
* Is the host reachable? `ping `
* Is SSH running? `ssh -p @`
* Is the SSH key correct? Check `bare_metal_ssh_private_key_path` points to the right file
* Does the user have sudo? Bare metal provisioner needs `sudo` for Docker operations
#### Docker not installed
**Symptom:** Provisioner fails with "docker: command not found"
**Fix:** The bare metal provider expects Docker and Docker Compose pre-installed on the host. Install them:
```bash
# On Ubuntu/Debian
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
```
## Variable Reference
All configuration variables for evm-cloud, organized by category. Variables marked **sensitive** are redacted in Terraform plan/apply output.
### Most Impactful Variables
These are the variables that most affect your deployment's cost, performance, and architecture:
| Variable | Impact | Default | What It Controls |
| ----------------------------------------- | ------------------ | -------------- | ---------------------------------------------------------------- |
| `compute_engine` | Architecture | `ec2` | Entire deployment model: EC2+Docker, EKS, k3s, or bare metal |
| `ec2_instance_type` / `k3s_instance_type` | Cost + Performance | `t3.medium` | Instance size — biggest cost driver |
| `ec2_indexer_mem_limit` | Performance | `2g` | How much RAM rindexer can use (increase for large indexing jobs) |
| `workload_mode` | Workflow | `terraform` | Whether Terraform or external tools manage workloads |
| `postgres_instance_class` | Cost | `db.t4g.micro` | RDS instance size (when using managed Postgres) |
| `network_enable_nat_gateway` | Cost | `false` | NAT gateway adds \~$35/mo — skip for dev |
***
### Core
| Variable | Type | Default | Description |
| ------------------------- | -------- | -------------- | ----------------------------------------------------------------------------------------------------- |
| `project_name` | `string` | (required) | Resource naming prefix. Must be non-empty. |
| `infrastructure_provider` | `string` | `"aws"` | Provider adapter: `aws` or `bare_metal`. |
| `compute_engine` | `string` | `"ec2"` | Compute engine: `ec2`, `eks`, `k3s`, or `docker_compose`. See [Concepts](./concepts.mdx) for details. |
| `workload_mode` | `string` | `"terraform"` | `terraform` (manage workloads) or `external` (output handoff only). |
| `deployment_target` | `string` | `"managed"` | Deployment mode: `managed`, `hybrid`, or `self_hosted`. |
| `runtime_arch` | `string` | `"multi"` | Architecture intent: `amd64`, `arm64`, or `multi`. |
| `database_mode` | `string` | `"managed"` | Database mode: `managed` or `self_hosted`. |
| `streaming_mode` | `string` | `"disabled"` | Streaming: `managed`, `self_hosted`, or `disabled`. |
| `ingress_mode` | `string` | `"managed_lb"` | Ingress: `managed_lb` or `self_hosted`. |
#### Valid Compute Engine Combinations
| Provider | Allowed Engines |
| ------------ | ----------------------- |
| `aws` | `ec2`, `eks`, `k3s` |
| `bare_metal` | `docker_compose`, `k3s` |
> **Note:** `k3s` requires `workload_mode = "external"` (two-phase deployment).
***
### Networking (AWS)
| Variable | Type | Default | Description |
| --------------------------------- | -------------- | ------------------------------ | ------------------------------------------------------------------------------ |
| `networking_enabled` | `bool` | `false` | Enable VPC provisioning. Required for most AWS deployments. |
| `network_environment` | `string` | `"dev"` | Profile: `dev` (minimal SGs), `production` (multi-AZ, NAT), `platform` (full). |
| `network_vpc_cidr` | `string` | `"10.42.0.0/16"` | VPC CIDR block. |
| `network_availability_zones` | `list(string)` | `["us-east-1a", "us-east-1b"]` | AZs for subnet placement. |
| `network_enable_nat_gateway` | `bool` | `false` | NAT gateway for private subnet egress. **Adds \~$35/mo.** Skip for dev. |
| `network_enable_vpc_endpoints` | `bool` | `false` | S3 gateway + interface endpoints (ECR, CloudWatch, SSM). |
| `aws_region` | `string` | `"us-east-1"` | AWS region. |
| `aws_skip_credentials_validation` | `bool` | `false` | Skip credential checks (for LocalStack testing). |
***
### EC2 Compute
Required when `compute_engine = "ec2"`.
| Variable | Type | Default | Sensitive | Description |
| ------------------------------------ | -------- | ------------- | --------- | ----------------------------------------------------------------------- |
| `ssh_public_key` | `string` | `""` | Yes | SSH public key content for the EC2 key pair. Required for EC2 and k3s. |
| `ec2_instance_type` | `string` | `"t3.medium"` | No | EC2 instance type. See [sizing guide](#instance-sizing) below. |
| `ec2_rpc_proxy_mem_limit` | `string` | `"1g"` | No | Docker memory limit for eRPC container. |
| `ec2_indexer_mem_limit` | `string` | `"2g"` | No | Docker memory limit for rindexer container. |
| `ec2_secret_recovery_window_in_days` | `number` | `7` | No | Secrets Manager deletion window. Set to `0` for dev (immediate delete). |
***
### k3s Compute
Required when `compute_engine = "k3s"`.
| Variable | Type | Default | Sensitive | Description |
| -------------------------- | -------------- | ---------------- | --------- | -------------------------------------------------------------------------------------------------------------- |
| `k3s_instance_type` | `string` | `"t3.medium"` | No | EC2 instance type for the k3s host. |
| `k3s_version` | `string` | `"v1.30.4+k3s1"` | No | Pinned k3s version. |
| `k3s_ssh_private_key_path` | `string` | `""` | Yes | Path to SSH private key for host provisioning. |
| `k3s_api_allowed_cidrs` | `list(string)` | `[]` | No | CIDRs allowed to access k3s API (port 6443). Defaults to VPC CIDR. **Add your IP for Terraform provisioning.** |
| `ssh_public_key` | `string` | `""` | Yes | SSH public key for the k3s EC2 key pair. |
> **Important:** `k3s_api_allowed_cidrs` must include your local IP (or `0.0.0.0/0` for dev) so Terraform can SSH into the instance during provisioning.
***
### PostgreSQL (Managed RDS)
Enable with `postgres_enabled = true` and `indexer_storage_backend = "postgres"`.
| Variable | Type | Default | Description |
| --------------------------- | -------- | ---------------- | --------------------------------------------------------- |
| `postgres_enabled` | `bool` | `false` | Enable managed PostgreSQL (RDS). |
| `postgres_instance_class` | `string` | `"db.t4g.micro"` | RDS instance class. See [sizing guide](#instance-sizing). |
| `postgres_engine_version` | `string` | `"16.4"` | PostgreSQL version. |
| `postgres_db_name` | `string` | `"rindexer"` | Database name. |
| `postgres_db_username` | `string` | `"rindexer"` | Master username. |
| `postgres_backup_retention` | `number` | `7` | Backup retention in days. |
***
### ClickHouse (BYODB)
Required when `indexer_storage_backend = "clickhouse"`. You provide the external ClickHouse endpoint.
| Variable | Type | Default | Sensitive | Description |
| ----------------------------- | -------- | ----------- | --------- | --------------------------------------------------------------------- |
| `indexer_clickhouse_url` | `string` | `""` | Yes | HTTP(S) endpoint, e.g. `https://your-instance.clickhouse.cloud:8443`. |
| `indexer_clickhouse_user` | `string` | `"default"` | No | ClickHouse username. |
| `indexer_clickhouse_password` | `string` | `""` | Yes | ClickHouse password. |
| `indexer_clickhouse_db` | `string` | `"default"` | No | Database name. |
***
### RPC Proxy (eRPC)
| Variable | Type | Default | Description |
| ------------------- | -------- | ---------------------------- | ------------------------------------------------------------------------ |
| `rpc_proxy_enabled` | `bool` | `false` | Enable eRPC proxy deployment. |
| `rpc_proxy_image` | `string` | `"ghcr.io/erpc/erpc:latest"` | Container image. Override for pinned versions or private registries. |
| `erpc_config_yaml` | `string` | `""` | Full eRPC config YAML content. Required when `rpc_proxy_enabled = true`. |
eRPC aggregates multiple upstream RPC endpoints with automatic failover, caching, and hedged requests. See [eRPC documentation](https://docs.erpc.cloud/) for config syntax.
***
### Indexer (rindexer)
| Variable | Type | Default | Description |
| ------------------------- | ------------- | ----------------------------------------- | --------------------------------------------------------------------------------------------- |
| `indexer_enabled` | `bool` | `false` | Enable rindexer deployment. |
| `indexer_image` | `string` | `"ghcr.io/joshstevens19/rindexer:latest"` | Container image. |
| `indexer_rpc_url` | `string` | `""` | RPC endpoint URL. Auto-resolves to eRPC internal URL when both are enabled. |
| `indexer_storage_backend` | `string` | `"postgres"` | Storage: `postgres` (managed RDS) or `clickhouse` (BYODB). |
| `rindexer_config_yaml` | `string` | `""` | Full rindexer.yaml content. Use `${RPC_URL}` and `${DATABASE_URL}` for runtime interpolation. |
| `rindexer_abis` | `map(string)` | `{}` | ABI files: `{ "ERC20.json" = file("abis/ERC20.json") }`. |
> **Tip:** When both `rpc_proxy_enabled` and `indexer_enabled` are true, the indexer's RPC URL automatically points to the eRPC internal service — no manual wiring needed.
***
### Bare Metal
Required when `infrastructure_provider = "bare_metal"`.
| Variable | Type | Default | Description |
| --------------------------------- | -------- | ---------- | --------------------------------- |
| `bare_metal_host` | `string` | `""` | IP or hostname of your VPS. |
| `bare_metal_ssh_user` | `string` | `"ubuntu"` | SSH user. |
| `bare_metal_ssh_private_key_path` | `string` | `""` | Path to SSH private key. |
| `bare_metal_ssh_port` | `number` | `22` | SSH port. |
| `bare_metal_rpc_proxy_mem_limit` | `string` | `"1g"` | Docker memory limit for eRPC. |
| `bare_metal_indexer_mem_limit` | `string` | `"2g"` | Docker memory limit for rindexer. |
***
### Instance Sizing
#### EC2 / k3s Host
| Instance | vCPU | RAM | Monthly Cost | Use Case |
| ----------- | ---- | ----- | ------------ | ------------------------------------------------ |
| `t3.micro` | 2 | 1 GB | \~$8 | Testing only (eRPC barely fits) |
| `t3.small` | 2 | 2 GB | \~$16 | Light dev (1 small indexer) |
| `t3.medium` | 2 | 4 GB | \~$33 | **Default.** Fits eRPC (1G) + rindexer (2G) + OS |
| `t3.large` | 2 | 8 GB | \~$67 | Heavy indexing, multiple contracts |
| `t3.xlarge` | 4 | 16 GB | \~$133 | Large-scale backfills |
**Rule of thumb:** `ec2_indexer_mem_limit` + `ec2_rpc_proxy_mem_limit` + 1 GB (OS) should not exceed instance RAM.
#### RDS PostgreSQL
| Instance | vCPU | RAM | Monthly Cost | Use Case |
| --------------- | ---- | ----- | ------------ | ------------------------ |
| `db.t4g.micro` | 2 | 1 GB | \~$13 | **Default.** Dev/staging |
| `db.t4g.small` | 2 | 2 GB | \~$26 | Small production |
| `db.t4g.medium` | 2 | 4 GB | \~$52 | Medium production |
| `db.r6g.large` | 2 | 16 GB | \~$175 | Heavy read workloads |
#### Memory Limits (Docker)
| Component | Default | Recommended Minimum | Heavy Use |
| --------- | ------- | ------------------- | --------- |
| eRPC | `1g` | `512m` | `2g` |
| rindexer | `2g` | `1g` | `4g` |
Increase `ec2_indexer_mem_limit` if rindexer is OOM-killed during large backfills. Check with `docker stats` on the instance.
## Updating Configuration After Deployment
How to change eRPC, rindexer, and ABI configurations after your initial deployment, for each compute engine.
### Overview
After the first `terraform apply` (or deployer run), you will need to update configuration for reasons like:
* Adding new contracts or events to index
* Changing RPC endpoints or adding upstreams to eRPC
* Updating ABI files for upgraded contracts
* Tuning performance settings (batch sizes, polling intervals)
The update procedure depends on your compute engine and workload mode.
***
### EC2 (Docker Compose)
#### terraform mode (default)
SSH into the instance, edit config files, and restart services:
```bash
# Get the instance IP
terraform output -json workload_handoff | jq -r '.runtime.ec2.instance_ip'
# SSH in
ssh -i ~/.ssh/evm-cloud ubuntu@
# Edit configurations
sudo vim /opt/evm-cloud/config/rindexer.yaml
sudo vim /opt/evm-cloud/config/erpc.yaml
# Restart affected services
sudo docker compose -f /opt/evm-cloud/docker-compose.yml restart rindexer
sudo docker compose -f /opt/evm-cloud/docker-compose.yml restart erpc
```
> **Note:** Terraform uses `lifecycle { ignore_changes = [user_data] }` on EC2 instances. This means config changes made directly on the instance will not be overwritten by a subsequent `terraform apply`, and config changes in your Terraform variables will not trigger instance recreation. This is intentional -- it prevents accidental downtime from instance replacement when only config content changes.
#### external mode
Use the EC2 deployer to push updated configs:
```bash
# Edit local config files
vim config/rindexer.yaml
# Redeploy
terraform output -json workload_handoff | ./deployers/ec2/deploy.sh /dev/stdin --config-dir ./config
```
The deployer copies files via SCP and restarts Docker Compose services.
***
### EKS
#### terraform mode
Update the config variables in your `.tfvars` file and run `terraform apply`:
```hcl
# Update rindexer config content
rindexer_config_yaml = file("config/rindexer.yaml")
erpc_config_yaml = file("config/erpc.yaml")
```
```bash
terraform apply -var-file=eks.tfvars
```
Terraform detects the content hash change on the ConfigMap, updates it, and triggers a pod rollout. Pods restart with the new configuration automatically.
#### external mode
Update Helm values or re-run the deployer:
```bash
# Edit config files
vim config/rindexer.yaml
# Redeploy via deployer
terraform output -json workload_handoff | ./deployers/eks/deploy.sh /dev/stdin --config-dir ./config
```
Or if using ArgoCD, commit the updated config to your GitOps repository and sync:
```bash
git add config/rindexer.yaml
git commit -m "Add new contract to indexer config"
git push
# ArgoCD detects the change and syncs
```
***
### k3s
k3s always uses external mode. Edit your config files and re-run the deployer:
```bash
# Edit config files in your config directory
vim config/rindexer.yaml
vim config/erpc.yaml
# Re-run the deployer
terraform output -json workload_handoff | ./../../deployers/k3s/deploy.sh /dev/stdin --config-dir ./config
```
The deployer runs `helm upgrade`, which is idempotent. Only changed values trigger pod restarts. If the config content has not changed, Helm detects this and skips the upgrade -- no pods are restarted.
***
### Bare Metal (Docker Compose)
Update the config variables and run `terraform apply`:
```hcl
rindexer_config_yaml = file("config/rindexer.yaml")
erpc_config_yaml = file("config/erpc.yaml")
```
```bash
terraform apply -var-file=bare_metal.tfvars
```
Terraform detects the content hash change, re-provisions the config files via SSH file provisioner, and restarts Docker Compose services. The `.env` file and Docker Compose definition are updated in place.
Alternatively, SSH in directly and edit configs on the host, same as the EC2 procedure.
***
### ABI Updates
Contract ABIs are delivered alongside the rindexer config. When you add or modify ABIs, the update procedure follows the same path as config updates for your engine.
#### Using the `rindexer_abis` variable (terraform mode)
```hcl
rindexer_abis = {
"ERC20.json" = file("abis/ERC20.json")
"MyContract.json" = file("abis/MyContract.json")
"NewContract.json" = file("abis/NewContract.json") # newly added
}
```
```bash
terraform apply -var-file=my-deployment.tfvars
```
#### Using the config directory (external mode)
Place ABI files in your config directory:
```
config/
abis/
ERC20.json
MyContract.json
NewContract.json # newly added
rindexer.yaml # references the ABIs
erpc.yaml
```
Re-run the deployer:
```bash
terraform output -json workload_handoff | ./deployers/k3s/deploy.sh /dev/stdin --config-dir ./config
```
> **Note:** When adding a new ABI, you also need to update `rindexer.yaml` to reference the new contract and specify which events to index. The ABI file alone does not trigger indexing.
***
### Quick Reference
| Engine | Mode | Update Method | Restarts |
| ---------- | --------- | ---------------------------------------- | ------------------- |
| EC2 | terraform | SSH + edit + `docker compose restart` | Manual |
| EC2 | external | `deployers/ec2/deploy.sh` | Automatic |
| EKS | terraform | Update variables + `terraform apply` | Automatic (rolling) |
| EKS | external | `deployers/eks/deploy.sh` or ArgoCD sync | Automatic (rolling) |
| k3s | external | `deployers/k3s/deploy.sh` | Automatic (rolling) |
| Bare metal | terraform | Update variables + `terraform apply` | Automatic |
***
### Related Pages
* **[Core Concepts -- Config Injection](../concepts.mdx#config-injection)** -- How configs reach containers
* **[Core Concepts -- Lifecycle Behavior](../concepts.mdx#lifecycle-behavior)** -- What Terraform does on changes
* **[Two-Phase Deployment](./two-phase-workflow.mdx)** -- k3s deployer workflow
* **[External Deployers](./external-deployers.mdx)** -- All deployer scripts in detail
* **[Variable Reference](../variable-reference.mdx)** -- Config-related variables
## External Deployers
When `workload_mode = "external"` (or when using `k3s`), Terraform provisions infrastructure only and outputs a structured `workload_handoff`. External deployers consume this handoff to deploy workloads independently.
### When to Use External Mode
Set `workload_mode = "external"` when:
* **CI/CD pipelines** deploy your containers (GitHub Actions, GitLab CI, Jenkins)
* **GitOps tools** manage workloads (ArgoCD, Flux)
* **Different deploy cadences** -- infrastructure changes monthly, application deploys daily
* **Team separation** -- infra team manages Terraform, app team manages Helm or ArgoCD
* **Testing** -- you want to iterate on workload configs without touching infrastructure
```hcl
module "evm_cloud" {
source = "github.com/evm-cloud/evm-cloud"
workload_mode = "external"
compute_engine = "eks"
# ... other variables
}
```
> **Note:** `k3s` always operates in external mode regardless of the `workload_mode` setting. See [Two-Phase Deployment](./two-phase-workflow.mdx) for why.
***
### The Workload Handoff v1 Contract
The `workload_handoff` output is a versioned JSON contract between Terraform (infrastructure) and deployers (workloads). The schema is stable within a major version.
```json
{
"version": "v1",
"mode": "external",
"compute_engine": "k3s",
"project_name": "my-indexer",
"identity": {
"iam_role_arn": "arn:aws:iam::123456789012:role/evm-cloud-my-indexer",
"instance_profile": "evm-cloud-my-indexer-profile"
},
"network": {
"vpc_id": "vpc-0abc123def456",
"private_subnet_ids": ["subnet-0aaa", "subnet-0bbb"],
"security_group_ids": ["sg-0ccc"]
},
"runtime": {
"ec2": {
"ssh_command": "ssh -i key.pem ubuntu@203.0.113.10",
"instance_ip": "203.0.113.10",
"config_path": "/opt/evm-cloud/config/"
},
"eks": {
"cluster_name": "evm-cloud-prod",
"cluster_endpoint": "https://ABCDEF.gr7.us-east-1.eks.amazonaws.com",
"oidc_provider_arn": "arn:aws:iam::123456789012:oidc-provider/..."
},
"k3s": {
"kubeconfig_base64": "YXBpVmVyc2lvbjogdjEK...",
"cluster_endpoint": "https://203.0.113.10:6443",
"host_ip": "203.0.113.10"
}
},
"services": {
"erpc": { "name": "erpc", "port": 4000, "internal_url": "http://erpc:4000" },
"rindexer": { "name": "rindexer", "port": 3001, "internal_url": "http://rindexer:3001" }
},
"data": {
"database_type": "postgresql",
"host": "evm-cloud-db.abc123.us-east-1.rds.amazonaws.com",
"port": 5432,
"username": "rindexer",
"password": "generated-password",
"database": "rindexer"
},
"artifacts": {
"config_channel": "helm",
"cloudwatch_log_group": "/evm-cloud/my-indexer"
}
}
```
Only the `runtime` block matching the active `compute_engine` is populated. The others are `null`.
***
### Handoff Consumption Pattern
#### Pipe directly (recommended)
Avoids writing credentials to disk:
```bash
terraform output -json workload_handoff | ./deployers/k3s/deploy.sh /dev/stdin --config-dir ./config
```
#### Save to file
When the deployer will run from a different machine or CI step:
```bash
terraform output -json workload_handoff > handoff.json
chmod 0600 handoff.json # contains credentials!
```
Then on the deploy machine:
```bash
./deployers/k3s/deploy.sh handoff.json --config-dir ./config
```
> **Warning:** `handoff.json` contains database passwords and potentially a full kubeconfig. Treat it as a credential. Do not commit it to version control. Delete it after use. See [Secrets Management](./secrets-management.mdx) for production handling.
#### Extract individual fields
For custom pipelines, extract specific values with `jq`:
```bash
# Get kubeconfig for kubectl
terraform output -json workload_handoff | jq -r '.runtime.k3s.kubeconfig_base64' | base64 -d > /tmp/kubeconfig
# Get database connection string
DB_HOST=$(terraform output -json workload_handoff | jq -r '.data.host')
DB_PASS=$(terraform output -json workload_handoff | jq -r '.data.password')
DB_PORT=$(terraform output -json workload_handoff | jq -r '.data.port')
```
***
### Three Deployer Paths
evm-cloud ships with deployer scripts for each compute engine. All deployers live in the `deployers/` directory and use shared Helm charts from `deployers/charts/`.
#### EC2 SSH Deployer (`deployers/ec2/`)
For `compute_engine = "ec2"` with `workload_mode = "external"`.
**What it does:**
1. Reads SSH connection details from the handoff (`runtime.ec2.ssh_command`)
2. Copies config files to the instance at `/opt/evm-cloud/config/`
3. Updates the `.env` file with current credentials
4. Restarts Docker Compose services
```bash
terraform output -json workload_handoff | ./deployers/ec2/deploy.sh /dev/stdin --config-dir ./config
```
**Update workflow:**
```bash
# Edit config locally, then redeploy
vim config/rindexer.yaml
terraform output -json workload_handoff | ./deployers/ec2/deploy.sh /dev/stdin --config-dir ./config
```
The deployer uses `scp` to copy files and `ssh` to restart services. No Helm or Kubernetes involved.
#### EKS Helm Deployer (`deployers/eks/`)
For `compute_engine = "eks"` with `workload_mode = "external"`.
**What it does:**
1. Configures kubectl via `aws eks update-kubeconfig` using the cluster name from the handoff
2. Renders Helm values from handoff data (database connection, service configuration)
3. Injects config files from `--config-dir`
4. Runs `helm upgrade --install` for eRPC and rindexer using shared charts
```bash
terraform output -json workload_handoff | ./deployers/eks/deploy.sh /dev/stdin --config-dir ./config
```
**ArgoCD integration:**
The EKS deployer also includes ArgoCD Application manifests at `deployers/eks/argocd/`. Point ArgoCD at your config repository and it will sync workloads automatically:
```bash
# Apply ArgoCD Application resources
kubectl apply -f deployers/eks/argocd/
```
#### k3s Helm Deployer (`deployers/k3s/`)
For `compute_engine = "k3s"` (always external mode).
**What it does:**
1. Extracts kubeconfig from `runtime.k3s.kubeconfig_base64` and decodes it
2. Renders Helm values from handoff data
3. Injects real configs from `--config-dir`
4. Runs `helm upgrade --install` for eRPC and rindexer
```bash
terraform output -json workload_handoff | ./deployers/k3s/deploy.sh /dev/stdin --config-dir ./config
```
**Teardown:**
```bash
terraform output -json workload_handoff | ./deployers/k3s/teardown.sh /dev/stdin
```
See [Two-Phase Deployment](./two-phase-workflow.mdx) for the full k3s workflow.
***
### Shared Helm Charts
All Kubernetes deployers (k3s and EKS) use the same Helm charts at `deployers/charts/`:
| Chart | Purpose |
| ----------------------------- | ------------------------------------------------------- |
| `deployers/charts/rpc-proxy/` | eRPC proxy deployment, service, and ConfigMap |
| `deployers/charts/indexer/` | rindexer deployment, service, ConfigMap, and ABI volume |
The charts are designed to be generic. Deployer scripts render values from the handoff and config files, then pass them to `helm upgrade --install`.
***
### CI/CD Examples
#### GitHub Actions
```yaml
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::123456789012:role/deploy
aws-region: us-east-1
- name: Setup Terraform
uses: hashicorp/setup-terraform@v3
- name: Get handoff
run: |
cd examples/minimal_aws_k3s_byo_clickhouse
terraform init
terraform output -json workload_handoff > /tmp/handoff.json
chmod 0600 /tmp/handoff.json
- name: Deploy workloads
run: |
./deployers/k3s/deploy.sh /tmp/handoff.json --config-dir ./config
```
#### GitOps (ArgoCD)
For teams using ArgoCD, the infra pipeline outputs the handoff and the app pipeline reads it:
1. Terraform provisions infrastructure and writes handoff to an S3 bucket (encrypted)
2. ArgoCD Application resources reference the shared Helm charts
3. ArgoCD syncs workloads using values derived from the handoff
***
### Related Pages
* **[Core Concepts -- Workload Modes](../concepts.mdx#workload-modes)** -- terraform vs external mode
* **[Two-Phase Deployment](./two-phase-workflow.mdx)** -- Full k3s walkthrough
* **[Secrets Management](./secrets-management.mdx)** -- Securing the handoff
* **[Updating Configuration](./config-updates.mdx)** -- Post-deploy config changes per engine
* **[Architecture](../architecture.mdx)** -- Module map and dependency graph
## Production Checklist
A step-by-step checklist for hardening your evm-cloud deployment before running in production.
***
### Remote State Backend
Terraform state contains all variable values, including secrets, in plaintext. Never use local state for production.
#### S3 + KMS + DynamoDB
```hcl
# backend.tf (add to your example directory)
terraform {
backend "s3" {
bucket = "myorg-terraform-state"
key = "evm-cloud/production/terraform.tfstate"
region = "us-east-1"
encrypt = true
kms_key_id = "alias/terraform-state"
dynamodb_table = "terraform-locks"
}
}
```
Create the prerequisite resources:
```bash
# S3 bucket with versioning
aws s3api create-bucket --bucket myorg-terraform-state --region us-east-1
aws s3api put-bucket-versioning --bucket myorg-terraform-state \
--versioning-configuration Status=Enabled
# KMS key for encryption
aws kms create-alias --alias-name alias/terraform-state \
--target-key-id $(aws kms create-key --query KeyMetadata.KeyId --output text)
# DynamoDB table for state locking
aws dynamodb create-table \
--table-name terraform-locks \
--attribute-definitions AttributeName=LockID,AttributeType=S \
--key-schema AttributeName=LockID,KeyType=HASH \
--billing-mode PAY_PER_REQUEST
```
After adding the backend configuration, migrate existing state:
```bash
terraform init -migrate-state
```
***
### Environment Isolation
Use separate state files (or separate directories) for each environment. Never share Terraform state across dev/staging/production.
#### Option A: Separate directories
```
environments/
dev/
main.tf # source = "../../"
dev.tfvars
secrets.auto.tfvars
backend.tf # key = "evm-cloud/dev/terraform.tfstate"
staging/
main.tf
staging.tfvars
secrets.auto.tfvars
backend.tf # key = "evm-cloud/staging/terraform.tfstate"
production/
main.tf
production.tfvars
secrets.auto.tfvars
backend.tf # key = "evm-cloud/production/terraform.tfstate"
```
#### Option B: Terraform workspaces
```bash
terraform workspace new production
terraform workspace select production
terraform apply -var-file=production.tfvars
```
Option A is recommended -- it provides stronger isolation and makes it impossible to accidentally apply a dev config to production.
***
### CI/CD Gates
#### On every pull request
Run `make qa` to catch formatting issues, validation errors, linting violations, and security misconfigurations:
```bash
make qa
# Runs: fmt-check + validate + lint (tflint) + security (checkov)
```
#### Before merge
Run `make verify` to validate all examples plan successfully:
```bash
make verify
# Runs: qa + plan for all examples against LocalStack
```
#### Plan-then-apply workflow
Never run `terraform apply` directly in CI. Generate a plan artifact, review it, then apply the specific plan file:
```bash
# CI: generate plan
terraform plan -var-file=production.tfvars -out=tfplan
# Upload tfplan as a build artifact
# After human review: apply the exact plan
terraform apply tfplan
```
This ensures the applied changes match exactly what was reviewed. No surprises from concurrent state changes.
***
### Instance Sizing
Use `t3.medium` or larger for production. The default `t3.medium` (4 GB RAM) fits eRPC (1 GB) + rindexer (2 GB) + OS (1 GB) with no headroom.
| Workload | Recommended Instance | Memory Config |
| ---------------------------- | -------------------- | -------------------- |
| Single chain, few contracts | `t3.medium` (4 GB) | rindexer 2g, eRPC 1g |
| Single chain, many contracts | `t3.large` (8 GB) | rindexer 4g, eRPC 2g |
| Multi-chain or backfill | `t3.xlarge` (16 GB) | rindexer 8g, eRPC 2g |
Monitor resource usage to right-size:
```bash
# EC2: check Docker container memory
ssh ubuntu@ "sudo docker stats --no-stream"
# K8s: check pod resource usage
kubectl top pods
```
See [Variable Reference -- Instance Sizing](../variable-reference.mdx#instance-sizing) for the full sizing table.
***
### NAT Gateway
Enable the NAT gateway for production if your workloads run in private subnets and need outbound internet access (for RPC calls, ClickHouse Cloud connections, etc.):
```hcl
network_enable_nat_gateway = true
```
> **Warning:** NAT gateways add approximately $35/month plus data transfer charges. Skip for dev environments where instances can use public subnets.
***
### Database Backup
#### Managed PostgreSQL (RDS)
Set the backup retention period:
```hcl
# production.tfvars
postgres_backup_retention = 30 # days (default is 7)
```
RDS handles automated daily backups and point-in-time recovery. Verify backups are running:
```bash
aws rds describe-db-instances \
--query "DBInstances[?DBInstanceIdentifier=='evm-cloud-prod'].{Retention:BackupRetentionPeriod,LatestRestore:LatestRestorableTime}"
```
#### ClickHouse (BYODB)
If using ClickHouse Cloud, backups are managed by the service. For self-hosted ClickHouse, configure backups on the ClickHouse side -- evm-cloud does not manage external database backups.
***
### Secrets Rotation
To rotate database credentials or other secrets:
1. Update the password in `secrets.auto.tfvars` (or your `TF_VAR_*` environment variable)
2. Run `terraform apply`
3. Verify the new credentials are propagated:
| Engine | Post-rotation Step |
| --------------- | ------------------------------------------------------------------- |
| EC2 | SSH in, run `pull-secrets.sh`, then `docker compose restart` |
| EKS (terraform) | Terraform updates the K8s Secret and triggers rollout automatically |
| EKS (external) | Re-run `deployers/eks/deploy.sh` |
| k3s | Re-run `deployers/k3s/deploy.sh` |
| Bare metal | Terraform re-provisions `.env` via SSH automatically |
Set the Secrets Manager recovery window appropriately:
```hcl
# production.tfvars
ec2_secret_recovery_window_in_days = 30 # default is 7, use 0 only for dev
```
See [Secrets Management](./secrets-management.mdx) for the full secrets lifecycle.
***
### Destroy Safety
#### Kubernetes deployments (k3s, EKS external)
Always remove workloads before destroying infrastructure:
```bash
# Step 1: Remove workloads
./deployers/k3s/teardown.sh handoff.json
# or: helm uninstall rindexer erpc
# Step 2: Destroy infrastructure
terraform destroy -var-file=production.tfvars
```
Skipping step 1 will not leak AWS resources (the EC2 instance is terminated along with everything on it), but Helm release state and any persistent volumes will be lost without clean shutdown.
#### All deployments
Before running `terraform destroy` in production:
1. Generate a destroy plan and review it:
```bash
terraform plan -destroy -var-file=production.tfvars -out=destroy-plan
# Review the plan carefully
terraform apply destroy-plan
```
2. Verify no unexpected resources are being destroyed (especially RDS instances or EBS volumes)
3. Confirm database backups are current before destroying any database resources
> **Warning:** Never run `terraform destroy` in production without reviewing the destroy plan first. RDS instances have deletion protection enabled by default, but other resources do not.
***
### Monitoring
#### EC2
Terraform creates a CloudWatch log group (available in the handoff as `artifacts.cloudwatch_log_group`). View logs:
```bash
# From the instance
ssh ubuntu@ "sudo docker compose -f /opt/evm-cloud/docker-compose.yml logs rindexer --tail 100"
ssh ubuntu@ "sudo docker compose -f /opt/evm-cloud/docker-compose.yml logs erpc --tail 100"
# Via CloudWatch (if configured)
aws logs tail /evm-cloud/my-indexer --follow
```
#### Kubernetes (k3s, EKS)
```bash
kubectl logs -l app=rindexer --tail=100
kubectl logs -l app=erpc --tail=100
# Watch for restarts
kubectl get pods -w
```
#### Health checks
Verify the indexer is making progress:
```bash
# rindexer health endpoint
curl http://:3001/health
# eRPC health endpoint
curl http://:4000/health
```
> **Note:** A future release will include a Prometheus + Grafana monitoring addon for dashboards and alerting. For now, use CloudWatch (EC2) or standard Kubernetes monitoring tooling.
***
### Summary Checklist
Use this as a pre-launch checklist:
* [ ] Remote state backend configured (S3 + KMS + DynamoDB)
* [ ] Environment isolation in place (separate directories or workspaces)
* [ ] CI runs `make qa` on every PR
* [ ] Plan-then-apply workflow enforced (no direct `terraform apply` in CI)
* [ ] Instance sized for workload (`t3.medium` minimum, monitor and adjust)
* [ ] NAT gateway enabled if private subnets need outbound access
* [ ] Database backup retention set (`postgres_backup_retention = 30`)
* [ ] Secrets Manager recovery window set to 30 days
* [ ] `secrets.auto.tfvars` and `handoff.json` are in `.gitignore`
* [ ] Destroy procedure documented (workload teardown before `terraform destroy`)
* [ ] Monitoring and log access verified
* [ ] Health check endpoints reachable
***
### Related Pages
* **[Secrets Management](./secrets-management.mdx)** -- Full secrets lifecycle and rotation
* **[Updating Configuration](./config-updates.mdx)** -- Post-deploy config changes
* **[Two-Phase Deployment](./two-phase-workflow.mdx)** -- k3s teardown procedure
* **[Variable Reference](../variable-reference.mdx)** -- All configuration options with defaults
* **[Getting Started](../getting-started.mdx)** -- First deployment walkthrough
## Secrets Management
How evm-cloud handles credentials across compute engines, and how to manage them securely in development and production.
### The `secrets.auto.tfvars` Pattern
Every [example](../examples/index.mdx) includes a `secrets.auto.tfvars.example` template. This is the primary way to provide sensitive values during local development.
```bash
cd examples/minimal_aws_byo_clickhouse
cp secrets.auto.tfvars.example secrets.auto.tfvars
```
Edit `secrets.auto.tfvars` with your values:
```hcl
# SSH key (public key content, not the file path)
ssh_public_key = "ssh-ed25519 AAAA... your-email"
# ClickHouse connection
indexer_clickhouse_url = "https://your-instance.clickhouse.cloud:8443"
indexer_clickhouse_password = "your-password"
```
**How it works:**
1. The file is named `*.auto.tfvars`, so Terraform loads it automatically -- no `-var-file` flag needed
2. The file is listed in `.gitignore` -- it never enters version control
3. The `.example` template shows which variables are required without exposing real values
#### What Goes in Secrets
| Variable | Description | Used By |
| ----------------------------- | ---------------------------------------- | ---------------- |
| `ssh_public_key` | Public key content for EC2/k3s key pair | EC2, k3s |
| `k3s_ssh_private_key_path` | Path to private key for k3s provisioning | k3s |
| `indexer_clickhouse_password` | ClickHouse database password | ClickHouse BYODB |
| `indexer_clickhouse_url` | ClickHouse HTTP(S) endpoint | ClickHouse BYODB |
***
### Secrets Delivery by Compute Engine
Each compute engine has a different mechanism for getting credentials from Terraform into running containers.
#### EC2 (Docker Compose)
```
terraform apply
→ Stores credentials in AWS Secrets Manager
→ cloud-init installs pull-secrets.sh on the instance
Instance boot:
pull-secrets.sh
→ Fetches secrets from Secrets Manager via AWS CLI
→ Writes /opt/evm-cloud/.env (chmod 0600)
→ Docker Compose reads .env file
```
The `pull-secrets.sh` script runs at instance startup and whenever you need to refresh credentials. Secrets Manager ARNs are passed to the instance via cloud-init user data.
#### EKS (terraform mode)
Terraform creates Kubernetes Secrets directly from variable values. Pod specs reference these secrets as environment variables or mounted files.
```
terraform apply
→ Creates K8s Secret resource (via kubernetes provider)
→ Pod spec references secret by name
→ kubelet mounts secret into container
```
No intermediate storage or scripts. Terraform manages the full lifecycle.
#### k3s (external mode)
The database password flows through the [workload handoff](../concepts.mdx#the-workload-handoff) into Helm values, which create a Kubernetes Secret.
```
terraform apply
→ Outputs workload_handoff JSON (contains password)
deploy.sh
→ Reads handoff JSON
→ Renders Helm values (password → values.yaml)
→ helm upgrade --install
→ Chart creates K8s Secret from values
→ Pod mounts secret
```
> **Note:** This means the password exists in the Terraform state and in the handoff JSON. See [Production Recommendations](#production-recommendations) for how to secure both. A future improvement will add External Secrets Operator (ESO) support to sync secrets directly from AWS Secrets Manager into the cluster, removing the password from the handoff entirely.
#### Bare Metal (Docker Compose)
Terraform's SSH file provisioner delivers a `.env` file directly to the remote host.
```
terraform apply
→ Connects via SSH
→ Writes /opt/evm-cloud/.env (chmod 0600)
→ Docker Compose reads .env file
```
No cloud secrets service. The `.env` file on disk is the only storage location.
***
### CI/CD Integration
In CI/CD pipelines, use `TF_VAR_*` environment variables instead of `secrets.auto.tfvars` files. Terraform automatically maps environment variables prefixed with `TF_VAR_` to input variables.
```bash
# GitHub Actions example
export TF_VAR_ssh_public_key="$SSH_PUBLIC_KEY"
export TF_VAR_indexer_clickhouse_password="$CH_PASSWORD"
export TF_VAR_indexer_clickhouse_url="$CH_URL"
export TF_VAR_k3s_ssh_private_key_path="$K3S_KEY_PATH"
terraform apply -var-file=minimal_clickhouse.tfvars
```
The non-sensitive `.tfvars` file (instance types, region, project name) can be committed to the repository. Secrets come from the environment.
```yaml
# .github/workflows/deploy.yml (snippet)
jobs:
deploy:
steps:
- name: Terraform Apply
env:
TF_VAR_ssh_public_key: ${{ secrets.SSH_PUBLIC_KEY }}
TF_VAR_indexer_clickhouse_password: ${{ secrets.CH_PASSWORD }}
TF_VAR_indexer_clickhouse_url: ${{ secrets.CH_URL }}
run: |
terraform init
terraform apply -auto-approve -var-file=minimal_clickhouse.tfvars
```
***
### Sensitive Outputs
The `workload_handoff` output is marked `sensitive = true` because it may contain:
* kubeconfig data (k3s) -- grants full cluster access
* Database passwords
* Internal service endpoints
Access it explicitly:
```bash
# View the full handoff
terraform output -json workload_handoff
# Extract specific fields
terraform output -json workload_handoff | jq -r '.runtime.k3s.kubeconfig_base64'
terraform output -json workload_handoff | jq -r '.data.password'
```
Terraform will not display sensitive outputs in plan/apply logs. You must use `terraform output -json` to retrieve them.
> **Warning:** If you write the handoff to a file (`> handoff.json`), treat that file as a credential. Set `chmod 0600` and do not commit it. Prefer piping directly to the deployer to avoid writing credentials to disk -- see [External Deployers](./external-deployers.mdx#handoff-consumption-pattern).
***
### Production Recommendations
#### Encrypted Terraform State
Terraform state contains all secret values in plaintext. Use an encrypted remote backend:
```hcl
# backend.tf
terraform {
backend "s3" {
bucket = "my-terraform-state"
key = "evm-cloud/production/terraform.tfstate"
region = "us-east-1"
encrypt = true
kms_key_id = "alias/terraform-state"
dynamodb_table = "terraform-locks"
}
}
```
See the [Production Checklist](./production-checklist.mdx#remote-state-backend) for the full backend setup.
#### Secrets Manager Recovery Window
The `ec2_secret_recovery_window_in_days` variable controls how long AWS Secrets Manager retains a deleted secret before permanent removal.
| Environment | Recommended Value | Why |
| ----------- | ----------------- | ---------------------------------------------------------------------- |
| Dev | `0` | Immediate deletion -- no cost for stale secrets during rapid iteration |
| Staging | `7` (default) | One week recovery window |
| Production | `30` | Full recovery period in case of accidental deletion |
```hcl
# production.tfvars
ec2_secret_recovery_window_in_days = 30
```
#### Credential Rotation
To rotate database credentials:
1. Update the password in `secrets.auto.tfvars` (or `TF_VAR_*` environment variable)
2. Run `terraform apply`
3. **EC2**: Terraform updates the Secrets Manager entry. SSH into the instance and run `pull-secrets.sh`, then `docker compose restart`
4. **k3s/EKS external**: Re-run the deployer script -- Helm upgrade will update the Kubernetes Secret and trigger a pod rollout
5. **EKS terraform**: Terraform updates the K8s Secret directly and triggers a rollout
6. **Bare metal**: Terraform re-provisions the `.env` file via SSH and restarts Docker Compose
#### General Best Practices
* Never commit `secrets.auto.tfvars` or `handoff.json` files
* Use separate AWS accounts (or at minimum, separate state files) for dev/staging/production
* Restrict Secrets Manager IAM policies to the specific secrets your deployment needs
* Audit Secrets Manager access via CloudTrail
* For k3s deployments, delete `handoff.json` after deploying workloads, or pipe directly to avoid writing to disk
***
### Related Pages
* **[Core Concepts -- Secrets Delivery](../concepts.mdx#secrets-delivery)** -- How secrets reach containers by engine
* **[Variable Reference](../variable-reference.mdx)** -- All secret-related variables with types and defaults
* **[External Deployers](./external-deployers.mdx)** -- Consuming the workload handoff securely
* **[Production Checklist](./production-checklist.mdx)** -- Full production hardening guide
* **[Two-Phase Deployment](./two-phase-workflow.mdx)** -- How k3s handles secrets across phases
## Two-Phase Deployment
k3s and EKS external mode require a two-phase deployment. This guide walks through the full workflow from provisioning to teardown.
### Why Two Phases
The Terraform `kubernetes` and `helm` providers need a kubeconfig to connect to the cluster. But the kubeconfig does not exist until Terraform finishes creating the cluster. This is a chicken-and-egg problem:
```
terraform plan
→ kubernetes provider tries to connect
→ kubeconfig doesn't exist yet
→ plan fails
```
Separating into two phases resolves this. Phase 1 creates the cluster and outputs the kubeconfig. Phase 2 uses that kubeconfig to deploy workloads.
### When It Applies
| Compute Engine | Workload Mode | Two-Phase Required |
| ---------------- | ------------- | -------------------------------------------------------------------- |
| `ec2` | `terraform` | No -- single `terraform apply` does everything |
| `ec2` | `external` | Yes -- Terraform provisions, external tool deploys |
| `eks` | `terraform` | No -- Terraform kubernetes provider uses `exec` auth |
| `eks` | `external` | Yes -- same pattern as k3s |
| `k3s` | any | **Always** -- k3s forces external mode regardless of `workload_mode` |
| `docker_compose` | `terraform` | No -- single `terraform apply` |
***
### Phase 1: Infrastructure
`terraform apply` provisions all infrastructure and outputs the `workload_handoff`.
```bash
cd examples/minimal_aws_k3s_byo_clickhouse
terraform init
terraform apply -var-file=minimal_k3s.tfvars
```
Terraform creates:
* VPC with public subnets and security groups
* EC2 instance with k3s installed and running
* IAM roles and Secrets Manager entries
* Kubeconfig extracted from the k3s host
After apply completes, the `workload_handoff` output contains everything Phase 2 needs:
```bash
terraform output -json workload_handoff
```
```json
{
"version": "v1",
"mode": "external",
"compute_engine": "k3s",
"project_name": "my-indexer",
"runtime": {
"k3s": {
"kubeconfig_base64": "YXBpVmVyc2lvbjogdjEK...",
"cluster_endpoint": "https://203.0.113.10:6443",
"host_ip": "203.0.113.10"
}
},
"services": {
"erpc": { "name": "erpc", "port": 4000 },
"rindexer": { "name": "rindexer", "port": 3001 }
},
"data": {
"database_type": "clickhouse",
"host": "your-instance.clickhouse.cloud",
"port": 8443,
"password": "your-password"
}
}
```
> **Warning:** The handoff contains credentials (kubeconfig, database password). See [Secrets Management](./secrets-management.mdx#sensitive-outputs) for how to handle it securely.
***
### Phase 2: Workloads
The deployer script reads the handoff, extracts the kubeconfig, renders Helm values, and deploys workloads onto the cluster.
```bash
terraform output -json workload_handoff | ./../../deployers/k3s/deploy.sh /dev/stdin --config-dir ./config
```
What the deployer does:
1. Extracts `kubeconfig_base64` from the handoff and decodes it
2. Renders Helm values from the handoff data (database connection, service ports)
3. Injects real config files from the `--config-dir` directory (rindexer.yaml, erpc.yaml, ABIs)
4. Runs `helm upgrade --install` for each service using shared charts at `deployers/charts/`
The `--config-dir` flag points to a directory with your actual configuration files:
```
config/
erpc.yaml # eRPC proxy configuration
rindexer.yaml # rindexer indexer configuration
abis/ # Contract ABI files
ERC20.json
MyContract.json
```
***
### Full Walkthrough
Complete sequence from zero to running indexer:
#### 1. Configure
```bash
cd examples/minimal_aws_k3s_byo_clickhouse
cp secrets.auto.tfvars.example secrets.auto.tfvars
# Edit secrets.auto.tfvars with your SSH key, ClickHouse credentials
```
#### 2. Phase 1 -- Provision Infrastructure
```bash
terraform init
terraform apply -var-file=minimal_k3s.tfvars
```
#### 3. Phase 2 -- Deploy Workloads
```bash
terraform output -json workload_handoff | ./../../deployers/k3s/deploy.sh /dev/stdin --config-dir ./config
```
#### 4. Verify
```bash
# Extract kubeconfig
KUBECONFIG=$(mktemp)
terraform output -json workload_handoff | jq -r '.runtime.k3s.kubeconfig_base64' | base64 -d > "$KUBECONFIG"
# Check pods are running
kubectl --kubeconfig="$KUBECONFIG" get pods
# Check logs
kubectl --kubeconfig="$KUBECONFIG" logs -l app=rindexer --tail=50
kubectl --kubeconfig="$KUBECONFIG" logs -l app=erpc --tail=50
```
You should see both `erpc` and `rindexer` pods in `Running` state.
***
### Teardown (Reverse Order)
Always remove workloads before destroying infrastructure. This ensures Helm release state is cleaned up properly.
#### 1. Remove Workloads (Phase 2)
```bash
./../../deployers/k3s/teardown.sh handoff.json
```
Or if you did not save the handoff to a file:
```bash
terraform output -json workload_handoff | ./../../deployers/k3s/teardown.sh /dev/stdin
```
#### 2. Destroy Infrastructure (Phase 1)
```bash
terraform destroy -var-file=minimal_k3s.tfvars
```
> **Note:** If you skip the workload teardown and go straight to `terraform destroy`, the EC2 instance (and all its containers) will be terminated. No resources leak on AWS, but Helm release state is lost. This is acceptable for dev environments but not recommended for production -- see [Production Checklist](./production-checklist.mdx#destroy-safety).
***
### Config Updates
To update configuration after initial deployment (new contracts, changed RPC endpoints, modified eRPC settings):
1. Edit your config files in the `config/` directory
2. Re-run the deployer:
```bash
terraform output -json workload_handoff | ./../../deployers/k3s/deploy.sh /dev/stdin --config-dir ./config
```
This runs `helm upgrade`, which is idempotent. Only changed values trigger pod restarts. Unchanged services remain running without interruption.
See [Updating Configuration After Deployment](./config-updates.mdx) for config update procedures across all compute engines.
***
### EKS External Mode
The same two-phase pattern applies to EKS with `workload_mode = "external"`. The difference is in how you authenticate:
```bash
# Phase 1: Terraform provisions EKS cluster
terraform apply -var-file=eks_external.tfvars
# Phase 2: Deploy using EKS deployer
terraform output -json workload_handoff | ./../../deployers/eks/deploy.sh /dev/stdin --config-dir ./config
```
EKS external mode uses the cluster endpoint and OIDC provider from the handoff instead of a kubeconfig file. The EKS deployer configures `kubectl` via `aws eks update-kubeconfig` rather than a base64-encoded kubeconfig.
***
### Related Pages
* **[Core Concepts -- Two-Phase Deployment](../concepts.mdx#two-phase-deployment)** -- Architectural overview
* **[External Deployers](./external-deployers.mdx)** -- All three deployer paths in detail
* **[Secrets Management](./secrets-management.mdx)** -- How credentials flow through phases
* **[Updating Configuration](./config-updates.mdx)** -- Post-deploy config changes
* **[Getting Started](../getting-started.mdx)** -- Simpler single-phase example (EC2)
## Bare Metal + ClickHouse (BYODB)
Deploy evm-cloud on any VPS or dedicated server. No AWS account required. Terraform connects via SSH and deploys Docker Compose services directly on your host. Works with Hetzner, OVH, DigitalOcean, Vultr, or any server with Docker installed.
### Architecture
```
┌────────────────── Your VPS ─────────────────┐
│ │
│ Docker Compose │
│ ├── eRPC proxy (port 4000) │
│ └── rindexer indexer │
│ └── writes to ────────────────────────────→ ClickHouse (external)
│ │
│ Config: /opt/evm-cloud/config/ │
│ Secrets: /opt/evm-cloud/.env │
└──────────────────────────────────────────────┘
Terraform connects via SSH (no AWS provider used)
```
### What Gets Deployed
* Docker Compose stack on your existing server (via SSH provisioner)
* eRPC container -- RPC proxy with failover and caching
* rindexer container -- EVM event indexer
* Config files deployed to `/opt/evm-cloud/config/`
* Environment file with ClickHouse credentials
**Not deployed:** No VPC, no IAM, no cloud-specific resources. The AWS Terraform provider is initialized but credentials are skipped.
### Prerequisites
* Terraform >= 1.5.0
* A VPS or server with Docker + Docker Compose installed
* SSH access to the server (key-based authentication)
* A ClickHouse instance ([ClickHouse Cloud](https://clickhouse.cloud/) or self-hosted)
### Quick Start
```bash
git clone https://github.com/ExoMonk/evm-cloud.git
cd evm-cloud/examples/bare_metal_byo_clickhouse
cp secrets.auto.tfvars.example secrets.auto.tfvars
# Edit secrets.auto.tfvars:
# bare_metal_host = "203.0.113.10"
# bare_metal_ssh_user = "root"
# bare_metal_ssh_private_key_path = "~/.ssh/id_ed25519"
# indexer_clickhouse_url = "https://your-instance.clickhouse.cloud:8443"
# indexer_clickhouse_password = "your-password"
terraform init
terraform apply
# Verify
ssh root@203.0.113.10 'docker compose -f /opt/evm-cloud/docker-compose.yml ps'
```
### Key Variables
| Variable | Type | Default | Description |
| --------------------------------- | ------ | ------- | ------------------------------------ |
| `bare_metal_host` | string | - | Server IP or hostname |
| `bare_metal_ssh_user` | string | - | SSH username |
| `bare_metal_ssh_private_key_path` | string | - | Path to SSH private key |
| `bare_metal_ssh_port` | number | `22` | SSH port |
| `bare_metal_rpc_proxy_mem_limit` | string | - | eRPC container memory limit |
| `bare_metal_indexer_mem_limit` | string | - | rindexer container memory limit |
| `indexer_clickhouse_url` | string | - | ClickHouse HTTP endpoint (sensitive) |
| `indexer_clickhouse_password` | string | - | ClickHouse password (sensitive) |
### When to Use This
**Choose this example when:**
* You have no AWS account or prefer not to use AWS
* You have an existing VPS (Hetzner, OVH, DigitalOcean, etc.)
* You want the absolute lowest cost (\~$5-20/mo for a small VPS)
* You want full control over the host operating system
**Consider alternatives when:**
* You want managed networking and IAM -- use any AWS example
* You want Kubernetes -- use [EKS](./eks-clickhouse.mdx) or [k3s](./k3s-clickhouse.mdx)
* You need managed PostgreSQL -- use [minimal\_aws\_rds](./ec2-docker-compose-rds.mdx)
See [examples/bare\_metal\_byo\_clickhouse](https://github.com/ExoMonk/evm-cloud/tree/main/examples/bare_metal_byo_clickhouse) for complete details.
## EC2 + Docker Compose + ClickHouse (BYODB)
Low-cost AWS deployment. A single EC2 instance running eRPC and rindexer with your own external ClickHouse database. Best for analytics workloads on a budget.
### Architecture
```
┌────────────────── AWS VPC ──────────────────┐
│ │
│ EC2 Instance (Docker Compose) │
│ ├── eRPC proxy (port 4000) │
│ └── rindexer indexer │
│ └── writes to ────────────────────────────→ ClickHouse (external)
│ │
│ Secrets Manager (CH creds, RPC URL) │
│ CloudWatch Logs (30-day retention) │
└──────────────────────────────────────────────┘
```
### What Gets Deployed
* VPC with public/private subnets, Internet Gateway, security groups
* EC2 instance with Docker + Compose pre-installed (cloud-init)
* eRPC container -- RPC proxy with failover and caching (256m memory)
* rindexer container -- EVM event indexer (512m memory)
* IAM role (CloudWatch Logs + Secrets Manager access)
* Secrets Manager secret for ClickHouse credentials
* CloudWatch Log Group with 30-day retention
**Not deployed:** No database -- you bring your own ClickHouse instance (ClickHouse Cloud free tier works).
### Prerequisites
* Terraform >= 1.5.0
* AWS CLI v2 with configured credentials (EC2, VPC, IAM, Secrets Manager)
* SSH key pair
* A ClickHouse instance ([ClickHouse Cloud](https://clickhouse.cloud/) free tier or self-hosted)
### Quick Start
```bash
git clone https://github.com/ExoMonk/evm-cloud.git
cd evm-cloud/examples/minimal_aws_byo_clickhouse
cp secrets.auto.tfvars.example secrets.auto.tfvars
# Edit secrets.auto.tfvars:
# ssh_public_key = "ssh-ed25519 AAAA..."
# indexer_clickhouse_url = "https://your-instance.clickhouse.cloud:8443"
# indexer_clickhouse_password = "your-password"
terraform init
terraform plan -var-file=minimal_clickhouse.tfvars
terraform apply -var-file=minimal_clickhouse.tfvars
# Verify
terraform output -json workload_handoff | jq -r '.runtime.ec2.public_ip'
ssh -i ~/.ssh/evm-cloud ubuntu@ 'sudo docker compose -f /opt/evm-cloud/docker-compose.yml ps'
```
### Key Variables
| Variable | Type | Default | Description |
| ----------------------------- | ------ | ---------- | ------------------------------------ |
| `ec2_instance_type` | string | `t3.micro` | EC2 instance size |
| `indexer_clickhouse_url` | string | - | ClickHouse HTTP endpoint (sensitive) |
| `indexer_clickhouse_user` | string | `default` | ClickHouse username |
| `indexer_clickhouse_password` | string | - | ClickHouse password (sensitive) |
| `indexer_clickhouse_db` | string | `default` | ClickHouse database name |
| `ec2_rpc_proxy_mem_limit` | string | `256m` | eRPC container memory limit |
| `ec2_indexer_mem_limit` | string | `512m` | rindexer container memory limit |
### When to Use This
**Choose this example when:**
* You want the cheapest AWS deployment (\~$15-35/mo for the EC2 instance alone)
* Your workload is analytics-heavy (ClickHouse excels at wide scans and aggregations)
* You already have a ClickHouse instance or want to use the free tier
**Consider alternatives when:**
* You want zero external dependencies -- use [minimal\_aws\_rds](./ec2-docker-compose-rds.mdx) (managed PostgreSQL)
* You need Kubernetes -- see [EKS](./eks-clickhouse.mdx) or [k3s](./k3s-clickhouse.mdx)
* You have no AWS account -- see [bare\_metal\_byo\_clickhouse](./bare-metal-clickhouse.mdx)
See [examples/minimal\_aws\_byo\_clickhouse](https://github.com/ExoMonk/evm-cloud/tree/main/examples/minimal_aws_byo_clickhouse) for complete details.
## EC2 + Docker Compose + Managed PostgreSQL
The simplest evm-cloud deployment. A single EC2 instance running eRPC and rindexer with an AWS-managed RDS PostgreSQL database. No external database required.
### Architecture
```
┌────────────────── AWS VPC ──────────────────┐
│ │
│ EC2 Instance (Docker Compose) │
│ ├── eRPC proxy (port 4000) │
│ └── rindexer indexer │
│ └── writes to ─────────────────┐ │
│ │ │
│ RDS PostgreSQL (managed) <─────────┘ │
│ │
│ Secrets Manager (DB creds, RPC URL) │
│ CloudWatch Logs (30-day retention) │
└──────────────────────────────────────────────┘
```
### What Gets Deployed
* VPC with public/private subnets, Internet Gateway, security groups
* EC2 instance with Docker + Compose pre-installed (cloud-init)
* eRPC container -- RPC proxy with failover and caching
* rindexer container -- EVM event indexer
* RDS PostgreSQL instance (single-AZ, configurable class)
* IAM role (CloudWatch Logs + Secrets Manager access)
* Secrets Manager secret for database credentials
* CloudWatch Log Group with 30-day retention
### Prerequisites
* Terraform >= 1.5.0
* AWS CLI v2 with configured credentials (EC2, VPC, IAM, RDS, Secrets Manager)
* SSH key pair
### Quick Start
```bash
git clone https://github.com/ExoMonk/evm-cloud.git
cd evm-cloud/examples/minimal_aws_rds
cp secrets.auto.tfvars.example secrets.auto.tfvars
# Edit secrets.auto.tfvars: set ssh_public_key
terraform init
terraform plan
terraform apply
# Verify
terraform output -json workload_handoff | jq -r '.runtime.ec2.public_ip'
ssh -i ~/.ssh/evm-cloud ubuntu@ 'sudo docker compose -f /opt/evm-cloud/docker-compose.yml ps'
```
### Key Variables
| Variable | Type | Default | Description |
| --------------------------- | ------ | ------------- | ---------------------------------- |
| `ec2_instance_type` | string | `t2.micro` | EC2 instance size |
| `postgres_instance_class` | string | `db.t3.micro` | RDS instance class |
| `postgres_engine_version` | string | - | PostgreSQL engine version |
| `postgres_db_name` | string | - | Database name |
| `postgres_backup_retention` | number | - | Backup retention in days |
| `ec2_indexer_mem_limit` | string | `512m` | rindexer container memory limit |
| `indexer_storage_backend` | string | `postgres` | Set to `postgres` for this example |
### When to Use This
**Choose this example when:**
* You want the simplest possible setup with zero external dependencies
* You prefer a managed database (AWS handles backups, patching, failover)
* Your workload fits PostgreSQL (transactional queries, moderate write volume)
**Consider alternatives when:**
* You need analytics-oriented queries (wide scans, aggregations) -- use [ClickHouse examples](./ec2-docker-compose-clickhouse.mdx) instead
* You want Kubernetes orchestration -- see [EKS](./eks-clickhouse.mdx) or [k3s](./k3s-clickhouse.mdx)
* You want to minimize AWS spend -- RDS adds \~$15-30/mo; the [BYODB ClickHouse example](./ec2-docker-compose-clickhouse.mdx) avoids this
See [examples/minimal\_aws\_rds](https://github.com/ExoMonk/evm-cloud/tree/main/examples/minimal_aws_rds) for complete details.
## EKS + ClickHouse (BYODB)
Production-grade Kubernetes deployment on AWS EKS. Runs eRPC and rindexer as Kubernetes pods with your own external ClickHouse database. Best for teams already using Kubernetes.
### Architecture
```
┌────────────────── AWS VPC ──────────────────┐
│ │
│ EKS Cluster (managed control plane) │
│ ├── eRPC pod (RPC proxy) │
│ └── rindexer pod (indexer) │
│ └── writes to ────────────────────────────→ ClickHouse (external)
│ │
│ K8s Secrets (CH creds, RPC URL) │
│ ECS cluster (service orchestration) │
└──────────────────────────────────────────────┘
```
### What Gets Deployed
* VPC with public/private subnets, NAT Gateway, security groups
* EKS cluster with managed control plane (\~$73/mo)
* ECS cluster for service orchestration
* eRPC deployment -- RPC proxy with failover and caching
* rindexer deployment -- EVM event indexer
* Kubernetes Secrets for ClickHouse credentials
* ConfigMap volume mounts for erpc.yaml, rindexer.yaml, and ABIs
* IAM roles for EKS service accounts
### Prerequisites
* Terraform >= 1.5.0
* AWS CLI v2 with configured credentials (EC2, VPC, IAM, EKS, Secrets Manager)
* `kubectl` configured for EKS access
* A ClickHouse instance ([ClickHouse Cloud](https://clickhouse.cloud/) or self-hosted)
### Quick Start
```bash
git clone https://github.com/ExoMonk/evm-cloud.git
cd evm-cloud/examples/aws_eks_BYO_clickhouse
cp secrets.auto.tfvars.example secrets.auto.tfvars
# Edit secrets.auto.tfvars:
# indexer_clickhouse_url = "https://your-instance.clickhouse.cloud:8443"
# indexer_clickhouse_password = "your-password"
terraform init
terraform plan
terraform apply
# Configure kubectl
aws eks update-kubeconfig --name --region us-east-1
kubectl get pods
```
### Key Variables
| Variable | Type | Default | Description |
| ----------------------------- | ------ | --------- | ------------------------------------ |
| `compute_engine` | string | `eks` | Must be `eks` for this example |
| `indexer_clickhouse_url` | string | - | ClickHouse HTTP endpoint (sensitive) |
| `indexer_clickhouse_user` | string | `default` | ClickHouse username |
| `indexer_clickhouse_password` | string | - | ClickHouse password (sensitive) |
| `indexer_clickhouse_db` | string | `default` | ClickHouse database name |
| `network_enable_nat_gateway` | bool | `true` | Required for private subnet egress |
### When to Use This
**Choose this example when:**
* You want production-grade Kubernetes with managed control plane
* Your team already operates EKS clusters
* You need pod autoscaling, rolling deployments, and health checks
* You are comfortable with the \~$73/mo EKS control plane cost
**Consider alternatives when:**
* You want Kubernetes without the EKS fee -- use [k3s](./k3s-clickhouse.mdx) (\~$35-50/mo)
* You want the simplest setup -- use [EC2 + Docker Compose](./ec2-docker-compose-clickhouse.mdx)
* You want to manage workload deployment separately -- see [external EKS](./external-eks.mdx)
See [examples/aws\_eks\_BYO\_clickhouse](https://github.com/ExoMonk/evm-cloud/tree/main/examples/aws_eks_BYO_clickhouse) for complete details.
## EC2 External Mode (Infrastructure Only)
Terraform provisions the EC2 instance and networking, but does not deploy workloads. You get a `workload_handoff` output containing SSH connection details and configuration, then deploy eRPC and rindexer using your own CI/CD pipeline, Ansible, or scripts.
### Architecture
```
┌── Terraform manages ────────────────────────┐
│ │
│ AWS VPC + subnets + security groups │
│ EC2 Instance (Docker installed, no apps) │
│ IAM role + Secrets Manager │
│ │
└──────────────────────────────────────────────┘
│
│ workload_handoff output (SSH IP, configs, secrets)
▼
┌── Your CI/CD deploys ───────────────────────┐
│ │
│ eRPC proxy container │
│ rindexer indexer container │
│ Config files + docker-compose.yml │
│ │
└──────────────────────────────────────────────┘
```
### What Gets Deployed (by Terraform)
* VPC with public/private subnets, Internet Gateway, security groups
* EC2 instance with Docker pre-installed (no application containers)
* IAM role (CloudWatch Logs + Secrets Manager access)
* Secrets Manager secret with ClickHouse credentials
* CloudWatch Log Group
**Not deployed by Terraform:** No Docker Compose services. The `workload_handoff` output provides everything your deploy pipeline needs.
### Prerequisites
* Terraform >= 1.5.0
* AWS CLI v2 with configured credentials (EC2, VPC, IAM, Secrets Manager)
* SSH key pair
* A deployment pipeline (GitHub Actions, GitLab CI, Ansible, or scripts)
* A ClickHouse instance
### Quick Start
```bash
git clone https://github.com/ExoMonk/evm-cloud.git
cd evm-cloud/examples/minimal_aws_external_ec2_byo
cp secrets.auto.tfvars.example secrets.auto.tfvars
# Edit secrets.auto.tfvars with your values
terraform init
terraform apply
# Get the handoff output for your deploy pipeline
terraform output -json workload_handoff
# Contains: SSH IP, instance ID, security group IDs, secret ARNs, config payloads
```
### Key Variables
| Variable | Type | Default | Description |
| ----------------------------- | ------ | ---------- | ------------------------------------ |
| `workload_mode` | string | `external` | Must be `external` for this example |
| `compute_engine` | string | `ec2` | EC2 compute backend |
| `ec2_instance_type` | string | `t2.micro` | EC2 instance size |
| `ssh_public_key` | string | - | SSH public key for EC2 key pair |
| `indexer_clickhouse_url` | string | - | ClickHouse HTTP endpoint (sensitive) |
| `indexer_clickhouse_password` | string | - | ClickHouse password (sensitive) |
### The workload\_handoff Output
When `workload_mode = "external"`, Terraform outputs a `workload_handoff` object containing everything your deploy pipeline needs:
* SSH connection details (IP, user, key name)
* Instance metadata (ID, security groups)
* Secret ARNs for credentials
* Config payloads (erpc.yaml, rindexer.yaml, ABIs) if provided at plan time
Your pipeline reads this output and handles container deployment independently.
### When to Use This
**Choose this example when:**
* You want Terraform to manage infrastructure but not application deployment
* Your team uses CI/CD pipelines (GitHub Actions, GitLab CI) for deployments
* You need separation of concerns between infra and app teams
* You want to test infrastructure changes without affecting running workloads
**Consider alternatives when:**
* You want Terraform to handle everything end-to-end -- use [EC2 + Docker Compose](./ec2-docker-compose-clickhouse.mdx)
* You want external mode with Kubernetes -- see [external EKS](./external-eks.mdx)
See [examples/minimal\_aws\_external\_ec2\_byo](https://github.com/ExoMonk/evm-cloud/tree/main/examples/minimal_aws_external_ec2_byo) for complete details.
## EKS External Mode (Infrastructure Only)
Terraform provisions an EKS cluster and networking, but does not deploy workloads. Designed for GitOps workflows where ArgoCD, Flux, or a CI/CD pipeline manages Kubernetes deployments separately from infrastructure.
### Architecture
```
┌── Terraform manages ────────────────────────┐
│ │
│ AWS VPC + subnets + NAT Gateway │
│ EKS Cluster (managed control plane) │
│ ECS Cluster (service orchestration) │
│ IAM roles for service accounts │
│ │
└──────────────────────────────────────────────┘
│
│ workload_handoff output (kubeconfig, secret ARNs, configs)
▼
┌── Your GitOps / CI/CD deploys ──────────────┐
│ │
│ ArgoCD / Flux / Helm / kubectl │
│ ├── eRPC deployment │
│ └── rindexer deployment │
│ └── writes to ClickHouse (external) │
│ │
└──────────────────────────────────────────────┘
```
### What Gets Deployed (by Terraform)
* VPC with public/private subnets, NAT Gateway, security groups
* EKS cluster with managed control plane
* ECS cluster for service orchestration
* IAM roles and policies for EKS service accounts
* Networking configuration for pod-to-internet egress
**Not deployed by Terraform:** No Kubernetes Deployments, Services, ConfigMaps, or Secrets. The `workload_handoff` output provides the cluster endpoint and configuration for your GitOps tool.
### Prerequisites
* Terraform >= 1.5.0
* AWS CLI v2 with configured credentials (EC2, VPC, IAM, EKS)
* `kubectl` and a GitOps tool (ArgoCD, Flux) or CI/CD pipeline
* A ClickHouse instance
### Quick Start
```bash
git clone https://github.com/ExoMonk/evm-cloud.git
cd evm-cloud/examples/minimal_aws_external_eks_byo
cp secrets.auto.tfvars.example secrets.auto.tfvars
# Edit secrets.auto.tfvars with your values
terraform init
terraform apply
# Get the handoff output for your GitOps pipeline
terraform output -json workload_handoff
# Configure kubectl
aws eks update-kubeconfig --name --region us-east-1
# Deploy workloads via your preferred method (ArgoCD, Helm, kubectl apply)
```
### Key Variables
| Variable | Type | Default | Description |
| ----------------------------- | ------ | ---------- | ------------------------------------ |
| `workload_mode` | string | `external` | Must be `external` for this example |
| `compute_engine` | string | `eks` | EKS compute backend |
| `indexer_clickhouse_url` | string | - | ClickHouse HTTP endpoint (sensitive) |
| `indexer_clickhouse_password` | string | - | ClickHouse password (sensitive) |
| `network_enable_nat_gateway` | bool | `true` | Required for private subnet egress |
### The workload\_handoff Output
When `workload_mode = "external"` with EKS, the `workload_handoff` output contains:
* EKS cluster endpoint and certificate authority
* Cluster name for `aws eks update-kubeconfig`
* IAM role ARNs for service accounts
* Config payloads (erpc.yaml, rindexer.yaml, ABIs) if provided at plan time
Feed this into ArgoCD Application manifests, Flux HelmReleases, or your CI/CD pipeline.
### When to Use This
**Choose this example when:**
* Your organization uses GitOps (ArgoCD, Flux) for Kubernetes deployments
* You want strict separation between infrastructure provisioning and application delivery
* Multiple teams share the EKS cluster and manage their own workloads
* You need audit trails for who changed what (infra vs. app changes in separate repos)
**Consider alternatives when:**
* You want Terraform to deploy workloads too -- use [EKS managed](./eks-clickhouse.mdx)
* You want external mode without EKS cost -- see [external EC2](./external-ec2.mdx)
* You do not need Kubernetes -- Docker Compose examples are simpler
See [examples/minimal\_aws\_external\_eks\_byo](https://github.com/ExoMonk/evm-cloud/tree/main/examples/minimal_aws_external_eks_byo) for complete details.
## Examples
7 ready-to-use deployment examples. Each is a complete Terraform configuration you can `terraform apply`.
### Which Example Should I Use?
| Example | Provider | Compute | Database | Workload Mode | Monthly Cost | Best For |
| -------------------------------------------------------------------- | ---------- | -------------- | ------------------ | ------------- | ------------ | -------------------------- |
| [minimal\_aws\_rds](./ec2-docker-compose-rds.mdx) | AWS | EC2 + Docker | RDS PostgreSQL | terraform | \~$45 | Simplest start, managed DB |
| [minimal\_aws\_byo\_clickhouse](./ec2-docker-compose-clickhouse.mdx) | AWS | EC2 + Docker | ClickHouse (BYODB) | terraform | \~$15-35 | Analytics, low cost |
| [aws\_eks\_BYO\_clickhouse](./eks-clickhouse.mdx) | AWS | EKS | ClickHouse (BYODB) | terraform | \~$110+ | Production K8s |
| [minimal\_aws\_k3s\_byo\_clickhouse](./k3s-clickhouse.mdx) | AWS | k3s on EC2 | ClickHouse (BYODB) | external | \~$35-50 | K8s without EKS cost |
| [bare\_metal\_byo\_clickhouse](./bare-metal-clickhouse.mdx) | Bare Metal | Docker Compose | ClickHouse (BYODB) | terraform | \~$5-20 | Any VPS, no AWS |
| [minimal\_aws\_external\_ec2\_byo](./external-ec2.mdx) | AWS | EC2 | ClickHouse (BYODB) | external | \~$15-35 | CI/CD owns workloads |
| [minimal\_aws\_external\_eks\_byo](./external-eks.mdx) | AWS | EKS | ClickHouse (BYODB) | external | \~$110+ | GitOps / ArgoCD |
### How to Choose
* **Need managed database?** Use [minimal\_aws\_rds](./ec2-docker-compose-rds.mdx) -- RDS PostgreSQL with zero external dependencies.
* **Want cheapest AWS?** Use [minimal\_aws\_byo\_clickhouse](./ec2-docker-compose-clickhouse.mdx) -- single EC2 instance with your own ClickHouse.
* **Want Kubernetes?** Use [EKS](./eks-clickhouse.mdx) for production or [k3s](./k3s-clickhouse.mdx) for budget K8s without the \~$73/mo control plane fee.
* **No AWS account?** Use [bare\_metal\_byo\_clickhouse](./bare-metal-clickhouse.mdx) -- works on any VPS (Hetzner, OVH, DigitalOcean).
* **Separate infra from app deploy?** Use [external\_ec2](./external-ec2.mdx) or [external\_eks](./external-eks.mdx) -- Terraform provisions infrastructure, your CI/CD deploys workloads.
### Common Prerequisites
All examples require:
* Terraform >= 1.5.0
* An SSH key pair (`ssh-keygen -t ed25519`)
* A ClickHouse instance (except `minimal_aws_rds` which uses managed PostgreSQL)
AWS examples additionally require:
* AWS CLI v2 with configured credentials
* Permissions for EC2, VPC, IAM, Secrets Manager (and EKS for K8s examples)
See [Getting Started](../getting-started.mdx) for full prerequisite setup.
## k3s + ClickHouse (BYODB)
Lightweight Kubernetes on a single EC2 instance using k3s. Gets you Kubernetes semantics (Helm, kubectl, pods) without the \~$73/mo EKS control plane fee. Uses a two-phase deployment workflow.
### Architecture
```
┌────────────────── AWS VPC ──────────────────┐
│ │
│ EC2 Instance (k3s single-node cluster) │
│ ├── eRPC pod (RPC proxy) │
│ └── rindexer pod (indexer) │
│ └── writes to ────────────────────────────→ ClickHouse (external)
│ │
│ k3s API server (port 6443) │
│ SSH access for provisioning │
└──────────────────────────────────────────────┘
```
### What Gets Deployed
**Phase 1 (Terraform):**
* VPC with public subnets, Internet Gateway, security groups
* EC2 instance with k3s installed via SSH provisioner
* SSH key pair for access
* Security group allowing SSH + k3s API (port 6443) from allowed CIDRs
**Phase 2 (Helm/kubectl -- you run this):**
* eRPC and rindexer deployed as Kubernetes pods via Helm charts or manifests
* Config and secrets injected via the `workload_handoff` output
### Two-Phase Workflow
This example uses `workload_mode = "external"`. Terraform provisions the infrastructure (Phase 1), then you deploy workloads yourself (Phase 2) using the `workload_handoff` output.
See the [two-phase deployment guide](../guides/two-phase-workflow) for the full workflow.
### Prerequisites
* Terraform >= 1.5.0
* AWS CLI v2 with configured credentials (EC2, VPC, IAM)
* SSH key pair (both public and private key paths)
* `kubectl` and `helm` (for Phase 2)
* A ClickHouse instance ([ClickHouse Cloud](https://clickhouse.cloud/) or self-hosted)
### Quick Start
```bash
git clone https://github.com/ExoMonk/evm-cloud.git
cd evm-cloud/examples/minimal_aws_k3s_byo_clickhouse
# Phase 1: Provision infrastructure
cp secrets.auto.tfvars.example secrets.auto.tfvars
# Edit secrets.auto.tfvars:
# ssh_public_key = "ssh-ed25519 AAAA..."
# k3s_ssh_private_key_path = "~/.ssh/evm-cloud"
# indexer_clickhouse_url = "https://your-instance.clickhouse.cloud:8443"
# indexer_clickhouse_password = "your-password"
terraform init
terraform apply
# Phase 2: Deploy workloads
terraform output -json workload_handoff # contains kubeconfig + connection details
# Use the output to configure kubectl and deploy via Helm
```
### Key Variables
| Variable | Type | Default | Description |
| ----------------------------- | ------------ | ------- | ------------------------------------------ |
| `k3s_instance_type` | string | - | EC2 instance size for k3s node |
| `k3s_ssh_private_key_path` | string | - | Path to SSH private key for provisioner |
| `k3s_version` | string | - | k3s release version |
| `k3s_api_allowed_cidrs` | list(string) | - | CIDRs allowed to reach k3s API (port 6443) |
| `ssh_public_key` | string | - | SSH public key content for EC2 key pair |
| `indexer_clickhouse_url` | string | - | ClickHouse HTTP endpoint (sensitive) |
| `indexer_clickhouse_password` | string | - | ClickHouse password (sensitive) |
### When to Use This
**Choose this example when:**
* You want Kubernetes features (Helm, rolling updates, health checks) without EKS cost
* You are comfortable managing a single-node k3s cluster
* Your workload fits on one instance (dev/staging or small-scale production)
**Consider alternatives when:**
* You want a managed Kubernetes control plane -- use [EKS](./eks-clickhouse.mdx)
* You want the simplest possible setup -- use [EC2 + Docker Compose](./ec2-docker-compose-clickhouse.mdx)
* You do not need Kubernetes at all -- Docker Compose examples are simpler to operate
See [examples/minimal\_aws\_k3s\_byo\_clickhouse](https://github.com/ExoMonk/evm-cloud/tree/main/examples/minimal_aws_k3s_byo_clickhouse) for complete details.