Secrets Management

How evm-cloud handles credentials across compute engines, and how to manage them securely in development and production.

The secrets.auto.tfvars Pattern

Every example includes a secrets.auto.tfvars.example template. This is the primary way to provide sensitive values during local development.

cd examples/minimal_aws_byo_clickhouse
cp secrets.auto.tfvars.example secrets.auto.tfvars

Edit secrets.auto.tfvars with your values:

# SSH key (public key content, not the file path)
ssh_public_key = "ssh-ed25519 AAAA... your-email"
 
# ClickHouse connection
indexer_clickhouse_url      = "https://your-instance.clickhouse.cloud:8443"
indexer_clickhouse_password = "your-password"

1. The file is named *.auto.tfvars, so Terraform loads it automatically -- no -var-file flag needed
2. The file is listed in .gitignore -- it never enters version control
3. The .example template shows which variables are required without exposing real values

What Goes in Secrets

Secrets Delivery by Compute Engine

Each compute engine has a different mechanism for getting credentials from Terraform into running containers.

EC2 (Docker Compose)

terraform apply
  → Stores credentials in AWS Secrets Manager
  → cloud-init installs pull-secrets.sh on the instance
 
Instance boot:
  pull-secrets.sh
  → Fetches secrets from Secrets Manager via AWS CLI
  → Writes /opt/evm-cloud/.env (chmod 0600)
  → Docker Compose reads .env file

The pull-secrets.sh script runs at instance startup and whenever you need to refresh credentials. Secrets Manager ARNs are passed to the instance via cloud-init user data.

EKS (terraform mode)

Terraform creates Kubernetes Secrets directly from variable values. Pod specs reference these secrets as environment variables or mounted files.

terraform apply
  → Creates K8s Secret resource (via kubernetes provider)
  → Pod spec references secret by name
  → kubelet mounts secret into container

No intermediate storage or scripts. Terraform manages the full lifecycle.

k3s (external mode)

The database password flows through the workload handoff into Helm values, which create a Kubernetes Secret.

terraform apply
  → Outputs workload_handoff JSON (contains password)
 
deploy.sh
  → Reads handoff JSON
  → Renders Helm values (password → values.yaml)
  → helm upgrade --install
  → Chart creates K8s Secret from values
  → Pod mounts secret

Note: This means the password exists in the Terraform state and in the handoff JSON. See Production Recommendations for how to secure both. A future improvement will add External Secrets Operator (ESO) support to sync secrets directly from AWS Secrets Manager into the cluster, removing the password from the handoff entirely.

Bare Metal (Docker Compose)

Terraform's SSH file provisioner delivers a .env file directly to the remote host.

terraform apply
  → Connects via SSH
  → Writes /opt/evm-cloud/.env (chmod 0600)
  → Docker Compose reads .env file

No cloud secrets service. The .env file on disk is the only storage location.

CI/CD Integration

In CI/CD pipelines, use TF_VAR_* environment variables instead of secrets.auto.tfvars files. Terraform automatically maps environment variables prefixed with TF_VAR_ to input variables.

# GitHub Actions example
export TF_VAR_ssh_public_key="$SSH_PUBLIC_KEY"
export TF_VAR_indexer_clickhouse_password="$CH_PASSWORD"
export TF_VAR_indexer_clickhouse_url="$CH_URL"
export TF_VAR_k3s_ssh_private_key_path="$K3S_KEY_PATH"
 
terraform apply -var-file=minimal_clickhouse.tfvars

The non-sensitive .tfvars file (instance types, region, project name) can be committed to the repository. Secrets come from the environment.

# .github/workflows/deploy.yml (snippet)
jobs:
  deploy:
    steps:
      - name: Terraform Apply
        env:
          TF_VAR_ssh_public_key: ${{ secrets.SSH_PUBLIC_KEY }}
          TF_VAR_indexer_clickhouse_password: ${{ secrets.CH_PASSWORD }}
          TF_VAR_indexer_clickhouse_url: ${{ secrets.CH_URL }}
        run: |
          terraform init
          terraform apply -auto-approve -var-file=minimal_clickhouse.tfvars

Sensitive Outputs

The workload_handoff output is marked sensitive = true because it may contain:

• kubeconfig data (k3s) -- grants full cluster access
• Database passwords
• Internal service endpoints

Access it explicitly:

# View the full handoff
terraform output -json workload_handoff
 
# Extract specific fields
terraform output -json workload_handoff | jq -r '.runtime.k3s.kubeconfig_base64'
terraform output -json workload_handoff | jq -r '.data.password'

Terraform will not display sensitive outputs in plan/apply logs. You must use terraform output -json to retrieve them.

Warning: If you write the handoff to a file (> handoff.json), treat that file as a credential. Set chmod 0600 and do not commit it. Prefer piping directly to the deployer to avoid writing credentials to disk -- see External Deployers.

Production Recommendations

Encrypted Terraform State

Terraform state contains all secret values in plaintext. Use an encrypted remote backend:

# backend.tf
terraform {
  backend "s3" {
    bucket         = "my-terraform-state"
    key            = "evm-cloud/production/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    kms_key_id     = "alias/terraform-state"
    dynamodb_table = "terraform-locks"
  }
}

See the Production Checklist for the full backend setup.

Secrets Manager Recovery Window

The ec2_secret_recovery_window_in_days variable controls how long AWS Secrets Manager retains a deleted secret before permanent removal.

# production.tfvars
ec2_secret_recovery_window_in_days = 30

Credential Rotation

To rotate database credentials:

1. Update the password in secrets.auto.tfvars (or TF_VAR_* environment variable)
2. Run terraform apply
3. EC2: Terraform updates the Secrets Manager entry. SSH into the instance and run pull-secrets.sh, then docker compose restart
4. k3s/EKS external: Re-run the deployer script -- Helm upgrade will update the Kubernetes Secret and trigger a pod rollout
5. EKS terraform: Terraform updates the K8s Secret directly and triggers a rollout
6. Bare metal: Terraform re-provisions the .env file via SSH and restarts Docker Compose

General Best Practices

• Never commit secrets.auto.tfvars or handoff.json files
• Use separate AWS accounts (or at minimum, separate state files) for dev/staging/production
• Restrict Secrets Manager IAM policies to the specific secrets your deployment needs
• Audit Secrets Manager access via CloudTrail
• For k3s deployments, delete handoff.json after deploying workloads, or pipe directly to avoid writing to disk

Related Pages

• Core Concepts -- Secrets Delivery -- How secrets reach containers by engine
• Variable Reference -- All secret-related variables with types and defaults
• External Deployers -- Consuming the workload handoff securely
• Production Checklist -- Full production hardening guide
• Two-Phase Deployment -- How k3s handles secrets across phases