# Deploying to Amazon EKS Complete guide to deploying self-hosted Spacelift Flows on Amazon EKS using OpenTofu and Helm. This guide provides a way to quickly get Spacelift Flows up and running on an Elastic Kubernetes Service (EKS) cluster. The infrastructure is deployed using OpenTofu, and the application services are deployed using Helm charts. Currently, agents have to run outside the Kubernetes cluster, e.g., on an EC2 Auto Scaling Group. We will soon provide a fully Kubernetes-native solution. ## Overview [Section titled “Overview”](#overview) This deployment creates a complete Spacelift Flows instance with the following components: - **EKS Auto Mode cluster** for container orchestration - **RDS Aurora PostgreSQL** for the database - **S3 bucket** for object storage - **KMS encryption** for data at rest - **ACM certificates** for SSL/TLS - **Agent pool** deployed via Terraform module The following services will be deployed as Kubernetes pods using the Helm charts: - The server. - The worker. - The gateway. The server hosts the Spacelift Flows HTTP API and serves the embedded frontend assets. The server is exposed to the outside world through an Application Load Balancer for HTTP traffic, including the OAuth and MCP endpoints required for external integrations. The worker is the component that handles recurring tasks and asynchronous jobs. The gateway is a service that hosts the WebSocket server for agents and routes JavaScript evaluations to the right agents/runtimes. The agent pool is deployed as an EC2 ASG and consists of an agent service. Agent services handle requests from the gateway and distribute execution commands. ## Requirements [Section titled “Requirements”](#requirements) Before starting, ensure you have: ### AWS Prerequisites [Section titled “AWS Prerequisites”](#aws-prerequisites) - AWS CLI configured with appropriate permissions - Access to an AWS account with the following service limits: - EKS clusters: At least 1 available - RDS Aurora clusters: At least 1 available - VPC: At least 1 available (or use existing) - NAT Gateways: At least 1 available per AZ - Elastic IPs: At least 1 available per NAT Gateway ### Tools Required [Section titled “Tools Required”](#tools-required) - [OpenTofu](https://opentofu.org/docs/intro/install/) >= 1.6.0 (or [Terraform](https://www.terraform.io/downloads) >= 1.5.0) - [kubectl](https://kubernetes.io/docs/tasks/tools/) for Kubernetes management - [Helm](https://helm.sh/docs/intro/install/) >= 3.0 for application deployment - [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) >= 2.0 ### Domain Requirements [Section titled “Domain Requirements”](#domain-requirements) - A registered domain name with DNS management access - Ability to create DNS records for certificate validation ### Optional Requirements [Section titled “Optional Requirements”](#optional-requirements) - SMTP server for email notifications (recommended). You can enable Amazon SES by setting the Terraform variable `enable_ses = true`. - [Anthropic API key](https://console.anthropic.com/) for AI features (recommended) - OpenTelemetry collector endpoint for observability ## Deploy Infrastructure [Section titled “Deploy Infrastructure”](#deploy-infrastructure) The infrastructure deployment uses a modular approach with OpenTofu to provision all necessary AWS resources. ### 1. Prepare the Environment [Section titled “1. Prepare the Environment”](#1-prepare-the-environment) First, ensure your AWS CLI is configured with the appropriate credentials and region: *Terminal window* ``` # Verify AWS CLI configurationaws sts get-caller-identity # Set your preferred region and export as environment variableexport TF_VAR_aws_region=us-west-2 ``` ### 2. Create Working Directory [Section titled “2. Create Working Directory”](#2-create-working-directory) Create a new directory for your infrastructure deployment: *Terminal window* ``` mkdir spacelift-flows-infracd spacelift-flows-infra ``` ### 3. Create Infrastructure Configuration [Section titled “3. Create Infrastructure Configuration”](#3-create-infrastructure-configuration) Create a `main.tf` file that references the Spacelift Flows infrastructure module: ** ``` terraform { required_version = ">= 1.5.0" required_providers { aws = { source = "hashicorp/aws" version = "~> 6.0" } random = { source = "hashicorp/random" version = "~> 3.4" } }} provider "aws" { region = var.aws_region} module "spacelift_flows" { source = "github.com/spacelift-io/terraform-aws-eks-spacelift-flows-selfhosted?ref=v0.2.0" # Required variables app_domain = var.app_domain organization_name = var.organization_name admin_email = var.admin_email aws_region = var.aws_region license_token = var.license_token anthropic_api_key = var.anthropic_api_key # optional expose_gateway = true # Email configuration (choose one of the following options): # Option 1: Dev mode - emails logged to server logs (testing only) # Useful for initial setup, but not suitable for production # email_dev_enabled = true # Option 2: Custom SMTP server # smtp_host = "smtp.example.com" # smtp_port = 587 # smtp_username = "" # smtp_password = "" # smtp_from_address = "noreply@yourcompany.com" # Option 3: Amazon SES (recommended for AWS deployments) # Requires domain verification and production access request (see Verify Deployment section) # enable_ses = true # Optional variables k8s_namespace = var.k8s_namespace admin_password = var.admin_password} # Uncomment to deploy the agent pool when the spacelift flows backend services are deployed.# module "spacelift_flows_agent_pool" {# source = "github.com/spacelift-io/terraform-aws-spacelift-flows-agentpool-ec2?ref=v0.4.0"# agent_pool_id = module.spacelift_flows.agent_pool_id# agent_pool_token = module.spacelift_flows.agent_pool_token# backend_endpoint = "https://${var.app_domain}"# gateway_endpoint = "https://gateway.${var.app_domain}"# agent_image_tag = var.spacelift_flows_image_tag## reuse_vpc_id = module.spacelift_flows.vpc_id# reuse_vpc_subnet_ids = module.spacelift_flows.vpc_private_subnet_ids# aws_region = var.aws_region# min_size = 1# desired_capacity = 5# max_size = 10# } ``` See more examples with different configurations in [the GitHub repository](https://github.com/spacelift-io/terraform-aws-eks-spacelift-flows-selfhosted). If you want to reuse Spacelift Self-Hosted RDS, create a Postgres database: ** ``` CREATE DATABASE flows; ``` We also suggest creating a separate user. And then you should build a connection URL like this: ** ``` database_connection_url = format("postgres://%s:%s@%s:5432/flows", module.spacelift.rds_username, urlencode(module.spacelift.rds_password), module.spacelift.rds_cluster_endpoint) ``` ### 4. Create Variables File [Section titled “4. Create Variables File”](#4-create-variables-file) Create a `variables.tf` file with variable definitions: ** ``` # Required variablesvariable "app_domain" { description = "The domain name for the Spacelift Flows instance" type = string} variable "organization_name" { description = "Name of the organization" type = string} variable "admin_email" { description = "Email address for the admin user" type = string} variable "aws_region" { description = "AWS region for deployment" type = string} variable "license_token" { description = "The JWT token for using Spacelift flows. Only required for generating the kubernetes_secrets output." type = string sensitive = true} variable "k8s_namespace" { type = string} variable "spacelift_flows_image_tag" { type = string} variable "anthropic_api_key" { description = "Anthropic API key for AI features" type = string default = ""} variable "admin_password" { description = "Admin password for initial login without SMTP (min 32 characters)" type = string sensitive = true default = ""} ``` ### 5. Create Outputs File [Section titled “5. Create Outputs File”](#5-create-outputs-file) Create an `outputs.tf` file to expose important values: ** ``` output "config_secret_manifest" { description = "Outputs manifests that are needed to configure a secret for the Flows app." value = module.spacelift_flows.config_secret_manifest sensitive = true} output "ingress_manifest" { description = "Outputs manifests that are needed to configure AWS ingress." value = module.spacelift_flows.ingress_manifest} # Uncomment if deploying in a private VPC# output "internal_ingress_manifest" {# description = "Outputs internal manifests that are needed to configure internal AWS ingress."# value = module.spacelift_flows.internal_ingress_manifest# } output "shell" { value = module.spacelift_flows.shell} ``` ### 6. Set Environment Variables [Section titled “6. Set Environment Variables”](#6-set-environment-variables) Configure your deployment using environment variables. Start with the minimum required variables: Choose a domain you control and can create DNS records for. Subdomains work well (e.g., `flows.yourcompany.com`). *Terminal window* ``` # Required configurationexport TF_VAR_app_domain="flows.yourcompany.com"export TF_VAR_organization_name="Your Organization"export TF_VAR_admin_email="admin@yourcompany.com"export TF_VAR_license_token=""export TF_VAR_k8s_namespace="spacelift-flows"export TF_VAR_spacelift_flows_image_tag="0.6.0"# Optional configurationexport TF_VAR_anthropic_api_key="" # optionalexport TF_VAR_admin_password="" # optional, at least 32 chars, enables login without SMTP ``` Flows has an incredibly powerful and helpful AI assistant, which can help you quickly build and debug Flows. The AI assistant requires an API key from Anthropic to work, and you can get it by signing up at [https://console.anthropic.com/](https://console.anthropic.com/) . While Flows will work without it, we strongly advise not skipping this. ### 7. Initialize OpenTofu [Section titled “7. Initialize OpenTofu”](#7-initialize-opentofu) Initialize the working directory and download required providers: *Terminal window* ``` tofu init ``` ### 8. Review the Deployment Plan [Section titled “8. Review the Deployment Plan”](#8-review-the-deployment-plan) Generate and review the execution plan: *Terminal window* ``` tofu plan ``` ### 9. Deploy the Infrastructure [Section titled “9. Deploy the Infrastructure”](#9-deploy-the-infrastructure) Apply the configuration to create the infrastructure: *Terminal window* ``` tofu apply ``` When prompted, type `yes` to confirm the deployment. ### 10. Verify Infrastructure Deployment [Section titled “10. Verify Infrastructure Deployment”](#10-verify-infrastructure-deployment) Once applied, you should grab all variables that need to be exported in the shell that will be used in next steps. We expose a shell output in tofu that you can source directly for convenience. *Terminal window* ``` # Source in your shell all the required env vars to continue the installation process$(tofu output -raw shell) ``` ### 11. Configure kubectl Access [Section titled “11. Configure kubectl Access”](#11-configure-kubectl-access) Configure kubectl to access your new EKS cluster: *Terminal window* ``` # Update kubeconfigaws eks update-kubeconfig --region $TF_VAR_aws_region --name $EKS_CLUSTER_NAME # Create a namespacekubectl create namespace $TF_VAR_k8s_namespace ``` ### 12. Validate the certificates [Section titled “12. Validate the certificates”](#12-validate-the-certificates) You can skip this step if you already provided an issued certificate as a `cert_arn` variable. After the infrastructure is deployed, you should validate that the ACM certificates are properly issued and configured: *Terminal window* ``` # Get the certificate ARN from OpenTofu outputsCERT_ARN=$(aws acm list-certificates --region $TF_VAR_aws_region --query "CertificateSummaryList[?DomainName=='$TF_VAR_app_domain'].CertificateArn" --output text) # Check certificate statusaws acm describe-certificate --certificate-arn $CERT_ARN --region $TF_VAR_aws_region --query "Certificate.Status" --output text ``` The certificate status should show `ISSUED`. If it shows `PENDING_VALIDATION`, you need to validate the certificate by creating the required DNS records: *Terminal window* ``` # Get DNS validation recordsaws acm describe-certificate --certificate-arn $CERT_ARN --region $TF_VAR_aws_region --query "Certificate.DomainValidationOptions[*].ResourceRecord" --output table ``` Create the CNAME records shown in the output in your DNS provider. The certificate will automatically be issued once DNS validation is complete (usually within a few minutes). You can monitor the validation status: *Terminal window* ``` # Check validation statusaws acm describe-certificate --certificate-arn $CERT_ARN --region $TF_VAR_aws_region --query "Certificate.DomainValidationOptions[*].ValidationStatus" --output text ``` Once all domains show `SUCCESS`, the certificate is ready to use. ### Infrastructure Deployment Complete [Section titled “Infrastructure Deployment Complete”](#infrastructure-deployment-complete) At this point, you have successfully deployed: - ✅ **VPC** with public and private subnets - ✅ **EKS Auto Mode cluster** ready for workloads - ✅ **RDS Aurora PostgreSQL** database cluster - ✅ **S3 bucket** for object storage with encryption - ✅ **KMS key** for encryption at rest - ✅ **IAM roles and policies** for service access The next step is to deploy the application services using the generated Kubernetes manifests. ## Deploy Application Services [Section titled “Deploy Application Services”](#deploy-application-services) ### 1. Apply Configuration Secret [Section titled “1. Apply Configuration Secret”](#1-apply-configuration-secret) Create the main configuration secret: *Terminal window* ``` tofu output -raw config_secret_manifest | kubectl apply -f - ``` ### 2. Apply Ingress Configuration [Section titled “2. Apply Ingress Configuration”](#2-apply-ingress-configuration) Create an AWS ingress class: *Terminal window* ``` tofu output -raw ingress_manifest | kubectl apply -f - ``` **For private VPC deployments**: Use the internal ingress class instead: *Terminal window* ``` tofu output -raw internal_ingress_manifest | kubectl apply -f - ``` ### 3. Deploy Core Services [Section titled “3. Deploy Core Services”](#3-deploy-core-services) Create helm values file for the spacelift flows installation: *Terminal window* ``` cat > flows-values.yaml <