Installing Kestra: Your First Steps to Powerful Orchestration.

Complete Setup Guide for Local, Docker, and Cloud Deployments.

Introduction: Why Installation Matters

Think of Kestra as the conductor of your data orchestra. Just as a great conductor needs the right stage setup, Kestra needs proper installation to perform at its best. In this guide, we'll walk through every installation option—from a quick local setup to enterprise-grade deployments.

Quick Decision Guide:

Just exploring? → Use Docker standalone (5 minutes)
Developing workflows? → Use Docker Compose (10 minutes)
Production deployment? → Use Kubernetes/Helm (30 minutes)
Enterprise scale? → Use Terraform on cloud (60 minutes)

Option 1: Local Development (Fastest Path)

Prerequisites Checklist

Before we begin, ensure you have:

✅ Docker installed (Get Docker)
✅ 2GB free RAM
✅ 10GB free disk space
✅ Internet connection

Method A: Quick Start with Docker (5 Minutes)

# Run this single command
docker run --rm \
  -p 8080:8080 \
  -v $(pwd)/flows:/app/flows \
  -v $(pwd)/storage:/app/storage \
  kestra/kestra:latest \
  standalone

# Or for Windows PowerShell
docker run --rm `
  -p 8080:8080 `
  -v ${PWD}/flows:/app/flows `
  -v ${PWD}/storage:/app/storage `
  kestra/kestra:latest `
  standalone

What this does:

Starts Kestra with embedded databases
Maps local directories for flows and storage
Opens port 8080 for the web UI
Uses temporary containers (removed on stop)

Access the UI:

Open http://localhost:8080
Default credentials: admin / password (change immediately!)

Method B: Docker Compose for Development (Recommended)

Create a docker-compose.yml file:

version: '3.8'

services:
  kestra:
    image: kestra/kestra:latest
    container_name: kestra-server
    ports:
      - "8080:8080"
      - "8081:8081"  # For metrics if needed
    environment:
      - KESTRA_CONFIGURATION= |
          kestra:
            server:
              basic-auth:
                enabled: true
                username: admin
                password: ${KESTRA_PASSWORD:-changeme}
            repository:
              type: postgres
            storage:
              type: local
            queue:
              type: postgres
    volumes:
      - ./flows:/app/flows      # Your workflow definitions
      - ./plugins:/app/plugins  # Custom plugins
      - ./logs:/app/logs        # Execution logs
      - ./storage:/app/storage  # File storage
    depends_on:
      postgres:
        condition: service_healthy
    restart: unless-stopped

  postgres:
    image: postgres:15-alpine
    container_name: kestra-postgres
    environment:
      POSTGRES_DB: kestra
      POSTGRES_USER: kestra
      POSTGRES_PASSWORD: kestra
    volumes:
      - postgres_data:/var/lib/postgresql/data
      - ./init-db:/docker-entrypoint-initdb.d  # Optional: initial scripts
    ports:
      - "5432:5432"  # Optional: expose for external tools
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U kestra"]
      interval: 10s
      timeout: 5s
      retries: 5
    restart: unless-stopped

volumes:
  postgres_data:

Start everything with one command:

# Create necessary directories
mkdir -p flows plugins logs storage init-db

# Start all services
docker-compose up -d

# Check status
docker-compose ps

# View logs
docker-compose logs -f kestra

What you get:

✅ Persistent database (PostgreSQL)
✅ File storage that survives restarts
✅ Proper service health checks
✅ Easy log viewing
✅ Production-like setup locally

Option 2: Production-Ready Kubernetes Setup

Prerequisites:

Kubernetes cluster (Minikube, Docker Desktop, EKS, AKS, GKE)
Helm 3+
kubectl configured

Step 1: Create Namespace and Secrets

# Create namespace
kubectl create namespace kestra

# Create a secret for credentials
kubectl create secret generic kestra-secrets \
  --namespace kestra \
  --from-literal=postgres-password='YourSecurePassword123!' \
  --from-literal=admin-password='AnotherSecurePassword456!'

# Create storage class (if needed)
cat <<EOF | kubectl apply -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: kestra-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
EOF

Step 2: Deploy with Helm

# Add Kestra Helm repository
helm repo add kestra https://kestra.io/helm
helm repo update

# Install Kestra with custom values
cat > kestra-values.yaml << EOF
# kestra-values.yaml
global:
  storageClass: kestra-storage

server:
  replicaCount: 2
  autoscaling:
    enabled: true
    minReplicas: 2
    maxReplicas: 10
  resources:
    requests:
      memory: "1Gi"
      cpu: "500m"
    limits:
      memory: "2Gi"
      cpu: "1000m"
  extraEnv:
    - name: JAVA_OPTS
      value: "-Xmx1g -Xms512m"

postgresql:
  enabled: true
  auth:
    database: kestra
    username: kestra
    existingSecret: kestra-secrets
    secretKeys:
      adminPasswordKey: postgres-password
  persistence:
    size: 50Gi
    storageClass: kestra-storage

ingress:
  enabled: true
  className: nginx
  hosts:
    - host: kestra.yourcompany.com
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: kestra-tls
      hosts:
        - kestra.yourcompany.com

redis:
  enabled: true
  architecture: standalone
  auth:
    existingSecret: kestra-secrets
    secretKeys:
      redis-passwordKey: redis-password

elasticsearch:
  enabled: true  # For enhanced logging and search
EOF

# Install the chart
helm install kestra kestra/kestra \
  --namespace kestra \
  --values kestra-values.yaml \
  --wait

# Check deployment status
kubectl get all -n kestra

Step 3: Configure Ingress and TLS

# If using cert-manager for TLS
cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: kestra-tls
  namespace: kestra
spec:
  secretName: kestra-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
    - kestra.yourcompany.com
EOF

Step 4: Verify Deployment

# Check pods
kubectl get pods -n kestra -w

# Check services
kubectl get svc -n kestra

# View logs
kubectl logs -n kestra deployment/kestra-server -f

# Port-forward for local access (if needed)
kubectl port-forward -n kestra svc/kestra-server 8080:80

Option 3: Cloud-Specific Deployments

AWS ECS Setup (Fargate)

# ecs-task-definition.json
{
  "family": "kestra-server",
  "networkMode": "awsvpc",
  "taskRoleArn": "arn:aws:iam::123456789012:role/ecsTaskRole",
  "executionRoleArn": "arn:aws:iam::123456789012:role/ecsTaskExecutionRole",
  "cpu": "2048",
  "memory": "4096",
  "containerDefinitions": [
    {
      "name": "kestra",
      "image": "kestra/kestra:latest",
      "portMappings": [
        {
          "containerPort": 8080,
          "protocol": "tcp"
        }
      ],
      "environment": [
        {
          "name": "KESTRA_CONFIGURATION",
          "value": "kestra.repository.type=postgres\nkestra.storage.type=s3"
        }
      ],
      "secrets": [
        {
          "name": "DB_PASSWORD",
          "valueFrom": "arn:aws:secretsmanager:region:account:secret:db_password"
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/kestra",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "ecs"
        }
      }
    }
  ]
}

Deploy with AWS CLI:

# Register task definition
aws ecs register-task-definition \
  --cli-input-json file://ecs-task-definition.json

# Create service
aws ecs create-service \
  --cluster production \
  --service-name kestra \
  --task-definition kestra-server:1 \
  --desired-count 3 \
  --launch-type FARGATE \
  --network-configuration "awsvpcConfiguration={subnets=[subnet-xxx,subnet-yyy],securityGroups=[sg-zzz],assignPublicIp=ENABLED}"

Google Cloud Run (Serverless)

# Deploy to Cloud Run
gcloud run deploy kestra \
  --image=kestra/kestra:latest \
  --platform=managed \
  --region=us-central1 \
  --allow-unauthenticated \
  --memory=2Gi \
  --cpu=2 \
  --set-env-vars="KESTRA_CONFIGURATION=..."

# Set up Cloud SQL connection
gcloud run services update kestra \
  --add-cloudsql-instances=PROJECT:REGION:INSTANCE \
  --set-env-vars="DB_HOST=/cloudsql/PROJECT:REGION:INSTANCE"

Azure Container Instances

# Create container instance
az container create \
  --resource-group kestra-rg \
  --name kestra-server \
  --image kestra/kestra:latest \
  --ports 8080 \
  --memory 4 \
  --cpu 2 \
  --environment-variables \
    KESTRA_CONFIGURATION='kestra.repository.type=postgres' \
  --dns-name-label kestra-unique \
  --azure-file-volume-share-name kestra-data \
  --azure-file-volume-account-name mystorageaccount \
  --azure-file-volume-account-key $STORAGE_KEY

Option 4: Bare Metal / VM Installation

For environments where containers aren't an option:

Step 1: Install Java and Dependencies

# Ubuntu/Debian
sudo apt update
sudo apt install -y openjdk-17-jre-headless postgresql-15 redis-server

# CentOS/RHEL
sudo yum install -y java-17-openjdk postgresql15-server redis

# macOS
brew install openjdk@17 postgresql@15 redis

Step 2: Configure Database

# Initialize PostgreSQL
sudo postgresql-setup initdb
sudo systemctl start postgresql
sudo systemctl enable postgresql

# Create Kestra database and user
sudo -u postgres psql << EOF
CREATE DATABASE kestra;
CREATE USER kestra WITH PASSWORD 'secure_password';
GRANT ALL PRIVILEGES ON DATABASE kestra TO kestra;
ALTER DATABASE kestra SET timezone TO 'UTC';
EOF

# Update pg_hba.conf for password auth
echo "host    kestra          kestra          127.0.0.1/32            md5" | sudo tee -a /etc/postgresql/15/main/pg_hba.conf
sudo systemctl restart postgresql

Step 3: Install Kestra

# Download Kestra
wget https://github.com/kestra-io/kestra/releases/latest/download/kestra-standalone.jar -O /opt/kestra/kestra.jar

# Create systemd service
cat << EOF | sudo tee /etc/systemd/system/kestra.service
[Unit]
Description=Kestra Workflow Orchestrator
After=network.target postgresql.service redis.service

[Service]
Type=simple
User=kestra
Group=kestra
WorkingDirectory=/opt/kestra
ExecStart=/usr/bin/java -Xmx2g -jar /opt/kestra/kestra.jar server standalone

# Security hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ReadWritePaths=/var/lib/kestra

[Install]
WantedBy=multi-user.target
EOF

# Create user and directories
sudo useradd -r -s /bin/false kestra
sudo mkdir -p /opt/kestra /var/lib/kestra/{flows,storage,logs,plugins}
sudo chown -R kestra:kestra /opt/kestra /var/lib/kestra

# Start service
sudo systemctl daemon-reload
sudo systemctl enable kestra
sudo systemctl start kestra
sudo systemctl status kestra

Verification and Testing

Health Check

# Using curl
curl http://localhost:8080/api/v1/health

# Expected response
{
  "status": "UP",
  "components": {
    "database": {
      "status": "UP"
    },
    "storage": {
      "status": "UP"
    }
  }
}

# Using API
curl -X GET "http://localhost:8080/api/v1/configs" \
  -H "Authorization: Basic $(echo -n 'admin:password' | base64)"

Create Your First Flow Programmatically

# test_kestra.py
import requests
import json
import yaml

# Configure
BASE_URL = "http://localhost:8080"
USERNAME = "admin"
PASSWORD = "password"
NAMESPACE = "demo"

# Create namespace
auth = (USERNAME, PASSWORD)
headers = {"Content-Type": "application/json"}

namespace_payload = {
    "namespace": NAMESPACE,
    "description": "Demo namespace"
}

response = requests.post(
    f"{BASE_URL}/api/v1/namespaces",
    json=namespace_payload,
    auth=auth,
    headers=headers
)

# Create a simple flow
flow_yaml = """
id: hello-world
namespace: demo
description: My first Kestra flow

tasks:
  - id: hello
    type: io.kestra.plugin.core.log.Log
    message: "Hello, Kestra! 🎉"

  - id: get-time
    type: io.kestra.plugin.core.debug.Return
    format: "Flow started at {{ execution.startDate }}"

triggers:
  - id: schedule
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "*/5 * * * *"  # Every 5 minutes
"""

# Create flow
response = requests.post(
    f"{BASE_URL}/api/v1/flows/{NAMESPACE}",
    data=flow_yaml,
    auth=auth,
    headers={"Content-Type": "application/yaml"}
)

# Execute flow
response = requests.post(
    f"{BASE_URL}/api/v1/executions/trigger/{NAMESPACE}/hello-world",
    auth=auth,
    headers=headers
)

print("Flow created and triggered successfully!")

Test with Kestra CLI

bash

# Install CLI
pip install kestra

# Configure
kestra config set-endpoint http://localhost:8080
kestra config set-username admin
kestra config set-password password

# Test commands
kestra namespace list
kestra flow list --namespace demo
kestra flow execute --namespace demo --flow hello-world
kestra execution list --namespace demo

Configuration Deep Dive

Configuration File Structure

Create kestra.yml for custom configurations:

# kestra.yml
kestra:
  server:
    basic-auth:
      enabled: true
      username: admin
      password: ${KESTRA_PASSWORD:?Password required}

    cors:
      enabled: true
      allowed-origins: ["https://*.yourcompany.com"]

    metrics:
      enabled: true
      port: 8081
      path: /metrics

  repository:
    type: postgres
    postgres:
      host: ${DB_HOST:localhost}
      port: ${DB_PORT:5432}
      database: kestra
      username: kestra
      password: ${DB_PASSWORD:?DB password required}
      pool-size: 10

  storage:
    type: s3  # or local, gcs, azure
    s3:
      region: ${AWS_REGION:us-east-1}
      bucket: kestra-storage
      endpoint: ${AWS_ENDPOINT:}  # For MinIO or other S3-compatible
      access-key-id: ${AWS_ACCESS_KEY_ID:}
      secret-access-key: ${AWS_SECRET_ACCESS_KEY:}

  queue:
    type: postgres  # or redis, kafka

  cache:
    type: redis
    redis:
      host: ${REDIS_HOST:localhost}
      port: ${REDIS_PORT:6379}
      password: ${REDIS_PASSWORD:}

  plugins:
    location: /app/plugins
    repositories:
      - id: kestra-io
        url: https://repo.kestra.io/repository/maven-public/

  tasks:
    default-retry:
      type: exponential
      max-attempt: 3
      delay: PT5S

  security:
    secret-key: ${SECRET_KEY:?Secret key required for encryption}
    encryption:
      enabled: true

Environment Variables

# .env file example
KESTRA_PASSWORD=SuperSecurePassword123!
DB_HOST=postgres
DB_PORT=5432
DB_PASSWORD=db_password_456
AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=AKIA...
AWS_SECRET_ACCESS_KEY=...
REDIS_HOST=redis
REDIS_PASSWORD=redis_pass_789
SECRET_KEY=$(openssl rand -hex 32)

Run with Custom Config

# Docker with custom config
docker run --rm \
  -p 8080:8080 \
  -v $(pwd)/kestra.yml:/app/kestra.yml \
  -v $(pwd)/.env:/app/.env \
  --env-file .env \
  kestra/kestra:latest \
  server standalone

# Or with environment variables
docker run --rm \
  -p 8080:8080 \
  -e KESTRA_PASSWORD=password \
  -e DB_HOST=postgres \
  kestra/kestra:latest \
  server standalone

Troubleshooting Guide

Common Issues and Solutions

1. Port Already in Use

# Find what's using port 8080
sudo lsof -i :8080

# Kill the process
sudo kill -9 <PID>

# Or change Kestra port
docker run --rm -p 9090:8080 kestra/kestra:latest

2. Database Connection Issues

# Test database connection
docker exec -it kestra-postgres psql -U kestra -d kestra

# Reset database (development only)
docker-compose down -v
docker-compose up -d

3. Permission Denied for Storage

# Fix volume permissions
sudo chown -R 1000:1000 ./flows ./storage ./logs

# Or run as current user
docker run --rm \
  -u $(id -u):$(id -g) \
  -v $(pwd)/flows:/app/flows \
  kestra/kestra:latest

4. Out of Memory Errors

# Increase memory in docker-compose
services:
  kestra:
    deploy:
      resources:
        limits:
          memory: 2G
        reservations:
          memory: 1G

5. Flow Not Appearing

# Check if flows are being loaded
curl http://localhost:8080/api/v1/flows

# Check logs for errors
docker-compose logs kestra | grep -i flow

# Verify directory mapping
docker exec -it kestra-server ls -la /app/flows

Debug Commands

# Get detailed system info
curl http://localhost:8080/api/v1/configs/debug

# Check plugin availability
curl http://localhost:8080/api/v1/plugins

# View server metrics
curl http://localhost:8080/api/v1/metrics

# Check queue status
curl http://localhost:8080/api/v1/queues

Post-Installation Checklist

✅ Security Hardening

# 1. Change default password
curl -X PUT "http://localhost:8080/api/v1/users/admin" \
  -H "Authorization: Basic $(echo -n 'admin:password' | base64)" \
  -H "Content-Type: application/json" \
  -d '{"password": "NewSecurePassword123!"}'

# 2. Enable HTTPS
# 3. Set up firewall rules
# 4. Configure audit logging

✅ Backup Strategy

# Backup flows
cp -r ./flows ./backups/flows-$(date +%Y%m%d)

# Backup database
docker exec kestra-postgres pg_dump -U kestra kestra > backup-$(date +%Y%m%d).sql

# Backup configuration
cp kestra.yml kestra.yml.backup-$(date +%Y%m%d)

✅ Monitoring Setup

# Install monitoring stack
docker-compose -f monitoring/docker-compose.yml up -d

# Configure Prometheus
echo "
  - job_name: 'kestra'
    static_configs:
      - targets: ['kestra:8081']
" >> prometheus/prometheus.yml

Production Deployment Checklist

Before Going Live:

Performance Testing: Load test with 100+ concurrent flows
Disaster Recovery: Test backup/restore procedures
High Availability: Ensure multiple replicas are running
Monitoring: Set up alerts for critical metrics
Security Audit: Review access controls and encryption
Documentation: Document deployment procedures
Rollback Plan: Test downgrade procedure
Load Balancer: Configure proper load balancing
DNS Setup: Configure proper DNS records
SSL/TLS: Set up proper certificates

Critical Metrics to Monitor:

Flow execution success rate (>99%)
Average execution time
Queue backlog size
Database connection pool usage
Memory and CPU utilization
Storage usage
API response times

Next Steps: What to Build First

Now that Kestra is installed, here are some practical projects to start with:

Project 1: Data Pipeline Template

# Create project structure
mkdir -p flows/{extract,transform,load}
mkdir -p scripts/{python,sql}
mkdir -p config

# Create your first real flow
cat > flows/extract/daily-sales.yml << 'EOF'
id: daily-sales-extract
namespace: sales.pipeline
description: Extract daily sales data

tasks:
  - id: extract-api
    type: io.kestra.plugin.core.http.Download
    uri: "https://api.company.com/sales/{{ execution.startDate | date('yyyy-MM-dd') }}"
    outputFile: "sales_{{ execution.startDate | date('yyyy-MM-dd') }}.json"
EOF

Project 2: Monitoring Dashboard

Deploy Grafana alongside Kestra
Import Kestra dashboard templates
Set up alerts for failed executions
Create custom metrics dashboards

Project 3: CI/CD Pipeline

# .github/workflows/kestra-deploy.yml
name: Deploy Kestra Flows
on:
  push:
    paths:
      - 'flows/**'

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Validate flows
        run: |
          docker run --rm \
            -v $(pwd)/flows:/flows \
            kestra/kestra:latest \
            flow validate /flows

      - name: Deploy to Kestra
        run: |
          curl -X POST "https://kestra.company.com/api/v1/flows/batch" \
            -H "Authorization: Bearer ${{ secrets.KESTRA_TOKEN }}" \
            -F "files=@flows/sales-pipeline.yml" \
            -F "namespace=sales"

Community and Support

Getting Help:

Documentation: kestra.io/docs
GitHub Issues: github.com/kestra-io/kestra/issues
Community Slack: kestra.io/slack (Join 3000+ members)
Stack Overflow: Use tag kestra

Contributing:

# Clone the repository
git clone https://github.com/kestra-io/kestra.git
cd kestra

# Build from source
./gradlew build

# Run tests
./gradlew test

# Submit pull request

Conclusion

You now have Kestra running in your chosen environment! Whether you opted for the quick Docker setup or a full Kubernetes deployment, you're ready to start building powerful data workflows.

Key Takeaways:

Start simple: Use Docker standalone for exploration
Scale gradually: Move to Docker Compose, then Kubernetes
Configure properly: Use environment variables and config files
Monitor everything: Set up metrics and alerts from day one
Join the community: The Kestra community is active and helpful

What's Next?

In the next article, we'll dive into Building Your First ETL Pipeline. We'll cover:

Extracting data from multiple sources
Transforming with Python and SQL
Loading to databases and data warehouses
Error handling and monitoring
Best practices for production pipelines

Before the next article, try:

Create a namespace for your project
Build a simple flow that downloads and processes a public dataset
Explore the web UI and understand the execution view
Join the Slack community and introduce yourself

Remember: The goal of installation isn't just to get software running—it's to create a foundation that's secure, scalable, and maintainable. Take the time to do it right, and your future self will thank you.

Happy orchestrating! 🚀

Command Palette

Comments

Kestra-DataEngineering

Kestra 101: Why It's Revolutionizing Data Orchestration.

More from this blog