Complete guide to automated infrastructure and application deployment workflows
This directory contains GitHub Actions workflows for automated MOSIP deployment:
- Terraform workflows for infrastructure provisioning (AWS complete, Azure/GCP placeholders)
- Helmsman workflows for application deployment on provisioned infrastructure
- GPG encrypted state management with branch isolation
- Sequential workflow execution with parallel optimization within Helmsman phase
| Workflow | Purpose | Trigger | State Management | PostgreSQL |
|---|---|---|---|---|
terraform.yml |
Deploy/Update Infrastructure | Manual Dispatch | GPG encrypted local | Integrated via Terraform |
terraform-destroy.yml |
Destroy Infrastructure | Manual Dispatch | Uses encrypted state | Handles PostgreSQL cleanup |
helmsman_external.yml |
Deploy Prerequisites & External Dependencies | Manual Dispatch | Uses deployed infra | Parallel deployment |
helmsman_mosip.yml |
Deploy MOSIP Services | Manual Dispatch | Uses deployed infra | Uses deployed PostgreSQL |
helmsman_esignet.yml |
Deploy eSignet Stack | Manual/Push | Uses deployed infra | Uses deployed PostgreSQL |
helmsman_testrigs.yml |
Deploy Test Rigs | Manual Dispatch | Uses deployed infra | Testing components |
| Provider | Status | Implementation |
|---|---|---|
| AWS | Complete | Full infrastructure, VPC, RKE2, PostgreSQL integration |
| Azure | Placeholder | Basic structure - community contributions welcome |
| GCP | Placeholder | Basic structure - community contributions welcome |
- Navigate: Actions → "terraform plan / apply"
- Configure Parameters:
CLOUD_PROVIDER: aws # Currently only AWS fully supported
TERRAFORM_COMPONENT: base-infra # base-infra | infra | observ-infra
BACKEND_TYPE: local # local with GPG encryption (recommended)
REMOTE_BACKEND_CONFIG: "" # Not used with local backend
SSH_PRIVATE_KEY: SSH_PRIVATE_KEY # GitHub secret name
TERRAFORM_APPLY: true # false = plan only
ENABLE_STATE_LOCKING: false # Optional DynamoDB state locking- Execute: Click "Run workflow"
- PostgreSQL Configuration: Set in
terraform/implementations/aws/infra/aws.tfvars:
enable_postgresql_setup = true # External PostgreSQL via Terraform + AnsiblePrerequisites: Complete Terraform infrastructure deployment first
- Navigate: Actions → "helmsman external dependencies"
- Configure: Select deployed infrastructure environment
- Parallel Execution: Prerequisites and External Dependencies deploy simultaneously
- Duration: ~70-110 minutes (20% faster than sequential approach)
- Navigate: Actions → "helmsman mosip services"
- PostgreSQL Integration: Automatically uses deployed PostgreSQL
- Duration: ~25-35 minutes
- Navigate: Actions → "terraform destroy"
- Configure: Use same parameters as deployment
- Confirm: Set
TERRAFORM_DESTROY: true - PostgreSQL Cleanup: Automatically handled
- Execute: Click "Run workflow"
PostgreSQL is configured in Terraform variables, not as a workflow input:
# terraform/implementations/aws/infra/aws.tfvars
enable_postgresql_setup = true # Enable external PostgreSQL
nginx_node_ebs_volume_size_2 = 200 # EBS volume size for PostgreSQL data
postgresql_version = "15" # PostgreSQL version
postgresql_port = "5433" # PostgreSQL port- External Database (
enable_postgresql_setup = true): PostgreSQL on dedicated instances via Terraform + Ansible - Container Database (
enable_postgresql_setup = false): In-cluster PostgreSQL as microservice container via Helmsman
Component Deployment Order
==========================
Step 1: Foundation (One-time)
├── Component: base-infra
├── Resources: VPC, Subnets, WireGuard
└── Purpose: Network foundation
Step 2: Management (Optional)
├── Component: observ-infra
├── Resources: Rancher UI, Keycloak
├── Dependencies: base-infra
└── Purpose: Cluster management
Step 3: Application (Repeatable)
├── Component: infra
├── Resources: MOSIP K8s clusters
├── Dependencies: base-infra
└── Purpose: MOSIP deployments
graph LR
A[base-infra<br/>Foundation] --> B[observ-infra<br/>Management<br/>Optional]
A --> C[infra<br/>MOSIP Clusters]
B -.->|Import Clusters| C
C --> D[Multiple MOSIP<br/>Deployments]
style A fill:#e1f5fe,stroke:#01579b,stroke-width:2px,color:#000000
style B fill:#fff3e0,stroke:#f57c00,stroke-width:2px,color:#000000
style C fill:#f3e5f5,stroke:#4a148c,stroke-width:2px,color:#000000
style D fill:#e8f5e8,stroke:#1b5e20,stroke-width:2px,color:#000000
| Component | Purpose | Deployment Order | Dependencies | Lifecycle |
|---|---|---|---|---|
| base-infra | VPC, Networking, WireGuard VPN | 1st (Foundation) | None | One-time setup |
| observ-infra | Rancher UI, Keycloak, RBAC management | 2nd (Optional) | base-infra | One-time setup |
| infra | MOSIP Kubernetes clusters (RKE2, NGINX, NFS) | 3rd (Multiple) | base-infra | Multiple deployments |
graph TD
START[GitHub Actions Trigger] --> VALIDATE[Validate Parameters]
VALIDATE --> SETUP[Setup Terraform & Cloud Credentials]
SETUP --> BACKEND[Configure Local Backend with GPG]
BACKEND --> DECRYPT{Encrypted State Exists?}
DECRYPT -->|Yes| DECRYPTSTATE[Decrypt State with GPG]
DECRYPT -->|No| INIT[terraform init]
DECRYPTSTATE --> INIT
INIT --> PLAN[terraform plan]
PLAN --> DECISION{Apply or Plan Only?}
DECISION -->|Plan Only| OUTPUT[Show Plan Output]
DECISION -->|Apply| APPLY[terraform apply]
APPLY --> ENCRYPT[Encrypt State with GPG]
ENCRYPT --> SUCCESS[Deployment Complete]
OUTPUT --> COMPLETE[Workflow Complete]
SUCCESS --> COMPLETE
style START fill:#e8f5e8,stroke:#1b5e20,stroke-width:2px,color:#000000
style APPLY fill:#f3e5f5,stroke:#4a148c,stroke-width:2px,color:#000000
style SUCCESS fill:#e1f5fe,stroke:#01579b,stroke-width:2px,color:#000000
Note: PostgreSQL setup is handled by Terraform modules and Ansible during the terraform apply step based on enable_postgresql_setup configuration in .tfvars files, not as a separate workflow step.
- Zero-configuration GPG: Uses GPG_PRIVATE_KEY secret automatically
- AES256 encryption: Local state files encrypted with GPG
- Custom naming: Pattern:
{provider}-{component}-{branch}-terraform.tfstate - Git safety: Encrypted state files tracked in repository
- Branch isolation: Complete separation of environment states
# Repository Structure (Encrypted)
.terraform-state/
├── aws-base-infra-testgrid-terraform.tfstate.gpg
├── aws-infra-testgrid-terraform.tfstate.gpg
└── aws-observ-infra-testgrid-terraform.tfstate.gpg
# Decrypted for Terraform Use (Temporary)
terraform/base-infra/aws-base-infra-testgrid-terraform.tfstate
terraform/infra/aws-infra-testgrid-terraform.tfstate
terraform/observ-infra/aws-observ-infra-testgrid-terraform.tfstate# Required GitHub Secret
GPG_PRIVATE_KEY: |
-----BEGIN PGP PRIVATE KEY BLOCK-----
<your-gpg-private-key>
-----END PGP PRIVATE KEY BLOCK-----| Parameter | Description | Values | Example |
|---|---|---|---|
CLOUD_PROVIDER |
Target cloud platform | aws (fully supported) | azure | gcp (placeholders) |
aws |
TERRAFORM_COMPONENT |
Infrastructure component | base-infra | observ-infra | infra |
base-infra |
BACKEND_TYPE |
State storage method | local (recommended) | remote |
local |
SSH_PRIVATE_KEY |
GitHub secret for SSH access | Secret name | SSH_PRIVATE_KEY |
GPG_PRIVATE_KEY |
GitHub secret for state encryption | Secret name | GPG_PRIVATE_KEY |
BACKEND_TYPE: local
# State files encrypted with GPG and stored in repository
# Custom naming: aws-infra-testgrid-terraform.tfstateBACKEND_TYPE: remote
REMOTE_BACKEND_CONFIG: aws:bucket-name:region
# Format: <cloud>:<bucket-name>:<region>
# Example: aws:mosip-terraform-bucket:us-west-2| Parameter | Description | Default | Options |
|---|---|---|---|
TERRAFORM_APPLY |
Execute apply after plan | false |
true | false |
TERRAFORM_DESTROY |
Destroy infrastructure | false |
true | false |
ENABLE_STATE_LOCKING |
Enable DynamoDB state locking | false |
true | false |
Note: PostgreSQL configuration is set in Terraform .tfvars files, not as workflow parameters.
# GPG Encryption for State Files
GPG_PRIVATE_KEY: |
-----BEGIN PGP PRIVATE KEY BLOCK-----
<your-gpg-private-key>
-----END PGP PRIVATE KEY BLOCK-----
# SSH Access for jumpserver
SSH_PRIVATE_KEY: |
-----BEGIN OPENSSH PRIVATE KEY-----
<your-private-key-content>
-----END OPENSSH PRIVATE KEY-----
# AWS Credentials (Complete Implementation)
AWS_ACCESS_KEY_ID: AKIA...
AWS_SECRET_ACCESS_KEY: wJalr...
# Azure Credentials (Placeholder Implementation)
AZURE_CLIENT_ID: 12345678-1234-1234-1234-123456789012
AZURE_CLIENT_SECRET: secret-value
AZURE_SUBSCRIPTION_ID: 12345678-1234-1234-1234-123456789012
AZURE_TENANT_ID: 12345678-1234-1234-1234-123456789012
# GCP Credentials (Placeholder Implementation)
GOOGLE_CREDENTIALS: |
{
"type": "service_account",
"project_id": "your-project",
...
}
# Optional: Slack notifications
SLACK_WEBHOOK_URL: https://hooks.slack.com/services/...- GPG encryption: All state files encrypted before commit
- Least privilege access: IAM roles with minimal required permissions
- Secret rotation: Regular rotation of access keys and GPG keys
- Audit logging: CloudTrail/Activity logs enabled for all operations
- Network isolation: Resources deployed in private subnets
- Database security: PostgreSQL with encrypted storage and secure access
# Deploy base infrastructure
CLOUD_PROVIDER: aws
TERRAFORM_COMPONENT: base-infra
BACKEND_TYPE: local
SSH_PRIVATE_KEY: SSH_PRIVATE_KEY
GPG_PRIVATE_KEY: GPG_PRIVATE_KEY
TERRAFORM_APPLY: true
# PostgreSQL configured in terraform/implementations/aws/infra/aws.tfvars:
# enable_postgresql_setup = true# Production deployment (main branch)
CLOUD_PROVIDER: aws
TERRAFORM_COMPONENT: infra
BACKEND_TYPE: local
# State: aws-infra-main-terraform.tfstate.gpg
# Staging deployment (staging branch)
CLOUD_PROVIDER: aws
TERRAFORM_COMPONENT: infra
BACKEND_TYPE: local
# State: aws-infra-staging-terraform.tfstate.gpg# Step 1: Deploy Terraform infrastructure (must complete first)
Workflow: terraform.yml
Component: base-infra → Deploy foundational infrastructure
Component: observ-infra → Deploy monitoring cluster (optional)
Component: infra → Deploy MOSIP cluster + PostgreSQL
# Step 2: Deploy Helmsman components (after Terraform complete)
Workflow: helmsman_external.yml → Parallel deployment:
- Prerequisites: Monitoring, Istio, Logging
- External Dependencies: PostgreSQL connection, MinIO, Keycloak, Kafka
# Step 3: Deploy MOSIP services (after external dependencies ready)
Workflow: helmsman_mosip.yml → Deploy MOSIP applications
# Step 4: Deploy eSignet stack (after MOSIP services or standalone)
Workflow: helmsman_esignet.yml → Deploy eSignet authentication stack
# Optional: Deploy test components
Workflow: helmsman_testrigs.yml → Deploy testing infrastructure| Issue | Cause | Solution |
|---|---|---|
| GPG Decryption Failed | Invalid GPG private key | Verify GPG_PRIVATE_KEY secret is correct |
| State File Not Found | Missing encrypted state | Check .terraform-state/ directory for .gpg files |
| Provider Auth Failed | Invalid credentials | Check cloud provider secrets configuration |
| Module Not Found | Git access issues | Verify SSH key has repository access |
| PostgreSQL Connection Failed | Database not ready | Wait for Ansible PostgreSQL setup to complete |
| Azure/GCP Placeholder Error | Incomplete implementation | Use AWS or contribute to Azure/GCP modules |
- Check workflow logs: Review GitHub Actions execution logs
- Validate GPG key: Ensure GPG_PRIVATE_KEY secret is configured
- Verify cloud credentials: Check AWS/Azure/GCP secrets
- Test connectivity: Verify network access to cloud APIs
- State inspection: Decrypt and inspect state files manually
- PostgreSQL logs: Check Ansible output for database setup
- Check IAM permissions for EC2, VPC, RDS
- Verify region availability and quota limits
- Test AWS CLI connectivity with provided credentials
- Current implementations use
null_resourceplaceholders - No actual cloud resources are created
- Community contributions needed for full implementation
sequenceDiagram
participant User
participant Terraform Workflows
participant Helmsman Workflows
participant AWS Infrastructure
participant PostgreSQL
User->>Terraform Workflows: 1. Deploy base-infra
Terraform Workflows->>AWS Infrastructure: Create VPC, WireGuard
AWS Infrastructure-->>User: Base infrastructure ready
User->>Terraform Workflows: 2. Deploy infra + PostgreSQL
Terraform Workflows->>AWS Infrastructure: Create RKE2 cluster
Terraform Workflows->>PostgreSQL: Setup PostgreSQL 15 via Ansible
AWS Infrastructure-->>User: MOSIP cluster + PostgreSQL ready
User->>Helmsman Workflows: 3. Deploy Prerequisites (parallel)
User->>Helmsman Workflows: 3. Deploy External Dependencies (parallel)
Helmsman Workflows-->>AWS Infrastructure: Install Prerequisites & Dependencies
AWS Infrastructure-->>User: Prerequisites & Dependencies ready (70-110 min)
User->>Helmsman Workflows: 4. Deploy MOSIP Services
Helmsman Workflows->>PostgreSQL: Connect to external database
AWS Infrastructure-->>User: MOSIP platform operational
User->>Helmsman Workflows: 5. Deploy eSignet Stack (optional)
Helmsman Workflows->>PostgreSQL: Init eSignet database
Helmsman Workflows-->>AWS Infrastructure: Install eSignet, Redis, SoftHSM, OIDC UI
AWS Infrastructure-->>User: eSignet stack operational
- Terraform Workflows (Sequential - Infrastructure Setup):
terraform.yml→ Deploy base-infraterraform.yml→ Deploy observ-infra (optional)terraform.yml→ Deploy infra + PostgreSQL
- Helmsman Workflows (Can run in parallel after Terraform complete):
helmsman_external.yml→ Prerequisites + External Dependencies (simultaneous)helmsman_mosip.yml→ MOSIP Services (after external dependencies ready)helmsman_esignet.yml→ eSignet Stack (after MOSIP DSF or standalone)helmsman_testrigs.yml→ Test components (optional)
graph TD
A[Terraform Infrastructure Complete] --> B[Helmsman Workflows Triggered]
B --> C[Prerequisites Workflow]
B --> D[External Dependencies Workflow]
C -->|Parallel| E[Monitoring Stack]
C -->|Parallel| F[Istio Service Mesh]
C -->|Parallel| G[Logging Infrastructure]
D -->|Parallel| H[PostgreSQL Connection]
D -->|Parallel| I[MinIO Storage]
D -->|Parallel| J[Keycloak IAM]
D -->|Parallel| K[Kafka Messaging]
E --> L[MOSIP Services Deployment]
F --> L
G --> L
H --> L
I --> L
J --> L
K --> L
L --> M{Deploy eSignet?}
M -->|Yes| N[eSignet Stack Deployment<br/>Redis, SoftHSM, Keycloak Init,<br/>eSignet, OIDC UI, Mock Identity]
M -->|No| O[Optional: Test Rigs]
N --> O
style A fill:#e1f5fe,stroke:#01579b,stroke-width:2px,color:#000000
style B fill:#e8f5e8,stroke:#1b5e20,stroke-width:2px,color:#000000
style C fill:#fff3e0,stroke:#f57c00,stroke-width:2px,color:#000000
style D fill:#fff3e0,stroke:#f57c00,stroke-width:2px,color:#000000
style L fill:#f3e5f5,stroke:#4a148c,stroke-width:2px,color:#000000
style N fill:#e0f2f1,stroke:#00695c,stroke-width:2px,color:#000000
Sequential Dependency: Terraform workflows must complete before Helmsman workflows Parallel Optimization: Prerequisites and External Dependencies run simultaneously via separate Helmsman workflows
- Workflow Maintenance: Keep workflows updated with latest Terraform versions
- State Management: GPG encrypted state with branch-based isolation
- Security Reviews: Regular rotation of GPG keys and cloud credentials
- PostgreSQL Management: Automated setup via Terraform + Ansible integration
- Performance Optimization: Use parallel deployment for 20% faster setup times
- AWS - Production ready with full feature set
- Azure - Placeholder implementation - contributions welcome
- GCP - Placeholder implementation - contributions welcome
Community contributions needed for Azure and GCP implementations
Deploys the complete eSignet authentication stack including Redis, SoftHSM, Keycloak, Mock Identity System, Mock Relying Party, and Partner Onboarder.
| Trigger | Condition |
|---|---|
| Manual | workflow_dispatch - Run from Actions tab |
| Push | When Helmsman/dsf/esignet-dsf.yaml is modified |
| Input | Description | Required | Default |
|---|---|---|---|
mode |
Helmsman mode: dry-run or apply |
Yes | dry-run |
skip_mosip_dsf_check |
Skip MOSIP DSF completion check for standalone deployment | No | false |
Configure in Repository → Settings → Secrets and variables → Actions → Secrets:
| Secret | Description | Required |
|---|---|---|
KUBECONFIG |
Kubernetes config file content | ✅ Yes |
CLUSTER_WIREGUARD_WG0 |
WireGuard configuration for cluster access | ✅ Yes |
MOCK_RELYING_PARTY_CLIENT_PRIVATE_KEY |
Mock Relying Party client private key (base64 encoded PEM) | ✅ Yes |
MOCK_RELYING_PARTY_JWE_PRIVATE_KEY |
JWE userinfo private key (base64 encoded PEM) | ✅ Yes |
Configure in Repository → Settings → Secrets and variables → Actions → Variables:
| Variable | Description | Values |
|---|---|---|
ESIGNET_STANDALONE_MODE |
Enable standalone mode for push-triggered runs | true / false |
# Encode client private key (no line wrapping)
cat client-private-key.pem | base64 -w 0
# Copy output → Add as MOCK_RELYING_PARTY_CLIENT_PRIVATE_KEY secret
# Encode JWE userinfo private key
cat jwe-userinfo-private-key.pem | base64 -w 0
# Copy output → Add as MOCK_RELYING_PARTY_JWE_PRIVATE_KEY secretRequires MOSIP DSF to be completed first. Checks for mosip-dsf=completed label on default namespace.
MOSIP DSF → eSignet DSF
Skips MOSIP DSF check. Enable via:
| Trigger Type | How to Enable |
|---|---|
| Manual Run | Set skip_mosip_dsf_check to true in workflow UI |
| Push Triggered | Set repository variable ESIGNET_STANDALONE_MODE=true |
| Priority | Component | Description |
|---|---|---|
| -19 | redis | Cache layer |
| -18 | softhsm-esignet | Hardware security module for eSignet |
| -15 | postgres-init-esignet | Database initialization |
| -14 | keycloak | Identity and access management |
| -13 | esignet-keycloak-init | Keycloak realm configuration |
| -12 | esignet | Core eSignet service |
| -11 | softhsm-mock-identity-system | HSM for mock identity |
| -11 | oidc-ui | OpenID Connect UI |
| -10 | mock-identity-system | Mock identity provider |
| -10 | partner-onboarder | Partner onboarding service |
| -9 | mock-relying-party-ui | Mock relying party frontend |
| -8 | mock-relying-party-service | Mock relying party backend |
- ✅ All secrets automatically masked in GitHub Actions logs
- ✅ Private keys passed as environment variables (never written to disk)
- ✅ WireGuard config written with restricted permissions (600)
- ✅ Kubeconfig stored with 400 permissions
- ✅ Explicit
::add-mask::for additional protection
# Check all eSignet pods
kubectl get pods -n esignet
# Verify services
kubectl get svc -n esignet
# Check eSignet logs
kubectl logs -n esignet -l app=esignet
# Verify namespace label
kubectl get ns default --show-labels | grep esignet-dsf| Issue | Solution |
|---|---|
MOSIP DSF not completed |
Run MOSIP DSF workflow first OR enable standalone mode |
Secret not configured |
Add required secrets in repository settings |
WireGuard connection failed |
Verify CLUSTER_WIREGUARD_WG0 contains valid config |
| Pods not starting | Check kubectl logs -n esignet <pod-name> for errors |
Professional infrastructure automation with enterprise-grade security, PostgreSQL integration, and parallel deployment capabilities