diff --git a/docs/setup/deploymentnew/v3-installation/1.2.1.0/README.md b/docs/setup/deploymentnew/v3-installation/1.2.1.0/README.md new file mode 100644 index 000000000..463b05bfd --- /dev/null +++ b/docs/setup/deploymentnew/v3-installation/1.2.1.0/README.md @@ -0,0 +1,2 @@ +# 1.2.0.2 + diff --git a/docs/setup/deploymentnew/v3-installation/1.2.1.0/aws-deployment.md b/docs/setup/deploymentnew/v3-installation/1.2.1.0/aws-deployment.md new file mode 100644 index 000000000..aa5efadd8 --- /dev/null +++ b/docs/setup/deploymentnew/v3-installation/1.2.1.0/aws-deployment.md @@ -0,0 +1,462 @@ +# AWS Deployment + +## 1. Wireguard + +* Wireguard Bastion server provides secure private channel to access MOSIP cluster. +* Bastian server restricts public access, and enables access to only those clients who have their public key listed with it. +* Bastian server listens on UDP port 51820. + +**Setup Wirguard VM** + +* Create a Wireguard server VM from aws console with mentioned '[**Hardware and Network Requirements**](pre-requisites.md)'. +* Edit the security group and add the following inbound rules in aws console + * type ‘custom TCP', port range ‘51820’ and source '0.0.0.0/0’ + * type ‘custom UDP', port range ‘51820’ and source '0.0.0.0/0’ +* Install docker in the Wireguard machine as given [here](https://docs.docker.com/engine/install/ubuntu/). + +**Setup Wireguard Bastian server** + +* SSH to wireguard VM +* Create directory for storing wireguard config files.`mkdir -p wireguard/config` +* Install and start wireguard server using docker as given below: + + ```sh + sudo docker run -d \ + --name=wireguard \ + --cap-add=NET_ADMIN \ + --cap-add=SYS_MODULE \ + -e PUID=1000 \ + -e PGID=1000 \ + -e TZ=Asia/Calcutta\ + -e PEERS=30 \ + -p 51820:51820/udp \ + -v /home/ubuntu/wireguard/config:/config \ + -v /lib/modules:/lib/modules \ + --sysctl="net.ipv4.conf.all.src_valid_mark=1" \ + --restart unless-stopped \ + ghcr.io/linuxserver/wireguard + ``` + + **Note:**\ + \* Increase the no of peers above in case needed more than 30 wireguard client confs. (`-e PEERS=30`)\ + \* Change the directory to be mounted to wireguard docker in case needed.\ + \* All your wireguard confs will be generated in the mounted directory. (`-v /home/ubuntu/wireguard/config:/config`) + +**Setup Wireguard Client in your PC** + +* Install [Wireguard client](https://www.wireguard.com/install/) in your PC. +* Assign `wireguard.conf`: + * SSH to the wireguard server VM. + * `cd /home/ubuntu/wireguard/config` + * assign one of the PR for yourself and use the same from the PC to connect to the server. + * create `assigned.txt` file to assign the keep track of peer files allocated and update everytime some peer is allocated to someone. + * ```java + peer1 : peername + peer2 : xyz + ``` + * Use `ls` cmd to see the list of peers. + * get inside your selected peer directory, and add mentioned changes in peer.conf: + * `cd peer1` + * `nano peer1.conf` + * Delete the DNS IP. + * Update the allowed IP's to subnets CIDR ip . e.g. 10.10.20.0/23 + * Share the updated `peer.conf` with respective peer to connect to wireguard server from Personel PC. +* add `peer.conf` in your PC’s `/etc/wireguard` directory as `wg0.conf`. +* start the wireguard client and check the status: + * ```sh + sudo systemctl start wg-quick@wg0 + sudo systemctl status wg-quick@wg0 + ``` +* Once Connected to wireguard you should be now able to login using private ip’s. + +## 2. Observation K8s Cluster setup + +### Pre-requisites +* Apps to be downloaded on users personal computer. + * [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) client version > 1.23.6 + * [helm](https://helm.sh/docs/intro/install/) client version > 3.8.2 and add `mosip` and `bitnami` repo as well. + * [eksctl](https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html) version 0.121.0. +* AWS account and credentials with permissions to create EKS cluster. +* AWS credentials in `~/.aws/` folder as given [here](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html). +* Save `~/.kube/config` file with another name. + * **IMPORTANT NOTE* : As part of EKS cluster creation steps your existing kubeconfig (`~/.kube/config`) file will be overridden. +* Save `.pem` file from AWS console and store it in `~/.ssh/` folder. (Generate a new one if you do not have this key file). + +## EKS cluster setup +* Setup Observation cluster: + * `cd $K8_ROOT/k8-cluster/csp/aws` + * Copy `observation.cluster.config.sample` to `rancher.cluster.config`. + * Review and update the below mentioned parameters of `rancher.cluster.config` carefully. + * name : cluster name. + * region : AWS region for EKS cluster. + * version : "1.24" + * EC2 instance related details + * instanceName + * instanceType + * desiredcapacity + * volumeSize + * volumeType + * publicKeyName + * update the details of the subnets to be used from vpc + * Execute `eksctl` command to create cluster + ```sh + eksctl create cluster -f rancher.cluster.config + ``` + * Wait for the cluster creation to complete, generally it takes around 30 minutes to create or update cluster. + * Once EKS K8 cluster is ready below mentioned output will be displayed in the console screen.`EKS cluster "my-cluster" in "region-code" region is ready` + * The config file for the new cluster will be created on `~/.kube/config` + * Make sure to backup and store the `~/.kube/config` with new name. e.g. `~/.kube/obs-cluster.config`. + * Change file permission using below command:`chmod 400 ~/.kube/obs-cluster.config` + * Set the `KUBECONFIG` properly so that you can access the cluster.`export KUBECONFIG=~/.kube/obs-cluster.config` + * Test cluster access: + * `kubectl get nodes` + * Command will result in details of the nodes of the rancher cluster. + +## 3. Observation K8s Cluster’s Ingress and Storage class setup + +Once the rancher cluster is ready we need ingress and storage class to be set for other applications to be installed. + +**3.a.** [**Nginx Ingress Controller**](https://kubernetes.github.io/ingress-nginx/deploy/) : used for ingress in rancher cluster. + +* ```sh + cd $K8_ROOT/ingress/ingress-nginx + helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx + helm repo update + helm install \ + ingress-nginx ingress-nginx/ingress-nginx \ + --namespace ingress-nginx \ + --create-namespace \ + -f ingress-nginx-lb.values.yaml + ``` + + The above will automatically spawn an [Internal AWS Network Load Balancer (L4)](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-network-load-balancer.html). +* Check the following on AWS console: + * An NLB has been created. You may also see the DNS of NLB with + + ```sh + kubectl -n ingress-nginx get svc + ``` + * Obtain AWS TLS certificate as given [here](https://docs.aws.amazon.com/acm/latest/userguide/dns-validation.html) + * Edit listner "443". Select "TLS". + * Note, the target group name of listner 80. Set target group of 443 to target group of 80. Basically, we want TLS termination at the LB and it must forward HTTP traffic (not HTTPS) to port 80 of ingress controller. So + * Input of LB: HTTPS + * Output of LB: HTTP --> port 80 of ingress nginx controller + * Enable "Proxy Protocol v2" in the target group settings + * Make sure all subnets are selected in LB -->Description-->Edit subnets. + * Check health check of target groups. + * Remove listner 80 from LB as we will receive traffic only on 443. +* **3.b. Storage class setup:** + * Default storage class on EKS is `gp2` . `GP2` by default is in `Delete` mode which means if PVC is deleted, the underlying storage PV is also deleted. + * Enable volume expansion for the existing `gp2` storage class, modify the YAML configuration by adding `allowVolumeExpansion: true` to the `gp2` storage class configuration. + * `kubectl edit sc gp2` : to edit the yaml configuration. + * Create storage class `gp2-retain` by running `sc.yaml` for PV in Retain mode. Set the storage class as gp2-retain in case you want to retain PV. + * ```sh + cd $K8_ROOT/storage-class/ebs/ + kubectl apply -f gp2-sc-retain.yaml + ``` + * EBS driver installation for our storage class to work, follow the steps [here](https://www.stacksimplify.com/aws-eks/kubernetes-storage/install-aws-ebs-csi-driver-on-aws-eks-for-persistent-storage/) to setup EBS driver. + +## 4. Domain name mapping + +* Create the following domain names: + * Rancher: `rancher.xyz.net` + * Keycloak: `keycloak.xyz.net` +* Point the above to **internal** ip address of the NLB. +* This assumes that you have a [Wireguard Bastion Host](https://docs.mosip.io/1.2.0/deployment/wireguard/wireguard-bastion) has been installed. +* Perform DNS mapping on Route 53 console. + +## 5. Rancher K8s Cluster Apps Installation + +* **5.a. Rancher UI** : Rancher provides full CRUD capability of creating and managing kubernetes cluster. + * Install rancher using Helm, update `hostname` in `rancher-values.yaml` and run the following command to install. + * ```sh + cd $K8_ROOT/observation/rancher-ui/ + helm repo add rancher-latest https://releases.rancher.com/server-charts/latest + helm repo update + helm install rancher rancher-latest/rancher \ + --namespace cattle-system \ + --create-namespace \ + -f rancher-values.yaml \ + --version=2.6.9 + ``` + * Login: + * Open Rancher page `https://rancher.org.net`. + * Get Bootstrap password using + + ```java + kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{ .data.bootstrapPassword|base64decode}}{{ "\n" }}' + ``` + * Assign a password. IMPORTANT: makes sure this password is securely saved and retrievable by Admin. +* **5.b.** [**Keycloak**](https://www.keycloak.org/) : Keycloak is an OAuth 2.0 compliant Identity Access Management (IAM) system used to manage the access to Rancher for cluster controls. + * ```sh + cd $K8_ROOT/observation/keycloak/ + ./install.sh + ``` + * `keycloak_client.json`: Used to create SAML client on Keycloak for Rancher integration. +* **5.c. Keycloak - Rancher Integration** + * Login as `admin` user in Keycloak and make sure an email id, and first name field is populated for `admin` user. This is important for Rancher authentication as given below. + * Enable authentication with Keycloak using the steps given [here](https://rancher.com/docs/rancher/v2.6/en/admin-settings/authentication/keycloak-saml/). + * In Keycloak add another Mapper for the rancher client (in Master realm) with following fields: + * Protocol: saml + * Name: username + * Mapper Type: User Property + * Property: username + * Friendly Name: username + * SAML Attribute Name: username + * SAML Attribute NameFormat: Basic + * Specify the following mappings in Rancher's Authentication Keycloak form: + * Display Name Field: givenName + * User Name Field: email + * UID Field: username + * Entity ID Field: [https://your-rancher-domain/v1-saml/keycloak/saml/metadata](https://your-rancher-domain/v1-saml/keycloak/saml/metadata) + * Rancher API Host: [https://your-rancher-domain](https://your-rancher-domain/) + * Groups Field: member +* **RBAC :** + * For users in Keycloak assign roles in Rancher - **cluster** and **project** roles. Under `default` project add all the namespaces. Then, to a non-admin user you may provide Read-Only role (under projects). + * If you want to create custom roles, you can follow the steps given [here](https://github.com/mosip/k8s-infra/blob/main/docs/create-custom-role.md). + * Add a member to cluster/project in Rancher: + * Give member name exactly as `username` in Keycloak + * Assign appropriate role like Cluster Owner, Cluster Viewer etc. + * You may create new role with fine grained acccess control. +* **Certificates expiry** + * In case you see certificate expiry message while adding users, on **local** cluster run these commands: + + [https://rancher.com/docs/rancher/v2.6/en/troubleshooting/expired-webhook-certificates/](https://rancher.com/docs/rancher/v2.6/en/troubleshooting/expired-webhook-certificates/) + +## 6. MOSIP K8s Cluster setup + +* Pre-requisites: + * Apps to be downloaded on users personal computer. + * [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) client version > 1.23.6 + * [helm](https://helm.sh/docs/intro/install/) client version > 3.8.2 and add `mosip` and `bitnami` repo as well. + * [eksctl](https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html) version 0.121.0. + * AWS account and credentials with permissions to create EKS cluster. + * AWS credentials in `~/.aws/` folder as given [here](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html). + * Save `~/.kube/config` file with another name. + * **IMPORTANT NOTE* : As part of EKS cluster creation steps your existing kubeconfig (`~/.kube/config`) file will be overridden. + * Save `.pem` file from AWS console and store it in `~/.ssh/` folder. (Generate a new one if you do not have this key file). + +* Setup mosip cluster + + * `cd $K8_ROOT/k8-cluster/csp/aws` + * Copy `cluster.config.sample` to `mosip.cluster.config`. + * Review and update the below mentioned parameters of `mosip.cluster.config` carefully. + * name : cluster name. + * region : AWS region for EKS cluster. + * version : "1.24" + * EC2 instance related details + * instanceName + * instanceType + * desiredcapacity + * volumeSize + * volumeType + * publicKeyName + * update the details of the subnets to be used from vpc + * Install + + `eksctl create cluster -f mosip.cluster.config` + + * Wait for the cluster creation to complete, generally it takes around 30 minutes to create or update cluster. + * Once EKS K8 cluster is ready below mentioned output will be displayed in the console screen.\ + `EKS cluster "my-cluster" in "region-code" region is ready` + * The config file for the new cluster will be created on `~/.kube/config` + * Make sure to backup and store the `~/.kube/config` with new name. e.g. `~/.kube/mosip-cluster.config`. + * Change file permission using below command:\ + `chmod 400 ~/.kube/mosip-cluster.config` + * Set the `KUBECONFIG` properly so that you can access the cluster.\ + `export KUBECONFIG=~/.kube/mosip-cluster.config` + * Test cluster access: + * `kubectl get nodes` + * Command will result in details of the nodes of the MOSIP cluster. + +## 7. Import Mosip Cluster into Rancher UI + +* Login as admin in Rancher console +* Select `Import Existing` for cluster addition. +* Select the `Generic` as cluster type to add. +* Fill the `Cluster Name` field with unique cluster name and select `Create`. +* You will get the kubectl commands to be executed in the kubernetes cluster. Copy the command and execute from your PC. (make sure your kube-config file is correctly set to Mosip cluster) + +```sh +eg. +kubectl apply -f https://rancher.e2e.mosip.net/v3/import/pdmkx6b4xxtpcd699gzwdtt5bckwf4ctdgr7xkmmtwg8dfjk4hmbpk_c-m-db8kcj4r.yaml +``` + +* Wait for few seconds after executing the command for the cluster to get verified. +* Your cluster is now added to the rancher management server. + +## 8. MOSIP K8 Cluster Global configmap, Storage Class, Ingress and Loadbalancer setup + +* **8.a. Global configmap**: Global configmap contains list of necesary details to be used throughout the namespaces of the cluster for common details. + * `cd $INFRA_ROOT/deployment/v3/external/` + * Copy `global_configmap.yaml.sample` to `global_configmap.yaml.` + * Update the domain names in `global_configmap.yaml` and run. + * `kubectl apply -f global_configmap.yaml` +* **8.b. Storage class setup**: + * Default storage class on EKS is `gp2` . `GP2` by default is in `Delete` mode which means if PVC is deleted, the underlying storage PV is also deleted. + * To enable volume expansion for the existing `gp2` storage class, modify the YAML configuration by adding `allowVolumeExpansion: true` to the `gp2` storage class configuration. + * `kubectl edit sc gp2` : to edit the yaml configuration. + * Create storage class `gp2-retain` by running `sc.yaml` for PV in Retain mode. Set the storage class as gp2-retain in case you want to retain PV. + * ```sh + cd $K8_ROOT/storage-class/ebs/ + kubectl apply -f gp2-sc.yaml + ``` + * We need the EBS driver for our storage class to work, follow the steps [here](https://www.stacksimplify.com/aws-eks/kubernetes-storage/install-aws-ebs-csi-driver-on-aws-eks-for-persistent-storage/) to setup EBS driver. + * Also we need EFS CSI driver for the regproc services, because EBS driver only supports RWO but we need RWX, follow these [steps](https://docs.aws.amazon.com/eks/latest/userguide/efs-csi.html) to setup EFS CSI driver. +* **8.c. Ingress and load balancer (LB)** : + * Ingress is not installed by default on EKS. We use Istio ingress gateway controller to allow traffic in the cluster. Two channels are created - public and internal. See [architecture](https://github.com/mosip/k8s-infra/blob/1.2.0.1-B1/README.md). + * Install [istioctl](https://istio.io/latest/docs/ops/diagnostic-tools/istioctl/#install-hahahugoshortcode-s2-hbhb) as given here in your system. + * Install ingress as given here: + * ```sh + cd $K8_ROOT/ingress/istio-mesh/loadbalancer/ + ./install.sh + ``` +* **8.d. Load Balancers setup for istio-ingress** + + * The above istio installation will automatically spawn an [Internal AWS Network Load Balancer (L4)](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-network-load-balancer.html). + * These may be also seen with + + ```sh + kubectl -n istio-system get svc + ``` + + * You may view them on AWS console in Loadbalancer section. + * TLS termination is supposed to be on LB. So all our traffic coming to ingress controller shall be HTTP. + * Obtain AWS TLS certificate as given [here](https://docs.aws.amazon.com/acm/latest/userguide/dns-validation.html) + * Add the certificates and 443 access to the LB listener. + * Update listener TCP->443 to **TLS->443** and point to the certificate of domain name that belongs to your cluster. + * Forward TLS->443 listner traffic to target group that corresponds to listener on port 80 of respective Loadbalancers. This is because after TLS termination the protocol is HTTP so we must point LB to HTTP port of ingress controller. + * Update health check ports of LB target groups to node port corresponding to port 15021. You can see the node ports with + * ```sh + kubectl -n istio-system get svc + ``` + * Enable Proxy Protocol v2 on target groups. + * Make sure all subnets are included in Availability Zones for the LB. Description --> Availability Zones --> Edit Subnets + * Make sure to delete the listeners for port 80 and 15021 from each of the loadbalancers as we restrict unsecured port 80 access over http. +* **8.e. DNS mapping** + * Initially all the services will be accesible only over the internal channel. + * Point all your domain names to internal LoadBalancers DNS/IP intially till testing is done. + * On AWS this may be done on Route 53 console. + * After Go live decision enable [public access](https://github.com/mosip/k8s-infra/blob/main/docs/public-access.md). +* **8.f. Check Overall if nginx and istio wiring is set correctly** + * Install httpbin: This utility docker returns http headers received inside the cluster. You may use it for general debugging - to check ingress, headers etc. + * ```sh + cd $K8_ROOT/utils/httpbin + ./install.sh + ``` + * To see what's reaching httpbin (example, replace with your domain name): + + ```sh + curl https://api-internal.sandbox.xyz.net/httpbin/get?show_env=true + Once public access is enabled also check this: + curl https://api.sandbox.xyz.net/httpbin/get?show_env=true + ``` + +## 9. Monitoring Module deployment + +* Prometheus and Grafana and Alertmanager tools are used for cluster monitoring. +* **Note** : This is optional for sandbox to be deployed in case monitoring is necesary and for production they can always go with alternate tools option. + * Select 'Monitoring' App from Rancher console -> _Apps & Marketplaces_. + * In Helm options, open the YAML file and disable Nginx Ingress. + * ```sh + ingressNginx: + enabled: false + ``` + * Click on `Install`. + +## 10. Alerting Setup + +* Alerting is part of cluster monitoring, where alert notifications are sent to the configured email or slack channel. +* **Note** : This is optional for sandbox to be deployed in case alerting is necesary and for production they can always go with alternate tools option. + * Monitoring should be deployed which includes deployment of prometheus, grafana and alertmanager. + * Create [slack incoming webhook](https://api.slack.com/messaging/webhooks). + * After setting slack incoming webhook update `slack_api_url` and `slack_channel_name` in `alertmanager.yml`. + * `cd $K8_ROOT/alerting/` + * `nano alertmanager.yml` + * Update + + ```sh + global: + resolve_timeout: 5m + slack_api_url: + ... + slack_configs: + - channel: '' + send_resolved: true + ``` + * Update `Cluster_name` in `patch-cluster-name.yaml`. + * `cd $K8_ROOT/alerting/` + * `nano patch-cluster-name.yaml` + * Update + + ```sh + spec: + externalLabels: + cluster: + ``` + * Install Default alerts along some of the defined custom alerts: + * ```sh + cd $K8_ROOT/alerting/ + ./install.sh + ``` + * Alerting is Installed. + +## 11. Logging Module Setup and Installation + +* Mosip uses Rancher Fluentd and elasticsearch to collect logs from all services and reflect the same in Kibana Dashboard. +* **Note** : This is optional for sandbox to be deployed in case logging is necesary and for production they can always go with alternate tools option. +* Install Rancher FluentD system : for screpping logs outs of all the microservices from Mosip k8 cluster. + * Install Logging from Apps and marketplace within the Rancher UI. + * Select Chart Version `100.1.3+up3.17.7` from Rancher console -> _Apps & Marketplaces_. +* Configure Rancher FluentD + * Create `clusteroutput` : + * `kubectl apply -f clusteroutput-elasticsearch.yaml` + * start `clusterFlow` + * `kubectl apply -f clusterflow-elasticsearch.yaml` +* Install elasticsearch, kibana and Istio addons + * `cd $K8_ROOT/logging` + * `./install.sh` +* set `min_age` in `elasticsearch-ilm-script.sh` and execute the same. `min_age` : is the min no. of days for which indices will be stored in elasticsearch. + * `cd $K8_ROOT/logging` + * `./elasticsearch-ilm-script.sh` +* Mosip provides set of Kibana Dashboards for checking logs and throughputs . + * Brief description of these dashboards are as follows: + * [01-logstash.ndjson](https://github.com/mosip/k8s-infra/blob/main/logging/dashboards/01-logstash.ndjson) contains the `logstash` _Index Pattern_ required by the rest of the dashboards. + * [02-error-only-logs.ndjson](https://github.com/mosip/k8s-infra/blob/main/logging/dashboards/02-error-only-logs.ndjson) contains a _Search_ dashboard which shows only the error logs of the services, called `MOSIP Error Logs` dashboard. + * [03-service-logs.ndjson](https://github.com/mosip/k8s-infra/blob/main/logging/dashboards/03-service-logs.ndjson) contains a _Search_ dashboard which show all logs of a particular service, called `MOSIP Service Logs` dashboard. + * [04-insight.ndjson](https://github.com/mosip/k8s-infra/blob/main/logging/dashboards/04-insight.ndjson) contains dashboards which show insights into MOSIP processes, like the number of UINs generated (total and per hr), the number of Biometric deduplications processed, number of packets uploaded etc, called `MOSIP Insight` dashboard. + * [05-response-time.ndjson](https://github.com/mosip/k8s-infra/blob/main/logging/dashboards/05-response-time.ndjson) contains dashboards which show how quickly different MOSIP Services are responding to different APIs, over time, called `Response Time` dashboard. + * Import dashboards: + * `cd $K8_ROOT/logging/dashboard` + * `./load_kibana_dashboards.sh ./dashboards ` + * View dashbords + * Open kibana dashboard from: `https://kibana.sandbox.xyz.net`. + * _Kibana_ --> _Menu_ (on top left) --> _Dashboard_ --> Select the dashboard. + +## 12. Mosip External Dependencies setup + +* External Dependencies: are set of external requirements needed for funtioning of MOSIP’s core services like DB, object store, hsm etc. +* ```sh + cd $INFRA_ROOT/deployment/v3/external/all + ./install-all.sh + ``` +* Check detailed installation instruction of all the [external componets](https://docs.mosip.io/1.2.0/deploymentnew/v3-installation/mosip-external-dependencies). + +## 13. MOSIP Modules Deployment + +* Now that all the Kubernetes cluster and external dependencies are already installed, you can continue with MOSIP service deployment. + * ```sh + cd $INFRA_ROOT/deployment/v3/mosip/all + ./install-all.sh + ``` +* Check the detailed MOSIP Modules Deployment [MOSIP Modular installation](https://docs.mosip.io/1.2.0/deploymentnew/v3-installation/mosip-modules-deployment) steps. + +## 14. API Testrig + +* MOSIP’s successfull deployment can be verified by comparing the results of api testrig with testrig benchmark. + * ```sh + cd $INFRA_ROOT/deployment/v3/apitestrig + ./install.sh + ``` + * When prompted input the hour of the day to execute the api-testrig. + * Daily api testrig cron jon will be executed at the very opted hour of the day. diff --git a/docs/setup/deploymentnew/v3-installation/1.2.1.0/on-premises-deployment-without-dns.md b/docs/setup/deploymentnew/v3-installation/1.2.1.0/on-premises-deployment-without-dns.md new file mode 100644 index 000000000..04cee4eef --- /dev/null +++ b/docs/setup/deploymentnew/v3-installation/1.2.1.0/on-premises-deployment-without-dns.md @@ -0,0 +1,1353 @@ +# On-Premises Deployment Without DNS + +## 1. Wireguard + +* Wireguard bastian server provides secure private channel to access MOSIP cluster. +* Bastian server restricts public access, and enables access to only those clients who have their public key listed in Wireguard server. +* Bastion server listens on UDP port 51820. +* In case you already have VPN configured to access nodes privately please skip Wireguard installation and continue to use the same VPN. + +**Setup Wireguard VM and wireguard bastion server** + +* Create a Wireguard server VM with mentioned '[**Hardware and Network Requirements**'](pre-requisites.md). +* Open required ports in the Bastian server VM. + + * `cd $K8_ROOT/wireguard/` + * Create copy of `hosts.ini.sample` as `hosts.ini` and update the required details for wireguard VM. + * `cp hosts.ini.sample hosts.ini` + + > Note : + > + > * Remove `[Cluster]` complete section from copied `hosts.ini` file. + > * Add below mentioned details: + > * ansible\_host : public IP of Wireguard Bastion server. eg. 100.10.20.56 + > * ansible\_user : user to be used for installation. In this ref-impl we use Ubuntu user. + > * ansible\_ssh\_private\_key\_file : path to pem key for ssh to wireguard server. eg. `~/.ssh/wireguard-ssh.pem`![hosts.ini](../../../../.gitbook/assets/wireguard-hosts-ini.png) + + * Execute ports.yml to enable ports on VM level using ufw: + * `ansible-playbook -i hosts.ini ports.yaml` + +> Note: +> +> * Permission of the pem files to access nodes should have 400 permission. `sudo chmod 400 ~/.ssh/privkey.pem` +> * These ports are only needed to be opened for sharing packets over UDP. +> * Take necessary measure on firewall level so that the Wireguard server can be reachable on 51820/udp. + +* Install docker + * execute docker.yml to install docker and add user to docker group: + + ``` + ansible-playbook -i hosts.ini docker.yaml + ``` +* Setup Wireguard server + * SSH to wireguard VM + * `ssh -i ubuntu@` + * Create directory for storing wireguard config files.`mkdir -p wireguard/config` + * Install and start wireguard server using docker as given below: + + ``` + sudo docker run -d \ + --name=wireguard \ + --cap-add=NET_ADMIN \ + --cap-add=SYS_MODULE \ + -e PUID=1000 \ + -e PGID=1000 \ + -e TZ=Asia/Calcutta \ + -e PEERS=30 \ + -p 51820:51820/udp \ + -v /home/ubuntu/wireguard/config:/config \ + -v /lib/modules:/lib/modules \ + --sysctl="net.ipv4.conf.all.src_valid_mark=1" \ + --restart unless-stopped \ + ghcr.io/linuxserver/wireguard + ``` + +> Note: +> +> * Increase the no. of peers above in case more than 30 wireguard client confs (-e PEERS=30) are needed. +> * Change the directory to be mounted to wireguard docker as per need. All your wireguard confs will be generated in the mounted directory (`-v /home/ubuntu/wireguard/config:/config`). + +**Setup Wireguard Client in your PC** + +* Install Wireguard client in your PC using [steps](https://www.wireguard.com/install/). +* Assign `wireguard.conf`: + + * SSH to the wireguard server VM. + * `cd /home/ubuntu/wireguard/config` + * assign one of the PR for yourself and use the same from the PC to connect to the server. + + * create `assigned.txt` file to assign the keep track of peer files allocated and update everytime some peer is allocated to someone. + + ``` + peer1 : peername + peer2 : xyz + ``` + + * use `ls` cmd to see the list of peers. + * get inside your selected peer directory, and add mentioned changes in `peer.conf`: + * `cd peer1` + * `nano peer1.conf` + + * Delete the DNS IP. + * Update the allowed IP's to subnets CIDR ip . e.g. 10.10.20.0/23. + + > Note: + > + > * CIDR Range will be shared by the Infra provider. + > * Make sure all the nodes are covered in the provided CIDR range. (nginx server, K8 cluster nodes for observation as well as mosip). + * Share the updated `peer.conf` with respective peer to connect to wireguard server from Personel PC. + * Add `peer.conf` in your PC’s `/etc/wireguard` directory as `wg0.conf`. + * Start the wireguard client and check the status: + + ``` + sudo systemctl start wg-quick@wg0 + sudo systemctl status wg-quick@wg0 + ``` +* Once connected to wireguard, you should be now able to login using private IP’s. + +## 2. Observation K8s Cluster setup and configuration + +* Install all the required tools mentioned in '[Personel Computer Setup](pre-requisites.md#personal-computer-requirements)' section. +* Setup Observation Cluster node VM’s hardware and network configuration as per ([requirements](pre-requisites.md#hardware-requirements)). +* Setup passwordless SSH into the cluster nodes via pem keys. (Ignore if VM’s are accessible via pem’s). + * Generate keys on your PC`ssh-keygen -t rsa` + * Copy the keys to remote observation node VM’s`ssh-copy-id @` + * SSH into the node to check password-less SSH`ssh -i ~/.ssh/ @` + +> Note: +> +> * Make sure the permission for `privkey.pem` for ssh is set to 400. + +* Open ports and install docker on Observation K8 Cluster node VM’s. + + * `cd $K8_ROOT/k8-cluster/on-prem/rke2/ansible` + * Copy `hosts.ini.sample` to `hosts.ini` and update required details. + * `cp hosts.ini.sample hosts.ini` + + > Note: + > + > * Ensure you are inside `ansible` directory as mentioned above. + > * Make sure incase `hosts.ini` is already used for other cluster creation the same is saved with new name and new `hosts.ini` is created. + > * ansible\_host : internal IP of nodes. eg. 100.10.20.56, 100.10.20.57 ... + > * ansible\_user : user to be used for installation. In this ref-implementation we use Ubuntu user. + > * ansible\_ssh\_private\_key\_file : path to pem key for ssh to wireguard server. eg. `~/.ssh/nodes-ssh.pem`![hosts.ini](../../../../.gitbook/assets/nodes-hosts-ini.png) + + * Execute `ports.yml` to enable ports on VM level using ufw: + * `ansible-playbook -i hosts.ini ports.yaml` + * Disable swap in cluster nodes. (Ignore if swap is already disabled) + * `ansible-playbook -i hosts.ini swap.yaml` + + > Caution: Always verify swap status with `swapon --show` before running the playbook to avoid unnecessary operations. + * execute `docker.yml` to install docker and add user to docker group: + * `ansible-playbook -i hosts.ini docker.yaml` +* RKE2 cluster creation + * Cluster creation using RKE2 and ansible script: + ```bash + ansible-playbook -i hosts.ini main.yaml + ``` + * After the successful creation of the RKE2 cluster, the kubeconfig file will be saved in the [k8s-infra/k8-cluster/on-prem/rke2/ansible/playbook/](k8s-infra/k8-cluster/on-prem/rke2/ansible/playbook/) directory as `{{ cluster_domain }}-{{ inventory_hostname }}.yaml`. Store it securely: + * As part of the Kubernetes creation process, a `kubeconfig` file has been created and written at `kube_config_cluster.yml`, which can be used to start interacting with your Kubernetes cluster. + * Copy the kubeconfig files to `.kube` directory and change the permission. + + ``` + chmod 400 $HOME/.kube/_config + ``` + + * To access the cluster using `kubeconfig` file use any one of the below method: + * `cp $HOME/.kube/_config $HOME/.kube/config`\ + **Alternatively** + * `export KUBECONFIG="$HOME/.kube/_config` + * Test cluster access: + * `kubectl get nodes` + * Command will result in details of the nodes of the Observation cluster. + * `cluster.rkestate`: The [Kubernetes Cluster State file](https://rancher.com/docs/rke/latest/en/installation/#kubernetes-cluster-state), this file contains credentials for full access to the cluster. + * In case not having Public DNS system add the custom DNS configuration for all the hostnames to be used for testing in Cluster's coredns configmap. + * Check whether coredns pods are up and running in your cluster via the below command: + + ``` + kubectl -n kube-system get pods -l k8s-app=kube-dns + ``` + * To update the coredns configmap, use the below command. + + ``` + kubectl -n kube-system edit cm coredns + ``` + + > Note:\ + > Default editor in WSL and Ubuntu is `vi`. Incase not familiar with `vi` change the editor to your prefered one:`export EDITOR=` + * Update the IP address and domain name in the below DNS hosts template and add it in the coredns configmap Corefile key in the kube-system namespace. + + ``` + hosts { + rancher.xyz.net keycloak.xyz.net + fallthrough + } + ``` + * Check whether the DNS changes are correctly updated in coredns configmap. + + ``` + kubectl -n kube-system get cm coredns -o yaml + ``` + * Restart the `coredns` pod in the `kube-system` namespace. + + ``` + kubectl -n kube-system rollout restart deploy coredns coredns-autoscaler + ``` + * Check status of coredns restart. + + ``` + kubectl -n kube-system rollout status deploy coredns + kubectl -n kube-system rollout status coredns-autoscaler + ``` + +## 3. Observation K8s Cluster Ingress and Storage class setup + +Once the rancher cluster is ready, we need ingress and storage class to be set for other applications to be installed. + +### 3.a. [Nginx Ingress Controller](https://kubernetes.github.io/ingress-nginx/deploy/): used for ingress in rancher cluster. + +``` +cd $K8_ROOT/rancher/on-prem +helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx +helm repo update +helm install ingress-nginx ingress-nginx/ingress-nginx \ +--namespace ingress-nginx \ +--version 4.0.18 \ +--create-namespace \ +-f ingress-nginx.values.yaml +``` + +> Note: +> +> * This will install ingress in namespace `ingress-nginx` of Observation cluster. +> * Crosscheck using below mentioned command: +> * `kubectl get all -n ingress-nginx` +> * Command should result with list of all the pods, deployments etc in ingress nginx namespace. + +### 3.b. Storage classes + +Multiple storage classes options are available for onprem K8's cluster. In this reference deployment will continue to use NFS as a storage class. + +* Move to nfs directory in your personel computer. + + ``` + cd $K8_ROOT/mosip/nfs + ``` +* Create a copy of hosts.ini.sample as hosts.ini. + + ``` + cp hosts.ini.sample hosts.ini + ``` +* Update the NFS machine details in `hosts.ini` file. + + > Note : + > + > * Add below mentioned details: + > * ansible\_host : internal IP of NFS server. eg. 10.12.23.21 + > * ansible\_user : user to be used for installation, in this ref-impl we use Ubuntu user. + > * ansible\_ssh\_private\_key\_file : path to pem key for ssh to wireguard server. eg. `~/.ssh/wireguard-ssh.pem` ![hosts.ini](../../../../.gitbook/assets/nfs-hosts-ini.png). +* Make sure Kubeconfig file is set correctly to point to required Observation cluster. + + ``` + kubectl config view + ``` + + Note: + + * Output should show the cluster name to confirm you are pointing to right kubernetes cluster. + * If not pinting to right K8 cluster change the kubeconfig to connect to right K8 cluster. +* Enable firewall with required ports: + + ``` + ansible-playbook -i ./hosts.ini nfs-ports.yaml + ``` +* SSH to the nfs node: + + ``` + ssh -i ~/.ssh/nfs-ssh.pem ubuntu@ + ``` +* Clone `k8s-infra` repo in nginx VM: + + ``` + git clone https://github.com/mosip/k8s-infra -b v1.2.0.1 + ``` +* Move to the nfs directory: + + ``` + cd /home/ubuntu/k8s-infra/mosip/nfs/ + ``` +* Execute script to install nfs server: + + ``` + sudo ./install-nfs-server.sh + ``` + + Note: + + > * Script prompts for below mentioned user inputs: + > + > ``` + > ..... + > Please Enter Environment Name: + > ..... + > ..... + > ..... + > [ Export the NFS Share Directory ] + > exporting *:/srv/nfs/mosip/ + > NFS Server Path: /srv/nfs/mosip/ + > ``` + > + > * envName: env name eg. dev/qa/uat... +* Switch to your personel computer and excute below mentioned commands: + + ``` + cd $K8_ROOT/mosip/nfs/ + + ./install-nfs-client-provisioner.sh + ``` + + Note: + + > * Script prompts for: + > * NFS Server: NFS server ip for persistence. + > * NFS Path : NFS path for storing the persisted data. eg. /srv/nfs/mosip/ +* Post installation check: + * Check status of NFS Client Provisioner. + + ``` + kubectl -n nfs get deployment.apps/nfs-client-provisioner + ``` + * Check status of nfs-client storage class. + + ``` + kubectl get storageclass + NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE + longhorn (default) driver.longhorn.io Delete Immediate true 57d + nfs-client cluster.local/nfs-client-provisioner Delete Immediate true 40s + ``` + * Set `nfs-client` as default storage class. + + ``` + kubectl patch storageclass nfs-client -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}' + ``` + +## 4. Setting up nginx server for Observation K8s Cluster + +### 4.a. SSL Certificate setup for TLS termination + +* For Nginx server setup we need ssl certificate, add the same into Nginx server. +* SSL certificates can be generated in multiple ways, either via lets encrypt if you have public DNS or via openssl certs when you don't have Public DNS. + * Openssl : Generate wildcard ssl certificate using openssl in case you don't have public DNS using below steps. (Ensure to use this only in development env, not suggested for Production env). + * Generate a self-signed certificate for your domain, such as \*.sandbox.xyz.net. + * SSH to the nginx VM. + + ``` + ssh -i ~/.ssh/ ubuntu@ + ``` + * Execute the following command to generate a self-signed SSL certificate. Prior to execution, kindly ensure to update environmental variables & rancher domain passed to openssl command: + + ``` + mkdir -p /etc/ssl/certs/ + export VALIDITY=700 + export COUNTRY=IN + export STATE=KAR + export LOCATION=BLR + export ORG=MOSIP + export ORG_UNIT=MOSIP + export COMMON_NAME=*.xyz.net + ``` + + ``` + + openssl req -x509 -nodes -days $VALIDITY \ + -newkey rsa:2048 -keyout /etc/ssl/private/tls.key -out /etc/ssl/certs/tls.crt \ + -subj "/C=$COUNTRY/ST=$STATE/L=$LOCATION/O=$ORG/OU=$ORG_UNIT/CN=$COMMON_NAME" \ + -addext "subjectAltName = DNS:rancher.xyz.net, DNS:*.xyz.net" + ``` + * Above command will generate certs in below specified location. Use it when prompted during nginx installation. + * fullChain path: `/etc/ssl/certs/tls.crt`. + * privKey path: `/etc/ssl/private/tls.key`. + +### 4.b. Install Nginx : + +* Move to nginx directory in your local +* `cd $K8_ROOT/mosip/on-prem/nginx/` +* Open required ports : + * Use any editor to create new `hosts.ini` file: + * ``` + nano hosts.ini + ``` + * Add below mentioned lines with updated details of nginx server to the `hosts.ini` and save. + + ``` + [nginx] + node-nginx ansible_host= ansible_user=root ansible_ssh_private_key_file= + ``` + * Execute below mentoned command to open required ports: + + ``` + ansible-playbook -i hosts.ini mosip/on-prem/nginx/nginx_ports.yaml + ``` +* Login to nginx server node. + + ``` + ssh -i ~/.ssh/ ubuntu@ + ``` + + * Clone [k8s-infra](https://github.com/mosip/k8s-infra) + + ``` + cd $K8_ROOT/rancher/on-prem/nginx + sudo ./install.sh + ``` + * Provide below mentioned inputs as and when promted + * Rancher nginx ip : internal ip of the nginx server VM. + * SSL cert path : path of the ssl certificate to be used for ssl termination. + * SSL key path : path of the ssl key to be used for ssl termination. + * Cluster node IPs : IPs of the rancher cluster node +* Post installation check: + * `sudo systemctl status nginx` +* Steps to Uninstall nginx (in case required). `sudo apt purge nginx nginx-common`. +* DNS mapping: + * Once nginx server is installed successfully, create DNS mapping for rancher cluster related domains as mentioned in DNS requirement section. (rancher.org.net, keycloak.org.net)\\ + * Add DNS entries in local hosts file of your system. + * For example: `/etc/hosts` files for Linux machines. + * `nano /etc/hosts` + * Update the domain and IP address. + + ``` + rancher.xyz.net keycloak.xyz.net + ``` + +## 5. Observation K8's Cluster Apps Installation + +### 5.a. Rancher UI + +* Rancher provides full CRUD capability of creating and managing kubernetes cluster. +* Install rancher using Helm, update `hostname`, & add `privateCA` to `true` in `rancher-values.yaml`, and run the following command to setup secrets for installation. + +``` + cd $K8_ROOT/rancher/rancher-ui` + helm repo add rancher https://releases.rancher.com/server-charts/stable` + helm repo update` + kubectl create ns cattle-system` +``` + +* Create a secret containing the observation nginx self-signed public certificate (i.e. `tls.crt` ) generated in [openssl section](on-prem-without-dns.md#setting-up-nginx-server-for-observation-k8s-cluster). + +``` +kubectl -n cattle-system create secret generic tls-ca --from-file=cacerts.pem= +``` + +* USe below command to install Rancher UI: + +``` +helm install rancher rancher/rancher --version 2.6.9 \ +--namespace cattle-system \ +--create-namespace \ +--set privateCA=true \ +-f rancher-values.yaml +``` + +* Login: + + * Connect to Wireguard Bastion. (Incase using WSL make sure to connect using Windows Wireguard instead of WSL one). + * Open Rancher page. + * Get Bootstrap password using + + ``` + kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{ .data.bootstrapPassword|base64decode}}{{ "\n" }}' + ``` + +> Note: Assign a password. IMPORTANT: makes sure this password is securely saved and retrievable by Admin. + +### 5.b. Keycloak + +* [Keycloak](https://www.keycloak.org/): Keycloak is an OAuth 2.0 compliant Identity Access Management (IAM) system used to manage the access to Rancher for cluster controls. + +``` +cd $K8_ROOT/rancher/keycloak +./install.sh +``` + +* Post installation access the keycloak using `iam.mosip.net` and get the credentials as per the post installation steps definedAdd commentMore actions ![keycloak-access](../../../../.gitbook/assets/keycloak-login.png). + +### 5.c. Keycloak - Rancher UI Integration + +* Login as `admin` user in Keycloak and make sure `email` and `firstName` fields are populated for the admin user. These are required for Rancher authentication to work properly. +* In Keycloak (in the `master` realm), create a new client with the following values: + * `Client ID`: `https:///v1-saml/keycloak/saml/metadata` + * `Client Protocol`: `saml` + * `Root URL`: _(leave empty)_ + * After saving, configure the client with: + * `Name`: `rancher` + * `Enabled`: `ON` + * `Login Theme`: `keycloak` + * `Sign Documents`: `ON` + * `Sign Assertions`: `ON` + * `Encrypt Assertions`: `OFF` + * `Client Signature Required`: `OFF` + * `Force POST Binding`: `OFF` + * `Front Channel Logout`: `OFF` + * `Force Name ID Format`: `OFF` + * `Name ID Format`: `username` + * `Valid Redirect URIs`: `https:///v1-saml/keycloak/saml/acs` + * `IDP Initiated SSO URL Name`: `IdPSSOName` + * Save the client +* In the same client, go to the `Mappers` tab and create the following: + * **Mapper 1**: + * `Protocol`: `saml` + * `Name`: `username` + * `Mapper Type`: `User Property` + * `Property`: `username` + * `Friendly Name`: `username` + * `SAML Attribute Name`: `username` + * `SAML Attribute NameFormat`: `Basic` + * **Mapper 2**: + * `Protocol`: `saml` + * `Name`: `groups` + * `Mapper Type`: `Group List` + * `Group Attribute Name`: `member` + * `Friendly Name`: (Leave empty) + * `SAML Attribute NameFormat`: `Basic` + * `Single Group Attribute`: `ON` + * `Full Group Path`: `OFF` + * Click `Add Builtin` → select all → `Add Selected` +* Download the Keycloak SAML descriptor XML file from: + * `https:///auth/realms/master/protocol/saml/descriptor` +* Generate a self-signed SSL certificate and private key (if not already available): + + ```bash + openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout myservice.key -out myservice.cert + + ``` +* **Rancher UI Configuration** + + * **In Rancher UI, go to :** + + `Users & Authentication` → `Auth Providers` → `Keycloak (SAML)` + + * **Configure the fields as follows :** + + `Display Name Field`: `givenName` + + `User Name Field`: `email` or `uid` + + `UID Field`: `username` + + `Groups Field`: `member` + + `Entity ID Field`: (Leave empty) + + `Rancher API Host`: `https://` + * **Upload the following files:** + + `Private Key`: `myservice.key` + + `Certificate`: `myservice.cert` + + `SAML Metadata XML`: (from the Keycloak descriptor link) + + * Click **Enable** to activate Keycloak authentication. + * **After successful integration, Rancher users should be able to log in using their Keycloak** + +### 5.d. RBAC for Rancher using Keycloak + +* For users in Keycloak assign roles in Rancher - **cluster** and **project** roles. Under `default` project add all the namespaces. Then, to a non-admin user you may provide Read-Only role (under projects). +* In case your cluster is used by multiple peoples with multiple roles and you want to create custom roles, you can follow the steps given [here](https://github.com/mosip/k8s-infra/blob/v1.2.0.1/docs/create-custom-role.md). +* Add a member to cluster/project in Rancher: + * Navigate to RBAC cluster members + * Add member name exactly as `username` in Keycloak + * Assign appropriate role like Cluster Owner, Cluster Viewer etc. + * You may create new role with fine grained acccess control. +* Add group to to cluster/project in Rancher: + * Navigate to RBAC cluster members + * Click on `Add` and select a group from the displayed drop-down. + * Assign appropriate role like Cluster Owner, Cluster Viewer etc. + * To add groups, the user must be a member of the group. +* Creating a Keycloak group involves the following steps: + * Go to the "Groups" section in Keycloak and create groups with default roles. + * Navigate to the "Users" section in Keycloak, select a user, and then go to the "Groups" tab. From the list of groups, add the user to the required group. + +## 6. MOSIP K8s Cluster setup + +* Pre-requisites: +* Install all the required tools mentioned in Pre-requisites for PC. + * kubectl + * helm + +``` +helm repo add bitnami https://charts.bitnami.com/bitnami +helm repo add mosip https://mosip.github.io/mosip-helm +``` + +* ansible +* rke (version 1.3.10) +* Setup MOSIP K8 Cluster node VM’s as per '[**Hardware and Network Requirements**'](pre-requisites.md). +* Run `env-check.yaml` to check if cluster nodes are fine and don't have known issues in it. + * cd $K8\_ROOT/rancher/on-prem + * create copy of `hosts.ini.sample` as `hosts.ini` and update the required details for MOSIP k8 cluster nodes. + + * `cp hosts.ini.sample hosts.ini` + + > Note: + > + > * Ensure you are inside `on-prem` directory as mentioned above. + > * ansible\_host : internal IP of nodes. eg. 100.10.20.56, 100.10.20.57 ... + > * ansible\_user : user to be used for installation. In this ref-implementation we use Ubuntu user. + > * ansible\_ssh\_private\_key\_file : path to pem key for ssh to wireguard server. eg. `~/.ssh/nodes-ssh.pem`![hosts.ini](../../../../.gitbook/assets/nodes-hosts-ini.png) + + * `ansible-playbook -i hosts.ini env-check.yaml` + * This ansible checks if localhost mapping is already present in `/etc/hosts` file in all cluster nodes, if not it adds the same. +* Setup passwordless ssh into the cluster nodes via pem keys. (Ignore if VM’s are accessible via pem’s). + * Generate keys on your PC + * `ssh-keygen -t rsa` + * Copy the keys to remote rancher node VM’s: + * `ssh-copy-id @` + * SSH into the node to check password-less SSH + * `ssh -i ~/.ssh/ @` + * Rancher UI : (deployed in Observation K8 cluster). +* Open ports and Install docker on MOSIP K8 Cluster node VM’s. + * `cd $K8_ROOT/mosip/on-prem` + * create copy of `hosts.ini.sample` as `hosts.ini` and update the required details for wireguard VM. + * `cp hosts.ini.sample hosts.ini` + * Update `vpc_ip` variable in `ports.yaml` with `vpc CIDR ip` to allow access only from machines inside same vpc. + + > Note: + > + > * CIDR Range will be shared by the Infra provider. + > * Make sure all the nodes are covered in the provided CIDR range. (nginx server, K8 cluster nodes for observation as well as mosip). + * execute `ports.yml` to enable ports on VM level using ufw: + * `ansible-playbook -i hosts.ini ports.yaml` + * Disable swap in cluster nodes. (Ignore if swap is already disabled) + + * `ansible-playbook -i hosts.ini swap.yaml` + + > Caution: Always verify swap status with `swapon --show` before running the playbook to avoid unnecessary operations. + * execute `docker.yml` to install docker and add user to docker group: + * ansible-playbook -i hosts.ini docker.yaml +* Creating RKE Cluster Configuration file + * rke config + * Command will prompt for nodal details related to cluster, provide inputs w.r.t below mentioned points: + + * `SSH Private Key Path` : + * `Number of Hosts`: + * `SSH Address of host` : + * `SSH User of host` : + + ``` + Is host () a Control Plane host (y/n)? [y]: y + Is host () a Worker host (y/n)? [n]: y + Is host () an etcd host (y/n)? [n]: y + ``` + + * Make all the nodes Worker `host` by default. + * To create an HA cluster, specify more than one host with role `Control Plane` and `etcd host`. + * `Network Plugin Type` : Continue with canal as default network plugin. + * For rest for other configuration opt the required or default value. +* As result of rke config command `cluster.ymlfile` will be generated inside same directory, update the below mentioned fields: + * `nano cluster.yml` + * Remove the default Ingress install + + ``` + ingress: + provider: none + ``` + * Add the name of the kubernetes cluster + + ``` + cluster_name: + ``` + * For production deployments edit the `cluster.yml`, according to this [RKE Cluster Hardening Guide](https://github.com/mosip/k8s-infra/blob/v1.2.0.1-B1/docs/rke-cluster-hardening.md). +* Setup up the cluster: + * Once `cluster.yml` is ready, you can bring up the kubernetes cluster using simple command. + * This command assumes the `cluster.yml` file is in the same directory as where you are running the command. + + * rke up + + ``` + INFO[0000] Building Kubernetes cluster + INFO[0000] [dialer] Setup tunnel for host [10.0.0.1] + INFO[0000] [network] Deploying port listener containers + INFO[0000] [network] Pulling image [alpine:latest] on host [10.0.0.1] + ... + INFO[0101] Finished building Kubernetes cluster successfully + ``` + * The last line should read `Finished building Kubernetes cluster successfully` to indicate that your cluster is ready to use. + * Copy the kubeconfig files + + ``` + cp kube_config_cluster.yml $HOME/.kube/_config + chmod 400 $HOME/.kube/_config + ``` + * To access the cluster using kubeconfig filr use any one of the below method: + * `cp $HOME/.kube/_config $HOME/.kube/config` + +**Alternatively** + +``` +* `export KUBECONFIG="$HOME/.kube/_config` +``` + +* Test cluster access: + * `kubect get nodes` + * Command will result in details of the nodes of the rancher cluster. +* Save Your files + * Save a copy of the following files in a secure location, they are needed to maintain, troubleshoot and upgrade your cluster.: + * `cluster.yml`: The RKE cluster configuration file. + * `kube_config_cluster.yml`: The [Kubeconfig file](https://rke.docs.rancher.com/kubeconfig) for the cluster, this file contains credentials for full access to the cluster. + * `cluster.rkestate`: The [Kubernetes Cluster State file](https://rke.docs.rancher.com/installation#kubernetes-cluster-state), this file contains credentials for full access to the cluster. +* In case not having Public DNS system add the custom DNS configuration for all the hostnames to be used for testing in Cluster's coredns configmap. + * Check whether coredns pods are up and running in your cluster via the below command: + + ``` + kubectl -n kube-system get pods -l k8s-app=kube-dns + ``` + * To update the coredns configmap, use the below command. + + ``` + kubectl -n kube-system edit cm coredns + ``` + + > Note:\ + > Default editor in WSL and Ubuntu is `vi`. Incase not familiar with `vi` change the editor to your prefered one:`export EDITOR=` + * Update the IP address and domain name in the below DNS hosts template and add it in the coredns configmap Corefile key in the kube-system namespace. + + ``` + hosts { + api.sandbox.xyz.net resident.sandbox.xyz.net esignet.sandbox.xyz.net prereg.sandbox.xyz.net healthservices.sandbox.xyz.net + sandbox.xyz.net api-internal.sandbox.xyz.net activemq.sandbox.xyz.net kibana.sandbox.xyz.net regclient.sandbox.xyz.net admin.sandbox.xyz.net minio.sandbox.xyz.net iam.sandbox.xyz.net kafka.sandbox.xyz.net postgres.sandbox.xyz.net pmp.sandbox.xyz.net onboarder.sandbox.xyz.net smtp.sandbox.xyz.net compliance.sandbox.xyz.net + + ## Observation + rancher.xyz.net keycloak.xyz.net + fallthrough + } + ``` + * Check whether the DNS changes are correctly updated in coredns configmap. + + ``` + kubectl -n kube-system get cm coredns -o yaml + ``` + * Restart the `coredns` pod in the `kube-system` namespace. + + ``` + kubectl -n kube-system rollout restart deploy coredns coredns-autoscaler + ``` + * Check status of coredns restart. + + ``` + kubectl -n kube-system rollout status deploy coredns + kubectl -n kube-system rollout status coredns-autoscaler + ``` + + > Note: + > + > * Since this deployment is without proper DNS and using self signed certificate please be informed that while accessing the UI components in browsr the sites will be in `Not-secure` state. + +## 7. MOSIP K8 Cluster Global configmap, Ingress and Storage Class setup + +### 7.a. Global configmap: + +* Global configmap Contains the list of neccesary details to be used throughout the namespaces of the cluster for common details. +* `cd $K8_ROOT/mosip` +* Copy `global_configmap.yaml.sample` to `global_configmap.yaml`. +* Update the domain names in `global_configmap.yaml` and run. +* `kubectl apply -f global_configmap.yaml` + +### 7.b. [Istio](https://istio.io/) Ingress setup: + +* It is a service mesh for the MOSIP K8 cluster which provides transparent layers on top of existing microservices along with powerful features enabling a uniform and more efficient way to secure, connect, and monitor services. + * `cd $K8_ROOT/mosip/on-prem/istio` + * `./install.sh` + * This will bring up all the Istio components and the Ingress Gateways. + * Check Ingress Gateway services: + + * `kubectl get svc -n istio-system` + + > Note: Response should contain service names as mentioned below. + > + > * `istio-ingressgateway`: external facing istio service. + > * `istio-ingressgateway-internal`: internal facing istio service. + > * `istiod`: Istio daemon for replicating the changes to all envoy filters. + +### 7.c. Storage classes + +Multiple storage classes options are available for onprem K8's cluster. In this reference deployment will continue to use NFS as a storage class. + +* Move to nfs directory in your personel computer. + + ``` + cd $K8_ROOT/mosip/nfs + ``` +* Create a copy of hosts.ini.sample as hosts.ini. + + ``` + cp hosts.ini.sample hosts.ini + ``` +* Update the NFS machine details in `hosts.ini` file. + + > Note : + > + > * Add below mentioned details: + > * ansible\_host : internal IP of NFS server. eg. 10.12.23.21 + > * ansible\_user : user to be used for installation, in this ref-impl we use Ubuntu user. + > * ansible\_ssh\_private\_key\_file : path to pem key for ssh to wireguard server. eg. `~/.ssh/wireguard-ssh.pem` ![hosts.ini](../../../../.gitbook/assets/nfs-hosts-ini.png). +* Make sure Kubeconfig file is set correctly to point to required mosip cluster. + + ``` + kubectl config view + ``` + + Note: + + * Output should show the cluster name to confirm you are pointing to right kubernetes cluster. + * If not pinting to right K8 cluster change the kubeconfig to connect to right K8 cluster. +* Enable firewall with required ports: + + ``` + ansible-playbook -i ./hosts.ini nfs-ports.yaml + ``` +* SSH to the nfs node: + + ``` + ssh -i ~/.ssh/nfs-ssh.pem ubuntu@ + ``` +* Clone `k8s-infra` repo in nginx VM: + + ``` + git clone https://github.com/mosip/k8s-infra -b v1.2.0.1 + ``` +* Move to the nfs directory: + + ``` + cd /home/ubuntu/k8s-infra/mosip/nfs/ + ``` +* Execute script to install nfs server: + + ``` + sudo ./install-nfs-server.sh + ``` + + Note: + + > * Script prompts for below mentioned user inputs: + > + > ``` + > ..... + > Please Enter Environment Name: + > ..... + > ..... + > ..... + > [ Export the NFS Share Directory ] + > exporting *:/srv/nfs/mosip/ + > NFS Server Path: /srv/nfs/mosip/ + > ``` + > + > * envName: env name eg. dev/qa/uat... +* Switch to your personel computer and excute below mentioned commands: + + ``` + cd $K8_ROOT/mosip/nfs/ + + ./install-nfs-client-provisioner.sh + ``` + + Note: + + > * Script prompts for: + > * NFS Server: NFS server ip for persistence. + > * NFS Path : NFS path for storing the persisted data. eg. /srv/nfs/mosip/ +* Post installation check: + * Check status of NFS Client Provisioner. + + ``` + kubectl -n nfs get deployment.apps/nfs-client-provisioner + ``` + * Check status of nfs-client storage class. + + ``` + kubectl get storageclass + NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE + longhorn (default) driver.longhorn.io Delete Immediate true 57d + nfs-client cluster.local/nfs-client-provisioner Delete Immediate true 40s + ``` + +## 8. Import MOSIP Cluster into Rancher UI + +* Login as admin in Rancher console +* Select `Import` Existing for cluster addition. +* Select `Generic` as cluster type to add. +* Fill the `Cluster Name` field with unique cluster name and select `Create`. +* You will get the kubectl commands to be executed in the kubernetes cluster. Copy the command and execute from your PC (make sure your `kube-config` file is correctly set to MOSIP cluster). + + ``` + e.g.: + kubectl apply -f https://rancher.e2e.mosip.net/v3/import/pdmkx6b4xxtpcd699gzwdtt5bckwf4ctdgr7xkmmtwg8dfjk4hmbpk_c-m-db8kcj4r.yaml + ``` +* Wait for few seconds after executing the command for the cluster to get verified. +* Your cluster is now added to the rancher management server. + +## 9. MOSIP K8 cluster Nginx server setup + +### 9.a. Openssl certificates creation + +* For Nginx server setup, we need ssl certificate, add the same into Nginx server. +* SSL certificates can be generated in multiple ways. Either via lets encrypt if you have public DNS or via openssl certs when you don't have Public DNS. + * Openssl : Generate wildcard ssl certificate using openssl in case you don't have public DNS using below steps. (Ensure to use this only in development env, not suggested for Production env). + * SSH to the nginx VM. + + ``` + ssh -i ~/.ssh/ ubuntu@ + ``` + * Install docker on nginx node. + + ``` + sudo apt-get update --fix-missing + sudo apt install docker.io -y + sudo systemctl restart docker + ``` + * Generate a self-signed certificate for your domain, such as `*.sandbox.xyz.net`. + * Execute the following command to generate a self-signed SSL certificate. Prior to execution, kindly ensure that the environmental variables passed to the OpenSSL Docker container have been properly updated: + * ``` + docker volume create --name gensslcerts --opt type=none --opt device=/etc/ssl --opt o=bind docker run -it --mount type=volume,src='gensslcerts',dst=/home/mosip/ssl,volume-driver=local \ -e VALIDITY=700 \ -e COUNTRY=IN \ -e STATE=KAR \ -e LOCATION=BLR \ -e ORG=MOSIP \ -e ORG_UNIT=MOSIP \ -e COMMON_NAME=<*.sandbox.xyz.net> \ mosipdev/openssl:latest + ``` + * Above command will generate certs in below specified location. Use it when prompted during nginx installation. + * fullChain path: /etc/ssl/certs/tls.crt. + * privKey path: /etc/ssl/private/tls.key. + +### 9.b. Nginx server setup for MOSIP K8's cluster + +* Move to nginx directory in your local:Add commentMore actions +* `cd $K8_ROOT/mosip/on-prem/nginx/` +* Open required ports : + * Use any editor to create new `hosts.ini` file: + * ``` + nano hosts.ini + ``` + * Add below mentioned lines with updated details of nginx server to the `hosts.ini` and save. + + ``` + [nginx] + node-nginx ansible_host= ansible_user=root ansible_ssh_private_key_file= + ``` + * Execute below mentoned command to open required ports: + + ``` + ansible-playbook -i hosts.ini mosip/on-prem/nginx/nginx_ports.yaml + ``` +* Login to nginx server node. + + ``` + ssh -i ~/.ssh/ -i ubuntu@ + ``` + + * Clone [k8s-infra](https://github.com/mosip/k8s-infra). + + ``` + cd $K8_ROOT/mosip/on-prem/nginx + sudo ./install.sh + ``` + * Provide below mentioned inputs as and when prompted + * MOSIP nginx server internal ip + * MOSIP nginx server public ip + * Publically accessible domains (comma separated with no whitespaces) + * SSL cert path + * SSL key path + * Cluster node ip's (comma separated no whitespace) +* When utilizing an openssl wildcard SSL certificate, please add the following server block to the nginx server configuration within the http block. Disregard this if using SSL certificates obtained through letsencrypt or for publicly available domains. Please note that this should only be used in a development environment and is not recommended for production environments. + + * `nano /etc/nginx/nginx.conf` + * ``` + server{ + listen :80; + server_name iam.sandbox.xyz.net; + location /auth/realms/mosip/protocol/openid-connect/certs { + proxy_pass http://myInternalIngressUpstream; + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + proxy_set_header Host $host; + proxy_set_header Referer $http_referer; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_pass_request_headers on; + proxy_set_header Strict-Transport-Security "max-age=0;"; + } + location / { return 301 https://iam.sandbox.xyz.net; } + } + ``` + + > Note: HTTP access is enabled for IAM because MOSIP's keymanager expects to have valid SSL certificates. Ensure to use this only for development purposes, and it is not recommended to use it in production environments. + + * Restart nginx service. + + ``` + sudo systemctl restart nginx + ``` +* Post installation check:`sudo systemctl status nginx` +* Steps to Uninstall nginx (in case required)`sudo apt purge nginx nginx-common` +* DNS mapping: + * Once nginx server is installed successfully, create DNS mapping for rancher cluster related domains as mentioned in DNS requirement section. (rancher.org.net, keycloak.org.net) + * In case used Openssl for wildcard ssl certificate add DNS entries in local hosts file of your system. + + * For example: `/etc/hosts` files for Linux machines. + + ``` + api.sandbox.xyz.net resident.sandbox.xyz.net esignet.sandbox.xyz.net prereg.sandbox.xyz.net healthservices.sandbox.xyz.net + sandbox.xyz.net api-internal.sandbox.xyz.net activemq.sandbox.xyz.net kibana.sandbox.xyz.net regclient.sandbox.xyz.net admin.sandbox.xyz.net minio.sandbox.xyz.net iam.sandbox.xyz.net kafka.sandbox.xyz.net postgres.sandbox.xyz.net pmp.sandbox.xyz.net onboarder.sandbox.xyz.net smtp.sandbox.xyz.net compliance.sandbox.xyz.net + ``` + +### 9.c. Check Overall nginx and istio wiring + +* Install `httpbin`: This utility docker returns http headers received inside the cluster. +* `httpbin` can be used for general debugging - to check ingress, headers etc. + + ``` + cd $K8_ROOT/utils/httpbin + ./install.sh + ``` + + * To see what is reaching the httpbin (example, replace with your domain name): + + ``` + curl https://api.sandbox.xyz.net/httpbin/get?show_env=true + curl https://api-internal.sandbox.xyz.net/httpbin/get?show_env=true + ``` + +## 10. Monitoring module deployment + +> Note : +> +> * Monitoring in the sandbox environment is optional and can be deployed if required. +> * For production environments, alternative monitoring tools can be used. +> * These steps can also be skipped in development environments if monitoring is not needed. +> * Incase skipping execute below commands to install monitoring crd as the same is required by mosip services: +> +> ``` +> helm repo add prometheus-community https://prometheus-community.github.io/helm-charts +> helm repo update +> kubectl create ns cattle-monitoring-system +> helm -n cattle-monitoring-system install rancher-monitoring-crd mosip/rancher-monitoring-crd +> ``` + +Prometheus and Grafana and Alertmanager tools are used for cluster monitoring. + +* Select 'Monitoring' App from Rancher console -> `Apps & Marketplaces`. +* In Helm options, open the YAML file and disable Nginx Ingress. + + ``` + ingressNginx: + enabled: false + ``` +* Click on `Install`. + +## 11. Alerting setup + +> Note : +> +> * Alerting in the sandbox environment is optional and can be deployed if required. +> * For production environments, alternative alerting tools can be used. +> * These steps can also be skipped in development environments if alerting is not needed. + +* Alerting is part of cluster monitoring, where alert notifications are sent to the configured email or slack channel. +* Monitoring should be deployed which includes deployment of prometheus, grafana and alertmanager. +* Create [slack incoming webhook](https://api.slack.com/messaging/webhooks). +* After setting slack incoming webhook update `slack_api_url` and `slack_channel_name` in `alertmanager.yml`. + + * `cd $K8_ROOT/monitoring/alerting/` + * `nano alertmanager.yml` + * Update: + + ``` + global: + resolve_timeout: 5m + slack_api_url: + ... + slack_configs: + - channel: '' + send_resolved: true + ``` +* Update `Cluster_name` in `patch-cluster-name.yaml`. +* `cd $K8_ROOT/monitoring/alerting/` +* `nano patch-cluster-name.yaml` +* Update: + +``` +spec: +externalLabels: +cluster: +``` + +* Install Default alerts along some of the defined custom alerts: + +``` +cd $K8_ROOT/monitoring/alerting/ +./install.sh +``` + +* Alerting is installed. + +## 12. Logging module setup and installation + +> Note : +> +> * Logging in the sandbox environment is optional and can be deployed if required. +> * For production environments, alternative logging tools can be used. +> * These steps can also be skipped in development environments if logging is not needed. + +MOSIP uses [Rancher Fluentd](https://ranchermanager.docs.rancher.com/v2.0-v2.4/explanations/integrations-in-rancher/cluster-logging/fluentd) and elasticsearch to collect logs from all services and reflect the same in Kibana Dashboard. + +* Install Rancher FluentD system : for scraping logs outs of all the microservices from MOSIP k8 cluster. + * Install Logging from Apps and marketplace within the Rancher UI. + * Select Chart Version `100.1.3+up3.17.7` from Rancher console -> Apps & Marketplaces. +* Configure Rancher FluentD + * Create `clusteroutput` + * `kubectl apply -f clusteroutput-elasticsearch.yaml` + * Start `clusterFlow` + * `kubectl apply -f clusterflow-elasticsearch.yaml` + * Install elasticsearch, kibana and Istio addons\\ + + ``` + cd $K8_ROOT/logging + ./intall.sh + ``` + * set `min_age` in `elasticsearch-ilm-script.sh` and execute the same. + * `min_age` : is the minimum no. of days for which indices will be stored in elasticsearch. + + ``` + cd $K8_ROOT/logging + ./elasticsearch-ilm-script.sh + ``` + * MOSIP provides set of Kibana Dashboards for checking logs and throughput's. + * Brief description of these dashboards are as follows: + * [01-logstash.ndjson](https://github.com/mosip/k8s-infra/blob/main/logging/dashboards/01-logstash.ndjson) contains the logstash _Index_ Pattern required by the rest of the dashboards. + * [02-error-only-logs.ndjson](https://github.com/mosip/k8s-infra/blob/main/logging/dashboards/03-service-logs.ndjson) contains a Search dashboard which shows only the error logs of the services, called `MOSIP Error Logs` dashboard. + * [03-service-logs.ndjson](https://github.com/mosip/k8s-infra/blob/main/logging/dashboards/03-service-logs.ndjson) contains a Search dashboard which show all logs of a particular service, called MOSIP Service Logs dashboard. + * [04-insight.ndjson](https://github.com/mosip/k8s-infra/blob/main/logging/dashboards/04-insight.ndjson) contains dashboards which show insights into MOSIP processes, like the number of UINs generated (total and per hr), the number of Biometric deduplications processed, number of packets uploaded etc, called `MOSIP Insight` dashboard. + * [05-response-time.ndjson](on-prem-installation-guidelines.md) contains dashboards which show how quickly different MOSIP Services are responding to different APIs, over time, called `Response Time` dashboard. +* Import dashboards: + * `cd K8_ROOT/logging` + * `./load_kibana_dashboards.sh ./dashboards ` +* View dashboards + +Open kibana dashboard from `https://kibana.sandbox.xyz.net`. + +Kibana --> Menu (on top left) --> Dashboard --> Select the dashboard. + +## 13. MOSIP External Dependencies setup + +External Dependencies are set of external requirements that are needed for functioning of MOSIP’s core services like DB, Object Store, HSM etc. + +``` +cd $INFRA_ROOT/deployment/v3/external/all +./install-all.sh +``` + +Click [here](https://docs.mosip.io/1.2.0/deploymentnew/v3-installation/mosip-external-dependencies) to check the detailed installation instructions of all the external components. + +> Note: +> +> * Connect to `mosip_pms` DB in postgres and execute the query to change `valid_to_date` for `mpolicy-default-mobile` in `pms.auth_policy` table. +> * Open the terminal. +> * Use the psql command to connect to the PostgreSQL server. The general syntax is: +> +> ``` +> psql -h -p 5432 -U postgres -d mosip_pms +> ``` +> +> * : The server address (e.g., localhost or an IP address). +> * Assuming other details remain same like port and user. +> +> ``` +> UPDATE pms.auth_policy SET valid_to_date = valid_to_date + interval '1 year' WHERE name = 'mpolicy-default-mobile'; +> ``` + +### Configuration change in case using Openssl wildcard ssl certificate. (Only advised in development env, not recommended for Production setup) + +* Add/Update the below property in application-default.properties and comment on the below property in the \*-default.properties file in the config repo. + + ``` + mosip.iam.certs_endpoint=http://${keycloak.external.host}/auth/realms/mosip/protocol/openid-connect/certs + ``` +* Add/Update the below property in the esignet-default.properties file in the config repo. + + ``` + spring.security.oauth2.resourceserver.jwt.jwk-set-uri=http://${keycloak.external.host}/auth/realms/mosip/protocol/openid-connect/certs + ``` + +## 14. MOSIP Modules Deployment + +* Now that all the Kubernetes cluster and external dependencies are already installed, will continue with MOSIP service deployment. +* While installing a few modules, installation script prompts to check if you have public domain and valid SSL certificates on the server. Opt option n as we are using self-signed certificates. For example: + + ``` + ./install.sh + Do you have public domain & valid SSL? (Y/n) + Y: if you have public domain & valid ssl certificate + n: If you don't have a public domain and a valid SSL certificate. Note: It is recommended to use this option only in development environments. + ``` +* Start installing mosip modules: + + ``` + cd $INFRA_ROOT/deployment/v3/mosip/all + ./install-all.sh + ``` + +> Note: +> +> * In case of failure on execution of `install-all.sh` follow [MOSIP Modules Deployment](https://docs.mosip.io/1.2.0/deploymentnew/v3-installation/mosip-modules-deployment) installation steps from the failure point. +> * Installation for config-server and admin service is facing delay in this version hence need to execute below mentioned commands respectively for config-server and admin-service once the script fails. +> * For config server: +> * Update `failureThreshold` for `startupProbe` to 60. +> +> ``` +> kubectl -n config-server edit deployment config-server +> ``` +> * For admin service: +> +> * Update `failureThreshold` for `startupProbe` to 60. +> +> ``` +> kubectl -n admin edit deployment admin-service +> ``` +> +> * Once admin service is up and running re-execute `install.sh` script after commenting out below mentioned commands: +> +> ``` +> #echo Installing Admin-Proxy into Masterdata and Keymanager. +> #kubectl -n $NS apply -f admin-proxy.yaml +> #echo Installing admin hotlist service. +> #helm -n $NS install admin-hotlist mosip/admin-hotlist --version $CHART_VERSION +> #echo Installing admin service. Will wait till service gets installed. +> #helm -n $NS install admin-service mosip/admin-service --set istio.corsPolicy.allowOrigins\[0\].prefix=https://$ADMIN_HOST --wait --version $CHART_VERSION +> ``` + +## 15. API Testrig + +MOSIP’s successfull deployment can be verified by comparing the results of api testrig with testrig benchmark. + +* Navigate to the Infra Root Directory: + + ``` + cd $INFRA_ROOT + ``` +* Clone the Functional Tests Repository + + ``` + git clone -b v1.3.3 https://github.com/mosip/mosip-functional-tests.git + ``` +* After cloned successfully try to install apitestrig + + ``` + cd $INFRA_ROOT/mosip-functional-tests/deploy/apitestrig + ``` +* Make script executable + + ``` + chmod +x copy_cm_func.sh + ``` +* Run the Installer Script + + ``` + ./install.sh + ``` + + > Note: + > + > * Script prompts for below mentioned inputs please provide as and when needed: + > * Enter the time (hr) to run the cronjob every day (0–23): Specify the hour you want the cronjob to run (e.g., 6 for 6 AM) + > * Do you have a public domain and valid SSL certificate? (Y/n): + > * Y – If you have a public domain and valid SSL certificate + > * n – If you do not have one (recommended only for development environments) + > * Retention days to remove old reports (Default: 3): Press Enter to accept the default or specify another value (e.g., 5). + > * Provide Slack Webhook URL to notify server issues on your Slack channel: (change the URL to your channel one) + > + > ``` + > https://hooks.slack.com/services/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 + > ``` + > * Is the eSignet service deployed? (yes/no): + > * no – If eSignet is not deployed, related test cases will be skipped. + > * Is values.yaml for the apitestrig chart set correctly as part of the prerequisites? (Y/n): \* Enter Y if this step is already completed. + > * Do you have S3 details for storing API-Testrig reports? (Y/n): + > * Enter Y to proceed with S3 configuration. + > * S3 Host: eg. `http://minio.minio:9000` + > * S3 Region:(Leave blank or enter your specific region, if applicable) + > * S3 Access Key:admin + +## 16. DSL Rig + +* Install Packetcreater + * Navigate to the packetcreator directory: + + ``` + cd $INFRA_ROOT/deployment/v3/testrig/packetcreator + ``` + * Run the installation script: + + ``` + ./install.sh + ``` + * NFS server details provide the following inputs: + * NFS Host: + * NFS PEM File : + * User for SSH Login: ubuntu + * Select the Ingress Controller Type: \* + 1. Ingress + * + 2. Istio (Choose 2) +* Install DSLrig + * Navigate to the dslrig directory: + + ``` + cd $INFRA_ROOT/deployment/v3/testrig/dslrig + ``` +* Run the installation script: + + ``` + * ./install.sh + ``` + + > Note: Before running install.sh, please ensure the following is added in the Helm install command: + > + > ``` + > --set dslorchestrator.configmaps.dslorchestrator.servicesNotDeployed="esignet" \ + > ``` + > + > After adding the above, the full Helm installation command should look like this: + > + > ``` + > helm -n $NS install dslorchestrator mosip/dslorchestrator \ + > --set crontime="0 $time * * *" \ + > --version $CHART_VERSION \ + > --set dslorchestrator.configmaps.s3.s3-host='http://minio.minio:9000' \ + > --set dslorchestrator.configmaps.s3.s3-user-key='admin' \ + > --set dslorchestrator.configmaps.s3.s3-region='' \ + > --set dslorchestrator.configmaps.db.db-server="$DB_HOST" \ + > --set dslorchestrator.configmaps.db.db-su-user="postgres" \ + > --set dslorchestrator.configmaps.db.db-port="5432" \ + > --set dslorchestrator.configmaps.dslorchestrator.USER="$USER" \ + > --set dslorchestrator.configmaps.dslorchestrator.ENDPOINT="https://$API_INTERNAL_HOST" \ + > --set dslorchestrator.configmaps.dslorchestrator.packetUtilityBaseUrl="$packetUtilityBaseUrl" \ + > --set persistence.nfs.server="$NFS_HOST" \ + > --set persistence.nfs.path="/srv/nfs/mosip/dsl-scenarios/$ENV_NAME" \ + > --set dslorchestrator.configmaps.dslorchestrator.reportExpirationInDays="$reportExpirationInDays" \ + > --set dslorchestrator.configmaps.dslorchestrator.NS="$NS" \ + > --set dslorchestrator.configmaps.dslorchestrator.servicesNotDeployed="esignet" \ + > $ENABLE_INSECURE + > ``` + > + > * When prompted, provide the following inputs: + > * NFS Host: + > * NFS PEM : + > * User for SSH Login:ubuntu + > * Cronjob Time (hour of the day, 0–23): (e.g., enter 6 for 6 AM) + > * Do you have a public domain and valid SSL certificate? (Y/n): + > * Y: If you have a valid public domain and SSL certificate + > * n: Use only in development environments + > * Packet Utility Base URL: + > + > ``` + > https://packetcreator.packetcreator:80/v1/packetcreator + > ``` + > * Retention Days to Remove Old Reports (default is 3): (Press Enter to accept default or provide a custom value) diff --git a/docs/setup/deploymentnew/v3-installation/1.2.1.0/on-premises-deployment.md b/docs/setup/deploymentnew/v3-installation/1.2.1.0/on-premises-deployment.md new file mode 100644 index 000000000..acd97b5ff --- /dev/null +++ b/docs/setup/deploymentnew/v3-installation/1.2.1.0/on-premises-deployment.md @@ -0,0 +1,1294 @@ +# On-Premises Deployment + +## 1. Wireguard + +- Wireguard bastion server provides a secure private channel to access the MOSIP cluster. +- The bastion server restricts public access and enables access only to clients whose public keys are listed in the Wireguard server. +- The bastion server listens on UDP port 51820. +- If you already have a VPN configured to access nodes privately, skip Wireguard installation and continue using the same VPN. + +### Setup Wireguard VM and Wireguard Bastion Server + +- Create a Wireguard server VM as per the [**Hardware and Network Requirements**](https://github.com/mosip/documentation/blob/1.2.0/docs/setup/deploymentnew/v3-installation/1.2.0.2/pre-requisites.md). + +- Open the required ports on the bastion server VM: + +```bash +cd $K8_ROOT/wireguard/ +cp hosts.ini.sample hosts.ini +``` + +> **Note:** +> - Remove the `[Cluster]` section entirely from the copied `hosts.ini` file. +> - Add the following details: +> - `ansible_host` — public IP of the Wireguard bastion server. e.g. `100.10.20.56` +> - `ansible_user` — user for installation. In this reference implementation we use the Ubuntu user. +> - `ansible_ssh_private_key_file` — path to the PEM key for SSH to the Wireguard server. e.g. `~/.ssh/wireguard-ssh.pem` + +- Execute `ports.yaml` to open ports at the VM level using ufw: + +```bash +ansible-playbook -i hosts.ini ports.yaml +``` + +> **Note:** +> - PEM file permissions for SSH access must be set to 400: `sudo chmod 400 ~/.ssh/privkey.pem` +> - These ports only need to be opened for UDP packet sharing. +> - Ensure the Wireguard server is reachable on port `51820/udp` at the firewall level. + +- Install Docker on the Wireguard VM: + +```bash +ansible-playbook -i hosts.ini docker.yaml +``` + +- Set up the Wireguard server. SSH into the Wireguard VM: + +```bash +ssh -i ubuntu@ +``` + +- Create a directory for storing Wireguard config files: + +```bash +mkdir -p wireguard/config +``` + +- Install and start the Wireguard server using Docker: + +```bash +sudo docker run -d \ + --name=wireguard \ + --cap-add=NET_ADMIN \ + --cap-add=SYS_MODULE \ + -e PUID=1000 \ + -e PGID=1000 \ + -e TZ=Asia/Calcutta \ + -e PEERS=30 \ + -p 51820:51820/udp \ + -v /home/ubuntu/wireguard/config:/config \ + -v /lib/modules:/lib/modules \ + --sysctl="net.ipv4.conf.all.src_valid_mark=1" \ + --restart unless-stopped \ + ghcr.io/linuxserver/wireguard +``` + +> **Note:** +> - Increase the number of peers if more than 30 Wireguard client configs are needed (`-e PEERS=30`). +> - Change the mounted directory as needed. All Wireguard configs will be generated in the mounted directory (`-v /home/ubuntu/wireguard/config:/config`). + +### Setup Wireguard Client on your PC + +- Install the Wireguard client using the steps at [wireguard.com/install](https://www.wireguard.com/install/). + +- Assign a `wireguard.conf`. SSH into the Wireguard server VM and navigate to the config directory: + +```bash +ssh -i ubuntu@ +cd /home/ubuntu/wireguard/config +``` + +- Assign one peer to yourself and use it from your PC to connect to the server. + +- Create an `assigned.txt` file to track allocated peer files, and update it each time a peer is assigned: + +``` +peer1 : peername +peer2 : xyz +``` + +- Use `ls` to see the list of available peers. + +- Enter your selected peer directory and update `peer.conf`: + +```bash +cd peer1 +nano peer1.conf +``` + +- Make the following changes: + - Delete the DNS IP line. + - Update `AllowedIPs` to the subnets CIDR range. e.g. `10.10.20.0/23` + +> **Note:** +> - The CIDR range will be provided by the infrastructure provider. +> - Ensure all nodes are covered in the CIDR range (nginx server, Observation K8s cluster nodes, and MOSIP K8s cluster nodes). + +- Share the updated `peer.conf` with the respective peer to allow them to connect from their PC. + +- Add `peer.conf` to your PC's `/etc/wireguard/` directory as `wg0.conf`. + +- Start the Wireguard client and check the status: + +```bash +sudo systemctl start wg-quick@wg0 +sudo systemctl status wg-quick@wg0 +``` + +- Once connected to Wireguard, you can log in to cluster nodes using their private IPs. + +--- + +## 2. Observation K8s Cluster Setup and Configuration + +> **Note:** Cluster creation uses **RKE2** (RKE1 is EOL). The recommended approach is the Ansible-based automated setup from `$K8_ROOT/k8-cluster/on-prem/rke2/ansible`. See the [RKE2 documentation](https://docs.rke2.io/) for full reference. + +- Install all the required tools listed in the [Personal Computer Setup](https://github.com/mosip/documentation/blob/1.2.0/docs/setup/deploymentnew/v3-installation/1.2.0.2/pre-requisites.md#personal-computer-requirements) section. +- Ensure the following are installed on all cluster VMs and the client machine: `ufw`, `wget`, `curl`, `kubectl`, `istioctl`, `helm`, `jq`, `ansible` (version > 2.12.4). +- Set up Observation Cluster node VMs per the [hardware and network requirements](https://github.com/mosip/documentation/blob/1.2.0/docs/setup/deploymentnew/v3-installation/1.2.0.2/pre-requisites.md#hardware-requirements). +- Set up passwordless SSH to the cluster nodes via PEM keys (skip if VMs are already accessible via PEM): + - Generate keys on your PC: `ssh-keygen -t rsa` + - Copy the keys to remote observation node VMs: `ssh-copy-id @` + - Verify SSH access: `ssh -i ~/.ssh/ @` + +> **Note:** Ensure the PEM file permission is set to 400: `sudo chmod 400 ~/.ssh/privkey.pem` + +### Step 1 — Prepare Inventory File + +Navigate to the ansible directory and create a copy of the sample inventory: + +```bash +cd $K8_ROOT/k8-cluster/on-prem/rke2/ansible +cp hosts.ini.sample hosts.ini +``` + +Update `hosts.ini` with the details of all cluster VMs: + +> **Note:** +> - `ansible_host` — internal IP of each node. e.g. `100.10.20.56`, `100.10.20.57` +> - `ansible_user` — user for installation. In this reference implementation we use the Ubuntu user. +> - `ansible_ssh_private_key_file` — path to the PEM key. e.g. `~/.ssh/nodes-ssh.pem` +> - **Sandbox/Development:** comment out the `[etcd]` section in `hosts.ini`. +> - **Production:** ensure all sections including `[etcd]` are fully populated. + +### Step 2 — Enable Required Ports + +Open the required ports on all nodes using ufw. For the full list of RKE2 networking requirements see [docs.rke2.io](https://docs.rke2.io/install/requirements#networking): + +```bash +ansible-playbook -i hosts.ini ports.yaml +``` + +### Step 3 — Disable Swap + +Disable swap on all cluster nodes (skip if already disabled): + +```bash +ansible-playbook -i hosts.ini swap.yaml +``` + +> **Caution:** Verify swap status with `swapon --show` before running this playbook. + +> **Note:** RKE2 uses containerd and does **not** require Docker. There is no `docker.yaml` step for RKE2 clusters. + +### Step 4 — Run the Main Playbook + +Execute the main Ansible playbook to install and configure the RKE2 cluster across all nodes: + +```bash +ansible-playbook -i hosts.ini main.yaml +``` + +This provisions the RKE2 cluster — installing the binary, distributing config to each node type (primary server, secondary servers, etcd, agents), and starting all services automatically. + +### Step 5 — Securely Store the Kubeconfig File + +After the cluster is successfully created, the kubeconfig file is saved in the `ansible/playbook/` directory as `{{ cluster_domain }}-{{ inventory_hostname }}.yaml`. Copy it to your `.kube` directory: + +```bash +cp {{ cluster_domain }}-{{ inventory_hostname }}.yaml $HOME/.kube/_config +chmod 400 $HOME/.kube/_config +``` + +### Step 6 — Access the Cluster + +To use this kubeconfig, choose one of the following: + +```bash +cp $HOME/.kube/_config $HOME/.kube/config +``` + +Alternatively, set the `KUBECONFIG` environment variable: + +```bash +export KUBECONFIG="$HOME/.kube/_config" +``` + +### Step 7 — Verify Cluster Access + +```bash +kubectl get nodes +``` + +The output should list all nodes of the Observation cluster. + +> **Important:** Store the kubeconfig file and the `hosts.ini` used for this deployment in a secure location. They are required for cluster management, adding nodes, and recovery. + +--- + +## 3. Observation K8s Cluster Ingress and Storage Class Setup + +Once the cluster is ready, ingress and a storage class must be configured before other applications can be installed. + +### 3.a. Nginx Ingress Controller + +[Nginx Ingress Controller](https://kubernetes.github.io/ingress-nginx/) is used for ingress in the Rancher cluster. + +```bash +cd $K8_ROOT/ingress/ingress-nginx +helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx +helm repo update +helm install \ + ingress-nginx ingress-nginx/ingress-nginx \ + --namespace ingress-nginx \ + --version 4.0.18 \ + --create-namespace \ + -f ingress-nginx.values.yaml +``` + +> **Note:** +> - This installs ingress into the `ingress-nginx` namespace of the Observation cluster. +> - Verify the installation: `kubectl get all -n ingress-nginx` +> - The output should list all pods and deployments in the `ingress-nginx` namespace. + +### 3.b. Storage Classes + +Multiple storage class options are available for on-premises K8s clusters. This reference deployment uses NFS as the storage class. + +- Move to the NFS directory on your personal computer: + +```bash +cd $K8_ROOT/storage-class/nfs +``` + +- Create a copy of `hosts.ini.sample` as `hosts.ini`: + +```bash +cp hosts.ini.sample hosts.ini +``` + +- Update the NFS machine details in `hosts.ini`: + +> **Note:** +> - `ansible_host` — internal IP of the NFS server. e.g. `10.12.23.21` +> - `ansible_user` — user for installation. In this reference implementation we use the Ubuntu user. +> - `ansible_ssh_private_key_file` — path to the PEM key. e.g. `~/.ssh/nfs-ssh.pem` + +- Confirm the kubeconfig is pointing to the correct Observation cluster: + +```bash +kubectl config view +``` + +> **Note:** The output should show the expected cluster name. If not, update your kubeconfig to point to the correct cluster. + +- Open the required firewall ports on the NFS node: + +```bash +ansible-playbook -i hosts.ini nfs-ports.yaml +``` + +- SSH into the NFS node: + +```bash +ssh -i ~/.ssh/nfs-ssh.pem ubuntu@ +``` + +- Clone the `k8s-infra` repo on the NFS VM: + +```bash +git clone https://github.com/mosip/k8s-infra -b v1.2.1.3 +``` + +- Move to the NFS directory and run the install script: + +```bash +cd /home/ubuntu/k8s-infra/storage-class/nfs/ +sudo ./install-nfs-server.sh +``` + +> **Note:** The script will prompt for an environment name: +> +> ``` +> Please Enter Environment Name: +> ``` +> +> where `envName` is the environment name e.g. `dev`, `qa`, `uat`. The NFS share will be exported at `/srv/nfs/mosip/`. + +- Switch back to your personal computer and run the NFS client provisioner: + +```bash +cd $K8_ROOT/storage-class/nfs/ +./install-nfs-client-provisioner.sh +``` + +> **Note:** The script will prompt for: +> - NFS Server: IP of the NFS server. +> - NFS Path: Path for persisted data. e.g. `/srv/nfs/mosip/` + +- Post-installation checks: + + - Check the status of the NFS Client Provisioner: + +```bash +kubectl -n nfs get deployment.apps/nfs-client-provisioner +``` + + - Check the storage class is registered: + +```bash +kubectl get storageclass +``` + +Expected output: + +``` +NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE +longhorn (default) driver.longhorn.io Delete Immediate true 57d +nfs-client cluster.local/nfs-client-provisioner Delete Immediate true 40s +``` + + - Set `nfs-client` as the default storage class: + +```bash +kubectl patch storageclass nfs-client -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}' +``` + +--- + +## 4. Setting up Nginx Server for Observation K8s Cluster + +### 4.a. SSL Certificate Setup for TLS Termination + +An SSL certificate is required for the Nginx server. If a valid certificate is not available, generate one using Let's Encrypt: + +- SSH into the nginx server. + +- Install the prerequisites: + +```bash +sudo apt update -y +sudo apt-get install software-properties-common -y +sudo add-apt-repository ppa:deadsnakes/ppa +sudo apt-get update -y +sudo apt-get install python3.8 -y +sudo apt install letsencrypt -y +sudo apt install certbot python3-certbot-nginx -y +``` + +- Generate a wildcard SSL certificate for your domain: + +```bash +sudo certbot certonly --agree-tos --manual --preferred-challenges=dns -d *.org.net +``` + +> **Note:** Replace `org.net` with your actual domain. + +- The DNS challenge requires you to create a TXT record in your DNS service: + - Host: `_acme-challenge.org.net` + - Value: the string prompted by the script. + +- Wait a few minutes for the DNS entry to propagate. Verify with: + +```bash +host -t TXT _acme-challenge.org.net +``` + +- Press Enter in the `certbot` prompt to proceed. +- Certificates are created in `/etc/letsencrypt/` and are valid for 3 months. +- For certificate renewal, see the [wildcard SSL renewal guide](https://github.com/mosip/k8s-infra/blob/v1.2.1.3/docs/wildcard-ssl-certs-letsencrypt.md#ssl-certificate-renewal). + +### 4.b. Install Nginx + +- Move to the nginx directory: + +```bash +cd $K8_ROOT/nginx/observation +``` + +- Create a `hosts.ini` file for the nginx server: + +```bash +nano hosts.ini +``` + +- Add the following with the nginx server details and save: + +```ini +[nginx] +node-nginx ansible_host= ansible_user=root ansible_ssh_private_key_file= +``` + +- Open the required ports: + +```bash +ansible-playbook -i hosts.ini nginx_ports.yaml +``` + +- SSH into the nginx server node: + +```bash +ssh -i ~/.ssh/ ubuntu@ +``` + +- From the nginx directory, run the install script: + +```bash +cd $K8_ROOT/nginx/observation +sudo ./install.sh +``` + +- Provide the following inputs when prompted: + - Rancher nginx IP: internal IP of the nginx server VM. + - SSL cert path: path to the SSL certificate for TLS termination. + - SSL key path: path to the SSL key for TLS termination. + - Cluster node IPs: IPs of the Rancher cluster nodes. + +- Post-installation checks: + - `sudo systemctl status nginx` + - To uninstall nginx if required: `sudo apt purge nginx nginx-common` + - **DNS mapping:** Once nginx is installed successfully, create DNS records for the Rancher cluster domains as described in the DNS requirements section (e.g. `rancher.org.net`, `keycloak.org.net`). + +--- + +## 5. Observation K8s Cluster Apps Installation + +### 5.a. Rancher UI + +Rancher provides full management capability for creating and administering Kubernetes clusters. + +- Update `hostname` in `rancher-values.yaml`, then install Rancher using Helm: + +```bash +cd $K8_ROOT/observation/rancher-ui +helm repo add rancher-latest https://releases.rancher.com/server-charts/latest +helm repo update +helm install rancher rancher-latest/rancher \ + --version 2.6.9 \ + --namespace cattle-system \ + --create-namespace \ + -f rancher-values.yaml +``` + +- Connect to Wireguard (on Windows via WSL, connect to the Wireguard server from Windows, not from WSL). +- Open the Rancher page in a browser. +- Retrieve the bootstrap password: + +```bash +kubectl get secret --namespace cattle-system bootstrap-secret \ + -o go-template='{{ .data.bootstrapPassword|base64decode}}{{ "\n" }}' +``` + +> **Important:** Assign a strong password and store it securely. It will be needed by the admin. + +### 5.b. Keycloak + +Keycloak is an OAuth 2.0-compliant Identity and Access Management (IAM) system used to manage access to Rancher for cluster administration. + +```bash +cd $K8_ROOT/observation/keycloak +./install.sh +``` + +After installation, access Keycloak at `iam.mosip.net` and retrieve credentials as per the post-installation steps. + +### 5.c. Keycloak — Rancher UI Integration + +- Log in as the `admin` user in Keycloak and ensure the `email` and `firstName` fields are populated for the admin user. These fields are required for Rancher authentication. + +- In Keycloak (in the `master` realm), create a new client with the following values: + - `Client ID`: `https:///v1-saml/keycloak/saml/metadata` + - `Client Protocol`: `saml` + - `Root URL`: *(leave empty)* + +- After saving, configure the client with the following settings: + +| Field | Value | +|-------|-------| +| Name | `rancher` | +| Enabled | `ON` | +| Login Theme | `keycloak` | +| Sign Documents | `ON` | +| Sign Assertions | `ON` | +| Encrypt Assertions | `OFF` | +| Client Signature Required | `OFF` | +| Force POST Binding | `OFF` | +| Front Channel Logout | `OFF` | +| Force Name ID Format | `OFF` | +| Name ID Format | `username` | +| Valid Redirect URIs | `https:///v1-saml/keycloak/saml/acs` | +| IDP Initiated SSO URL Name | `IdPSSOName` | + +- In the same client, go to the `Mappers` tab and create the following mappers: + + **Mapper 1 — username:** + + | Field | Value | + |-------|-------| + | Protocol | `saml` | + | Name | `username` | + | Mapper Type | `User Property` | + | Property | `username` | + | Friendly Name | `username` | + | SAML Attribute Name | `username` | + | SAML Attribute NameFormat | `Basic` | + + **Mapper 2 — groups:** + + | Field | Value | + |-------|-------| + | Protocol | `saml` | + | Name | `groups` | + | Mapper Type | `Group List` | + | Group Attribute Name | `member` | + | Friendly Name | *(leave empty)* | + | SAML Attribute NameFormat | `Basic` | + | Single Group Attribute | `ON` | + | Full Group Path | `OFF` | + +- Click `Add Builtin` → select all → `Add Selected`. + +- Download the Keycloak SAML descriptor XML from: + +``` +https:///auth/realms/master/protocol/saml/descriptor +``` + +- Generate a self-signed SSL certificate and key (if not already available): + +```bash +openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 \ + -keyout myservice.key -out myservice.cert +``` + +- In Rancher UI, navigate to: `Users & Authentication` → `Auth Providers` → `Keycloak (SAML)` + +- Configure the fields as follows: + +| Field | Value | +|-------|-------| +| Display Name Field | `givenName` | +| User Name Field | `email` or `uid` | +| UID Field | `username` | +| Groups Field | `member` | +| Entity ID Field | *(leave empty)* | +| Rancher API Host | `https://` | + +- Upload the following files: + - `Private Key`: `myservice.key` + - `Certificate`: `myservice.cert` + - `SAML Metadata XML`: content from the Keycloak descriptor URL above + +- Click **Enable** to activate Keycloak authentication. After successful integration, Rancher users can log in using their Keycloak credentials. + +### 5.d. RBAC for Rancher using Keycloak + +- Assign **cluster** and **project** roles in Rancher for Keycloak users. Add all namespaces under the `default` project. Non-admin users can be granted the Read-Only role at the project level. +- To create custom roles, follow the steps [here](https://github.com/mosip/k8s-infra/blob/v1.2.1.3/docs/create-custom-role.md). + +- To add a member to a cluster or project in Rancher: + - Navigate to RBAC cluster members. + - Add the member name exactly as the `username` in Keycloak. + - Assign an appropriate role (e.g. Cluster Owner, Cluster Viewer). + +- To add a group to a cluster or project in Rancher: + - Navigate to RBAC cluster members. + - Click `Add` and select a group from the dropdown. + - Assign an appropriate role. + - Note: a user must be a member of the group to add it. + +- To create a Keycloak group: + - Go to the `Groups` section in Keycloak and create a group with default roles. + - In the `Users` section, select a user, go to the `Groups` tab, and add the user to the required group. + +--- + +## 6. MOSIP K8s Cluster Setup + +> **Note:** Cluster creation uses **RKE2** (RKE1 is EOL). The recommended approach is the Ansible-based automated setup from `$K8_ROOT/k8-cluster/on-prem/rke2/ansible`. See the [RKE2 documentation](https://docs.rke2.io/) for full reference. + +- Install the following tools on your PC and all cluster VMs: `kubectl`, `helm`, `ansible` (version > 2.12.4), `ufw`, `wget`, `curl`, `istioctl`, `jq`. + +- Add the required Helm repositories: + +```bash +helm repo add bitnami https://charts.bitnami.com/bitnami +helm repo add mosip https://mosip.github.io/mosip-helm +``` + +- Set up MOSIP K8s Cluster node VMs per the [Hardware and Network Requirements](https://github.com/mosip/documentation/blob/1.2.0/docs/setup/deploymentnew/v3-installation/1.2.0.2/pre-requisites.md). + +- Set up passwordless SSH to the cluster nodes via PEM keys (skip if VMs are already accessible via PEM): + - Generate keys on your PC: `ssh-keygen -t rsa` + - Copy keys to remote cluster node VMs: `ssh-copy-id @` + - Verify SSH access: `ssh -i ~/.ssh/ @` + +### Step 1 — Prepare Inventory File + +Navigate to the ansible directory and create a copy of the sample inventory: + +```bash +cd $K8_ROOT/k8-cluster/on-prem/rke2/ansible +cp hosts.ini.sample hosts.ini +``` + +Update `hosts.ini` with the details of all MOSIP cluster VMs: + +> **Note:** +> - `ansible_host` — internal IP of each node. e.g. `100.10.20.56`, `100.10.20.57` +> - `ansible_user` — user for installation. In this reference implementation we use the Ubuntu user. +> - `ansible_ssh_private_key_file` — path to the PEM key. e.g. `~/.ssh/nodes-ssh.pem` +> - **Sandbox/Development:** comment out the `[etcd]` section in `hosts.ini`. +> - **Production:** ensure all sections including `[etcd]` are fully populated. + +### Step 2 — Enable Required Ports + +Open the required ports on all nodes. For the full list of RKE2 networking requirements see [docs.rke2.io](https://docs.rke2.io/install/requirements#networking): + +```bash +ansible-playbook -i hosts.ini ports.yaml +``` + +### Step 3 — Disable Swap + +Disable swap on all cluster nodes (skip if already disabled): + +```bash +ansible-playbook -i hosts.ini swap.yaml +``` + +> **Caution:** Verify swap status with `swapon --show` before running this playbook. + +> **Note:** RKE2 uses containerd and does **not** require Docker. There is no `docker.yaml` step for RKE2 clusters. + +### Step 4 — Run the Main Playbook + +Execute the main Ansible playbook to install and configure the RKE2 cluster across all nodes: + +```bash +ansible-playbook -i hosts.ini main.yaml +``` + +This provisions the RKE2 cluster — installing the binary, distributing config to each node type (primary server, secondary servers, etcd, agents), and starting all services automatically. + +### Step 5 — Securely Store the Kubeconfig File + +After the cluster is successfully created, the kubeconfig file is saved in the `ansible/playbook/` directory as `{{ cluster_domain }}-{{ inventory_hostname }}.yaml`. Copy it to your `.kube` directory: + +```bash +cp {{ cluster_domain }}-{{ inventory_hostname }}.yaml $HOME/.kube/_config +chmod 400 $HOME/.kube/_config +``` + +### Step 6 — Access the Cluster + +To use this kubeconfig, choose one of the following: + +```bash +cp $HOME/.kube/_config $HOME/.kube/config +``` + +Alternatively, set the `KUBECONFIG` environment variable: + +```bash +export KUBECONFIG="$HOME/.kube/_config" +``` + +### Step 7 — Verify Cluster Access + +```bash +kubectl get nodes +``` + +The output should list all nodes of the MOSIP cluster. + +> **Important:** Store the kubeconfig file and the `hosts.ini` used for this deployment in a secure location. They are required for cluster management, adding nodes, and recovery. + +--- + +## 7. MOSIP K8s Cluster Global ConfigMap, Ingress, and Storage Class Setup + +### 7.a. Global ConfigMap + +The global ConfigMap contains common configuration values shared across all namespaces in the cluster. + +```bash +cd $INFRA_ROOT/deployment/v3/external +cp global_configmap.yaml.sample global_configmap.yaml +``` + +Update the domain names in `global_configmap.yaml`, then apply: + +```bash +kubectl apply -f global_configmap.yaml +``` + +### 7.b. Istio Ingress Setup + +[Istio](https://istio.io/) is a service mesh for the MOSIP K8s cluster. It provides transparent layers over existing microservices, enabling a uniform way to secure, connect, and monitor services. + +```bash +cd $K8_ROOT/ingress/istio-mesh/nodeport +./install.sh +``` + +This will bring up all Istio components and the Ingress Gateways. Verify the Ingress Gateway services: + +```bash +kubectl get svc -n istio-system +``` + +> **Note:** The response should include the following services: +> - `istio-ingressgateway` — external-facing Istio service. +> - `istio-ingressgateway-internal` — internal-facing Istio service. +> - `istiod` — Istio daemon that replicates changes to all Envoy filters. + +### 7.c. Storage Classes + +Multiple storage class options are available for on-premises K8s clusters. This reference deployment uses NFS as the storage class. + +- Move to the NFS directory on your personal computer: + +```bash +cd $K8_ROOT/storage-class/nfs +``` + +- Create a copy of `hosts.ini.sample` as `hosts.ini`: + +```bash +cp hosts.ini.sample hosts.ini +``` + +- Update the NFS machine details in `hosts.ini`: + +> **Note:** +> - `ansible_host` — internal IP of the NFS server. e.g. `10.12.23.21` +> - `ansible_user` — user for installation. In this reference implementation we use the Ubuntu user. +> - `ansible_ssh_private_key_file` — path to the PEM key. e.g. `~/.ssh/nfs-ssh.pem` + +- Confirm the kubeconfig is pointing to the correct MOSIP cluster: + +```bash +kubectl config view +``` + +> **Note:** The output should show the expected cluster name. If not, update your kubeconfig to point to the correct cluster. + +- Open the required firewall ports on the NFS node: + +```bash +ansible-playbook -i hosts.ini nfs-ports.yaml +``` + +- SSH into the NFS node: + +```bash +ssh -i ~/.ssh/nfs-ssh.pem ubuntu@ +``` + +- Clone the `k8s-infra` repo on the NFS VM: + +```bash +git clone https://github.com/mosip/k8s-infra -b v1.2.1.3 +``` + +- Move to the NFS directory and run the install script: + +```bash +cd /home/ubuntu/k8s-infra/storage-class/nfs/ +sudo ./install-nfs-server.sh +``` + +> **Note:** The script will prompt for an environment name: +> +> ``` +> Please Enter Environment Name: +> ``` +> +> where `envName` is the environment name e.g. `dev`, `qa`, `uat`. The NFS share will be exported at `/srv/nfs/mosip/`. + +- Switch back to your personal computer and run the NFS client provisioner: + +```bash +cd $K8_ROOT/storage-class/nfs/ +./install-nfs-client-provisioner.sh +``` + +> **Note:** The script will prompt for: +> - NFS Server: IP of the NFS server. +> - NFS Path: Path for persisted data. e.g. `/srv/nfs/mosip/` + +- Post-installation checks: + + - Check the status of the NFS Client Provisioner: + +```bash +kubectl -n nfs get deployment.apps/nfs-client-provisioner +``` + + - Check the storage class is registered: + +```bash +kubectl get storageclass +``` + +--- + +## 8. Import MOSIP Cluster into Rancher UI + +- Log in as admin in the Rancher console. +- Select `Import Existing` for cluster addition. +- Select `Generic` as the cluster type. +- Fill in the `Cluster Name` field with a unique name and click `Create`. +- Copy the generated `kubectl` command and execute it from your PC (ensure your kubeconfig is set to the MOSIP cluster): + +```bash +kubectl apply -f https://rancher.e2e.mosip.net/v3/import/.yaml +``` + +- Wait a few seconds for the cluster to be verified. The cluster will then appear in the Rancher management console. + +--- + +## 9. MOSIP K8s Cluster Nginx Server Setup + +### 9.a. SSL Certificate Creation + +An SSL certificate is required for the Nginx server. If a valid certificate is not available, generate one using Let's Encrypt: + +- SSH into the nginx server. + +- Install the prerequisites: + +```bash +sudo apt update -y +sudo apt-get install software-properties-common -y +sudo add-apt-repository ppa:deadsnakes/ppa +sudo apt-get update -y +sudo apt-get install python3.8 -y +sudo apt install letsencrypt -y +sudo apt install certbot python3-certbot-nginx -y +``` + +- Generate a wildcard SSL certificate for your domain: + +```bash +sudo certbot certonly --agree-tos --manual --preferred-challenges=dns \ + -d *.sandbox.mosip.net -d sandbox.mosip.net +``` + +> **Note:** Replace `sandbox.mosip.net` with your actual domain. + +- The DNS challenge requires you to create a TXT record in your DNS service: + - Host: `_acme-challenge.sandbox.mosip.net` + - Value: the string prompted by the script. + +- Wait a few minutes for the DNS entry to propagate. Verify with: + +```bash +host -t TXT _acme-challenge.sandbox.mosip.net +``` + +- Press Enter in the `certbot` prompt to proceed. +- Certificates are created in `/etc/letsencrypt/` and are valid for 3 months. +- For certificate renewal, see the [wildcard SSL renewal guide](https://github.com/mosip/k8s-infra/blob/v1.2.1.3/docs/wildcard-ssl-certs-letsencrypt.md#ssl-certificate-renewal). + +### 9.b. Nginx Server Setup for MOSIP K8s Cluster + +- Move to the nginx directory: + +```bash +cd $K8_ROOT/nginx/mosip +``` + +- Create a `hosts.ini` file for the nginx server: + +```bash +nano hosts.ini +``` + +- Add the following with nginx server details and save: + +```ini +[nginx] +node-nginx ansible_host= ansible_user=root ansible_ssh_private_key_file= +``` + +- Open the required ports: + +```bash +ansible-playbook -i hosts.ini nginx_ports.yaml +``` + +- SSH into the nginx server node: + +```bash +ssh -i ~/.ssh/ ubuntu@ +``` + +- From the nginx directory, run the install script: + +```bash +cd $K8_ROOT/nginx/mosip +sudo ./install.sh +``` + +- Provide the following inputs when prompted: + - MOSIP nginx server internal IP. + - MOSIP nginx server public IP. + - Publicly accessible domains (comma-separated, no whitespace). + - SSL cert path. + - SSL key path. + - Cluster node IPs (comma-separated, no whitespace). + +- Post-installation checks: + - `sudo systemctl status nginx` + - To uninstall nginx if required: `sudo apt purge nginx nginx-common` + - **DNS mapping:** Once nginx is installed successfully, create DNS records for the MOSIP cluster domains as described in the DNS requirements section. + +### 9.c. Verify Nginx and Istio Wiring + +Install `httpbin` — a utility that echoes HTTP headers received inside the cluster, useful for debugging ingress and header routing: + +```bash +cd $K8_ROOT/utils/httpbin +./install.sh +``` + +Test external and internal ingress (replace with your actual domain): + +```bash +curl https://api.sandbox.xyz.net/httpbin/get?show_env=true +curl https://api-internal.sandbox.xyz.net/httpbin/get?show_env=true +``` + +--- + +## 10. Monitoring Module Deployment + +> **Note:** +> - Monitoring is optional in sandbox environments. +> - For production environments, alternative monitoring tools may be used. +> - These steps can be skipped in development environments if monitoring is not needed. +> - If skipping, run the following commands to install the monitoring CRD, which is required by MOSIP services: +> +> ```bash +> helm repo add prometheus-community https://prometheus-community.github.io/helm-charts +> helm repo update +> kubectl create ns cattle-monitoring-system +> helm -n cattle-monitoring-system install rancher-monitoring-crd mosip/rancher-monitoring-crd +> ``` + +- Prometheus, Grafana, and Alertmanager are used for cluster monitoring. +- In Rancher console, go to `Apps & Marketplaces` and select the `Monitoring` app. +- In the Helm options, open the YAML file and disable the Nginx Ingress: + +```yaml +ingressNginx: + enabled: false +``` + +- Click `Install`. + +--- + +## 11. Alerting Setup + +> **Note:** +> - Alerting is optional in sandbox environments. +> - For production environments, alternative alerting tools may be used. +> - These steps can be skipped in development environments if alerting is not needed. + +- Alerting is part of cluster monitoring. Alert notifications are sent to a configured email address or Slack channel. +- Monitoring (Prometheus, Grafana, Alertmanager) must be deployed before setting up alerting. + +- Create a [Slack incoming webhook](https://api.slack.com/messaging/webhooks). + +- Update `slack_api_url` and the Slack channel name in `alertmanager.yml`: + +```bash +cd $K8_ROOT/alerting/ +nano alertmanager.yml +``` + +Update the following values: + +```yaml +global: + resolve_timeout: 5m + slack_api_url: +... +slack_configs: + - channel: '' + send_resolved: true +``` + +- Update the cluster name in `patch-cluster-name.yaml`: + +```bash +cd $K8_ROOT/alerting/ +nano patch-cluster-name.yaml +``` + +Update: + +```yaml +spec: + externalLabels: + cluster: +``` + +- Install the default and custom alerts: + +```bash +cd $K8_ROOT/alerting/ +./install.sh +``` + +--- + +## 12. Logging Module Setup and Installation + +> **Note:** +> - Logging is optional in sandbox environments. +> - For production environments, alternative logging tools may be used. +> - These steps can be skipped in development environments if logging is not needed. + +MOSIP uses [Rancher Fluentd](https://ranchermanager.docs.rancher.com/v2.0-v2.4/explanations/integrations-in-rancher/cluster-logging/fluentd) and Elasticsearch to collect logs from all services and display them in a Kibana dashboard. + +- Install the Rancher FluentD system (required to scrape logs from all MOSIP microservices): + - In Rancher UI, go to `Apps & Marketplaces` and install the Logging app. + - Select chart version `100.1.3+up3.17.7`. + +- Configure Rancher FluentD. Create the `clusteroutput`: + +```bash +kubectl apply -f clusteroutput-elasticsearch.yaml +``` + +- Create the `clusterflow`: + +```bash +kubectl apply -f clusterflow-elasticsearch.yaml +``` + +- Install Elasticsearch, Kibana, and Istio addons: + +```bash +cd $K8_ROOT/logging +./install.sh +``` + +- Set the `min_age` value in `elasticsearch-ilm-script.sh` (the minimum number of days to retain indices in Elasticsearch), then execute it: + +```bash +cd $K8_ROOT/logging +./elasticsearch-ilm-script.sh +``` + +- MOSIP provides the following Kibana dashboards: + +| Dashboard file | Description | +|----------------|-------------| +| [01-logstash.ndjson](https://github.com/mosip/k8s-infra/blob/v1.2.1.3/logging/dashboards/01-logstash.ndjson) | Logstash index pattern required by all other dashboards | +| [02-error-only-logs.ndjson](https://github.com/mosip/k8s-infra/blob/v1.2.1.3/logging/dashboards/02-error-only-logs.ndjson) | Shows only error-level logs (`MOSIP Error Logs` dashboard) | +| [03-service-logs.ndjson](https://github.com/mosip/k8s-infra/blob/v1.2.1.3/logging/dashboards/03-service-logs.ndjson) | Shows all logs for a specific service (`MOSIP Service Logs` dashboard) | +| [04-insight.ndjson](https://github.com/mosip/k8s-infra/blob/v1.2.1.3/logging/dashboards/04-insight.ndjson) | Shows MOSIP process insights such as UINs generated and packets uploaded (`MOSIP Insight` dashboard) | +| [05-response-time.ndjson](https://github.com/mosip/k8s-infra/blob/v1.2.1.3/logging/dashboards/05-response-time.ndjson) | Shows API response time trends across MOSIP services (`Response Time` dashboard) | + +- Import dashboards: + +```bash +cd $K8_ROOT/logging +./load_kibana_dashboards.sh ./dashboards +``` + +- View dashboards: open Kibana at `https://kibana.sandbox.xyz.net`, then navigate to **Menu** → **Dashboard** → select a dashboard. + +--- + +## 13. MOSIP External Dependencies Setup + +External dependencies include all external components required by MOSIP's core services: database (PostgreSQL), object store (MinIO), HSM, and others. + +```bash +cd $INFRA_ROOT/deployment/v3/external/all +./install-all.sh +``` + +See the [detailed installation instructions](https://docs.mosip.io/1.2.0/deploymentnew/v3-installation/mosip-external-dependencies) for all external components. + +> **Note:** After installation, connect to the `mosip_pms` PostgreSQL database and extend the `valid_to_date` for `mpolicy-default-mobile`: +> +> ```bash +> psql -h -p 5432 -U postgres -d mosip_pms +> ``` +> +> ```sql +> UPDATE pms.auth_policy +> SET valid_to_date = valid_to_date + interval '1 year' +> WHERE name = 'mpolicy-default-mobile'; +> ``` + +--- + +## 14. MOSIP Modules Deployment + +With the Kubernetes cluster and all external dependencies in place, proceed with MOSIP service deployment. + +```bash +cd $INFRA_ROOT/deployment/v3/mosip/all +./install-all.sh +``` + +> **Note:** +> - If `install-all.sh` fails at any point, follow the [MOSIP Modules Deployment](https://docs.mosip.io/1.2.0/deploymentnew/v3-installation/mosip-modules-deployment) guide from the point of failure. +> - The config-server and admin service may experience startup delays in this version. Apply the following fixes if needed: +> +> **For config-server** — increase `failureThreshold` for `startupProbe` to 60: +> +> ```bash +> kubectl -n config-server edit deployment config-server +> ``` +> +> **For admin-service** — increase `failureThreshold` for `startupProbe` to 60: +> +> ```bash +> kubectl -n admin edit deployment admin-service +> ``` +> +> Once admin-service is running, re-execute `install.sh` after commenting out the following lines: +> +> ```bash +> #echo Installing Admin-Proxy into Masterdata and Keymanager. +> #kubectl -n $NS apply -f admin-proxy.yaml +> #echo Installing admin hotlist service. +> #helm -n $NS install admin-hotlist mosip/admin-hotlist --version $CHART_VERSION +> #echo Installing admin service. Will wait till service gets installed. +> #helm -n $NS install admin-service mosip/admin-service \ +> # --set istio.corsPolicy.allowOrigins\[0\].prefix=https://$ADMIN_HOST \ +> # --wait --version $CHART_VERSION +> ``` + +--- + +## 15. API Testrig + +MOSIP's successful deployment can be verified by running the API testrig and comparing results against the testrig benchmark. + +- Navigate to the Infra Root Directory: + +```bash +cd $INFRA_ROOT +``` + +- Clone the Functional Tests repository: + +```bash +git clone -b v1.3.3 https://github.com/mosip/mosip-functional-tests.git +``` + +- Move to the apitestrig deployment directory: + +```bash +cd $INFRA_ROOT/mosip-functional-tests/deploy/apitestrig +``` + +- Make the script executable: + +```bash +chmod +x copy_cm_func.sh +``` + +- Run the installer: + +```bash +./install.sh +``` + +> **Note:** The script prompts for the following inputs: +> - Cronjob hour (0–23): e.g. `6` for 6 AM. +> - Do you have a public domain and valid SSL certificate? (Y/n) +> - Retention days for old reports (default: 3). +> - Slack Webhook URL for server issue notifications. +> - Is the eSignet service deployed? (yes/no) — if `no`, eSignet-related test cases are skipped. +> - Is `values.yaml` for the apitestrig chart configured as part of prerequisites? (Y/n) +> - Do you have S3 details for storing API-Testrig reports? (Y/n) + +--- + +## 16. DSL Rig + +### Install Packetcreator + +- Navigate to the packetcreator directory: + +```bash +cd $INFRA_ROOT/deployment/v3/testrig/packetcreator +``` + +- Run the installation script: + +```bash +./install.sh +``` + +- Provide the following inputs when prompted: + - NFS Host: + - NFS PEM File: + - User for SSH Login: `ubuntu` + - Ingress Controller Type: select `2` for Istio. + +### Install DSL Rig + +- Navigate to the dslrig directory: + +```bash +cd $INFRA_ROOT/deployment/v3/testrig/dslrig +``` + +- Run the installation script: + +```bash +./install.sh +``` + +> **Note:** Before running `install.sh`, ensure the following flag is included in the Helm install command: +> +> ``` +> --set dslorchestrator.configmaps.dslorchestrator.servicesNotDeployed="esignet" +> ``` +> +> The full Helm install command should look like: +> +> ```bash +> helm -n $NS install dslorchestrator mosip/dslorchestrator \ +> --set crontime="0 $time * * *" \ +> --version $CHART_VERSION \ +> --set dslorchestrator.configmaps.s3.s3-host='http://minio.minio:9000' \ +> --set dslorchestrator.configmaps.s3.s3-user-key='admin' \ +> --set dslorchestrator.configmaps.s3.s3-region='' \ +> --set dslorchestrator.configmaps.db.db-server="$DB_HOST" \ +> --set dslorchestrator.configmaps.db.db-su-user="postgres" \ +> --set dslorchestrator.configmaps.db.db-port="5432" \ +> --set dslorchestrator.configmaps.dslorchestrator.USER="$USER" \ +> --set dslorchestrator.configmaps.dslorchestrator.ENDPOINT="https://$API_INTERNAL_HOST" \ +> --set dslorchestrator.configmaps.dslorchestrator.packetUtilityBaseUrl="$packetUtilityBaseUrl" \ +> --set persistence.nfs.server="$NFS_HOST" \ +> --set persistence.nfs.path="/srv/nfs/mosip/dsl-scenarios/$ENV_NAME" \ +> --set dslorchestrator.configmaps.dslorchestrator.reportExpirationInDays="$reportExpirationInDays" \ +> --set dslorchestrator.configmaps.dslorchestrator.NS="$NS" \ +> --set dslorchestrator.configmaps.dslorchestrator.servicesNotDeployed="esignet" \ +> $ENABLE_INSECURE +> ``` +> +> When prompted, provide the following inputs: +> - NFS Host: +> - NFS PEM: +> - User for SSH Login: `ubuntu` +> - Cronjob hour (0–23). +> - Do you have a public domain and valid SSL certificate? (Y/n) +> - Packet Utility Base URL: `https://packetcreator.packetcreator:80/v1/packetcreator` +> - Retention days for old reports (default: 3). + +--- diff --git a/docs/setup/deploymentnew/v3-installation/1.2.1.0/overview-and-architecture.md b/docs/setup/deploymentnew/v3-installation/1.2.1.0/overview-and-architecture.md new file mode 100644 index 000000000..204f86abd --- /dev/null +++ b/docs/setup/deploymentnew/v3-installation/1.2.1.0/overview-and-architecture.md @@ -0,0 +1,39 @@ +# Overview and Architecture + +## Overview + +* MOSIP modules are deployed as microservices in a Kubernetes cluster. +* [Wireguard](https://www.wireguard.com/) is used as a trust network extension to access the admin, control, and observation panes. +* In case of on-field registration, WireGuard is extended to the registration client machines as one of the sources of trust. +* MOSIP uses [Nginx](https://www.nginx.com/) server for: + * SSL termination + * Reverse Proxy + * CDN/Cache management + * Load balancing +* Kubernetes cluster is administered using the [Rancher](https://rancher.com/docs/rancher/v1.3/en/kubernetes/#rancher-ui) and [rke](https://www.rancher.com/products/rke) tools. +* We have two Kubernetes clusters: + +**Observation cluster** - This cluster is part of the observation plane and assists with administrative tasks. By design, this is kept independent from the actual cluster as a good security practice and to ensure clear segregation of roles and responsibilities. As a best practice, this cluster or its services should be internal and should never be exposed to the external world. + +* [Rancher](https://rancher.com/docs/rancher/v1.3/en/kubernetes/#rancher-ui) is used for managing the MOSIP cluster. +* [Keycloak](https://www.keycloak.org/) in this cluster is used to manage user access and rights for the observation plane. +* It is recommended to configure log monitoring and network monitoring in this cluster. +* In case you have an internal container registry, then it should run here. + +**MOSIP cluster** - This cluster runs all the MOSIP components and certain third-party components like the kafka, keycloak etc. + +* [MOSIP External Components](https://github.com/mosip/mosip-infra/blob/v1.2.0.1-B1/deployment/v3/external/README.md#mosip-external-components) +* [MOSIP Services](https://github.com/mosip/mosip-infra/blob/v1.2.0.1-B1/deployment/v3/mosip/README.md#mosip-services) + +## Architecture diagram + +
+ +### Deployment repos + +These are the repos that we would use for installing and configuring the MOSIP deployment. For detailed steps, please follow the guide. This section is for reference. + +* [k8s-infra](https://github.com/mosip/k8s-infra/tree/v1.2.0.2) : contains the scripts to install and configure a Kubernetes cluster with required monitoring, logging, and alerting tools. +* [mosip-infra](https://github.com/mosip/mosip-infra/tree/v1.2.0.2/deployment/v3) : contains the deployment scripts to run charts in a defined sequence. +* [mosip-config](https://github.com/mosip/mosip-config/tree/v1.2.0.1) : contains all the configuration files required by the MOSIP modules. +* [mosip-helm](https://github.com/mosip/mosip-helm/tree/v1.2.0.2) : contains packaged Helm charts for all the MOSIP modules. diff --git a/docs/setup/deploymentnew/v3-installation/1.2.1.0/pre-requisites.md b/docs/setup/deploymentnew/v3-installation/1.2.1.0/pre-requisites.md new file mode 100644 index 000000000..88f81d44f --- /dev/null +++ b/docs/setup/deploymentnew/v3-installation/1.2.1.0/pre-requisites.md @@ -0,0 +1,90 @@ +# Pre-Requisites + +## Hardware requirements + +* VM’s required can be with any OS as per convenience. +* Here, we are referting to __Ubuntu OS (22.0.4)__ throughout this installation guide. +* Ensure to create Ubuntu user with its directory `/home/ubuntu` as the same user is referenced throughout the document. Else, change the user related commands as and when needed throughout the installation. + +
Sl no.PurposevCPU'sRAMStorage (HDD)no. ofVM'sHA
1.Wireguard Bastion Host24 GB8 GB1(ensure to setup active-passive)
2.Observation Cluster nodes28 GB32 GB22
3.Observation Nginx server (use Loadbalancer if required)24 GB16 GB1Nginx+
4.MOSIP Cluster nodes1232 GB128 GB66
5.MOSIP Nginx server ( use Loadbalancer if required)24 GB16 GB1Nginx+
+ +## Network requirements + +* All the VM's should be able to communicate with each other. +* Need stable Intra network connectivity between these VM's. +* All the VM's should have stable internet connectivity for docker image download (in case of local setup ensure to have a locally accessible docker registry). +* Server Interface requirement as mentioned in below table: + +
Sl no.PurposeNetwork Interfaces
1.Wireguard Bastion HostOne Private interface : that is on the same network as all the rest of nodes (e.g.: inside local NAT Network).

One public interface : Either has a direct public IP, or a firewall NAT (global address) rule that forwards traffic on 51820/udp port to this interface IP.
2.K8 Cluster nodesOne internal interface: with internet access and that is on the same network as all the rest of nodes (e.g.: inside local NAT Network )
3.Observation Nginx serverOne internal interface: with internet access and that is on the same network as all the rest of nodes (e.g.: inside local NAT Network).
4.Mosip Nginx serverOne internal interface : that is on the same network as all the rest of nodes (e.g.: inside local NAT Network).

One public interface : Either has a direct public IP, or a firewall NAT (global address) rule that forwards traffic on 443/tcp port to this interface IP.
+ +## DNS requirements +* Incase not using public DNS system please ignore this as the same mapping needs to be performed at node level and is mentioned in the deployment document. + +
Domain NameMapping detailsPurpose
1.rancher.xyz.netPrivate IP of Nginx server or load balancer for Observation clusterRancher dashboard to monitor and manage the kubernetes cluster.
2.keycloak.xyz.netPrivate IP of Nginx server for Observation clusterAdministrative IAM tool (keycloak). This is for the kubernetes administration.
3.sandbox.xyx.netPrivate IP of Nginx server for MOSIP clusterIndex page for links to different dashboards of MOSIP env. (This is just for reference, please do not expose this page in a real production or UAT environment)
4.api-internal.sandbox.xyz.netPrivate IP of Nginx server for MOSIP clusterInternal API’s are exposed through this domain. They are accessible privately over wireguard channel
5.api.sandbox.xyx.netPublic IP of Nginx server for MOSIP clusterAll the API’s that are publically usable are exposed using this domain.
6.prereg.sandbox.xyz.netPublic IP of Nginx server for MOSIP clusterDomain name for MOSIP's pre-registration portal. The portal is accessible publicly.
7.activemq.sandbox.xyx.netPrivate IP of Nginx server for MOSIP clusterProvides direct access to activemq dashboard. It is limited and can be used only over wireguard.
8.kibana.sandbox.xyx.netPrivate IP of Nginx server for MOSIP clusterOptional installation. Used to access kibana dashboard over wireguard.
9.regclient.sandbox.xyz.netPrivate IP of Nginx server for MOSIP clusterRegistration Client can be downloaded from this domain. It should be used over wireguard.
10.admin.sandbox.xyz.netPrivate IP of Nginx server for MOSIP clusterMOSIP's admin portal is exposed using this domain. This is an internal domain and is restricted to access over wireguard
11.object-store.sandbox.xyx.netPrivate IP of Nginx server for MOSIP clusterOptional- This domain is used to access the object server. Based on the object server that you choose map this domain accordingly. In our reference implementation, MinIO is used and this domain let's you access MinIO’s Console over wireguard
12.kafka.sandbox.xyz.netPrivate IP of Nginx server for MOSIP clusterKafka UI is installed as part of the MOSIP’s default installation. We can access kafka UI over wireguard. Mostly used for administrative needs.
13.iam.sandbox.xyz.netPrivate IP of Nginx server for MOSIP clusterMOSIP uses an OpenID Connect server to limit and manage access across all the services. The default installation comes with Keycloak. This domain is used to access the keycloak server over wireguard
14.postgres.sandbox.xyz.netPrivate IP of Nginx server for MOSIP clusterThis domain points to the postgres server. You can connect to postgres via port forwarding over wireguard
15.pmp.sandbox.xyz.netPrivate IP of Nginx server for MOSIP clusterMOSIP’s partner management portal is used to manage partners accessing partner management portal over wireguard
16.onboarder.sandbox.xyz.netPrivate IP of Nginx server for MOSIP clusterAccessing reports of MOSIP partner onboarding over wireguard
17.resident.sandbox.xyz.netPublic IP of Nginx server for MOSIP clusterAccessing resident portal publically
18.idp.sandbox.xyz.netPublic IP of Nginx server for MOSIP clusterAccessing IDP over public
19.smtp.sandbox.xyz.netPrivate IP of Nginx server for MOSIP clusterAccessing mock-smtp UI over wireguard
+ +## Certificate requirements + +As only secured https connections are allowed via nginx server will need below mentioned valid ssl certificates: + +* One valid wildcard ssl certificate related to domain used for accessing Observation cluster, this needs to be stored inside the nginx server VM for Observation cluster. In above e.g.: \*.org.net is the similar example domain. +* One valid wildcard ssl certificate related to domain used for accessing Mosip cluster, this needs to be stored inside the nginx server VM for mosip cluster. In above e.g.: \*.sandbox.xyz.net is the similar example domain. + +## Personal computer requirements +> Note: +> * In case unfamiliar with below mentioned tools and trying deployment from personel computer having Windows OS, suggestion is to use WSL for easy proceedings. +> * Please ensure the following tools with the specified versions are installed on your personal computer. + +* [git](https://github.com/git-guides/install-git) - version 2.25.1 or higher. +* [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl)- version 1.23.6 or higher +* [helm](https://helm.sh/docs/intro/install/)- any client version above 3.0.0 and add below repos as well: + +```bash +helm repo add bitnami https://charts.bitnami.com/bitnami +helm repo add mosip https://mosip.github.io/mosip-helm +``` + +* [Istioctl](https://istio.io/latest/docs/setup/getting-started/#download) : version: 1.15.0 +* [rke](https://rancher.com/docs/rke/latest/en/installation/) : version: [1.3.10](https://github.com/rancher/rke/releases/tag/v1.3.10) +* [Ansible](https://docs.ansible.com/ansible/latest/installation_guide/installation_distros.html) - version > 2.12.4 +* Openssl Need openssl version 1.1.1 specifically for regclient installation. + * Check the current OpenSSL version: + ```bash + openssl version + ``` + * If it's not OpenSSL 1.1.1f, remove the existing OpenSSL + ```bash + sudo apt remove openssl + ``` + * Manually install OpenSSL 1.1.1f by following this [guide](https://learnubuntu.com/install-openssl/#install-openssl-manually-in-ubuntu): + > Note : + > * While following the guide: + > * Replace all instances of openssl-1.1.1s with openssl-1.1.1f. + > * Skip the commands that create backups of binaries: + > ```bash + > sudo mv /usr/bin/c_rehash /usr/bin/c_rehash.backup sudo mv /usr/bin/openssl /usr/bin/openssl.backup + > ``` + > * After completing the installation, verify the OpenSSL version again: + > ```bash + > openssl version + > ``` + > * It should show: OpenSSL 1.1.1f + +* Once the above tools are installed, create a directory named `mosip` on your PC and follow the steps below: + + * clone k8’s infra repo with tag : 1.2.0.2 (**whichever is the latest version**) inside mosip directory. `git clone https://github.com/mosip/k8s-infra -b v1.2.0.2` + * clone mosip-infra with tag : 1.2.0.2 (**whichever is the latest version**) inside mosip directory. `git clone https://github.com/mosip/mosip-infra -b v1.2.0.2` + * Set below mentioned variables in bashrc + + ``` + export MOSIP_ROOT= + export K8_ROOT=$MOSIP_ROOT/k8s-infra + export INFRA_ROOT=$MOSIP_ROOT/mosip-infra + ``` + + `source ~/.bashrc` + + > Note: Above mentioned environment variables will be used throughout the installation to move between one directory to other to run install scripts. +* Ensure that nodes are accessible from your local machine for SSH authentication. +* Verify that the SSH private key for the nodes is available on your local machine. +* Supported operating systems include Ubuntu 24.04 and other Debian-based distributions. +