KubeObservability-Lab

A Go microservice running on Kubernetes with full observability — Prometheus metrics, Grafana dashboards, horizontal autoscaling, and a GitHub Actions CI pipeline.

Stack: Go · Docker · Kubernetes · Helm · Prometheus · Grafana · GitHub Actions

---

What's in here

app/
  main.go          ← the Go service (3 routes, Prometheus metrics)
  Dockerfile       ← multi-stage build (builder → tiny alpine image)
  go.mod / go.sum  ← dependencies

k8s/
  configmap.yaml   ← non-secret config (env vars)
  secret.yaml      ← secret config (API keys etc.)
  deployment.yaml  ← runs 2 pods, health checks, resource limits
  service.yaml     ← ClusterIP to reach the pods inside the cluster
  hpa.yaml         ← auto-scales pods 2→5 when CPU > 50%
  servicemonitor.yaml ← tells Prometheus to scrape this service

.github/workflows/
  ci.yaml          ← build → test → Docker build → CVE scan → manifest lint

---

The Go service

Three routes:

Route	What it does
GET /	Returns a hello message
GET /healthz	Returns ok — Kubernetes uses this for liveness/readiness probes
GET /roll	Sleeps 10–300ms and has a 10% error rate — good for dashboard testing

All routes track two Prometheus metrics:

• http_requests_total — counter by path and status code
• http_request_duration_seconds — histogram of latency

Prometheus scrapes these at /metrics.

---

Run locally (no Kubernetes)

cd app
go run .
# visit http://localhost:8080/roll  or  http://localhost:8080/metrics

---

Deploy to Kubernetes

Prerequisites

• Docker Desktop with Kubernetes enabled, or minikube
• kubectl
• Helm

Step 1 — Build and push the Docker image

docker build -t <your-dockerhub-username>/kubeobs-app:latest ./app
docker push <your-dockerhub-username>/kubeobs-app:latest

Update the image name in k8s/deployment.yaml line 21 to match.

Step 2 — Apply all Kubernetes manifests

kubectl apply -f k8s/

Check everything is running:

kubectl get pods
kubectl get hpa

Step 3 — Install Prometheus + Grafana via Helm

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace

This installs Prometheus, Grafana, and Alertmanager all at once.

Step 4 — Open Grafana

kubectl port-forward -n monitoring svc/kube-prometheus-stack-grafana 3000:80

Visit http://localhost:3000
Default login: admin / prom-operator

Step 5 — Add dashboards

In Grafana, go to Dashboards → Import and paste these IDs:

Dashboard	ID
Kubernetes cluster overview	315
Go runtime metrics	6671

Or build your own — in the query box type http_requests_total or http_request_duration_seconds and Grafana will autocomplete.

---

Generate load to test autoscaling

Open a terminal and run this loop — it hits /roll 100 times per second:

kubectl run load --image=busybox --restart=Never -- \
  /bin/sh -c "while true; do wget -q -O- http://kubeobs-app/roll; done"

Watch the HPA react:

kubectl get hpa -w

You'll see REPLICAS climb from 2 toward 5 as CPU goes above 50%.

When you're done:

kubectl delete pod load

---

GitHub Actions CI

Every push runs .github/workflows/ci.yaml:

1. go test — unit tests
2. go build — makes sure the code compiles
3. docker build — produces the image
4. Trivy — scans the image for HIGH/CRITICAL CVEs
5. kubeval — validates all YAML in k8s/ against the Kubernetes schema

---

How each Kubernetes piece fits together

You (laptop)
    │
    │  kubectl port-forward
    ▼
Service (ClusterIP)          ← routes traffic to healthy pods
    │
    ▼
Deployment                   ← manages the pods, rolls out updates
    │
    ├── Pod 1  (/healthz ← liveness + readiness probe)
    └── Pod 2  (/metrics ← Prometheus scrapes here)

HPA watches CPU → adds/removes pods from the Deployment

ConfigMap + Secret → injected as env vars into each pod

ServiceMonitor → tells Prometheus which Service to scrape

Prometheus → stores metrics
    │
    ▼
Grafana → visualizes them

---

Tear down

kubectl delete -f k8s/
helm uninstall kube-prometheus-stack -n monitoring