K8SAutoscale InCluster Configuration Problem

[lamchakchan]

I'm currently not able to get the k8sAutoscale() node to work. When the TICK script is enabled, I get an error returned.

cannot use the k8sAutoscale node, could not create kubernetes client: unknown kubernetes cluster ""

I've configured kapacitord using environment variables. With my kapacitor docker container launched within my Kubernetes cluster. I verified the environment variables exist.

printenv | grep KAPACITOR_KUBERNETES
KAPACITOR_KUBERNETES_ENABLED=true
KAPACITOR_KUBERNETES_IN_CLUSTER=true

I tried to find the initializer for the In-Cluster configuration and found this line. Should this reference the os.Getenv("KUBERNETES_SERVICE_HOST") instead of having http://kubernetes hardcoded in as the url?

https://github.com/influxdata/kapacitor/blob/master/services/k8s/client/client.go#L95

Version
Using official kapacitor:1.3 image

kapacitor version
Kapacitor 1.3.3 (git: HEAD ce586f35e89e75a1779e2b493caba15d66295a15)

Environment
Running inside Kubernetes 1.7.x within AWS

[nathanielc]

Could you share the configuration file being used by the process?

Unfortunately there are some weird edge cases with using env to configure Kapacitor where the configuration file needs to have the right place holders in order to work.

[lamchakchan]

The kapacitor.conf is the default one packaged with the kapacitor Docker image. In my setup, im not overriding this file. Just layering configurations with ENV vars.

[lamchakchan]

I'm installing this using helm. Here is the command with some values blanked out.

helm install <package-name> \
--name kapacitor \
--set controler.replicaCount=1 \
--set image.repository=kapacitor \
--set image.tag=1.3 \
--set imagePullPolicy=ifNotPresent \
--set service.type=NodePort \
--set service.networkProtocol=TCP \
--set service.internalPort=9092 \
--set service.externalPort=9092 \
--set service.nodePort=30992 \
--set "env[0].name=KAPACITOR_LOGGING_LEVEL,env[0].value=DEBUG" \
--set "env[1].name=KAPACITOR_KUBERNETES_ENABLED,env[1].value=true" \
--set "env[2].name=KAPACITOR_KUBERNETES_IN_CLUSTER,env[2].value=true" \
--set "env[3].name=KAPACITOR_DATA_DIR,env[3].value=/kapacitor" \
--set "env[4].name=KAPACITOR_REPLAY_DIR,env[4].value=/kapacitor/replay" \
--set "env[5].name=KAPACITOR_INFLUXDB_0_URLS_0,env[5].value=<influxdb-url>" \
--set "env[6].name=KAPACITOR_INFLUXDB_0_USERNAME,env[6].value=<influxdb-user>" \
--set "env[7].name=KAPACITOR_INFLUXDB_0_PASSWORD,env[7].value=<influxdb-password>" \
--set "env[8].name=KAPACITOR_INFLUXDB_0_DB,env[8].value=<influxdb-database>" \
--set "env[9].name=KAPACITOR_INFLUXDB_0_RP,env[9].value=<influxdb-rp>" \
--set "env[10].name=KAPACITOR_INFLUXDB_0_SUBSCRIPTION_PROTOCOL,env[10].value=http" \
--set "env[11].name=KAPACITOR_INFLUXDB_0_HTTP_PORT,env[11].value=30992" \
--set "env[12].name=KAPACITOR_INFLUXDB_0_KAPACITOR_HOSTNAME,env[12].value=<reachable-host-name>" \
--set "volumes[0].name=kapacitor-volume,volumes[0].hostPath.path=/data/kapacitor" \
--set "volumeMounts[0].mountPath=/kapacitor,volumeMounts[0].name=kapacitor-volume" \
--namespace kube-system

[lamchakchan]

My TICK script would also help too. I realized there is a node.cluster property for the k8autoscale node which I'm not using. But I also don't see it in the documentation for 1.3 either.

// database
var database string
// retention policy for database
var retentionPolicy string
// dataset collected within the retention policy
var measurement string
// Optional where filter
var where_filter = lambda: TRUE
// Optional list of group by dimensions
var groups = ['host']
// Which field to process
var field string
// The time scale to calculate the average against
var movingAverageCount = 60
// Deployment this is scaling for
var deploymentName = 'placeholder'
// Threshold for triggering
var target = 10.0
// time interval per scaling up
var scalingCooldown = 1m
// time interval per scaling down
var descalingCooldown = 2m
	
stream
	|from()
		.database(database)
		.retentionPolicy(retentionPolicy)
		.measurement(measurement)
		.where(where_filter)
		.groupBy(groups)
		.truncate(1s)
	// Compute the rate of requests per second per host
	|derivative(field)
		.as('point_per_second')
		.unit(1s)
		.nonNegative()
	|alert()
		.crit(lambda: "point_per_second" > target)
		.log('/var/log/test.log')
	|sum('point_per_second')
		.as('total_point_per_second')
	|movingAverage('total_point_per_second', movingAverageCount)
		.as('avg_point_per_second')
	// add window()
	|k8sAutoscale()
		// We are scaling a deployment.
		.kind('deployments')
		// The name of the replicaset to scale is found in the 'replicaset' tag.
		.resourceName(deploymentName)
		// Set the cool down timer values.
		.increaseCooldown(scalingCooldown)
		.decreaseCooldown(descalingCooldown)
		// Compute the desired number of replicas based on the
		// avg_point_per_second and target values.
		.replicas(lambda: int(ceil("avg_point_per_second" / target)))

[lamchakchan]

@nathanielc Is there a reason why k8s.NewConfig() isn't part of the initialization process in this code block?

[lamchakchan]

I think the above missing code is it. I'm going to build a local copy and test it and I'll make a PR for this.

[nathanielc]

@lamchakchan Sorry for the late response, I have been out. I believe the NewConfig() was missing because of a backwards compatibility issue. Something about going from having just one k8s configuration section to allowing a list of them. I'll dig into this today and let you know. I saw your PR #1616, I'll let you know what I find.

[nathanielc]

@lamchakchan Does it work for you if you use KAPACITOR_KUBERNETES_0_ENABLED and KAPACITOR_KUBERNETES_0_IN_CLUSTER?

[nathanielc]

@lamchakchan I have been unable to reproduce your error. Using either the index or non-index env var names. Is it possible to share your helm package?

[lamchakchan]

@nathanielc sorry for the delay in getting back to you. I'm still getting the error when trying the indexed _0_ environment key. I'm available for a call to expedite this.

I can share the --debug output from the helm processing. I <REDACTED> some values. If you put the contents below into a single file and then follow up with a kubectl create -f <file> will create you the kapacitor pods I have.

# Source: xo-app-manifest/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: kapacitor-test-1
  labels:
        app: kapacitor
        environment: test
        role: test-worker
        stack: maintainence

spec:
  type: ClusterIP
  ports:
    - port: 9092
      targetPort: 9092
      protocol: TCP
      name: kapacitor-test-1
  selector:
    environment: test
    stack: maintainence
    role: test-worker
---
# Source: xo-app-manifest/templates/deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: kapacitor-test-1-xo-app-manifest
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: kapacitor
        environment: test
        role: test-worker
        stack: maintainence

    spec:
      containers:
      - name: kapacitor-test-1
        image: kapacitor:1.3
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - mountPath: /kapacitor
          name: kapacitor-volume

        env:
          - name: "KAPACITOR_LOGGING_LEVEL"
            value: "DEBUG"
          - name: "KAPACITOR_KUBERNETES_0_ENABLED"
            value: "true"
          - name: "KAPACITOR_KUBERNETES_0_IN_CLUSTER"
            value: "true"
          - name: "KAPACITOR_DATA_DIR"
            value: "/kapacitor"
          - name: "KAPACITOR_REPLAY_DIR"
            value: "/kapacitor/replay"
          - name: "KAPACITOR_INFLUXDB_0_URLS_0"
            value: "<REDACTED>"
          - name: "KAPACITOR_INFLUXDB_0_USERNAME"
            value: "kapacitor"
          - name: "KAPACITOR_INFLUXDB_0_PASSWORD"
            value: "mink-U9UPje!"
          - name: "KAPACITOR_INFLUXDB_0_DB"
            value: "kubernetes"
          - name: "KAPACITOR_INFLUXDB_0_RP"
            value: "2-weeks"
          - name: "KAPACITOR_INFLUXDB_0_INSECURE_SKIP_VERIFY"
            value: "true"
          - name: "KAPACITOR_INFLUXDB_0_SUBSCRIPTION_PROTOCOL"
            value: "http"
          - name: "KAPACITOR_INFLUXDB_0_HTTP_PORT"
            value: "9092"
          - name: "KAPACITOR_INFLUXDB_0_KAPACITOR_HOSTNAME"
            value: "<REDACTED>"
        resources:
          limits:
            cpu: 100m
            memory: 128Mi
          requests:
            cpu: 100m
            memory: 128Mi

        ports:
        - containerPort: 9092
      volumes:
        - hostPath:
            path: /data/kapacitor
          name: kapacitor-volume

After the pod boots up, I kubectl exec -it into it and installed this TICK script using kapacitor define.

stream
    |from()
        .database('kubernetes')
        .retentionPolicy('2-weeks')
        .measurement('docker_container_cpu')
        .where(lambda: ("io.kubernetes.container.name" == 'kapacitor' OR "io.kubernetes.container.name" == 'heapster')  AND "environment" == 'preproduction')
        .truncate(1s)
    // |groupBy('container_name')
    |alert()
        .crit(lambda: "usage_percent" > 0)
        .log('/var/log/everything.log')
    |movingAverage('usage_percent', 30)
        .as('average_resource')
    |eval(lambda: int(ceil("average_resource" / float(19))))
        .as('replicaCount')
    |alert()
        .crit(lambda: "replicaCount" > 0)
        .log('/var/log/test.log')
    |k8sAutoscale()
	// 	// We are scaling a deployment.
 	.kind('deployments')
	// 	// The namespace of the deployment
 	.namespace('default')
	// 	// The name of the replicaset to scale is found in the 'replicaset' tag.
 	.resourceName('test-loader-xo-app-manifest')
	// 	// Set the cool down timer values.
 	.increaseCooldown(2m)
 	.decreaseCooldown(5m)
	// 	// The minimum amount of replica to have regardless of averages
 	.min(2)
	// 	// Compute the desired number of replicas based on the
	// 	// avg_point_per_second and target values.
 	.replicas(lambda: "replicaCount")

Here are the result of this action.

root@kapacitor-test-xo-app-manifest-3378918348-xf0qd:~# kapacitor define test-loader -dbrp kubernetes.2-weeks -tick ./ter.tick -type stream
root@kapacitor-test-xo-app-manifest-3378918348-xf0qd:~# kapacitor show test-loader
ID: test-loader
Error:
Template:
Type: stream
Status: disabled
Executing: false
Created: 13 Nov 17 16:08 UTC
Modified: 13 Nov 17 16:08 UTC
LastEnabled: 01 Jan 01 00:00 UTC
Databases Retention Policies: ["kubernetes"."2-weeks"]
TICKscript:
stream
    |from()
        .database('kubernetes')
        .retentionPolicy('2-weeks')
        .measurement('docker_container_cpu')
        .where(lambda: ("io.kubernetes.container.name" == 'kapacitor' OR "io.kubernetes.container.name" == 'heapster') AND "environment" == 'preproduction')
        .truncate(1s)
    // |groupBy('container_name')
    |alert()
        .crit(lambda: "usage_percent" > 0)
        .log('/var/log/everything.log')
    |movingAverage('usage_percent', 30)
        .as('average_resource')
    |eval(lambda: int(ceil("average_resource" / float(19))))
        .as('replicaCount')
    |alert()
        .crit(lambda: "replicaCount" > 0)
        .log('/var/log/test.log')
    |k8sAutoscale()
        //     // We are scaling a deployment.
        .kind('deployments')
        //     // The namespace of the deployment
        .namespace('default')
        //     // The name of the replicaset to scale is found in the 'replicaset' tag.
        .resourceName('test-loader-xo-app-manifest')
        //     // Set the cool down timer values.
        .increaseCooldown(2m)
        .decreaseCooldown(5m)
        //     // The minimum amount of replica to have regardless of averages
        .min(2)
        //     // Compute the desired number of replicas based on the
        //     // avg_point_per_second and target values.
        .replicas(lambda: "replicaCount")

DOT:
digraph test-loader {
stream0 -> from1;
from1 -> alert2;
alert2 -> movingAverage3;
movingAverage3 -> eval4;
eval4 -> alert5;
alert5 -> k8s_autoscale6;
}
root@kapacitor-test-xo-app-manifest-3378918348-xf0qd:~# kapacitor enable test-loader
enabling task test-loader: cannot use the k8sAutoscale node, could not create kubernetes client: unknown kubernetes cluster "", cannot get client

[lamchakchan]

Also, the printenv output for the related session from above.

root@kapacitor-test-xo-app-manifest-3378918348-xf0qd:~# printenv | grep KUBERNETES_0
KAPACITOR_KUBERNETES_0_IN_CLUSTER=true
KAPACITOR_KUBERNETES_0_ENABLED=true

[neuhausler]

Run in to the same issue .. fixed it by just adding enable kubernetes to kapacitor.conf .. rest of configuration can still be overwritten with environment variables

 [kubernetes]
    enabled = true

.. as a side note, that can all be done in Kubernetes using a configMap volume without the need of having to rebuild the docker image. That configMap can then be mounted in to /etc/kapacitor

apiVersion: v1
kind: ConfigMap
metadata:
  name: kapacitor-config
  namespace: crawler
data:
  kapacitor.conf: |
    data_dir = "/var/lib/kapacitor"

    [replay]
      dir = "/var/lib/kapacitor/replay"

    [storage]
      boltdb = "/var/lib/kapacitor/kapacitor.db"
  
    [kubernetes]
      enabled = true
      in-cluster = true

    [logging]
      file = "STDERR"
      level = "DEBUG"

    [config-override]
      enabled = true