September 19, 2018

The default way to scale applications in Kubernetes (K8s) is by CPU usage. But not all problems are CPU based and scaling on other metrics like requests or latency is sometimes more suitable.
In this blogpost I will explain how you can use any kind of metric to scale your application on K8s using Spring Boot and Prometheus.

Objective

At the end of this post we have a running Horizontal Pod Autoscale (HPA) that scales my SampleApp on Actuator metrics. To reach this goal we will deploy Prometheus, which is used to scrape all our Pods. After that we will deploy the k8s-prometheus-adapter, which enables K8s to use the provided metrics for scaling.

Architecture for scaling Kubernetes with custom metrics

Architecture Diagram showing the services needed to scale on custom metrics

Prerequisite

Before starting with deploying everything, I want to list and introduce the required and the recommended tools.

  1. minikube or any other way to run Kubernetes locally
  2. Any way to build a Spring Boot Application, I use maven
  3. go, to get cfssl and other Tools that are really great
  4. hey to generate some load and traffic
  5. Finally jq to make the K8s API readable

 

After all the preparation we can start with the hand-on part

Sample Application Spring Boot

As we want to scale on Spring Boot Actuator metrics (at least for the sake of this demo), we need an application that provides these metrics to an endpoint. For this you have to copy the following dependencies into your pom.xml.

Those dependencies add the Endpoint /actuator/prometheus, which publishes the actuator metrics in the Prometheus format.

Deploying everything to Kubernetes

Let’s get to the most interesting part – deploying everything to Kubernetes.

Namespace

First you will have to deploy the namespace custom-metrics. Most of the parts will run in this namespace, for example the Prometheus instance.

Prometheus

Prometheus can either be deployed via the prometheus-operator or (as I did) via some Kubernetes yml files (see here).  To keep this easy, you don’t have to configure anything for Prometheus to get your deployed pods, those will be detected later via Kubernetes annotations. The given configuration will add some metrics for Kubernetes objects like cAdvisor.

k8s-prometheus-adapter

This is where the magic happens. To use the provided metrics and Prometheus query tools as a scaling target you have to provide the K8s API Endpoint custom.metrics.k8s.io. This will be done by the adapter. Let’s get this deployed.
I highly recommend reading the documentation for each of the following steps as I can’t go into all the nitty-gritty.

Generate certificates

Yes, you will have to generate some certificates. I used the gencerts.sh script from the Prometheus Operator.  You will need to install cfssl and openssl (if not already done) for this.

Deploy Adapter

After creating the certificates you can deploy the k8s-prometheus-adapter with this yaml. This also provides one rule that the adapter will use to fetch metrics.

Configure Adapter

The configuration is provided as YAML within an ConfigMap (already deployed) in Kubernetes.

This provides the metric http_server_requests to use within any HPA. The Adapter provides this metric for each Pod, mapped by the label kubernetes_pod_name. The value of this metric is generated by the following Prometheus query:

Sample App

To deploy the Sample App you need a Container Image and a deployment yaml. I will not describe how to create the image needed, as there are mulitple online resources available. For the deployment you can use just copy the following deployment.yaml and enter your image. Note that the important thing are the annotations. You need those so Prometheus scrapes your Actuator metrics from the pods.

 

Test if everything works

Lets check if every deployed object works. First we check our Prometheus instance. It should run and in the web interface show a kubernetes-pods job scaping our deployed sample app.

Prometheus Interface showing scraped kubernetes Pods

Prometheus Interface showing scraped kubernetes Pods

If this works we can skip the check our Sample App deployment as Prometheus can only pick it up if everything works correctly. Next we check the adapter. For this, simply query the Kubernetes API:

If the output looks like this, everything is ready to go.

k8s output showing the registered API endpoints

Output showing the registered API Endpoints for custom metrics

HPA on custom Metrics

Lets deploy the HPA and see, how Kubernetes autoscales our application.

HPA Autoscaling based on custom metrics

Pitfalls that happened to me

  1. Internal Server Error
    My metricsQuery within the Config was invalid. You can see this by trying to query a metric using

    If this returns an Internal Error, than most likely your config is not correct.
  2. Spring Boot App not providing metrics
    Most likely this is caused by not exposing the endpoints. In your application-properties (or yaml) dont forget ro expose at least the prometheus endpoint
  3. HPA not picking up metrics
    For whatever reason, at some point the deployed HPA did not pick up any metrics. To fix this is redeployed it and it worked, but I don’t think this is the best idea. Check the API Endpoints first to see that the value for metrics is correctly written. After that, check you HPAs events and see what the tell you. Most likely a redeploy will help, but if not it might be a problem with your configurations

I hope this helps you to find your problems and maybe solve them. Thanks for the read.

P.S. All my Ressources can be found in my repository: https://github.com/Etone/SpringBootActuatorScalingK8s

Leave a Comment

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close