Scaling Redis Cluster via Kubernetes Horizontal Pod Autoscaler

In this article I will take you through demo of a Horizontally Auto Scaling Redis Cluster with the help of Kubernetes HPA configuration.

Note:

• I am using minikube for demo purpose, but the code and yaml can be used in a Kubernetes cluster with little to none modifications.
• This is not a production ready setup. It is only a prototype.

Introduction

Redis Cluster provides a way to shard data automatically between multiple nodes. This enables properties like horizontal scaling, high availability and partition tolerance in a system. More info about redis cluster is here.

Horizontal Pod Autoscaler automatically scales pods in a deployment/replication controller/replica set/stateful set based on current metrics. Metrics can be cpu, memory or custom metrics based on application properties. More info on HPA is here.

Autoscaling Redis Cluster Architecture

Following components are involved,

1. Redis Standalone Instances starts up in cluster mode.
2. Redis Cluster Coordinator written in GoLang that creates cluster and registers new pods as master or slave as they come.
3. Controller Manager and HPA configuration that tracks the average memory utilisation and creates new Pod.

Typical scenario is as follows,

• Create redis-cluster-coordinator deployment that starts a pod and runs the Redis Cluster Coordinator.
• Create redis-autoscale-cluster deployment with 6 replicas for redis-cluster.
• Each redis pod starts up as individual instance first and registers itself with Redis Cluster Coordinator.
• After seed nodes have joined, Redis Cluster Coordinator creates redis cluster.
• Create HPA configuration — redis-autoscale-cluster with 60% average utilisation target.
• If the average memory usage of pods in deployment redis-autoscale-cluster i.e. in redis cluster goes above certain threshold, Controller Manager creates a new redis pod.
• The new redis pod registers itself with redis cluster coordinator.
• Redis cluster coordinator adds it to the redis cluster and based on redis cluster state — if the memory is above threshold or if some slave is missing configures the new redis pod.

Setup

The complete code is available at Git repo HERE.

Redis Cluster Coordinator code is available HERE.

We will setup a 6 pod redis cluster where 3 are master pod and rest 3 are slaves. Slots are sharded between 3 master pods. Following which we will test the scale up feature of redis cluster by simulating high memory usage in one of the master pods.

Let’s first create all the docker images from docker folder HERE.

eval $(minikube docker-env)
cd docker/redis-autoscale-cluster
docker build -t redis-autoscale-cluster:0.1 .
cd docker/redis-cluster-controller
docker build -t redis-cluster-controller:0.1 .

By default minikube does not have metric server enabled, therefore we need to first enable it with below command.

minikube addons enable metrics-server

Next we will register all the needed services. Service YAML is available HERE.

kubectl create -f redis-autoscale-all-service.yaml

Next, we will create the HPA object. YAML is available HERE.

kubectl create -f redis-autoscale-cluster-hpa.yaml

I have set min replicas to 6 and max replicas to 12. Max replica can be updated based on estimate of how much scaling is expected in near future. The targets field shows <unknown> because redis cluster pods are not started yet.

Note, I am using minikube with kubernetes v1.17.3, in this version we cannot configure behavior section in HPA. Using behavior section we can disable scale down option. We don’t want our cluster to scale down automatically because that could lead to data loss. However, it will be interesting to see if we can configure what pods Controller Manager should delete in process of scale down operation.

Next, we will create all the deployments. YAML is HERE and HERE

kubectl create -f redis-cluster-coordinator-deployment.yaml
kubectl create -f redis-autoscale-cluster-deployment.yaml

After all the redis pods comes up the cluster is formed. The logs of redis cluster coordinator confirm that all the redis pods are now part of the redis cluster. From below we see 3 pods are master and rest 3 are slaves.

Redis Cluster Coordinator Log

At this point, we can already see metrics are being collected and HPA is showing the targets field correctly.

Testing

In the HPA YAML definition, I have set the average utilisation to 60 just for demo purpose. In production this value should be based on application load pattern.

Let’s test our first scenario, slave pod dies.

From the above redis cluster coordinator log, we will select slave pod redis-autoscale-cluster-5fc7b66d9c-22wk6 (IP: 172.17.0.9) which is replica of master pod redis-autoscale-cluster-5fc7b66d9c-fr9d9 (IP: 172.17.0.6).

kubectl delete pod redis-autoscale-cluster-5fc7b66d9c-22wk6

Upon deleting this pod, to maintain the desired state deployment controller will quickly start a new pod. This new pod is fresh Redis instance that is not part of the cluster yet. So, it should become part of the cluster and should be configured as slave to the same master pod “redis-autoscale-cluster-5fc7b66d9c-fr9d9” (IP: 172.17.0.6) .

From below redis cluster coordinator log, we can see new pod is configured replica of the same master node. Our first test case succeeds.

Let’s test our second scenario, master pod dies.

We will delete master pod redis-autoscale-cluster-5fc7b66d9c-fr9d9 ( IP: 172.17.0.6 ) from previous scenario.

kubectl delete pod redis-autoscale-cluster-5fc7b66d9c-fr9d9

Unlike previous scenario, in this scenario first redis cluster will complete failover of slave ( Pod with IP : 172.17.0.12 ) to master redis pod. Then redis cluster coordinator will configure the new pod as slave of this master pod.

Our second test case also succeeds. The new pod ( IP : 172.17.0.9 ) is configured slave to master pod ( IP: 172.17.0.12 ).

Our final test scenario is scaling based on high memory utilisation.

We start by adding dummy data into a redis master pod. Dummy data is HERE. For testing purpose, we can login to one of the nodes and run the shell script.

We can see memory usage of each pod with below command,

kubectl top pod

From the above screenshot, we can see the master pod “redis-autoscale-cluster-5fc7b66d9c-v45gs” takes 11 mb of memory and its replica takes 10 mb of memory. This pushes the average memory utilisation above 60%.

kubectl get hpa -o wide

Cluster controller at this point creates two new pods. From cluster controller logs,

Now, Redis cluster coordinator configures the first new pod as master and of-load 50% of the slot from the master pod “redis-autoscale-cluster-5fc7b66d9c-v45g” to new master pod “redis-autoscale-cluster-5fc7b66d9c-j4464”.

We can see that now the slots are shared between the two masters,

This concludes the demo.

Conclusion

In this article, we saw how we can configure autoscaling redis cluster via HPA. Kubernetes controller manager automatically creates new replicas of redis pod when average memory usage goes above the average utilisation. The redis cluster coordinator then detects and configures the new redis pod as part of redis cluster.

To note,

• The redis cluster coordinator code does not covers all the cases. Currently, it is just a prototype. If you noticed, the second new redis pod is not configured properly, it is a master without slots. In future I will be iterating over all the different cases and hardening the coordinator code.
• Scaling Down leads to data loss as kubernetes can delete any pod. We can disable scale down in kubernetes version ≥ v1.18.0-alpha.1. It will be interesting research to see how we can scale down redis cluster without data loss.
• None of YAML/code is close to production ready. It is intended to be a starting point for production ready YAML/code.