Deploying Redis HA cluster in kubernetes

Redis is an in-memory key value data store which is used as a database, cache and message broker.

In this blog we will look at how to deploy Redis HA cluster in kubernetes.

Why High Availability?

Reliability is one of the key fundamentals for designing a system. Reliability means that our system works correctly even if some fault has happened.

One of the most common fault is server fail-stop and to be tolerant of this fault , most intuitive approach is to replicate the services over multiple servers and to provide a coherent interface to clients, abstracting all the complexities underlying our system. Therefore, if any server goes down, the request can be served by another server. In simple terms, this is High Availability (HA).

Redis High Availability approach.

Redis provides HA by implementing master — slave architecture. Master accepts all the writes and it synchronously replicates the same to all connected slaves. However, when master goes down, slaves cannot select a new master from among themselves on their own. This logic is outsourced to sentinels.

Redis-Sentinel is bundled with default redis package so we don’t need to install any thing separately. Sentinels are a group of (2n + 1) (recommended no. to form quorum, where n can be 1,2,3 and so on) processes which is responsible for monitoring the redis cluster and initiating failovers in case master goes down. During failovers sentinels select new master from the group of redis slaves by vote. The slave with most votes becomes the next master. Sentinels automatically change the configuration of the selected redis instance to master’s configuration and directs all other redis slaves to follow the new master.

Meanwhile, if a client was in between any operation, it will timeout and redis client-library will automatically create fresh connection factory to the new master with the help of sentinels. Thus, abstracting away the complexity from clients.

In this blog we will look at how to setup redis HA (sentinel based) in a minikube cluster. The same setup can be used for kubernetes cluster also.

Go to the repo here.

Normally, we use CICD(like Ansible) tool to deploy this setup in dev, qa, preprod, prod but here, for simplicity purpose, I will manually submit yamls.

Typically, below are the steps we need to follow to setup redis HA cluster.

1. Start first redis pod and store its cluster ip. We call this master.
2. Start 3 sentinel pods in succession after configuring master’s ip in its configuration file.
3. We then start slave pods either in succession or all at once. We then ask each slave to follow master by issuing following command ,

slaveof %masterip%

All the steps above are automated, except submitting yaml for service and deployments.

Clone the git repo,

> git clone https://github.com/sarweshsuman/redis-sentinel-ha-k8s-deployment.git

Start minikube,

> minikube start

Enable minikube docker env

eval $(minikube docker-env)

Next we create docker image of our redis instance,

> cd redis-sentinel-ha-k8s-deployment/docker
> docker build -t redis-sentinel-ha-cluster:1.0 .
.....
Successfully built 05c1e517000a
Successfully tagged redis-sentinel-ha-cluster:1.0

Lets look at the docker file,

We need separate configuration files for master and slave, so that if a pod starts as a master it uses master configuration and if slave, then slave configuration. Both configuration files have placeholders which are filled up while the pod is starting up. These configuration files can be further customised for specific use.

Before we proceed lets quickly look at entrypoint.sh script. All the logic is in this script. It expects some env variable to be set when the pods are started like, whether the pod is master or sentinel, sentinel service host & port, master service host & port.

The algorithm for entrypoint script can be summed up as following,

[Non Sentinel Part]

1. After startup the pod first tries to connect to the sentinel service.
2. If it succeeds then, the pod understands that it is not a master pod, there already exists a master, it will retrieve the master’s ip and start as slave redis instance.
3. If it can’t connect to sentinel and if env variable “master” is set to true, then it starts as a master redis instance else start as redis slave.
4. When it starts as master redis instance, it sets a key “STARTUP_MASTER_IP” to its cluster ip which is later used by sentinel processes for startup.

[Sentinel Part]

1. Sentinel pods have “sentinel” env variable set to true.
2. When a sentinel pod starts up, it tries to see if there already exists a sentinel process running, if yes, then it connects to that sentinel process and retrieves master’s ip.
3. Else, it then connects to the master via the master service and retrieves the master’s ip.
4. Then starts up sentinel service using master’s ip.

This script takes care of startup, new membership and post failover scenarios.

Let’s proceed with next step, we first create all the needed kubernetes services.

> cd ..
> ls
Readme.md   
create-master-deployment.yaml 
create-sentinel-deployment.yaml 
create-service.yaml  
create-slave-deployment.yaml
docker
> kubectl apply -f create-service.yaml

This step comes first because once the service is created, corresponding env variables will be automatically injected by kubernetes in future redis pods.

Next we create master deployment first,

> kubectl apply -f create-master-deployment.yaml

Then, we create sentinel deployment, its has replication set to 3, as we need minimum 3 replicas to form a quorum.

> kubectl apply -f create-sentinel-deployment.yaml

Next, we create redis slave deployment.

> kubectl apply -f create-slave-deployment.yaml

Finally, our slave pod is up as well. This completes our deployment and this is all that we have to do.

Lets look at the log snippet from one of the sentinel pod,

• monitor master mymaster 172.17.0.2 6379 quorum 2 , tells us that sentinels are now monitoring the master.
• slave slave 172.17.0.6:6379 172.17.0.6 6379 @ mymaster 172.17.0.2 6379, tells us that sentinels knows about the slave after it joins the cluster.

Lets delete the master pod and see if we can trigger failover,

> kubectl delete pod redis-ha-cluster-master-d1-f7cd4cc58-8md4z

Yay! from the log, we can see that sentinels are able to detect that the master with ip “172.17.0.2” went down and then they select new master with ip “172.17.0.6”. *It is to note that the slave pod name might contain term master, after failover, pod name is only for demonstration purpose here.

The master, slave and sentinel deployment makes sure that the cluster is always in desired state. The sentinel service is used by clients to connect to the master for writes and reads.

It is important to note that here applications outside the kubernetes cluster cannot connect to this redis cluster as we are working with cluster ip. To enable external connections we need to do additional changes which is out of scope of this blog.

Master Slave approach is not always safe.

Naive implementation of master — slave synchronous replication is not always safe. It means that we can create failure sequences where the database becomes unavailable even with just one node down. For example, if a slave goes down, master continues to receive writes, then if the master goes down the cluster stops. Then if the former slave comes up and becomes master then the writes to previous master is lost.

However, redis master — slave approach is safe. Sentinels makes sure if a slave that joins the cluster with old state in absence of a master does not become new master, in this case the cluster halts and requires manual intervention. There are additional measures that can be enabled in redis configuration to make sure we don’t loose any data like setting min-slaves-to-write and min-slaves-max-lag.

With kubernetes deployment configured, redis cluster becomes even more reliable.

With this, I will end this blog here.