Technology Sharing

한어Русский языкEnglishFrançaisIndonesianSanskrit日本語DeutschPortuguêsΕλληνικάespañolItalianoSuomalainenLatina

Introduction

It is relatively troublesome to build stateful services in a k8s cluster. For example, the current mainstream approach to building redis is to use shared storage ceph and nas to achieve data persistence. Some are achieved by using node affinity and hostpath. Currently, most operators use shared storage. This article will use hostpath affinity weighted multiple copies to build a redis cluster based on the existing environment and different requirements to achieve data persistence and high availability of redis services.

1. Conditions and environment description:

The k8s version is k8s-1.29.4. The environment is built in the telecommunications room, with six worker nodes. Each node has a non-system SSD disk mounted to the /data/ path. It does not use shared storage such as ceph or nas, nor does it use operator. Currently, redis-operator basically uses shared storage.

2. Requirements:

• Build a redis sentinel cluster.
• High availability: Even if a k8s node dies or the pod is restarted, it will not affect the use.
• High performance: Reading and writing local disks achieves high-performance IO. If shared storage resources and hardware performance are insufficient, IO will become a performance bottleneck.
• Data security: After a switch or node failure, data should be kept as complete as possible.
• Simple installation, easy management and maintenance

3. Implementation principle and description

• 1. Build a redis sentinel cluster, one master, two slaves, and three sentinel processes.
• 2. Redis uses three independent statefulsets and one replica for construction, and the sentinel process uses one deployment and three replicas for construction.
• 3. The redis process uses node affinity and weight to implement master and backup nodes.
• 4. Redis uses service headless service, and Sentinel uses service service.
• 5. A group of two slaves is configured as master and backup based on weights. You need to plan the label names of the six nodes.

Note: The redis built by this solution can only be used in the k8s cluster and cannot be provided to the outside of the cluster through LoadBlancer or NodePort, because the IP of the master redis elected in the sentinel is the IP of the pod. When a switch occurs, it is impossible to determine which master redis is.

4. Detailed steps

4.1. Planning node labels

Note: The role division here means that since redis has one master and two slaves, each redis pod falls on three master nodes. When the master node fails, it switches to the standby node. For example, the pod named defaultapp-redis-a-0 falls on the node with the label storage-selector=node-a-master by default. When the node fails, it will be matched according to the weight and placed on the node with storage-selector=node-a-slve.

4.2. Create configmap configuration

The configuration contains four files, the distribution description is as follows:
sentinel.conf: Sentinel configuration file. By default, the first statefulset is used as the main redis. The configuration: resolve-hostnames supports host names.
redis-master.conf: main redis configuration file, memory is configured to 256M, port is configured to 6379, password is configured to redis#123, storage path is /data/redis.
redis.conf: From the redis configuration file, the memory is configured to 256M, the port is configured to 6380, the password is configured to: redis#123, the storage path is /data/redis-2, and the data is synchronized from the master redis.
run.sh: Its main function is to determine the role to run different configuration files, create a directory according to the host name, and then soft-link to the storage directory. This step is mainly to store the redis data of each service in its own directory, so that when a new redis is created, it will not cause directory conflicts on the node.

apiVersion: v1
kind: ConfigMap
metadata:
  name: defaultapp-redis-config
  namespace: default-ns
  labels:
    appname: default-app
    app: defaultapp-redis-config
data:
  sentinel.conf: |
    sentinel monitor mymaster defaultapp-redis-a-0.defaultapp-redis-a.default-ns.svc.cluster.local 6379 2
    sentinel auth-pass mymaster redis#123
    sentinel down-after-milliseconds mymaster 30000
    sentinel parallel-syncs mymaster 1
    sentinel failover-timeout mymaster 180000
    SENTINEL resolve-hostnames yes
    SENTINEL announce-hostnames yes
  redis-master.conf: |
    maxmemory 256mb
    requirepass redis#123
    save 3600 1 300 10 60 100
    dir /data/redis
    masterauth redis#123
  redis.conf: |
    maxmemory 256mb
    requirepass redis#123
    save 3600 1 300 10 60 100
    dir /data/redis
    replicaof defaultapp-redis-a-0.defaultapp-redis-a.default-ns.svc.cluster.local 6379
    masterauth redis#123
  run.sh: |
    #!/bin/sh
    role=$1
    hname=$(hostname)
    if [ $role == "master" ];then
        echo "run redis master"
        [ -d /home/redis/$hname ] || mkdir -p /home/redis/$hname
        [ -d /data ] || mkdir /data
        ln -s /home/redis/$hname /data/redis
        redis-server /etc/redis/redis-master.conf
    elif [ $role == "sentinel" ];then
        echo "run redis sentinel"
        mkdir -p /etc/redis
        cp /home/redis/sentinel.conf /etc/redis/
        redis-sentinel /etc/redis/sentinel.conf
    else
        echo "run redis slave"
        [ -d /home/redis/$hname ] || mkdir -p /home/redis/$hname
        [ -d /data ] || mkdir /data
        ln -s /home/redis/$hname /data/redis
        redis-server /etc/redis/redis.conf
    fi

                    上一篇
                    下一篇