Technology Sharing

7.11 Learning Check-in ---- Beginner's Guide to Redis (Part 6)

2024-07-12

한어Русский языкEnglishFrançaisIndonesianSanskrit日本語DeutschPortuguêsΕλληνικάespañolItalianoSuomalainenLatina

7.11 Study Check-in

insert image description here

1. redis transaction

The concept of transaction and ACID characteristics

insert image description here
Database level transactions

At the database level, a transaction is a set of operations that either all execute successfully or none execute at all.

Four characteristics of database transactions

  • A: Atomic, atomicity, executes all SQL as atomic work units, either all or none of them;
  • C: Consistent. After the transaction is completed, the status of all data is consistent. That is, as long as 100 is subtracted from account A, 100 must be added to account B.
  • I: Isolation. If multiple transactions are executed concurrently, the modifications made by each transaction must be isolated from other transactions.
  • D: Duration, persistence, that is, after the transaction is completed, the changes to the database data are persistently stored.

Redis Transactions

A Redis transaction is a collection of commands. All commands in a transaction will be serialized, and a series of commands will be executed one-time, sequentially, and exclusively.

insert image description here

Three major features of Redis transactions

  • Separate isolation operations:All commands in a transaction will be serialized and executed in order. During the execution of a transaction, it will not be interrupted by command requests sent by other clients;
  • No concept of isolation levels: The commands in the queue will not be actually executed before they are submitted, because no instructions will be actually executed before the transaction is submitted, so there is no such thing as "queries within a transaction will see updates in the transaction, but queries outside the transaction cannot see them".
  • No atomicity guaranteed: If a command fails to execute in the same redis transaction, the subsequent commands will still be executed without rollback;

Three phases of Redis transaction execution

insert image description here

  • Open:byMULTIStart a transaction;
  • Join the team: Multiple commands are queued into a transaction. These commands are not executed immediately upon receipt, but are placed in the transaction queue waiting to be executed;
  • implement:Depend onEXECCommand triggers a transaction;

Redis transaction basic operations

insert image description here
Multi、Exec、discard

Transaction from inputMultiAt the beginning of the command, the input commands will be pushed into the command buffer queue in sequence and will not be executed until the inputExecAfter that, Redis will execute the commands in the previous command buffer queue in sequence.discardTo give up teaming.

example

127.0.0.1:6379> set t1 1
OK
127.0.0.1:6379> MULTI
OK
127.0.0.1:6379(TX)> set id 12
QUEUED
127.0.0.1:6379(TX)> get id
QUEUED
127.0.0.1:6379(TX)> incr t1
QUEUED
127.0.0.1:6379(TX)> incr t1
QUEUED
127.0.0.1:6379(TX)> get t1
QUEUED
127.0.0.1:6379(TX)> EXEC
1) OK
2) "12"
3) (integer) 2
4) (integer) 3
5) "3"
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20

Abandoning a transaction

127.0.0.1:6379> MULTI
OK
127.0.0.1:6379(TX)> set name z3
QUEUED
127.0.0.1:6379(TX)> set age 29
QUEUED
127.0.0.1:6379(TX)> incr t1
QUEUED
127.0.0.1:6379(TX)> DISCARD
OK
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10

All sit together

127.0.0.1:6379> MULTI
OK
127.0.0.1:6379(TX)> set name z3
QUEUED
127.0.0.1:6379(TX)> get name
QUEUED
127.0.0.1:6379(TX)> incr t1
QUEUED
127.0.0.1:6379(TX)> get t1
QUEUED
127.0.0.1:6379(TX)> set email
(error) ERR wrong number of arguments for 'set' command
127.0.0.1:6379(TX)> exec
(error) EXECABORT Transaction discarded because of previous errors.
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14

Notice
If there are incorrect instructions in the command set (note that they are syntax errors), all of them will fail.

Every wrong has its perpetrator, every debt has its creditor

127.0.0.1:6379> MULTI
OK
127.0.0.1:6379(TX)> set age 11
QUEUED
127.0.0.1:6379(TX)> incr t1
QUEUED
127.0.0.1:6379(TX)> set email [email protected]
QUEUED
127.0.0.1:6379(TX)> incr email
QUEUED
127.0.0.1:6379(TX)> get age
QUEUED
127.0.0.1:6379(TX)> exec
1) OK
2) (integer) 5
3) OK
4) (error) ERR value is not an integer or out of range
5) "11"
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18

Notice
Runtime errors, that is, non-syntax errors, correct commands will be executed, and incorrect commands will return errors.

2. Redis cluster

Master-slave replication

insert image description here
Overview
Among existing enterprises, 80% of companies use Redis stand-alone services. In actual scenarios, single-node Redis is prone to risks.

Problems:

  • Machine failure. We deploy to a Redis server. When a machine failure occurs, we need to migrate to another server and ensure that the data is synchronized.
  • Capacity bottleneck. When we need to expand the Redis memory from 16G to 64G, a single machine will definitely not be able to meet the needs. Of course, you can buy a new machine with 128G.

Solution

To achieve larger storage capacity of the distributed database and withstand high concurrent access, we will store the data of the original centralized database on multiple other network nodes.

Notice
In order to solve this single node problem, Redis will also replicate multiple copies of the data and deploy them on other nodes for replication, achieving high availability of Redis and redundant backup of data to ensure high availability of data and services.

What is Master-Slave Replication

Master-slave replication refers to copying the data of a Redis server to other Redis servers. The former is called the master node, and the latter is called the slave node. Data replication is one-way, only from the master node to the slave node.

insert image description here
The role of master-slave replication

  • Data redundancy: Master-slave replication implements hot backup of data and is a data redundancy method besides persistence.
  • Recovery:When a problem occurs on the master node, the slave node can provide services to achieve fast fault recovery; it is actually a kind of service redundancy.
  • Load Balancing:On the basis of master-slave replication, with read-write separation, the master node can provide write services, and the slave node can provide read services (that is, the application connects to the master node when writing Redis data, and the application connects to the slave node when reading Redis data), sharing the server load; especially in the scenario of more reads than writes, sharing the read load by multiple slave nodes can greatly improve the concurrency of the Redis server.
  • High availability cornerstone: In addition to the above functions, master-slave replication is also the basis for the implementation of Sentinel and cluster. Therefore, master-slave replication is the basis of Redis high availability.

Master-slave replication environment construction

insert image description here
Writing configuration files
Create redis6379.conf

include /usr/local/redis-7.2.4/redis.config
pidfile /var/run/redis_6379.pid
port 6379
dbfilename dump6379.rdb
  • 1
  • 2
  • 3
  • 4

Create redis6380.conf

include /usr/local/redis-7.2.4/redis.config
pidfile /var/run/redis_6380.pid
port 6380
dbfilename dump6380.rdb
  • 1
  • 2
  • 3
  • 4

Create redis6381.conf

include /usr/local/redis-7.2.4/redis.config
pidfile /var/run/redis_6381.pid
port 6381
dbfilename dump6381.rdb
  • 1
  • 2
  • 3
  • 4

Start three redis servers

./redis-server ../redis6379.conf
./redis-server ../redis6380.conf
./redis-server ../redis6381.conf
  • 1
  • 2
  • 3

View system processes

[root@localhost src]# ps -ef |grep redis
root    40737    1  0 22:05 ?     00:00:00 ./redis-server *:6379
root    40743    1  0 22:05 ?     00:00:00 ./redis-server *:6380
root    40750    1  0 22:05 ?     00:00:00 ./redis-server *:6381
root    40758  40631  0 22:05 pts/0   00:00:00 grep --color=auto redis
  • 1
  • 2
  • 3
  • 4
  • 5

Check the operation status of the three hosts

#打印主从复制的相关信息
./redis-cli -p 6379
./redis-cli -p 6380
./redis-cli -p 6381
127.0.0.1:6379> info replication
127.0.0.1:6380> info replication
127.0.0.1:6381> info replication
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7

The slave database is not equipped with the master database

Syntax format:

slaveof  <ip> <port>
  • 1

Example:

Executed on 6380 and 6381.

127.0.0.1:6380> SLAVEOF 127.0.0.1 6379
OK
  • 1
  • 2

Write on the host and read data on the slave

set k1 v1
  • 1

Analysis of the master-slave replication principle

insert image description here
Master-slave replication can be divided into 3 stages

  • Connection establishment phase (preparation phase)
  • Data synchronization phase
  • Command propagation phase

The replication process is roughly divided into 6 steps

insert image description here

  1. Save the master node (master) information.
    Check the status information after executing slaveof
info replication
# Replication
role:slave
master_host:127.0.0.1
master_port:6379
master_link_status:up
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  1. The slave node maintains the replication logic through a scheduled task that runs every second. When the scheduled task finds a new master node, it will try to establish a network connection with the node.
    insert image description here
  2. Establish a network connection between the slave node and the master node
    The slave node will establish a socket with port 51234, which is specifically used to receive the replication commands sent by the master node.

insert image description here
4. Send ping command

After the connection is successfully established, the slave node sends a ping request for the first communication.

insert image description here

effect:

  • Check whether the network socket between the master and the slave is available.
  • Check whether the master node can currently accept commands
  1. ASD.
    If the requirepass parameter is set for the master node, password verification is required. The slave node must configure the masterauth parameter to ensure that the password is the same as that of the master node in order to pass the verification. If the verification fails, the replication will be terminated and the slave node will restart the replication process.

  2. Synchronize the dataset.
    After the master-slave replication connection is communicating normally, for the scenario of establishing replication for the first time, the master node will send all the data it holds to the slave node. This part of the operation is the longest step.
    insert image description here

Master-slave synchronization strategy

When the master and slave are just connected, full synchronization is performed; after full synchronization, incremental synchronization is performed. Of course, if necessary, the slave can initiate full synchronization at any time. The redis strategy is that no matter what, incremental synchronization will be tried first. If it fails, the slave will be required to perform full synchronization.

For example

Save a cache

set name jjy
  • 1

The recorded command is

$3 r n
set r n
$4 r n
name r n
$5  r n
jjy r n
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
Offset100010011002100310041005100610071008
Byte value$3rn$4nam

7. Command continuous replication.
When the master node synchronizes the current data to the slave node, the replication establishment process is completed. Next, the master node will continue to send write commands to the slave node to ensure the consistency of the master and slave data.

Sentinel Monitoring

insert image description here
Disadvantages of Redis master-slave replication

When the host Master goes down, we need to manually solve the switch.

insert image description here

Leakage issues:
Once the master node goes down and the write service is unavailable, you need to manually switch, reselect the master node, and manually set the master-slave relationship.

Master-slave switching technology

When the master server goes down, you need to manually switch a slave server to the master server, which requires manual intervention, is time-consuming and labor-intensive, and will cause service unavailability for a period of time. This is not a recommended method. More often, we give priority toSentry Mode

Sentinel Overview

Sentinel mode is a special mode. First, Redis provides the sentinel command. Sentinel is an independent process. As a process, it will run independently. The principle is that Sentinel monitors multiple running Redis instances by sending commands and waiting for the Redis server to respond.

insert image description here
Sentinel role

  • Cluster Monitoring: Responsible for monitoring whether the redis master and slave processes are working properly
  • notification: If a redis instance fails, the sentinel is responsible for sending a message as an alarm notification to the administrator
  • Failover:If the master node hangs up, it will automatically transfer to the slave node
  • Configuration Center: If a failover occurs, notify the client of the new master address

Sentinel monitoring environment construction

insert image description here

Create a new sentinel-26379.conf file

#端口
port 26379
#守护进程运行
daemonize yes
#日志文件
logfile "26379.log"
sentinel monitor mymaster localhost 6379 2
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7

parameter:
sentinel monitor mymaster 192.168.92.128 6379 2 The configuration means that the sentinel node monitors the master node 192.168.92.128:6379, which is named mymaster. The last 2 is related to the failure determination of the master node: at least two sentinel nodes must agree to determine that the master node has failed and perform a failover.

Create a new sentinel-26380.conf file

#端口
port 26380
#守护进程运行
daemonize yes
#日志文件
logfile "26380.log"
sentinel monitor mymaster localhost 6379 2
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7

Create a new sentinel-26381.conf file

#端口
port 26381
#守护进程运行
daemonize yes
#日志文件
logfile "26381.log"
sentinel monitor mymaster localhost 6379 2
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7

Sentinel node startup

redis-sentinel sentinel-26379.conf
  • 1

View the sentinel node status

[root@localhost src]# ./redis-cli -p 26379
127.0.0.1:26379> 
127.0.0.1:26379> 
127.0.0.1:26379> info sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=192.168.66.100:6379,slaves=2,sentinels=3
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11

Analysis of Sentinel Working Principle

insert image description here
Monitoring Phase
insert image description here

Notice:

  • Sentinel 1 sends an info message to the master and slave to get the full information.
  • Sentinel (Sentinel 2) -----&gt; Initiate info to the master (master), then you will know the information of the existing sentinel (Sentinel 1) and connect to the slave (slave).
  • Sentinel (Sentinel 2)-----&gt;Initiate a subscribe to sentinel (Sentinel 1).

Notification Phase

Sentinel continuously sends notifications to the master and slave to collect information.

Failover phase

In the notification phase, if the notification sent by sentinel does not receive a response from the master, the master will be marked as SRI_S_DOWN, and the master status will be sent to each sentinel. When other sentinels hear that the master has died, they will say, "I don't believe it, let me go and check it out too," and share the results with each sentinel. When half of the sentinels believe that the master has died, the master will be marked as SRI_0_DOWN.
insert image description here
Here comes the question:

At this time, the master needs to be replaced. Who will be the master?

Voting method

Way:

I will vote for the sentinel whose election notification I receive first.

Excluding some cases:

  • Not online
  • Slow response
  • Disconnected from the original master for a long time
  • Priority Principle

Failover

insert image description here
Overview
Demonstrates Sentinel's monitoring and automatic failover capabilities when the primary node fails.

Demonstrating Failover
Use the kill command to kill the master node

ps aux |grep redis
kill -9 pid
  • 1
  • 2

View sentinel node information

If you use the info Sentinel command in the sentinel node immediately, you can check it.

[root@localhost src]# ./redis-cli -p 26379
127.0.0.1:26379> info sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=127.0.0.1:6381,slaves=5,sentinels=3
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9

Notice
You will find that the master node has not yet switched over, because it takes some time for the Sentinel to detect the failure of the master node and transfer.

Restart 6379 nodes

[root@localhost src]# ./redis-cli info replication
# Replication
role:slave
master_host:127.0.0.1
master_port:6381
master_link_status:down
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6

Configuration files will be rewritten
During the failover phase, the configuration files of the sentinel and the master and slave nodes will be rewritten.

include /usr/local/redis/redis.conf
pidfile "/var/run/redis_6379.pid"
port 6379
dbfilename "dump6379.rdb"
# Generated by CONFIG REWRITE
daemonize yes
protected-mode no
appendonly yes
slowlog-max-len 1200
slowlog-log-slower-than 1000
save 5 1
user default on nopass ~* &* +@all
dir "/usr/local/redis"
replicaof 127.0.0.1 6381
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14

in conclusion

  • The master and slave nodes in the Sentinel system are no different from ordinary master and slave nodes. Fault detection and transfer are controlled and completed by the Sentinel.
  • Sentinel nodes are essentially redis nodes.
  • Each sentinel node only needs to be configured to monitor the master node, and it can automatically discover other sentinel nodes and slave nodes.
  • During the sentinel node startup and failover phase, the configuration files of each node will be rewritten (config rewrite).

Cluster Mode

Redis has three cluster modes

  • Master-slave mode
  • Sentinel Mode
  • Cluster Mode

Disadvantages of Sentry Mode
insert image description here
shortcoming

  • When the master fails, Sentinel will elect a new master. During the election, there is no way to access Redis, resulting in a momentary interruption of access.
  • In sentinel mode, only the master node can write to the outside world, and the slave node can only be used for reading. Although a single Redis node supports a maximum of 100,000 QPS, during e-commerce promotions, the pressure of writing data is all on the master.
  • The memory of a single Redis node cannot be set too large. If the data is too large, the master-slave synchronization will be very slow. When the node is started, it will take a particularly long time.

Cluster Mode Overview
insert image description here
Redis cluster is a distributed service cluster consisting of multiple master-slave node groups, which has replication, high availability and sharding features.

Advantages of Redis Cluster

  • Redis cluster has multiple masters, which can reduce the impact of access interruption problems.
  • Redis cluster has multiple masters, which can provide higher concurrency
  • Redis cluster can be stored in shards, so that more data can be stored

Cluster mode construction

A Redis cluster requires at least three master nodes. We build three masters here, each with a slave node, for a total of six Redis nodes.
insert image description here
Cluster construction

Create 6 different redis nodes with port numbers 6379, 6380, 6381, 6382, 6383, and 6384 respectively.

Notice: The dump.rdb and appendonly.aof files must be deleted before copying

1. Create a new configuration file
Create redis6379.config, redis6380.config, redis6381.config, redis6382.config, redis6383.config, and redis6384.config files and modify the port number in the configuration file to correspond to the file port number.

daemonize yes
dir /usr/local/redis-7.2.4/redis-cluster/6382/
bind 192.168.47.100
port 6382
dbfilename dump6382.rdb
cluster-enabled yes
cluster-config-file nodes-6382.conf
cluster-node-timeout 5000
appendonly yes
protected-mode no
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10

parameter

  • cluster-config-file: Cluster persistent configuration file, which contains the status of other nodes, persistent variables, etc., and is automatically generated in the dir directory configured above. Each node will maintain a cluster configuration file during operation; whenever the cluster information changes (such as adding or removing nodes), all nodes in the cluster will update the latest information to the configuration file; when the node restarts, it will re-read the configuration file to obtain the cluster information, and can easily rejoin the cluster. The cluster configuration file is maintained by Redis and does not need to be modified manually.
  • clouster-enabled: Enable clustering

Create a folder

mkdir -p /usr/local/redis-7.2.4/redis-cluster/6379/
mkdir -p /usr/local/redis-7.2.4/redis-cluster/6380/
mkdir -p /usr/local/redis-7.2.4/redis-cluster/6381/
mkdir -p /usr/local/redis-7.2.4/redis-cluster/6382/
mkdir -p /usr/local/redis-7.2.4/redis-cluster/6383/
mkdir -p /usr/local/redis-7.2.4/redis-cluster/6384/
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6

Start six nodes

[root@bogon src]# ./redis-server ../redis6379.config
[root@bogon src]# ./redis-server ../redis6380.config
[root@bogon src]# ./redis-server ../redis6381.config
[root@bogon src]# ./redis-server ../redis6382.config
[root@bogon src]# ./redis-server ../redis6383.config
[root@bogon src]# ./redis-server ../redis6384.config
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6

Check whether each node is started successfully

[root@bogon src]# ps -ef | grep redis
root    3889    1 0 09:56 ?    00:00:03 ./redis-server 0.0.0.0:6379 [cluster]
root    3895    1 0 09:56 ?    00:00:03 ./redis-server 0.0.0.0:6380 [cluster]
root    3901    1 0 09:57 ?    00:00:03 ./redis-server 0.0.0.0:6381 [cluster]
root    3907    1 0 09:57 ?    00:00:02 ./redis-server *:6382 [cluster]
root    3913    1 0 09:57 ?    00:00:02 ./redis-server 0.0.0.0:6383 [cluster]
root    3919    1 0 09:57 ?    00:00:02 ./redis-server 0.0.0.0:6384 [cluster]
root    4247  2418 0 10:22 pts/0  00:00:00 grep --color=auto redis
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8

Configuring the cluster

Command format: --cluster-replicas 1 means creating a slave node for each master

Notice:The IP here is the real IP of the machine where each node is located

[root@localhost src]# ./redis-cli --cluster create 192.168.47.100:6379 192.168.47.100:6380 192.168.47.100:6381 192.168.47.100:6382 192.168.47.100:6383 192.168.47.100:6384 --cluster-replicas 1
  • 1

insert image description here

Verify the cluster

Connect to any client

./redis-cli -h 192.168.47.100  -p 6379 -c
  • 1

parameter:

-h : host address
-p : port number
-c: indicates cluster mode

Data write test

[root@bogon src]# ./redis-cli -p 6379 -c
127.0.0.1:6379> set name zhangsan
-> Redirected to slot [5798] located at 192.168.47.100:6380
OK
192.168.47.100:6380> get name
"zhangsan"
192.168.47.100:6380>
[root@bogon src]# ./redis-cli -p 6383 -c
127.0.0.1:6383> get name
-> Redirected to slot [5798] located at 192.168.47.100:6380
"zhangsan"
192.168.47.100:6380>
[root@bogon src]# ./redis-cli -p 6383 -c
127.0.0.1:6383> readonly
OK
127.0.0.1:6383> get name
"zhangsan"
127.0.0.1:6383>
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18

Cluster mode principle analysis

Redis ClusterAll data is divided into 16384 slots, and each node is responsible for a part of the slots. The slot information is stored in each node. Only the master node will be assigned slots, and the slave node will not be assigned slots.

insert image description here

Slot location algorithm: k1 = 127001

By default, the Cluster uses the crc16 algorithm to hash the key value to get an integer value, and then uses this integer value to perform a modulo operation on 16384 to get the specific slot.

HASH_SLOT = CRC16(key) % 16384

Recovery
View Node

192.168.66.103:8001> cluster nodes
  • 1

Kill the Master Node

lsof -i:8001
kill -9 pid
  • 1
  • 2

Observe node information

insert image description here

Java Operation Redis Cluster

insert image description here
Modify the configuration file

spring.data.redis.cluster.nodes=192.168.47.100:6381,192.168.47.100:6383,192.168.47.100:6380
  • 1

Notice
1. A Redis cluster requires at least 3 nodes to ensure high availability.
2. You should try to avoid adding or deleting nodes during the operation of the Redis cluster, as this may cause data migration and affect the overall performance of the Redis cluster.

Code written in Java

@SpringBootTest
public class CluseterTest {

  @Autowired
  private RedisTemplate<String,Object> redisTemplate;

  @Test
  void string() {
    //  保存字符串
    redisTemplate.opsForValue().set("itbaizhan","itbaizhan123");

    System.out.println(redisTemplate.opsForValue().get("itbaizhan"));

   }
}
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15

If my content is helpful to you, pleaseLike, comment, favoriteCreation is not easy, your support is my motivation to keep going.
insert image description here