Friday, 16 August 2019

Restoring the failed member in MySQL InnoDB cluster with MySQL Enterprise Backup

Using MySQL Enterprise Backup to restore and rebuild the failed member in secondary node

Testing Environment:

Linux OS: CentOS 7.4
MySQL Server Version: 8.0
MySQL Enterprise Backup 8.0
MySQL Enterprise Transparent Encryption Encryption 
MySQL InnoDB cluster 8.0

Host: primary01 as primary node
Host: second02 as secondary node

A. Pre-Task: 

1. Taking Full-backup with Compression on secondary using MySQL Enterprise Backup

mysqlbackup --defaults-file=my.cnf --backup_image=/backup/mysqld/my.mbi_`date +%d%m_%H%M` --backup-dir=/backup/mysqld/backup_`date +%d%m_%H%M` --user=mysbackup -p --host=second02 --port=3306 --encrypt-password= --compress --compress-level=5  backup-to-image &>> `date +%d%m_%H%M`_full_compress_tde_mysqld_backup.log

2. check backup log with "completed" and please ignore the warning since the backup was taken on secondary node.

ySQL Enterprise Backup  Ver 8.0.15-commercial for linux-glibc2.12 on x86_64 (MySQL Enterprise - Commercial)
Copyright (c) 2003, 2019, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

190816 16:56:26 MAIN    INFO: A thread created with Id '140676524181376'
190816 16:56:26 MAIN    INFO: Starting with following command line ...
 mysqlbackup
        --defaults-file=my.cnf
        --backup_image=/backup/mysqld/my.mbi_1608_1656
        --backup-dir=/backup/mysqld/backup_1608_1656
        --user=mysbackup -pxxxxxxxxxxxx --host=second02 --port=3306
        --encrypt-password=xxxxxxxxxx --compress --compress-level=5
        backup-to-image

190816 16:56:26 MAIN    INFO:
190816 16:56:26 MAIN    INFO: No SSL options specified.
190816 16:56:26 MAIN    INFO: MySQL server version is '8.0.15-commercial'
190816 16:56:26 MAIN    INFO: MySQL server compile os version is 'linux-glibc2.12'
190816 16:56:26 MAIN    INFO: SSL/TLS version used for connection is TLSv1.2
190816 16:56:26 MAIN WARNING: This backup operation cannot write to backup progress. The MySQL server is running with the --super-read-only option.
....
mysqlbackup completed OK! with 1 warnings

B. Task: Restore the backup on secondary node
1.Stop MySQL server
mysql -u mysql -p password  -hsecond02 -P3306 -e "shutdown"

2. remove all data files under mysql database data directory
rm -rf /data/mysqld
mkdir -p /data/mysqld

3. remove all log files and the keyring files 
rm -rf /data/mysql-keyring
mkdir -p /data/mysql-keyring
rm -rf /data/log
mkdir -p /data/log

4. create restore directory
mkdir -p /backup/mysqld/restore

5. restore backup 
mysqlbackup --defaults-file=/my.cnf --backup-image=/backup/mysqld/ --backup-dir=/backup/mysqld/restore --datadir=/data/mysqld --user=mysbackup -p password  --host=second02 --port=3306 --encrypt-password= encryption_password  --uncompress copy-back-and-apply-log &>> `date +%d%m_%H%M`_full_compress_tde_mysqld_restore.log

C. Post-Task: Re-join as a New Member in MySQL InnoDB cluster
1. Start restored MySQL Server
mysqld_safe --defaults-file=my.cnf &

2. Reset group replication member status
mysql -u mysql -p password -hsecond02 -P3306
mysql> RESET MASTER;
mysql> RESET SLAVE ALL;


*Remark: please follow the below steps if enabled auto recover using group replication's built-in mechanism with the variable "gtid_executed
//start

mysql> SET sql_log_bin=off;
mysql> source /data/mysqld/backup_gtid_executed.sql
mysql> SET sql_log_bin=on;

//end

3. Re-join the restored server in MySQL InnoDB cluster
mysqlsh --uri=mysql:password @primary01:3306

MySQL  primary01:3306 ssl  JS > var x = dba.getCluster();
MySQL  primary01:3306 ssl  JS > x.rescan();
Rescanning the cluster...

Result of the rescanning operation for the 'default' ReplicaSet:
{
    "name": "default", 
    "newTopologyMode": null, 
    "newlyDiscoveredInstances": [], 
    "unavailableInstances": [
        {
            "host": "second02:3306", 
            "label": "second02:3306", 
            "member_id": "4e1eff7d-bf22-11e9-93f4-080027ea2dcf"
        }
    ]
}

The instance 'second02:3306' is no longer part of the ReplicaSet.
The instance is either offline or left the HA group. You can try to add it to the cluster again with the cluster.rejoinInstance('second02:3306') command or you can remove it from the cluster configuration.
Would you like to remove it from the cluster metadata? [Y/n]: y 
Removing instance from the cluster metadata...
The instance 'second02:3306' was successfully removed from the cluster metadata.

MySQL  primary01:3306 ssl  JS > dba.checkInstanceConfiguration('group_replication_user:group_replication_password@second02:3306');
MySQL  primary01:3306 ssl  JS > dba.configureInstance('mysadm:mysadm43d@second02:3306', {clusterAdmin:'group_replication_user',clusterAdminPassword:'group_replication_password'});
dba.checkInstanceConfiguration('group_replication_user:group_replication_password@second02:3306');
MySQL  primary01:3306 ssl  JS > x.status();
MySQL  primary01:3306 ssl  JS >x.addInstance('group_replication_user:group_replication_password>@second02:3306');
MySQL  primary01:3306 ssl  JS > x.status();
{
    "clusterName": "CC_CLU01", 
    "defaultReplicaSet": {
        "name": "default", 
        "primary": "primary01:3306", 
        "ssl": "REQUIRED", 
        "status": "OK_NO_TOLERANCE", 
        "statusText": "Cluster is NOT tolerant to any failures.", 
        "topology": {
            "primary01:3310": {
                "address": "primary01:3306", 
                "mode": "R/W", 
                "readReplicas": {}, 
                "role": "HA", 
                "status": "ONLINE"
            }, 
            "second02:3310": {
                "address": "second02:3306", 
                "mode": "R/O", 
                "readReplicas": {}, 
                "role": "HA", 
                "status": "ONLINE"
            }
        }, 
        "topologyMode": "Single-Primary"
    }, 
    "groupInformationSourceMember": "primary01:3306"
}


3. check member status after re-join the restored server in MySQL InnoDB cluster
mysql -u mysql -p password  -hsecond02 -P3306
mysql> SELECT member_host, member_port, member_state FROM performance_schema.replication_group_members;
+-------------+-------------+--------------+
| member_host | member_port | member_state |
+-------------+-------------+--------------+
| primary01   |        3306 | ONLINE       |
| second02    |        3306 | ONLINE       |
+-------------+-------------+--------------+
2 rows in set (0.00 sec)

mysql>