Thursday 2 July 2020

How to config and enable MySQL Enterprise Audit in MySQL

Here is the steps to show you how to config and enable the MySQL Enterprise Audit in MySQL.

A. Config and enable the MySQL Enterprise Audit in MySQL.
Step 1: Append the following lines in config file
plugin-load-add=authentication_pam.so
plugin-load-add=audit_log.so
audit_log_flush=on
audit_log_rotate_on_size=200MB
#Example of audit log name is audit_host01_3306.log
audit_log_file=/db_log/audit/audit_host01_3306.log 
#Log connection error only
audit_log_connection_policy=ERRORS

Step 2: Check plugin status
SELECT PLUGIN_NAME, PLUGIN_STATUS FROM INFORMATION_SCHEMA.PLUGINS 
WHERE PLUGIN_NAME LIKE 'audit%';
   
Step 3: Create audit function and tables
USE mysql;
CREATE TABLE IF NOT EXISTS audit_log_filter(NAME VARCHAR(64) BINARY NOT NULL PRIMARY KEY, FILTER JSON NOT NULL) engine= InnoDB CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_as_ci;
CREATE TABLE IF NOT EXISTS audit_log_user(USER VARCHAR(32) BINARY NOT NULL, 
HOST VARCHAR(60) BINARY NOT NULL, FILTERNAME VARCHAR(64) BINARY NOT NULL, 
PRIMARY KEY (USER, HOST), FOREIGN KEY (FILTERNAME) REFERENCES mysql.audit_log_filter(NAME)) 
engine= InnoDB CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_as_ci;
INSTALL PLUGIN audit_log SONAME 'audit_log.so';
CREATE FUNCTION audit_log_filter_set_filter RETURNS STRING SONAME 'audit_log.so';
CREATE FUNCTION audit_log_filter_remove_filter RETURNS STRING SONAME 'audit_log.so';
CREATE FUNCTION audit_log_filter_set_user RETURNS STRING SONAME 'audit_log.so';
CREATE FUNCTION audit_log_filter_remove_user RETURNS STRING SONAME 'audit_log.so';
CREATE FUNCTION audit_log_filter_flush RETURNS STRING SONAME 'audit_log.so';
CREATE FUNCTION audit_log_read_bookmark RETURNS STRING SONAME 'audit_log.so';
CREATE FUNCTION audit_log_read RETURNS STRING SONAME 'audit_log.so';
CREATE FUNCTION audit_log_encryption_password_set RETURNS INTEGER SONAME 'audit_log.so';
CREATE FUNCTION audit_log_encryption_password_get RETURNS STRING SONAME 'audit_log.so';
SELECT audit_log_filter_flush() AS 'Result';


Step 4: Rotate the log file when the audit log reached 200MB
SET GLOBAL audit_log_rotate_on_size = 1024*1024*1024;
B. Create the log filter and assigned it to specified user. 
Reference: 

Step 5: Create the log filter for logging specified DDL
set @log_ddl_filter='
{
"filter": {
  "class": {
    "name": "general",
    "event": {
      "name": "status",
      "log": {
        "and": [
        {
           "or": [
           {"field": { "name": "general_command.str", "value": "Query" }},
           {"field": { "name": "general_command.str", "value": "Execute" }}
           ]
        },
       {
          "or": [
          {"field": { "name": "general_sql_command.str", "value": "alter_db" }},
          {"field": { "name": "general_sql_command.str", "value": "alter_db_upgrade" }},
          {"field": { "name": "general_sql_command.str", "value": "alter_event" }},
          {"field": { "name": "general_sql_command.str", "value": "alter_function" }},
          {"field": { "name": "general_sql_command.str", "value": "alter_instance" }},
          {"field": { "name": "general_sql_command.str", "value": "alter_procedure" }},
          {"field": { "name": "general_sql_command.str", "value": "alter_server" }},
          {"field": { "name": "general_sql_command.str", "value": "alter_table" }},
          {"field": { "name": "general_sql_command.str", "value": "alter_tablespace" }},
          {"field": { "name": "general_sql_command.str", "value": "create_db" }},
          {"field": { "name": "general_sql_command.str", "value": "create_event" }},
          {"field": { "name": "general_sql_command.str", "value": "create_function" }},
          {"field": { "name": "general_sql_command.str", "value": "create_index" }},
          {"field": { "name": "general_sql_command.str", "value": "create_procedure" }},
          {"field": { "name": "general_sql_command.str", "value": "create_server" }},
          {"field": { "name": "general_sql_command.str", "value": "create_table" }},
          {"field": { "name": "general_sql_command.str", "value": "create_trigger" }},
          {"field": { "name": "general_sql_command.str", "value": "create_udf" }},
          {"field": { "name": "general_sql_command.str", "value": "create_view" }},
          {"field": { "name": "general_sql_command.str", "value": "drop_db" }},
          {"field": { "name": "general_sql_command.str", "value": "drop_event" }},
          {"field": { "name": "general_sql_command.str", "value": "drop_function" }},
          {"field": { "name": "general_sql_command.str", "value": "drop_index" }},
          {"field": { "name": "general_sql_command.str", "value": "drop_procedure" }},
          {"field": { "name": "general_sql_command.str", "value": "drop_server" }},
          {"field": { "name": "general_sql_command.str", "value": "drop_table" }},
          {"field": { "name": "general_sql_command.str", "value": "drop_trigger" }},
          {"field": { "name": "general_sql_command.str", "value": "drop_view" }},
          {"field": { "name": "general_sql_command.str", "value": "rename_table" }}
          ]
       }
    ]
    }
   }
  }
 }
}';

Step 6: Apply the filter for all DB users
Verify the filter is a valid JSON object:
use mysql;
select json_valid(@log_ddl_filter);
Remove any previous version of the filter:
select audit_log_filter_remove_filter('log_specified_ddl');
Set the new filter:
select audit_log_filter_set_filter('log_specified_ddl',@log_ddl_filter); 
Specify which user(s) the filter should apply to, e.g. apply the filter to all user.
Remark: The filter can be assigned to any user account if the user has no assigned filter.
select audit_log_filter_set_user('%','log_specified_ddl');
Enquiry the created audit filter 
select * from audit_log_filter where name = 'log_specified_ddl' \G
Enquiry the audit user account that has assigned filter.
select * from audit_log_user;

Wednesday 1 July 2020

Automate MongoDB daily compress backup with email notification via crontab

Automate taken full compress daily backup and housekeep the backup copied for MongoDB with email notification via crontab.

By separation of duty, I suggest to create the user for backup and restoration on MongoDB.
1. Create the user if not created
shell> mongo
> use admin;
> db.createUser(
        { user: "<username>",
           pwd: "<password>",
           roles: [
                     { role: "backup" },
                     { role: "restore" }
           ]
         }
   )

Enquiry the user after creation.
> use admin;
> db.getUser("<username>")

2. Create the backup script: 
Remark: The below script will housekeep the backup copy more than one day.
shell> vi backup_onlinefull_compress_mongodb.sh
## Declare variables
vstartdate=`date +%d-%m-%y_%H:%M`
vdate=`date +%d%m_%H%M`
vhost=$1
vmongodbport=$2
vbackuppathlog=/backup/$vmongodbport/mongod
vbackuppath=/backup/$vmongodbport/mongod
log=$(echo $vdate"_"$vhost"_"$vmongodbport"_full_backup_compress.log")
## Check if the user has input enough parameter
if [ ! $# -eq 2 ]; then
        echo ""
        echo "This is the operation script for Backup MongoDB Instance."
        echo "Error: Wrong Input!"
        echo ""
        echo "Example: ./backup_onlinefull_compress_mongodb.sh host01 27017"
        exit 1;
fi
#password file
pwd=$(cat passwordfile)
log_file=$vbackuppathlog/$log

echo $vstartdate
echo $vhost
echo $config_file
echo $socket_file
echo $log_file
find $vbackuppathlog/*$vmongodbport*.log -type f -mtime +2 -exec rm {} +
find $vbackuppath -mindepth 1 -maxdepth 1 -type d -mtime +0 -exec rm -r {} +
vbackupdir=$vbackuppath/`date +%y%m%d_%H%M`
mkdir -p $vbackupdir
echo "Host: " $vhost >> $log_file
echo "Date: " $vstartdate>> $log_file
echo "mongodb Port: " $vmongodbport >> $log_file
mongodump --archive="$vbackupdir/mongodump_backup_$vmongodbport_`date +%d%m_%H%M`.gz" --oplog  --gzip  -u <username> -p$pwd --authenticationDatabase=admin --port=$vmongodbport &>> $log_file
cp $log_file $vbackupdir
# Set mailing list and email content
MAILLIST="user01@dummy.com" 
HOSTNAME=$(hostname)
BODY=$vbackuppathlog/body.txt
echo "Host: " $vhost >> $BODY
echo "Date: " $vstartdate>> $BODY
echo "mongodb Port: " $vmongodbport >> $BODY
# Send Email - MongoDB Backup Report
    MAILSUB="[$HOSTNAME Port: $vmongodbport] - MongoDB Backup Report ($vstartdate)"
    mailx -a $log_file -s "$MAILSUB" "$MAILLIST" < $BODY
    rm $BODY

3. setup cron job
Example: start the backup job at 01:00 everyday, the backup script is located at $HOME.
shell> crontab -e
## Daily backup for MongoDB
0 1 * * * ($HOME/backup_onlinefull_compress_mongodb.sh host01 27017 > $HOME/run_backup_onlinefull_compress_mongodb.log)


Tuesday 31 March 2020

Add online redo log group in standby database

How to add online redo log group in standby database
1. Change the value of standby database state to "APPLY-OFF" in dgmgrl
DGMGRL> show database verbose testdgbdr;

Database - testdgbdr 

  Role:               PHYSICAL STANDBY
  Intended State:     APPLY-ON

DGMGRL> edit database testdgbdr set state='APPLY-OFF';
DGMGRL> show database verbose testdgbdr;

Database - testdgbdr 

  Role:               PHYSICAL STANDBY
  Intended State:     APPLY-OFF

2. Add online redo log group in standby database
SQL> select db_unique_name from v$database

DB_UNIQUE_NAME
------------------------------
testdgbdr 

SQL> select group#, bytes/1024/1024 MB from v$log;

    GROUP#         MB
---------- ----------
         1         50
         2         50
         3         50

SQL> alter database add logfile group 4 ('/u01/app/redo/testdgb/redo04.log') SIZE 50M;

Database altered.

SQL> select group#, bytes/1024/1024 MB from v$log;

    GROUP#         MB
---------- ----------
         1         50
         2         50
         3         50
         4         50

3. Change the value of standby database state to "APPLY-ON" in dgmgrl
DGMGRL> show database verbose testdgbdr;

Database - testdgbdr 

  Role:               PHYSICAL STANDBY
  Intended State:     APPLY-OFF

DGMGRL> edit database testdgbdr set state='APPLY-ON';
DGMGRL> show database verbose testdgbdr;

Database - testdgbdr 

  Role:               PHYSICAL STANDBY
  Intended State:     APPLY-ON

Show configuration with warning ORA-16792: configurable property value is inconsistent with database setting

Here is the example how to resolve the warning with ORA-16792: configurable property value is inconsistent with database setting in dgmgrl.

DGMGRL> show configuration;

Configuration - testdgb_dg_config

  Protection Mode: MaxAvailability
  Members:
  testdgb - Primary database
    testdgbdr - Physical standby database 
      Warning: ORA-16792: configurable property value is inconsistent with database setting

Fast-Start Failover: DISABLED

Configuration Status:
WARNING   (status updated 32 seconds ago)

DGMGRL> show database testdgbdr 'InconsistentProperties';
INCONSISTENT PROPERTIES
INSTANCE_NAME   PROPERTY_NAME          MEMORY_VALUE   SPFILE_VALUE    BROKER_VALUE 
testdgb         StandbyFileManagement  MANUAL         MANUAL          AUTO 

The broker value of "StandbyFileMangement" is inconsistent with database spfile and memory value, so we need to align the broker value "StandbyFileMangement" with database config.

DGMGRL> edit database testdgbdr set property StandbyFileManagement='MANUAL';
Property "standbyfilemanagement" updated

Check again the database DG broker inconsistent property after updated the value.

DGMGRL> show database testdgbdr 'InconsistentProperties';
INCONSISTENT PROPERTIES
INSTANCE_NAME   PROPERTY_NAME          MEMORY_VALUE   SPFILE_VALUE    BROKER_VALUE 

The warning ORA-16792 has been resolved after amended the value in database property. 
DGMGRL> show configuration;

Configuration - testdgb_dg_config

  Protection Mode: MaxAvailability
  Members:
  testdgb      - Primary database
    testdgbdr - Physical standby database 

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 41 seconds ago)