PostgreSQL
 sql >> Cơ Sở Dữ Liệu >  >> RDS >> PostgreSQL

Di chuyển cơ sở dữ liệu PostgreSQL từ On-Prem sang đám mây bằng AWS RDS

Sau đây là hướng dẫn từng bước để di chuyển cơ sở dữ liệu tại chỗ PostgreSQL của bạn sang đám mây bằng cách sử dụng Amazon RDS. Đối với hướng dẫn này, bạn cần chạy PostgreSQL 10 trở lên trên máy chủ Centos / RHEL Linux.

Bước 1 - Tạo thư mục liên kết 

#script directory
mkdir -p /opt/dba/postgres/migrate/
 
#backup directory
mkdir -p /dbbackup/
 
#log directory
mkdir -p /var/log/postgres/migrate/

Bước 2 - Tạo tập lệnh sao lưu 

cd /opt/dba/postgres/migrate/
vi migrateDB_parallel.sh

Dán nội dung bên dưới:

chmod +x migrateDB_parallel.sh
 
#!/bin/bash
#set -e
set -u #exit immediately upon error
 
#################################################################################################################################
 
# Declare variables
 
migrate_date=$(date +'%m%d%Y')
log_dir=/var/log/postgres/migrate
 
psql_version=$1
 
source_host=$2
source_port=$3
source_database=$4
source_schema=$5
source_dbowner=$6
source_superuser=$7
source_jobs=$8
 
dest_host=$9
dest_port=${10}
dest_database=${11}
dest_dbowner=${12}
dest_superuser=${13}
dest_jobs=${14}
 
schema_only=${15}
no_owner=${16}
no_privileges=${17}
 
backup_dir=${18}/${psql_version}/${source_database}
 
test_run=${19}
 
if [[ ${source_schema} == n ]]; then
  sql_schema=""
  log="${log_dir}/${source_database}_${migrate_date}.log"
else
  sql_schema="--schema=${source_schema}"
  log="${log_dir}/${source_database}_${source_schema}_${migrate_date}.log"
fi
 
if [[ $schema_only == y ]]; then
  sql_schema_only="--schema-only"
else
  sql_schema_only=""
fi
 
if [[ $no_owner == y ]]; then
  sql_no_owner="--no-owner"
else
  sql_no_owner=""
fi
 
if [[ $no_privileges == y ]]; then
  sql_no_privileges="--no-privileges"
else
  sql_no_privileges=""
fi
 
BINDIR="/usr/pgsql-${psql_version}/bin"
 
#BINDIR="/usr/bin"
#log_dir=~/
 
mkdir -p ${log_dir}
 
 
################################################################################################################################
echo "${0} "  > $log
echo "" >> $log
echo "" >> $log
echo "$(date +'%m-%d-%Y %T') - DURATION: BEGIN"  >> $log
echo "" >> $log
echo "" >> $log
 
 
################################################################################################################################
# Display Parameters
echo " - psql_version:          $1"        >> $log
echo "                                    "        >> $log
echo " - source_host:                   $2"        >> $log
echo " - source_port:                   $3"        >> $log
echo " - source_database:               $4"        >> $log
echo " - source_schema:         $5"     >> $log
echo " - source_dbowner:                $6"        >> $log
echo " - source_superuser:              $7"        >> $log
echo " - source_jobs:                   $8"        >> $log
echo "                                    "        >> $log
echo " - dest_host:                     $9"        >> $log
echo " - dest_port:                     ${10}"        >> $log
echo " - dest_database:         ${11}"     >> $log
echo " - dest_dbowner:          ${12}"     >> $log
echo " - dest_superuser:                ${13}"     >> $log
echo " - dest_jobs:                     ${14}"     >> $log
echo "                                    "        >> $log
echo " - schema_only:                   ${15}"     >> $log
echo " - no_owner:                      ${16}"     >> $log
echo " - no_privileges:         ${17}"     >> $log
echo "                                    "        >> $log
echo " - sql_schema_only:               ${sql_schema_only}"     >> $log
echo " - sql_schema:                    ${sql_schema}"     >> $log
echo " - sql_no_owner:          ${sql_no_owner}"     >> $log
echo " - sql_no_privileges:             ${sql_no_privileges}"     >> $log
echo "                                    "        >> $log
echo " - test_run:                      ${19}"     >> $log
echo "                                    "        >> $log
echo " - BINDIR:                      ${BINDIR}" >> $log
echo " - backup_dir:                    ${backup_dir}"     >> $log
echo " - log_file:                    ${log}"    >> $log
echo "" >> $log
echo "" >> $log
echo "" >> $log
 
# vi $log; exit 99;
 
#################################################################################################################################
# Backup database
echo "######################################################################################################"  >> $log
echo "$(date +'%m-%d-%Y %T') - beginning database backup..."  >> $log
echo "" >> $log
echo "" >> $log
 
if [[ ${source_schema} == n ]]; then
 
        echo "# Gather ${source_host}:${source_database} db size..." >> $log
        ${BINDIR}/psql -p ${source_port} postgres -h${source_host} -U${source_superuser} -c"\l+ ${source_database}" >> $log 2>&1
 
else
 
        echo "# Gather ${source_host}:${source_database}.${source_schema} schema size..." >> $log
        ${BINDIR}/psql -p ${source_port} -h${source_host} ${source_database} -U${source_superuser} -c"
                SELECT schema_name, pg_size_pretty(sum(table_size)::bigint) as diskSpace, (sum(table_size) / pg_database_size(current_database())) * 100 as percent
                FROM (
                        SELECT pgn.nspname as schema_name,pg_relation_size(pgc.oid) as table_size
                        FROM   pg_catalog.pg_class pgc
                        JOIN pg_catalog.pg_namespace pgn ON relnamespace = pgn.oid
                        WHERE pgn.nspname='${source_schema}'
                     ) t
                GROUP BY schema_name" >> $log 2>&1
 
fi
 
echo "" >> $log
echo "" >> $log
 
 
echo "# Refresh directories..." >> $log
echo "mkdir -p ${backup_dir}" >> $log 2>&1
echo "rm    -f ${backup_dir}/*" >> $log 2>&1
if [[ $test_run == n ]]; then
      mkdir -p ${backup_dir}  >> $log 2>&1
      rm    -f ${backup_dir}/*  >> $log 2>&1
fi
echo "" >> $log
echo "" >> $log
 
 
# vi $log; exit 99;
 
echo "# Backup database..." >> $log
echo "${BINDIR}/pg_dump -v ${source_database} -h ${source_host} -p ${source_port} -U ${source_dbowner} -Fd -j ${source_jobs} -f ${backup_dir} ${sql_schema} ${sql_schema_only} ${sql_no_owner} ${sql_no_privileges}" >> $log 2>&1
if [[ $test_run == n ]]; then
      ${BINDIR}/pg_dump -v ${source_database} -h ${source_host} -p ${source_port} -U ${source_dbowner} -Fd -j ${source_jobs} -f ${backup_dir} ${sql_schema} ${sql_schema_only} ${sql_no_owner} ${sql_no_privileges}  >> $log 2>&1
fi
echo "" >> $log
echo "" >> $log
echo "$(date +'%m-%d-%Y %T') - completed database backup..." >> $log
echo "" >> $log
echo "" >> $log
echo "" >> $log
 
 
#################################################################################################################################
echo "######################################################################################################"  >> $log
echo "$(date +'%m-%d-%Y %T') - beginning database restore..."  >> $log
echo "" >> $log
echo "" >> $log
 
 
 
 
if [[ ${source_schema} == n ]]; then
 
        echo "# Grant ${dest_dbowner} createdb..." >> $log
        echo "${BINDIR}/psql -E -e postgres -h${dest_host} -p${dest_port} -U${dest_superuser} -c\"alter role ${dest_dbowner} createdb\"" >> $log 2>&1
        if [[ $test_run == n ]]; then
              ${BINDIR}/psql -E -e postgres -h${dest_host} -p${dest_port} -U${dest_superuser} -c "alter role ${dest_dbowner} createdb"   >> $log 2>&1
        fi
        echo "" >> $log
        echo "" >> $log
 
        echo "# Create database..." >> $log
        echo "${BINDIR}/psql -E -e postgres -h${dest_host} -p${dest_port} -U${dest_dbowner} -c\"create database ${dest_database}\"" >> $log 2>&1
        if [[ $test_run == n ]]; then
              ${BINDIR}/psql -E -e postgres -h${dest_host} -p${dest_port} -U${dest_dbowner} -c "create database ${dest_database}"   >> $log 2>&1
        fi
        echo "" >> $log
        echo "" >> $log
        #vi $log; exit 99;
 
else
 
        echo "# Create schema..." >> $log
        echo "${BINDIR}/psql -E -e -h${dest_host} -p${dest_port} ${dest_database} -U${dest_dbowner} -c\"create schema ${source_schema}\"" >> $log 2>&1
        if [[ $test_run == n ]]; then
              ${BINDIR}/psql -E -e -h${dest_host} -p${dest_port} ${dest_database} -U${dest_dbowner} -c "create schema ${source_schema}"   >> $log 2>&1
        fi
        echo "" >> $log
        echo "" >> $log
        #vi $log; exit 99;
 
fi
 
echo "# Restore database..." >> $log
echo "${BINDIR}/pg_restore -v -d ${dest_database} -h${dest_host} -p${dest_port} -U ${dest_dbowner} -j ${dest_jobs} -Fd ${backup_dir} ${sql_schema} ${sql_schema_only} ${sql_no_owner} ${sql_no_privileges}" >> $log 2>&1
if [[ $test_run == n ]]; then
      ${BINDIR}/pg_restore -v -d ${dest_database} -h${dest_host} -p${dest_port} -U ${dest_dbowner} -j ${dest_jobs} -Fd ${backup_dir} ${sql_schema} ${sql_schema_only} ${sql_no_owner} ${sql_no_privileges}  >> $log 2>&1
fi
echo "" >> $log
echo "" >> $log
echo "$(date +'%m-%d-%Y %T') - completed database restore..." >> $log
echo "" >> $log
echo "" >> $log
 
if [[ $source_schema != n ]]; then
        ${BINDIR}/psql -h${dest_host} -p${dest_port} ${dest_database} -U${dest_dbowner} -c "\dn+ ${source_schema}"   >> $log 2>&1
        ${BINDIR}/psql -h${dest_host} -p${dest_port} ${dest_database} -U${dest_dbowner} -c "\dt+ ${source_schema}.*" >> $log 2>&1
fi
echo "" >> $log
echo "" >> $log
echo "" >> $log
echo "" >> $log
 
echo "$(date +'%m-%d-%Y %T') - DURATION: END" >> $log
vi $log

Bước 3 - Thực hiện Chạy thử nghiệm cho tất cả các Lược đồ và Lệnh xem lại

Để chuẩn bị đúng cách cho quá trình di chuyển, hãy thu thập tất cả thông tin chi tiết về chính xác những gì cần sao lưu và những gì cần khôi phục. Dữ liệu này sẽ đóng vai trò là các tham số được truyền vào shell script ở trên. Trong ví dụ này, chúng tôi sẽ di chuyển một cơ sở dữ liệu đầy đủ bao gồm tất cả các lược đồ của nó truyền vào các tham số sau:

Tham số Lệnh (Giá trị Tham số)
psql_version =$ 1
source_host =$ 2
source_port =$ 3
source_database =$ 4
source_schema =$ 5
source_dbowner =$ 6
source_superuser =$ 7
source_jobs =$ 8
dest_host =$ 9
dest_port =$ {10}
dest_database =$ {11}
dest_dbowner =$ {12}
dest_superuser =$ {13}
dest_jobs =$ {14}
schema_only =$ {15}
no_owner =$ {16}
no_privileges =$ {17}
backup_dir =$ {18} / $ {psql_version} / $ {source_database}
test_run =$ {19} 		/ opt / dba / migrate / migrateDB_parallel.sh \
10 \
onpremdbserver.domain.com \
5432 \
source_database \
n \
source_appuser \
postgres \
100 \
dest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com \
5432 \
dest_database_newdb \
dest_rds_superuser \
dest_rds_superuser \
120 \
n \
y \
y \
"/ dbbackup" \
Y

Đảm bảo xem lại các lệnh trước khi thực thi 

05-24-2019 22:14:08 - DURATION: BEGIN
  
 - psql_version:        10
                                    
 - source_host:        onpremdbserver.domain.com
 - source_port:        5432
 - source_database:        source_database
 - source_schema:        n
 - source_dbowner:        source_appuser
 - source_superuser:        postgres
 - source_jobs:        100
                                    
 - dest_host:            dest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com
 - dest_port:            5432
 - dest_database:        dest_database_newdb
 - dest_dbowner:        dest_rds_superuser
 - dest_superuser:        dest_rds_superuser
 - dest_jobs:            120
                                    
 - schema_only:        n
 - no_owner:            y
 - no_privileges:        y
                                    
 - sql_schema_only:               
 - sql_schema:                    
 - sql_no_owner:        --no-owner
 - sql_no_privileges:        --no-privileges
                                    
 - test_run:            y
                                    
 - BINDIR:            /usr/pgsql-10/bin
 - backup_dir:            /dbbackup/10/source_database
 - log_file:            /var/log/postgres/migrate/source_database_05242019.log
  
 
######################################################################################################
05-24-2019 22:14:08 - beginning database backup...
  
# Gather onpremdbserver.domain.com:source_database db size...
                                                        List of databases
      Name       |     Owner      | Encoding |   Collate   |    Ctype    | Access privileges |  Size   | Tablespace | Description
-----------------+----------------+----------+-------------+-------------+-------------------+---------+------------+-------------
 source_database | source_appuser | UTF8     | en_US.UTF-8 | en_US.UTF-8 |                   | 7992 kB | pg_default |
(1 row)
 
  
# Refresh directories...
mkdir -p /dbbackup/10/source_database
rm    -f /dbbackup/10/source_database/*
 
 
# Backup database...
/usr/pgsql-10/bin/pg_dump -v source_database -h onpremdbserver.domain.com -p 5432 -U source_appuser -Fd -j 100 -f /dbbackup/9.6/source_database   --no-owner --no-privileges
  
05-24-2019 22:14:09 - completed database backup...
 
  
######################################################################################################
05-24-2019 22:14:09 - beginning database restore...
  
# Grant dest_rds_superuser createdb...
/usr/pgsql-9.6/bin/psql -E -e postgres -hdest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com -p5432 -Udest_rds_superuser -c"alter role dest_rds_superuser createdb"
  
# Create database...
/usr/pgsql-9.6/bin/psql -E -e postgres -hdest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com -p5432 -Udest_rds_superuser -c"create database dest_database_newdb"
  
# Restore database...
/usr/pgsql-9.6/bin/pg_restore -v -d dest_database_newdb -hdest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com -p5432 -U dest_rds_superuser -j 120 -Fd /dbbackup/9.6/source_database   --no-owner --no-privileges
  
05-24-2019 22:14:09 - completed database restore...

Bước 4 - Thực thi tập lệnh (Tất cả các lược đồ)

Lưu ý:Nhập “n” cho tất cả các lược đồ:

Tham số Lệnh (Giá trị Tham số)
psql_version =$ 1
source_host =$ 2
source_port =$ 3
source_database =$ 4
source_schema =$ 5
source_dbowner =$ 6
source_superuser =$ 7
source_jobs =$ 8
dest_host =$ 9
dest_port =$ {10}
dest_database =$ {11}
dest_dbowner =$ {12}
dest_superuser =$ {13}
dest_jobs =$ {14}
schema_only =$ {15}
no_owner =$ {16}
no_privileges =$ {17}
backup_dir =$ {18} / $ {psql_version} / $ {source_database}
test_run =$ {19} 		/ opt / dba / migrate / migrateDB_parallel.sh \
10 \
onpremdbserver.domain.com \
5432 \
source_database \
n \
source_appuser \
postgres \
100 \
dest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com \
5432 \
dest_database_newdb \
dest_rds_superuser \
dest_rds_superuser \
120 \
n \
y \
y \
"/ dbbackup" \
Y

Đầu ra 

05-24-2019 23:16:40 - DURATION: BEGIN
 
 - psql_version:              10    
 - source_host:               onpremdbserver.domain.com
 - source_port:               5432
 - source_database:           source_database
 - source_schema:        n
 - source_dbowner:            source_appuser
 - source_superuser:          postgres
 - source_jobs:               100
                                    
 - dest_host:            dest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com
 - dest_port:            5432
 - dest_database:             dest_database_newdb
 - dest_dbowner:              dest_rds_superuser
 - dest_superuser:            dest_rds_superuser
 - dest_jobs:                 120
                                    
 - schema_only:               n
 - no_owner:                  y
 - no_privileges:             y
                                    
 - sql_schema_only:               
 - sql_schema:                    
 - sql_no_owner:              --no-owner
 - sql_no_privileges:         --no-privileges
                                    
 - test_run:                  n
                                    
 - BINDIR:            /usr/pgsql-10/bin
 - backup_dir:                /dbbackup/10/source_database
 - log_file:                  /var/log/postgres/migrate/source_database_05242019.log
 
  
######################################################################################################
05-24-2019 23:16:40 - beginning database backup...
 
 # Gather onpremdbserver.domain.com db size...
                                                        List of databases
      Name       |     Owner      | Encoding |   Collate   |    Ctype    | Access privileges |  Size   | Tablespace | Description
-----------------+----------------+----------+-------------+-------------+-------------------+---------+------------+-------------
 source_database | source_appuser | UTF8     | en_US.UTF-8 | en_US.UTF-8 |                   | 7992 kB | pg_default |
(1 row)
  
# Refresh directories...
mkdir -p /dbbackup/10/source_database
rm    -f /dbbackup/10/source_database/*
 
 # Backup database...
/usr/pgsql-10/bin/pg_dump -v source_database -h onpremdbserver.domain.com -p 5432 -U source_appuser -Fd -j 100 -f /dbbackup/10/source_database   --no-owner --no-privileges
pg_dump: last built-in OID is 16383
pg_dump: reading extensions
pg_dump: identifying extension members
pg_dump: reading schemas
pg_dump: reading user-defined tables
pg_dump: reading user-defined functions
pg_dump: reading user-defined types
pg_dump: reading procedural languages
pg_dump: reading user-defined aggregate functions
pg_dump: reading user-defined operators
pg_dump: reading user-defined access methods
pg_dump: reading user-defined operator classes
pg_dump: reading user-defined operator families
pg_dump: reading user-defined text search parsers
pg_dump: reading user-defined text search templates
pg_dump: reading user-defined text search dictionaries
pg_dump: reading user-defined text search configurations
pg_dump: reading user-defined foreign-data wrappers
pg_dump: reading user-defined foreign servers
pg_dump: reading default privileges
pg_dump: reading user-defined collations
pg_dump: reading user-defined conversions
pg_dump: reading type casts
pg_dump: reading transforms
pg_dump: reading table inheritance information
pg_dump: reading event triggers
pg_dump: finding extension tables
pg_dump: finding inheritance relationships
pg_dump: reading column info for interesting tables
pg_dump: finding the columns and types of table "source_schema.table1"
pg_dump: finding the columns and types of table "source_schema.table2"
pg_dump: finding the columns and types of table "source_schema.table3"
pg_dump: finding default expressions of table "source_schema.table3"
pg_dump: flagging inherited columns in subtables
pg_dump: reading indexes
pg_dump: reading extended statistics
pg_dump: reading constraints
pg_dump: reading triggers
pg_dump: reading rewrite rules
pg_dump: reading policies
pg_dump: reading row security enabled for table "source_schema.table1"
pg_dump: reading policies for table "source_schema.table1"
pg_dump: reading row security enabled for table "source_schema.table2"
pg_dump: reading policies for table "source_schema.table2"
pg_dump: reading row security enabled for table "source_schema.table3_id_seq"
pg_dump: reading policies for table "source_schema.table3_id_seq"
pg_dump: reading row security enabled for table "source_schema.table3"
pg_dump: reading policies for table "source_schema.table3"
pg_dump: reading publications
pg_dump: reading publication membership
pg_dump: reading publication membership for table "source_schema.table1"
pg_dump: reading publication membership for table "source_schema.table2"
pg_dump: reading publication membership for table "source_schema.table3"
pg_dump: reading subscriptions
pg_dump: reading large objects
pg_dump: reading dependency data
pg_dump: saving encoding = UTF8
pg_dump: saving standard_conforming_strings = on
pg_dump: saving search_path =
pg_dump: saving database definition
pg_dump: dumping contents of table "source_schema.table3"
pg_dump: finished item 3800 TABLE DATA table3
pg_dump: dumping contents of table "source_schema.table1"
pg_dump: finished item 3797 TABLE DATA table1
pg_dump: dumping contents of table "source_schema.table2"
pg_dump: finished item 3798 TABLE DATA table2
  
05-24-2019 23:16:41 - completed database backup...
   ######################################################################################################
05-24-2019 23:16:41 - beginning database restore...
  
# Grant dest_rds_superuser createdb...
/usr/pgsql-10/bin/psql -E -e postgres -hdest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com -p5432 -Udest_rds_superuser -c"alter role dest_rds_superuser createdb"
alter role dest_rds_superuser createdb
ALTER ROLE
  
# Create database...
/usr/pgsql-10/bin/psql -E -e postgres -hdest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com -p5432 -Udest_rds_superuser -c"create database dest_database_newdb"
create database dest_database_newdb
CREATE DATABASE
  
# Restore database...
/usr/pgsql-10/bin/pg_restore -v -d dest_database_newdb -hdest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com -p5432 -U dest_rds_superuser -j 120 -Fd /dbbackup/10/source_database   --no-owner --no-privileges
pg_restore: connecting to database for restore
pg_restore: processing item 3803 ENCODING ENCODING
pg_restore: processing item 3804 STDSTRINGS STDSTRINGS
pg_restore: processing item 3805 SEARCHPATH SEARCHPATH
pg_restore: processing item 3806 DATABASE source_database
pg_restore: processing item 3 SCHEMA public
pg_restore: creating SCHEMA "public"
pg_restore: processing item 3807 COMMENT SCHEMA public
pg_restore: creating COMMENT "SCHEMA public"
pg_restore: processing item 6 SCHEMA source_schema
pg_restore: creating SCHEMA "source_schema"
pg_restore: processing item 1 EXTENSION plpgsql
pg_restore: creating EXTENSION "plpgsql"
pg_restore: processing item 3808 COMMENT EXTENSION plpgsql
pg_restore: creating COMMENT "EXTENSION plpgsql"
pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 3808; 0 0 COMMENT EXTENSION plpgsql
pg_restore: [archiver (db)] could not execute query: ERROR:  must be owner of extension plpgsql
    Command was: COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';
 
 
 
pg_restore: processing item 197 TABLE table1
pg_restore: creating TABLE "source_schema.table1"
pg_restore: processing item 198 TABLE table2
pg_restore: creating TABLE "source_schema.table2"
pg_restore: processing item 200 TABLE table3
pg_restore: creating TABLE "source_schema.table3"
pg_restore: processing item 199 SEQUENCE table3_id_seq
pg_restore: creating SEQUENCE "source_schema.table3_id_seq"
pg_restore: processing item 3809 SEQUENCE OWNED BY table3_id_seq
pg_restore: creating SEQUENCE OWNED BY "source_schema.table3_id_seq"
pg_restore: processing item 3675 DEFAULT table3 id
pg_restore: creating DEFAULT "source_schema.table3 id"
pg_restore: entering main parallel loop
pg_restore: launching item 3797 TABLE DATA table1
pg_restore: launching item 3798 TABLE DATA table2
pg_restore: launching item 3800 TABLE DATA table3
pg_restore: launching item 3810 SEQUENCE SET table3_id_seq
pg_restore: processing data for table "source_schema.table2"
pg_restore: executing SEQUENCE SET table3_id_seq
pg_restore: finished item 3810 SEQUENCE SET table3_id_seq
pg_restore: processing data for table "source_schema.table1"
pg_restore: processing data for table "source_schema.table3"
pg_restore: finished item 3797 TABLE DATA table1
pg_restore: finished item 3798 TABLE DATA table2
pg_restore: finished item 3800 TABLE DATA table3
pg_restore: finished main parallel loop
WARNING: errors ignored on restore: 1
  
05-24-2019 23:16:43 - completed database restore...
  
05-24-2019 23:16:43 - DURATION: END

Bước 5 - Thực hiện Chạy thử cho Lược đồ

Tham số Lệnh (Giá trị Tham số)
psql_version =$ 1
source_host =$ 2
source_port =$ 3
source_database =$ 4
source_schema =$ 5
source_dbowner =$ 6
source_superuser =$ 7
source_jobs =$ 8
dest_host =$ 9
dest_port =$ {10}
dest_database =$ {11}
dest_dbowner =$ {12}
dest_superuser =$ {13}
dest_jobs =$ {14}
schema_only =$ {15}
no_owner =$ {16}
no_privileges =$ {17}
backup_dir =$ {18} / $ {psql_version} / $ {source_database}
test_run =$ {19} 		/ opt / dba / migrate / migrateDB_parallel.sh \
10 \
onpremdbserver.domain.com \
5432 \
source_database \
source_schema \
source_appuser \
postgres \
100 \
dest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com \
5432 \
dest_database_newdb \
dest_rds_superuser \
dest_rds_superuser \
120 \
n \
y \
y \
"/ dbbackup" \
y

Một lần nữa, Đảm bảo xem lại các lệnh trước khi thực thi 

05-24-2019 22:50:48 - DURATION: BEGIN
  
 - psql_version:        10
                                    
 - source_host:               onpremdbserver.domain.com
 - source_port:               5432
 - source_database:           source_database
 - source_schema:        source_schema
 - source_dbowner:        source_appuser
 - source_superuser:          postgres
 - source_jobs:               100
                                    
 - dest_host:                 dest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com
 - dest_port:                 5432
 - dest_database:             dest_database_newdb
 - dest_dbowner:              dest_rds_superuser
 - dest_superuser:            dest_rds_superuser
 - dest_jobs:                 120
                                    
 - schema_only:               n
 - no_owner:                  y
 - no_privileges:             y
                                    
 - sql_schema_only:               
 - sql_schema:                --schema=source_schema
 - sql_no_owner:              --no-owner
 - sql_no_privileges:         --no-privileges
                                    
 - test_run:                  y
                                    
 - BINDIR:                    /usr/pgsql-10/bin
 - backup_dir:                /dbbackup/10/source_database
 - log_file:            /var/log/postgres/migrate/source_database_source_schema_05242019.log
 
  
######################################################################################################
05-24-2019 22:50:48 - beginning database backup...
  
# Gather onpremdbserver.domain.com:source_database.source_schema schema size...
  schema_name  | diskspace  |        percent         
---------------+------------+------------------------
 source_schema | 8192 bytes | 0.10010060159337380900
(1 row)
 
 
# Refresh directories...
mkdir -p /dbbackup/10/source_database
rm    -f /dbbackup/10/source_database/*
  
# Backup database...
/usr/pgsql-10/bin/pg_dump -v source_database -h onpremdbserver.domain.com -p 5432 -U source_appuser -Fd -j 100 -f /dbbackup/10/source_database --schema=source_schema  --no-owner --no-privileges
  
05-24-2019 22:50:49 - completed database backup...
  
######################################################################################################
05-24-2019 22:50:49 - beginning database restore...
  
# Create schema...
/usr/pgsql-10/bin/psql -E -e -hdest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com -p5432 dest_database_newdb -Udest_rds_superuser -c"create schema source_schema"
  
# Restore database...
/usr/pgsql-10/bin/pg_restore -v -d dest_database_newdb -hdest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com -p5432 -U dest_rds_superuser -j 120 -Fd /dbbackup/10/source_database --schema=source_schema  --no-owner --no-privileges
  
05-24-2019 22:50:49 - completed database restore...
  
                           List of schemas
     Name      |       Owner        | Access privileges | Description
---------------+--------------------+-------------------+-------------
 source_schema | dest_rds_superuser |                   |
(1 row)
 
                               List of relations
    Schema     |  Name  | Type  |       Owner        |    Size    | Description
---------------+--------+-------+--------------------+------------+-------------
 source_schema | table1 | table | dest_rds_superuser | 0 bytes    |
 source_schema | table2 | table | dest_rds_superuser | 0 bytes    |
 source_schema | table3 | table | dest_rds_superuser | 8192 bytes |
(3 rows)
 
 
 
05-24-2019 22:50:49 - DURATION: END

Bước 6 - Tập lệnh thực thi (Một lược đồ)

Tham số Lệnh (Giá trị Tham số)
psql_version =$ 1
source_host =$ 2
source_port =$ 3
source_database =$ 4
source_schema =$ 5
source_dbowner =$ 6
source_superuser =$ 7
source_jobs =$ 8
dest_host =$ 9
dest_port =$ {10}
dest_database =$ {11}
dest_dbowner =$ {12}
dest_superuser =$ {13}
dest_jobs =$ {14}
schema_only =$ {15}
no_owner =$ {16}
no_privileges =$ {17}
backup_dir =$ {18} / $ {psql_version} / $ {source_database}
test_run =$ {19} 		/ opt / dba / migrate / migrateDB_parallel.sh \
10 \
onpremdbserver.domain.com \
5432 \
source_database \
source_schema \
source_appuser \
postgres \
100 \
dest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com \
5432 \
dest_database_newdb \
dest_rds_superuser \
dest_rds_superuser \
120 \
n \
y \
y \
"/ dbbackup" \
n

Lưu ý:Chuyển “source_schema” cho tên giản đồ

Xem lại các lệnh trước khi thực thi 

05-24-2019 23:22:13 - DURATION: BEGIN
  
 - psql_version:              10
                                    
 - source_host:               onpremdbserver.domain.com
 - source_port:            5432
 - source_database:           source_database
 - source_schema:             source_schema
 - source_dbowner:            source_appuser
 - source_superuser:          postgres
 - source_jobs:               100
                                    
 - dest_host:                 dest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com
 - dest_port:                 5432
 - dest_database:             dest_database_newdb
 - dest_dbowner:              dest_rds_superuser
 - dest_superuser:            dest_rds_superuser
 - dest_jobs:                 120
                                    
 - schema_only:               n
 - no_owner:                  y
 - no_privileges:             y
                                    
 - sql_schema_only:               
 - sql_schema:                --schema=source_schema
 - sql_no_owner:        --no-owner
 - sql_no_privileges:        --no-privileges
                                    
 - test_run:                  n
                                    
 - BINDIR:                    /usr/pgsql-10/bin
 - backup_dir:                /dbbackup/10/source_database
 - log_file:            /var/log/postgres/migrate/source_database_source_schema_05242019.log
 
 
######################################################################################################
05-24-2019 23:22:13 - beginning database backup...
 
 
# Gather onpremdbserver.domain.com:source_database.source_schema schema size...
  schema_name  | diskspace  |        percent         
---------------+------------+------------------------
 source_schema | 8192 bytes | 0.10010060159337380900
(1 row)
 
 
# Refresh directories...
mkdir -p /dbbackup/10/source_database
rm    -f /dbbackup/10/source_database/*
  
# Backup database...
/usr/pgsql-10/bin/pg_dump -v source_database -h onpremdbserver.domain.com -p 5432 -U source_appuser -Fd -j 100 -f /dbbackup/10/source_database --schema=source_schema  --no-owner --no-privileges
pg_dump: last built-in OID is 16383
pg_dump: reading extensions
pg_dump: identifying extension members
pg_dump: reading schemas
pg_dump: reading user-defined tables
pg_dump: reading user-defined functions
pg_dump: reading user-defined types
pg_dump: reading procedural languages
pg_dump: reading user-defined aggregate functions
pg_dump: reading user-defined operators
pg_dump: reading user-defined access methods
pg_dump: reading user-defined operator classes
pg_dump: reading user-defined operator families
pg_dump: reading user-defined text search parsers
pg_dump: reading user-defined text search templates
pg_dump: reading user-defined text search dictionaries
pg_dump: reading user-defined text search configurations
pg_dump: reading user-defined foreign-data wrappers
pg_dump: reading user-defined foreign servers
pg_dump: reading default privileges
pg_dump: reading user-defined collations
pg_dump: reading user-defined conversions
pg_dump: reading type casts
pg_dump: reading transforms
pg_dump: reading table inheritance information
pg_dump: reading event triggers
pg_dump: finding extension tables
pg_dump: finding inheritance relationships
pg_dump: reading column info for interesting tables
pg_dump: finding the columns and types of table "source_schema.table1"
pg_dump: finding the columns and types of table "source_schema.table2"
pg_dump: finding the columns and types of table "source_schema.table3"
pg_dump: finding default expressions of table "source_schema.table3"
pg_dump: flagging inherited columns in subtables
pg_dump: reading indexes
pg_dump: reading extended statistics
pg_dump: reading constraints
pg_dump: reading triggers
pg_dump: reading rewrite rules
pg_dump: reading policies
pg_dump: reading row security enabled for table "source_schema.table1"
pg_dump: reading policies for table "source_schema.table1"
pg_dump: reading row security enabled for table "source_schema.table2"
pg_dump: reading policies for table "source_schema.table2"
pg_dump: reading row security enabled for table "source_schema.table3_id_seq"
pg_dump: reading policies for table "source_schema.table3_id_seq"
pg_dump: reading row security enabled for table "source_schema.table3"
pg_dump: reading policies for table "source_schema.table3"
pg_dump: reading publications
pg_dump: reading publication membership
pg_dump: reading publication membership for table "source_schema.table1"
pg_dump: reading publication membership for table "source_schema.table2"
pg_dump: reading publication membership for table "source_schema.table3"
pg_dump: reading subscriptions
pg_dump: reading dependency data
pg_dump: saving encoding = UTF8
pg_dump: saving standard_conforming_strings = on
pg_dump: saving search_path =
pg_dump: dumping contents of table "source_schema.table1"
pg_dump: dumping contents of table "source_schema.table3"
pg_dump: finished item 3797 TABLE DATA table1
pg_dump: finished item 3800 TABLE DATA table3
pg_dump: dumping contents of table "source_schema.table2"
pg_dump: finished item 3798 TABLE DATA table2
 
 
05-24-2019 23:22:14 - completed database backup...
 
 
######################################################################################################
05-24-2019 23:22:14 - beginning database restore...
 
 # Create schema...
/usr/pgsql-10/bin/psql -E -e -hdest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com -p5432 dest_database_newdb -Udest_rds_superuser -c"create schema source_schema"
create schema source_schema
CREATE SCHEMA
  
# Restore database...
/usr/pgsql-10/bin/pg_restore -v -d dest_database_newdb -hdest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com -p5432 -U dest_rds_superuser -j 120 -Fd /dbbackup/10/source_database --schema=source_schema  --no-owner --no-privileges
pg_restore: connecting to database for restore
pg_restore: processing item 3803 ENCODING ENCODING
pg_restore: processing item 3804 STDSTRINGS STDSTRINGS
pg_restore: processing item 3805 SEARCHPATH SEARCHPATH
pg_restore: processing item 6 SCHEMA source_schema
pg_restore: processing item 197 TABLE table1
pg_restore: creating TABLE "source_schema.table1"
pg_restore: processing item 198 TABLE table2
pg_restore: creating TABLE "source_schema.table2"
pg_restore: processing item 200 TABLE table3
pg_restore: creating TABLE "source_schema.table3"
pg_restore: processing item 199 SEQUENCE table3_id_seq
pg_restore: creating SEQUENCE "source_schema.table3_id_seq"
pg_restore: processing item 3806 SEQUENCE OWNED BY table3_id_seq
pg_restore: creating SEQUENCE OWNED BY "source_schema.table3_id_seq"
pg_restore: processing item 3675 DEFAULT table3 id
pg_restore: creating DEFAULT "source_schema.table3 id"
pg_restore: entering main parallel loop
pg_restore: launching item 3797 TABLE DATA table1
pg_restore: launching item 3798 TABLE DATA table2
pg_restore: launching item 3800 TABLE DATA table3
pg_restore: launching item 3807 SEQUENCE SET table3_id_seq
pg_restore: pg_restore: processing data for table "source_schema.table2"
processing data for table "source_schema.table3"
pg_restore: executing SEQUENCE SET table3_id_seq
pg_restore: processing data for table "source_schema.table1"
pg_restore: finished item 3807 SEQUENCE SET table3_id_seq
pg_restore: finished item 3798 TABLE DATA table2
pg_restore: finished item 3800 TABLE DATA table3
pg_restore: finished item 3797 TABLE DATA table1
pg_restore: finished main parallel loop
 
 
05-24-2019 23:22:14 - completed database restore...
  
                           List of schemas
     Name      |       Owner        | Access privileges | Description
---------------+--------------------+-------------------+-------------
 source_schema | dest_rds_superuser |                   |
(1 row)
                                List of relations
    Schema     |  Name  | Type  |       Owner        |    Size    | Description
---------------+--------+-------+--------------------+------------+-------------
 source_schema | table1 | table | dest_rds_superuser | 0 bytes    |
 source_schema | table2 | table | dest_rds_superuser | 0 bytes    |
 source_schema | table3 | table | dest_rds_superuser | 8192 bytes |
(3 rows)
 
 
05-24-2019 23:22:14 - DURATION: END

Step 7 — Clean Up

After validating from the end user, remove any older databases and/or schemas. You have now migrated your database from on prem to AWS RDS! In the next part, I will discuss “Common Errors when Migrating PostgreSQL Databases From On-Prem to AWS RDS”.


  1. Database
    2.   
  2. Mysql
    3.   
  3. Oracle
    4.   
  4. Sqlserver
    5.   
  5. PostgreSQL
    6.   
  6. Access
    7.   
  7. SQLite
    8.   
  8. MariaDB

  1. Hiệu suất TPC-H kể từ PostgreSQL 8.3

  2. Cột PostgreSQL không tồn tại nhưng nó thực sự có

  3. PostgreSQL:Xóa thuộc tính khỏi cột JSON

  4. Làm cách nào để xuất bảng dưới dạng CSV với các tiêu đề trên Postgresql?

  5. Khởi động ứng dụng Spring Boot rất chậm