分类: PostgreSQL

ucloud云上部署GreenPlum4.3.12集群

一、机器环境、版本如下:
操作系统:CentOS6.5
数据库版本:
greenplum-db-4.3.12.0-rhel5-x86_64
greenplum-cc-web-3.2.0-LINUX-x86_64

节点信息:

主机名        Ip地址          说明
mdw          172.28.1.11     主库master
smdw         172.28.1.12     备库standby master
sdw1         172.28.1.13     segment库节点一
sdw2         172.28.1.14     segment库节点二

二、系统初始化设置(四台机器都修改)
1、修改系统参数/etc/sysctl.conf

kernel.shmmax = 500000000
kernel.shmmni = 4096
kernel.shmall = 4000000000
kernel.sem = 250 512000 100 2048
kernel.sysrq = 1
kernel.core_uses_pid = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.msgmni = 2048
net.ipv4.tcp_syncookies = 1
net.ipv4.ip_forward = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.arp_filter = 1
net.ipv4.conf.all.arp_filter = 1
net.ipv4.ip_local_port_range = 1025 65535
net.core.netdev_max_backlog = 10000
vm.overcommit_memory = 2

2、文件句柄数修改/etc/security/limits.conf

* soft nofile 65536
* hard nofile 65536
* soft nproc 131072
* hard nproc 131072
* soft core unlimited

3、关闭selinux、iptables

修改/etc/selinux/config中SELINUX=disabled
关闭iptables
service iptables stop
chkconfig iptables off

4、调整磁盘IO调度
Linux磁盘I/O调度器对磁盘的访问支持不同的策略,默认的为cfq,GreenPlum建议设置为deadline
查看磁盘的I/O调度策略,看到默认的为[cfq]

查看系统分区情况:
[root@dw01 ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        20G  3.3G   16G  18% /
tmpfs            12G     0   12G   0% /dev/shm
/dev/vdb        493G  311M  467G   1% /data
查看/分区的I/O调度策略
[root@dw01 ~]# cat /sys/block/vda/queue/scheduler  
noop anticipatory [deadline] cfq 
查看/data分区的I/O调度策略
[root@dw01 ~]# cat /sys/block/vdb/queue/scheduler 
noop anticipatory [deadline] cfq 

查看当前系统内核

[root@dw01 ~]# uname -r
2.6.32-696.18.7.1.el6.ucloud.x86_64

修改系统引导文件,在/boot/grub/menu.lst 文件里面关于kernel这一行的末尾添加elevator=deadline

 /boot/grub/menu.lst 
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You do not have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /, eg.
#          root (hd0,0)
#          kernel /boot/vmlinuz-version ro root=/dev/vda1
#          initrd /boot/initrd-[generic-]version.img
#boot=/dev/vda
default=0
timeout=1
splashimage=(hd0,0)/boot/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.32-696.18.7.1.el6.ucloud.x86_64)
        root (hd0,0)
        kernel /boot/vmlinuz-2.6.32-696.18.7.1.el6.ucloud.x86_64 ro root=/dev/vda1 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=512M-2G:64M,2G-4G:128M,4G-:192M  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM quiet console=tty1 console=ttyS0,115200n8 elevator=deadline
        initrd /boot/initramfs-2.6.32-696.18.7.1.el6.ucloud.x86_64.img
title CentOS (2.6.32-696.18.7.el6.x86_64.debug)
        root (hd0,0)
        kernel /boot/vmlinuz-2.6.32-696.18.7.el6.x86_64.debug ro root=/dev/vda1 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=512M-2G:64M,2G-4G:128M,4G-:192M  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM quiet console=tty1 console=ttyS0,115200n8
        initrd /boot/initramfs-2.6.32-696.18.7.el6.x86_64.debug.img
title CentOS (2.6.32-431.11.29.el6.ucloud.x86_64)
        root (hd0,0)
        kernel /boot/vmlinuz-2.6.32-431.11.29.el6.ucloud.x86_64 ro root=/dev/vda1 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=512M-2G:64M,2G-4G:128M,4G-:192M  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM quiet console=tty1 console=ttyS0,115200n8
        initrd /boot/initramfs-2.6.32-431.11.29.el6.ucloud.x86_64.img
title CentOS (2.6.32-431.11.25.el6.ucloud.x86_64)
        root (hd0,0)
        kernel /boot/vmlinuz-2.6.32-431.11.25.el6.ucloud.x86_64 ro root=/dev/vda1 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=512M-2G:64M,2G-4G:128M,4G-:192M  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM quiet console=tty1 console=ttyS0,115200n8
        initrd /boot/initramfs-2.6.32-431.11.25.el6.ucloud.x86_64.img

查看硬盘I/O预读扇区值:默认为256

[root@dw01 ~]# blockdev --getra /dev/vda1 
256
[root@dw01 ~]# blockdev --getra /dev/vdb
256

修改为65536

blockdev --setra 65536 /dev/vda1
blockdev --setra 65536 /dev/vdb

须将其写入开机配置文件/etc/rc.d/rc.local 否则重启就会失效。

/etc/rc.d/rc.local 
echo "blockdev --setra 65536 /dev/vda1" >> /etc/rc.d/rc.local 
echo "blockdev --setra 65536 /dev/vdb" >> /etc/rc.d/rc.local 

5、修改hostname

[root@dw01 ~]# vim /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=dw01
NOZEROCONF=yes

6、修改hosts

 /etc/hosts 
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

172.28.1.11   dw01 mdw
172.28.1.12   dw02 smdw
172.28.1.13   dw03 sdw1
172.28.1.14   dw04 sdw2

以上设置四台机器都需设置,重启生效;

7、创建配置文件
a、创建包含Greenplum部署的所有主机名,内容如下

[gpadmin@dw01 gpconfig]$ cat host_file 
mdw
smdw
sdw1
sdw2

b、创建包含备库standby mastersegment库节点一节点二的主机名,内容如下

[gpadmin@dw01 gpconfig]$ cat dw234 
smdw
sdw1
sdw2

c、创建包含segment库节点一节点二的主机名,内容如下

[gpadmin@dw01 gpconfig]$ cat seg_hosts 
sdw1
sdw2

三、下载GreenPlum安装包
根据操作系统版本下载:
https://network.pivotal.io/products/pivotal-gpdb#/releases/4540/file_groups/560
这里选择
Greenplum Database 4.3.12.0 for RedHat Entrerprise Linux 5, 6 and 7
122 MB4.3.12.0
注意 登陆Pivotal Network账号以后,才能下载

md5sum *zip
ee90c7a35c706404840be38ba1be557b  greenplum-cc-web-3.2.0-LINUX-x86_64.zip
edaa67d561653fbf81431c968e5f297f  greenplum-db-4.3.12.0-rhel5-x86_64.zip

四、解压安装

unzip greenplum-db-4.3.12.0-rhel5-x86_64.zip
./greenplum-db-4.3.12.0-rhel5-x86_64.bin

根据系统提示输入yes和回车,默认会按照在/usr/local/greenplum-db-4.3.12.0下,并创建软链接greenplum-db ---/greenplum-db-4.3.12.0
在/home/gpadmin/.bash_profile文件中添加

source /usr/local/greenplum-db/greenplum_path.sh
export GPHOME=/usr/local/greenplum-db
export MASTER_DATA_DIRECTORY=/data/gpmaster/gpseg-1

五、免key登陆
四台机器之间互相做免密码登陆,参考https://www.unixso.com/Linux/ssh-key.html

source  /usr/local/greenplum-db/greenplum_path.sh
gpssh-exkeys -f /usr/local/greenplum-db/gpconfig/host_file     #打通所有服务器       

六、创建配置文件gpinitsystem_config
在master机器上创建

mkdir -p /usr/local/greenplum-db/gpconfig
创建/usr/local/greenplum-db/gpconfig/gpinitsystem_config文件内容如下:
ARRAY_NAME="EMC Greenplum DW"
SEG_PREFIX=gpseg
PORT_BASE=40000
declare -a DATA_DIRECTORY=(/data/gpdata/gpdatap1 /data/gpdata/gpdatap1)
MASTER_HOSTNAME=dw01
MASTER_DIRECTORY=/data/gpmaster
MASTER_PORT=5432
TRUSTED_SHELL=ssh
CHECK_POINT_SEGMENTS=8
ENCODING=UNICODE
MIRROR_PORT_BASE=50000
REPLICATION_PORT_BASE=41000
MIRROR_REPLICATION_PORT_BASE=51000
declare -a MIRROR_DATA_DIRECTORY=(/data/gpdata/gpdatam1 /data/gpdata/gpdatam1)
MACHINE_LIST_FILE=/usr/local/greenplum-db/gpconfig/seg_hosts

gpinitsystem_config关键配置项参数说明:

DATA_DIRECTORY:决定了每个Segment上部署多少个Primary Instance,一个目录就是一个Instance。这里建议把所有的Instance都指向一个目录,例如部署两个实例,则为:/gpdata/primary /gpdata/primary。原因是我们在使用中会创建新的文件系统,这样便于和文件系统区分。
MASTER_HOSTNAME:Master的主机名。
MASTER_DIRECTORY:Master文件目录。
MASTER_PORT:该端口用于访问Greenplum数据库。
MIRROR_DATA_DIRECTORY:(如果需要部署Mirror),决定了每个Segment上部署多少个Mirror Instance。类似于DATA_DIRECTORY,例如:/gpdata/mirror /gpdata/mirror。
DATABASE_NAME:创建一个数据库,输入数据库名称。也可以之后手工创建。

六、分发、创建用户、目录、权限
安装包分发:master上操作

cd /usr/local
scp -r greenplum-db-4.3.12.0/ dw02:/usr/local/
scp -r greenplum-db-4.3.12.0/ dw03:/usr/local/
scp -r greenplum-db-4.3.12.0/ dw04:/usr/local/
gpssh -f /usr/local/greenplum-db/gpconfig/host_file
Note: command history unsupported on this machine ...
=>
依次执行以下命令
useradd -g gpadmin gpadmin
useradd -g gpmon gpmon
echo 123456 | passwd gpadmin --stdin
echo 123456 | passwd gpmon --stdin
mkdir -p /data/gpdata/gpdatap1
mkdir -p /data/gpdata/gpdatam1
mkdir -p /data/gpmaster
chown -R gpadmin.gpadmin /data/gpdata/
chown -R gpadmin.gpadmin /data/gpmaster/
chown -R gpadmin.gpadmin /usr/local/greenplum-db-4.3.12.0/

七、系统检查

source /usr/local/greenplum-db/greenplum_path.sh
gpcheck -f /usr/local/greenplum-db/gpconfig/host_file -m mdw -s smdw          
20180406:14:39:12:026168 gpcheck:dw01:root-[INFO]:-dedupe hostnames
20180406:14:39:13:026168 gpcheck:dw01:root-[INFO]:-Detected platform: Generic Linux Cluster
20180406:14:39:13:026168 gpcheck:dw01:root-[INFO]:-generate data on servers
20180406:14:39:13:026168 gpcheck:dw01:root-[INFO]:-copy data files from servers
20180406:14:39:13:026168 gpcheck:dw01:root-[INFO]:-delete remote tmp files
20180406:14:39:13:026168 gpcheck:dw01:root-[INFO]:-Using gpcheck config file: /usr/local/greenplum-db/./etc/gpcheck.cnf
20180406:14:39:13:026168 gpcheck:dw01:root-[INFO]:-GPCHECK_NORMAL
20180406:14:39:13:026168 gpcheck:dw01:root-[INFO]:-gpcheck completing...

如果系统检查有Error提示,需根据提示修改系统参数,然后重新检查
检查网络性能

gpcheckperf -f host_file -r N -d /tmp/ > checknet.out
cat checknet.out
-------------------
--  NETPERF TEST
-------------------

====================
==  RESULT
====================
Netperf bisection bandwidth test
mdw -> smdw = 287.060000
sdw1 -> sdw2 = 279.900000
smdw -> mdw = 299.840000
sdw2 -> sdw1 = 302.450000

Summary:
sum = 1169.25 MB/sec
min = 279.90 MB/sec
max = 302.45 MB/sec
avg = 292.31 MB/sec
median = 299.84 MB/sec

八、初始化数据库
seg_hosts是segment服务器列表,一行存一个hostname,smdw是standby master的hostname名字,在gpadmin账号下运行:

su - gpadmin
cd /usr/local/greenplum-db/gpconfig/
gpinitsystem -c /usr/local/greenplum-db/gpconfig/gpinitsystem_config  -h seg_hosts -s smdw

如有报错参考官方文档:http://gpdb.docs.pivotal.io/43120/install_guide/init_gpdb.html
安装完以后登陆

[gpadmin@dw01]$ psql -d postgres

psql (8.2.15)

Type "help" for help.

 

postgres=# help

You are using psql, the command-line interface to PostgreSQL.

Type:  \copyright for distribution terms

       \h for help with SQL commands

       \? for help with psql commands

       \g or terminate with semicolon to execute query

       \q to quit

postgres=#

查看各机器进程

gpssh -f /usr/local/greenplum-db/gpconfig/host_file 
Note: command history unsupported on this machine ...
=> netstat -nltp | grep postgres
[sdw2] (Not all processes could be identified, non-owned process info
[sdw2]  will not be shown, you would have to be root to see it all.)
[sdw2] tcp        0      0 0.0.0.0:40000               0.0.0.0:*                   LISTEN      10601/postgres      
[sdw2] tcp        0      0 0.0.0.0:40001               0.0.0.0:*                   LISTEN      10602/postgres      
[sdw2] tcp        0      0 172.28.64.190:41000         0.0.0.0:*                   LISTEN      10636/postgres      
[sdw2] tcp        0      0 172.28.64.190:41001         0.0.0.0:*                   LISTEN      10641/postgres      
[sdw2] tcp        0      0 0.0.0.0:50000               0.0.0.0:*                   LISTEN      10603/postgres      
[sdw2] tcp        0      0 0.0.0.0:50001               0.0.0.0:*                   LISTEN      10600/postgres      
[sdw2] tcp        0      0 172.28.64.190:51000         0.0.0.0:*                   LISTEN      10621/postgres      
[sdw2] tcp        0      0 172.28.64.190:51001         0.0.0.0:*                   LISTEN      10620/postgres      
[sdw2] tcp        0      0 :::40000                    :::*                        LISTEN      10601/postgres      
[sdw2] tcp        0      0 :::40001                    :::*                        LISTEN      10602/postgres      
[sdw2] tcp        0      0 :::50000                    :::*                        LISTEN      10603/postgres      
[sdw2] tcp        0      0 :::50001                    :::*                        LISTEN      10600/postgres      
[smdw] (Not all processes could be identified, non-owned process info
[smdw]  will not be shown, you would have to be root to see it all.)
[smdw] tcp        0      0 0.0.0.0:5432                0.0.0.0:*                   LISTEN      8501/postgres       
[smdw] tcp        0      0 :::5432                     :::*                        LISTEN      8501/postgres       
[ mdw] (Not all processes could be identified, non-owned process info
[ mdw]  will not be shown, you would have to be root to see it all.)
[ mdw] tcp        0      0 0.0.0.0:5432                0.0.0.0:*                   LISTEN      17489/postgres      
[ mdw] tcp        0      0 :::5285                     :::*                        LISTEN      17496/postgres      
[ mdw] tcp        0      0 :::5432                     :::*                        LISTEN      17489/postgres      
[sdw1] (Not all processes could be identified, non-owned process info
[sdw1]  will not be shown, you would have to be root to see it all.)
[sdw1] tcp        0      0 0.0.0.0:40000               0.0.0.0:*                   LISTEN      10662/postgres      
[sdw1] tcp        0      0 0.0.0.0:40001               0.0.0.0:*                   LISTEN      10660/postgres      
[sdw1] tcp        0      0 172.28.56.68:41000          0.0.0.0:*                   LISTEN      10685/postgres      
[sdw1] tcp        0      0 172.28.56.68:41001          0.0.0.0:*                   LISTEN      10684/postgres      
[sdw1] tcp        0      0 0.0.0.0:50000               0.0.0.0:*                   LISTEN      10661/postgres      
[sdw1] tcp        0      0 0.0.0.0:50001               0.0.0.0:*                   LISTEN      10663/postgres      
[sdw1] tcp        0      0 172.28.56.68:51000          0.0.0.0:*                   LISTEN      10693/postgres      
[sdw1] tcp        0      0 172.28.56.68:51001          0.0.0.0:*                   LISTEN      10694/postgres      
[sdw1] tcp        0      0 :::40000                    :::*                        LISTEN      10662/postgres      
[sdw1] tcp        0      0 :::40001                    :::*                        LISTEN      10660/postgres      
[sdw1] tcp        0      0 :::50000                    :::*                        LISTEN      10661/postgres      
[sdw1] tcp        0      0 :::50001                    :::*                        LISTEN      10663/postgres      
=> 

九、安装PerformanceMonitor数据收集Agent

source /usr/local/greenplum-db/greenplum_path.sh
gpperfmon_install --enable --password 123456 --port 5432
gpstop -r   #重启GP生效

查看进程

ps -ef |grep gpmmon
gpadmin  17498 17489  0 Apr04 ?        00:01:04 /usr/local/greenplum-db-4.3.12.0/bin/gpmmon -D /data/gpmaster/gpseg-1/gpperfmon/conf/gpperfmon.conf -p 5432

查看端口

netstat -nltp | grep gp
tcp        0      0 0.0.0.0:28080               0.0.0.0:*                   LISTEN      11984/gpmonws             
tcp        0      0 :::8888                     :::*                        LISTEN      17611/gpsmon 

查看监控数据是否写入数据库

psql -d gpperfmon -c 'select * from system_now;'
        ctime        | hostname |  mem_total  |  mem_used  | mem_actual_used | mem_actual_free | swap_total | swap_used | swap_page_in | swap_page_out | c
pu_user | cpu_sys | cpu_idle | load0 | load1 | load2 | quantum | disk_ro_rate | disk_wo_rate | disk_rb_rate | disk_wb_rate | net_rp_rate | net_wp_rate | n
et_rb_rate | net_wb_rate 
---------------------+----------+-------------+------------+-----------------+-----------------+------------+-----------+--------------+---------------+--
--------+---------+----------+-------+-------+-------+---------+--------------+--------------+--------------+--------------+-------------+-------------+--
-----------+-------------
 2018-04-06 14:59:15 | dw01     | 25132879872 | 1774948352 |       391598080 |     24741281792 |  536866816 |         0 |            0 |             0 |  
   0.22 |    0.34 |    99.44 |  0.02 |  0.04 |     0 |      15 |            0 |            3 |            0 |        47850 |          40 |          43 |  
      9267 |       21458
 2018-04-06 14:59:15 | dw02     |  8188444672 |  963731456 |       157970432 |      8030474240 |  536866816 |         0 |            0 |             0 |  
   0.13 |    0.32 |    99.55 |  0.05 |  0.03 |     0 |      15 |            0 |            2 |            0 |         9788 |           3 |           3 |  
       331 |         572
 2018-04-06 14:59:15 | dw03     |  8188444672 | 2338099200 |       239792128 |      7948652544 |  536866816 |         0 |            0 |             0 |  
   0.28 |    0.42 |    99.27 |     0 |     0 |     0 |      15 |            0 |            9 |            0 |       178355 |         169 |          66 |  
    174387 |      154813
 2018-04-06 14:59:15 | dw04     |  8188444672 | 2338926592 |       242188288 |      7946256384 |  536866816 |         0 |            0 |             0 |  
   0.28 |    0.61 |    99.07 |     0 |     0 |     0 |      15 |            0 |            8 |            0 |       175088 |         165 |          65 |  
    167569 |      162326
(4 rows)

至此GreenPlum4.3.12集群安装完毕·

相关文章

发表新评论