【每日一博】mfs 分布式存储 +master 端高可用

（master采用keepalived+drbd实现高可用 ）

mfsmaster 172.25.254.10

mfsbackup 172.25.254.11

VIP 是172.25.254.12

mfsmaster 和 mfsbackup 一定要主机名解析 , 同时主机名也必须和主机名解析一致。

uname -n 查看你的名字 ~

drbd 配置:

Mfsmaster mfsbackup 都做以下操作, 首先安装包，这是我自己打的 rpm 包。

安装即可

vim /etc/drbd.d/mfs.res

 1 resource mfs {                                                                                                2         meta-disk internal;  3         device /dev/drbd1;  4         syncer {  5                 verify-alg sha1;  6         }  7 on mfsmaster {  8         disk /dev/vda1;  9         address 172.25.254.10:7789; 10 } 11 on mfsbackup { 12         disk /dev/vda1; 13         address 172.25.254.11:7789; 14 } 15 } 

安装mfs master

[root@mfsmaster etc]# cd  /etc [root@mfsmaster etc]# mv mfsmaster.cfg.dist mfsmaster.cfg [root@mfsmaster etc]# mv mfsexports.cfg.dist mfsexports.cfg [root@mfsmaster etc]# mv mfsmetalogger.cfg.dist mfsmetalogger.cfg [root@mfsmaster etc]# mv mfstopology.cfg.dist mfstopology.cfg

vim mfsmaster.cfg

1 

WORKING_USER = nobody  2  WORKING_GROUP = nobody  3  SYSLOG_IDENT = mfsmaster  4  LOCK_MEMORY = 0  5  NICE_LEVEL = -19  6   7  EXPORTS_FILENAME = /etc/mfsexports.cfg  8   9  TOPOLOGY_FILENAME = /etc/mfstopology.cfg 10  11  DATA_PATH = /data1/drbd 12  13  BACK_LOGS = 50 14  BACK_META_KEEP_PREVIOUS = 1 15  16  REPLICATIONS_DELAY_INIT = 300 17  REPLICATIONS_DELAY_DISCONNECT = 3600 18  19  MATOML_LISTEN_HOST = * 20  MATOML_LISTEN_PORT = 9419 21  22  MATOCS_LISTEN_HOST = * 23  MATOCS_LISTEN_PORT = 9420 24  25  MATOCL_LISTEN_HOST = * 26  MATOCL_LISTEN_PORT = 9421 27  28  CHUNKS_LOOP_CPS = 100000 29  CHUNKS_LOOP_TIME = 300 30  31  CHUNKS_SOFT_DEL_LIMIT = 10 32  CHUNKS_HARD_DEL_LIMIT = 25 33  CHUNKS_WRITE_REP_LIMIT = 2 34  CHUNKS_READ_REP_LIMIT = 10 35  36  REJECT_OLD_CLIENTS = 0 

vim mfsmetalogger.cfg

 1  WORKING_USER = nobody                                                                                        2  WORKING_GROUP = nobody   3  SYSLOG_IDENT = mfsmetalogger  4  LOCK_MEMORY = 0  5  NICE_LEVEL = -19  6   7  DATA_PATH = /var/lib/mfs  8   9  BACK_LOGS = 50 10  BACK_META_KEEP_PREVIOUS = 3 11  META_DOWNLOAD_FREQ = 24 12  13  MASTER_RECONNECTION_DELAY = 5 14  15  MASTER_HOST = mfsmaster 16  MASTER_PORT = 9419 17  18  MASTER_TIMEOUT = 60 

ok 为了方便，直接把这几个文件复制到mfsbackup上就可以

[root@mfsmaster etc]# scp mfs* 172.25.254.11:/etc/ root@172.25.254.11's password:  mfsexports.cfg                                                                100% 4060     4.0KB/s   00:00     mfsmaster.cfg                                                                 100%  849     0.8KB/s   00:00     mfsmetalogger.cfg                                                             100%  401     0.4KB/s   00:00     mfstopology.cfg                                                               100% 1123     1.1KB/s   00:00

master上执行

[root@mfsmaster etc]# drbdsetup /dev/drbd1 primary --force [root@mfsmaster etc]# mkfs.ext4 /dev/drbd1

都执行

mkdir -p /data1/drbdchown -R nobody.nobody  /data1/drbd/

[root@mfsmaster mfs]# cp /var/lib/mfs/metadata.mfs.empty /data1/drbd/ [root@mfsmaster drbd]# mv metadata.mfs.empty metadata.mfs [root@mfsmaster drbd]# mfsmaster

ok master 已经可以正常运行了

,为了保证无误我们在backup上也测试一次。

[root@mfsmaster drbd]# mfsmaster stop [root@mfsmaster drbd]# cd[root@mfsmaster ~] # umount /dev/drbd1 [root@mfsmaster ~]# drbdadm secondary mfs

[root@mfsbackup drbd+mfs-rpm]# drbdadm primary mfs [root@mfsbackup drbd+mfs-rpm]# mount /dev/drbd1 /data1/drbd/ [root@mfsbackup drbd+mfs-rpm]# mfsmaster

这个时候我们的master端已经全线部署完成，接下来我们要实现他的高可用，这个很重要

首先说明，我使用自己的方法很简单的实现主断备启，但是是一次性的，也就是说，只支持一次主的mfsmaster断了，backup可以接替，并不能来回。需要手工操作。

脚本写的简单后续有时间修改，网上有很多成品的，虽然很菜还是写自己的。。

同样是keepalived +drbd大神们都是全自动的

我这其实是一次性的但是轻量简单呗。

1. 安装keepalived 两端执行：

yum install gcc openssl-devel popt-devel -y  ./configure --prefix=/usr/local/keepalived [root@mfsmaster keepalived-1.2.8]# make && make install  [root@mfsmaster ~]# ln -s /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/ [root@mfsmaster ~]# ln -s /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/ [root@mfsmaster ~]# ln -s /usr/local/keepalived/etc/keepalived /etc/ [root@mfsmaster ~]# ln -s /usr/local/keepalived/sbin/keepalived /usr/sbin [root@mfsmaster ~]# vim /etc/keepalived/keepalived.conf 

 1 ! Configuration File for keepalived  2   3 global_defs {  4    notification_email {  5      576786031@qq.com  6    }  7    notification_email_from zabbix@server2.com  8    smtp_server 172.25.254.2  9    smtp_connect_timeout 30 10    router_id mfs_master 11 } 12 vrrp_script check_drbd { 13         script "/etc/keepalived/check_drbd.sh" 14             interval 15      15     } 16 vrrp_instance mfs { 17     state MASTER 18     interface eth0 19     virtual_router_id 51                                                                                                                                                  20     priority 150 21     advert_int 1 22     authentication { 23         auth_type PASS 24         auth_pass 1111 25     } 26     virtual_ipaddress { 27         172.25.254.12 28     } 29     track_script { 30         check_drbd 31         } 32 notify_master /etc/keepalived/master.sh 33 notify_backup /etc/keepalived/backup.sh 34 } ~ 

[root@mfsmaster keepalived]# vim check_drbd.sh

 1 /bin/bash: Configuration: command not found  2 # File Name: check_drbd.sh  3 # Author: dty  4 # mail: 576786031@qq.com  5 # Created Time: Sun 02 Aug 2015 01:32:43 PM CST  6 #########################################################################  7 #!/bin/bash      8 A=`ps -C mfsmaster --no-header |wc -l`  9 if [ $A -eq 0 ];then 10     umount /data1/drbd/ 11     drbdadm secondary mfs 12     killall keepalived      13 fi                14 

mfsbackup 上      cd /etc/keepalived                 写两个脚本 master.sh backup.sh

 [root@mfsbackup keepalived]# cat backup.sh master.sh  #!/bin/bash mfsmaster stop umount /dev/drbd1 drbdadm secondary mfs ######################################################################### # File Name: master.sh # Author: dty # mail: 576786031@qq.com # Created Time: Sun 02 Aug 2015 01:38:12 PM CST ######################################################################### #!/bin/bash drbdadm primary mfs mount /dev/drbd1 /data1/drbd mfsmaster start 

vim /etc/keepalived.conf

 1 ! Configuration File for keepalived  2     3 global_defs {   4    notification_email {   5      576786031@qq.com  6    }        7    notification_email_from zabbix@server2.com  8    smtp_server 172.25.254.2   9    smtp_connect_timeout 30 10    router_id mfs_master 11 }        12 vrrp_instance mfs { 13     state BACKUP    14     interface eth0 15     virtual_router_id 51 16     priority 80 17     advert_int 1 18     authentication { 19         auth_type PASS 20         auth_pass 1111 21     } 22     virtual_ipaddress { 23         172.25.254.12 24     } 25 notify_master /etc/keepalived/master.sh 26 notify_backup /etc/keepalived/backup.sh 27 }        ~ 

~                           

此时我们可以实验了

开启mfsmaster，

开启keepalived

mfsbackup上也启动 keepalived

[root@mfsbackup keepalived]# /etc/init.d/keepalived start

我们把主上的mfsmaster关掉

可以看到vip已经没了，转移到了备机上

服务已经完美转移了 。

这个时候我们来搞定客户端和 chunkserver 的安装

yum localinstall mfs-chunkserver-1.6.26-1.x86_64.rpm -y

mv mfschunkserver.cfg.dist mfschunkserver

mkdir -p /data1/mfs //设置个目录存放数据

chown nobody.nobody /data1/mfs

vim /etc/mfschunkserver

 1  WORKING_USER = nobody                                                        2  WORKING_GROUP = nobody   3  SYSLOG_IDENT = mfschunkserver  4  LOCK_MEMORY = 0  5  NICE_LEVEL = -19  6   7  DATA_PATH = /data1/mfs  8   9  MASTER_RECONNECTION_DELAY = 5 10  11  BIND_HOST = * 12  MASTER_HOST = 172.25.254.12 13  MASTER_PORT = 9420 14  15  MASTER_TIMEOUT = 60 16  17  CSSERV_LISTEN_HOST = * 18  CSSERV_LISTEN_PORT = 9422 19  20  HDD_CONF_FILENAME = /etc/mfshdd.cfg 21  HDD_TEST_FREQ = 10 

设置chunk存放路径

[root@server7 ~]# cat /etc/mfshdd.cfg # mount points of HDD drives # #/mnt/hd1 #/mnt/hd2 #etc. /data1/mfs/

chunkserver已经部署完成了有多个以此类推就好了.

下来就是客户端的安装及使用了

# yum localinstall -y mfs-client-1.6.26-1.x86_64.rpm # cd /etc# cp mfsmount.cfg.dist mfsmount.cfg # vi mfsmount.cfg定义客户端默认挂载mfsmaster=mfsmaster/mnt/mfs# mfsmount [root@server4 ~]# df -h Filesystem                    Size  Used Avail Use% Mounted on/dev/mapper/VolGroup-lv_root  6.7G  978M  5.4G  16% /tmpfs                         499M     0  499M   0% /dev/shm/dev/sda1                     485M   33M  427M   8% /boot172.25.254.12:9421             16G     0   16G   0% /mnt/mfs

inodes with permission denied:          0

[root@server4 mfs]# cp /etc/passwd leaf1 [root@server4 mfs]# cp /etc/passwd leaf2 [root@server4 mfs]#  mfsfileinfo leaf1/passwd

leaf1/passwd:  chunk 0: 0000000000000008_00000001 / (id:8 ver:1)   copy 1: 172.25.254.7:9422 [root@server4 mfs]#  mfsfileinfo leaf2/passwd leaf2/passwd:  chunk 0: 0000000000000009_00000001 / (id:9 ver:1)   copy 1: 172.25.254.8:9422   copy 2: 172.25.254.9:9422 ok 效果就这样粗来了 

其他的mfs的使用这里不做赘述，主要为了验证我们的高可用机制是否可行，就到这里了

ps：当出现问题调转到backup后，如果主的正常了，我们 手动切换必须遵守以下步骤

1. 停掉mfsmaster服务

   mfsmaster stop

umount /dev/drbd1

drbdadm secondary mfs

1. 主机：

   1.drbdadm  primary mfs

2.mount /dev/drbd1 /data1/drbd

3.mfsmaster start

4.keepalived start

还有要提到的就是可能会出现脑裂，这一般时操作不按顺序，也可能有其他原因，下面是网上大神写的，

脑裂的处理方法

drbd1主,drbd2辅

1，正常情况下状态：  [root@drbd1 ~]# cat /proc/drbd   version: 8.3.8 (api:88/proto:86-94)  : 299AFE04D7AFD98B3CA0AF9   0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----   ns:2144476 nr:0 dw:36468 dr:2115769 al:14 bm:129 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0 [root@drbd2 ~]# cat /proc/drbd  version: 8.3.8 (api:88/proto:86-94) srcversion: 299AFE04D7AFD98B3CA0AF9   0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----  ns:0 nr:2141684 dw:2141684 dr:0 al:0 bm:130 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0 2，drbd1故障后 drbd1状态： [root@drbd1 ~]# cat /proc/drbd  version: 8.3.8 (api:88/proto:86-94) srcversion: 299AFE04D7AFD98B3CA0AF9   0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown   r----  ns:4 nr:102664 dw:102668 dr:157 al:1 bm:8 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0 drbd2的状态： [root@drbd2 ~]# cat /proc/drbd  version: 8.3.8 (api:88/proto:86-94) srcversion: 299AFE04D7AFD98B3CA0AF9   0: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r----  ns:0 nr:2141684 dw:2141684 dr:0 al:0 bm:130 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0 3，处理方法： a，将secondary配置成primary角色 [root@drbd2 ~]# drbdsetup /dev/drbd0 primary -o [root@drbd2 ~]# cat /proc/drbd  version: 8.3.8 (api:88/proto:86-94) srcversion: 299AFE04D7AFD98B3CA0AF9   0: cs:WFConnection ro:Primary/Unknown ds:UpToDate/Outdated C r----  ns:0 nr:2141684 dw:2141684 dr:0 al:0 bm:130 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0 挂载： [root@drbd2 /]# mount /dev/drbd0 /data1 [root@drbd2 data1]# ll total 10272 -rw-r--r-- 1 root root 10485760 Feb 13 11:26 aa.img drwx------ 2 root root 16384 Feb 13 11:25 lost+found 这个时候drbd2开始提供服务，开始写数据 drbd1主恢复正常后： [root@drbd1 ~]# cat /proc/drbd version: 8.3.8 (api:88/proto:86-94) srcversion: 299AFE04D7AFD98B3CA0AF9   0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown   r----  ns:2144476 nr:0 dw:36484 dr:2115769 al:14 bm:129 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:8 drbd1状态是:StandAlone，此时，drbd1是不会和drbd2互相联系的 我们来查看下日志： [root@drbd1 ~]# tailf /var/log/messages Feb 13 16:14:27 drbd1 kernel: block drbd0: helper command: /sbin/drbdadm split-brain minor-0 Feb 13 16:14:27 drbd1 kernel: block drbd0: helper command: /sbin/drbdadm split-brain minor-0 exit code 0 (0x0) Feb 13 16:14:27 drbd1 kernel: block drbd0: conn( WFReportParams -> Disconnecting )  Feb 13 16:14:27 drbd1 kernel: block drbd0: error receiving ReportState, l: 4! Feb 13 16:14:27 drbd1 kernel: block drbd0: asender terminated Feb 13 16:14:27 drbd1 kernel: block drbd0: Terminating drbd0_asender Feb 13 16:14:27 drbd1 kernel: block drbd0: Connection closed Feb 13 16:14:27 drbd1 kernel: block drbd0: conn( Disconnecting -> StandAlone )  Feb 13 16:14:27 drbd1 kernel: block drbd0: receiver terminated Feb 13 16:14:27 drbd1 kernel: block drbd0: Terminating drbd0_receiver 脑裂出现！ 解决方法： 1>,我们需要将现在的drbd1角色修改为secondary [root@drbd1 ~]# drbdadm secondary r0 [root@drbd1 ~]# drbdadm -- --discard-my-data connect r0  ##该命令告诉drbd，secondary上的数据不正确，以primary上的数据为准。 2>，我们还需要在drbd2上执行下面操作 [root@drbd2 /]# drbdadm connect r0 这样drbd1就能和drbd2开始连接上了，并且保证数据不会丢失： [root@drbd1 ~]# cat /proc/drbd    version: 8.3.8 (api:88/proto:86-94) srcversion: 299AFE04D7AFD98B3CA0AF9   0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----  ns:0 nr:20592 dw:20592 dr:0 al:0 bm:4 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0 

ok 就这样了。 未完待续。。。。。