转载

最近的一次ASM diskgroup在线迁移记录

目的：一套11.2.0.4 双节点的RAC当前使用的是阵列里的FC(Fibre channel)盘，运行一段时间后发现数据库比较空闲，所以打算将其迁移到同阵列的SATA盘上，以将FC盘预留给对IO要求更高的应用。
迁移方法：由ASM实现存储管理的数据库，使用ASM diskgroup在线迁移方式是再好不过的选择
挑战：难度虽然不大，但前一次做diskgroup在线迁移还是3年以前的事情，这次特意从头到尾将整个步骤整理如下，供参考

//////////////////
// 第一部分：新盘准备
//////////////////
1、shajx1主机上当前使用的dg及盘
ASMCMD> lsdg
State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED EXTERN N 512 4096 1048576 743632 350106 0 350106 0 N AJXDG1/
MOUNTED EXTERN N 512 4096 1048576 743632 321451 0 321451 0 N AJXDG2/
MOUNTED EXTERN N 512 4096 1048576 7500 7134 0 7134 0 N SYSDG1/
MOUNTED EXTERN N 512 4096 1048576 7500 7103 0 7103 0 Y SYSDG2/
MOUNTED EXTERN N 512 4096 1048576 7500 7403 0 7403 0 N SYSDG3/
ASMCMD> lsdsk -G sysdg1
Path
/dev/rhdiskpower44
/dev/rhdiskpower45
ASMCMD> lsdsk -G sysdg2
Path
/dev/rhdiskpower46
/dev/rhdiskpower47
ASMCMD> lsdsk -G sysdg3
Path
/dev/rhdiskpower48
/dev/rhdiskpower49

2、shajx1主机上认到的新盘及大小如下：
/dev/rhdiskpower58 13FD 09E:0 05A:C8 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower59 1401 09E:0 06A:C7 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower60 1405 09E:0 05A:CA 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower61 1409 09E:0 06A:C9 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower62 140D 09E:0 05A:CC 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower63 1411 09E:0 06A:CB 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower64 1415 09E:0 06C:D0 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower65 1419 09E:0 05C:D1 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower66 141D 09E:0 06C:D2 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower67 1421 09E:0 05C:D3 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower68 1425 09E:0 06C:D4 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower69 1429 09E:0 06A:CD 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower70 142D 09E:0 05D:D6 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower71 1431 09E:0 06D:D5 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower72 1435 09E:0 05D:D8 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower73 1439 09E:0 06D:D7 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower74 143D 09E:0 05D:DA 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower75 1441 09E:0 06D:D9 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower76 1445 09E:0 05D:DC 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower77 1449 09E:0 06D:DB 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower78 144D 09E:0 06B:C0 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower79 1451 09E:0 05B:C1 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower80 1455 09E:0 06B:C2 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower81 1459 09E:0 05B:C3 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower82 145D 09E:0 06B:C4 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower83 1461 09E:0 06A:D6 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower84 1465 09E:0 05A:D5 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower85 1469 09E:0 06A:D8 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower86 146D 09E:0 05A:D7 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower87 1471 09E:0 06A:DA 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower88 1475 09E:0 05A:D9 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower89 1479 09E:0 06A:DC 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower90 147D 09E:0 05A:DB 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower91 1481 09E:0 05C:C0 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower92 1485 09E:0 06C:C1 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower93 1489 09E:0 05C:C2 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower94 148D 09E:0 06C:C3 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower95 1491 09E:0 05C:C4 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower96 1495 09E:0 06D:C6 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower97 1499 09E:0 05D:C5 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower98 149D 09E:0 06D:C8 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower99 14A1 09E:0 05D:C7 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower100 14A5 09E:0 06D:CA 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower101 14A9 09E:0 05D:C9 2-Way Mir N/Grp'd (M) RW 46478
/dev/rhdiskpower102 40DF 09E:0 10A:D18 2-Way Mir N/Grp'd RW 938
/dev/rhdiskpower103 40E0 09E:0 10C:D18 2-Way Mir N/Grp'd RW 938
/dev/rhdiskpower104 40E1 09E:0 09B:D18 2-Way Mir N/Grp'd RW 938
/dev/rhdiskpower105 40E2 09E:0 09D:D18 2-Way Mir N/Grp'd RW 938
/dev/rhdiskpower106 40E3 09E:0 09A:D19 2-Way Mir N/Grp'd RW 938
/dev/rhdiskpower107 40E4 09E:0 09C:D19 2-Way Mir N/Grp'd RW 938
/dev/rhdiskpower108 40DD 09E:0 09B:C1B 2-Way Mir N/Grp'd RW 938
/dev/rhdiskpower109 40DE 09E:0 10D:C1A 2-Way Mir N/Grp'd RW 938
/dev/rhdiskpower110 40D5 09E:0 08B:D1D 2-Way Mir N/Grp'd RW 938
/dev/rhdiskpower111 40D6 09E:0 08D:D1D 2-Way Mir N/Grp'd RW 938
/dev/rhdiskpower112 40D7 09E:0 10A:C1B 2-Way Mir N/Grp'd RW 938
/dev/rhdiskpower113 40D8 09E:0 09C:C1A 2-Way Mir N/Grp'd RW 938
/dev/rhdiskpower114 40D9 09E:0 10B:C1A 2-Way Mir N/Grp'd RW 938
/dev/rhdiskpower115 40DA 09E:0 09D:C19 2-Way Mir N/Grp'd RW 938
/dev/rhdiskpower116 40DB 09E:0 09A:C1C 2-Way Mir N/Grp'd RW 938
/dev/rhdiskpower117 40DC 09E:0 10C:C1B 2-Way Mir N/Grp'd RW 938

其中hdiskpower58~89用于数据文件，hdiskpower90~hdiskpower101保留用于archivelog文件系统，hdiskpower102~hdiskpower117用于新建OCR、vote盘所在DG

3、修改hdiskpower58~hdiskpower89、hdiskpower102~hdiskpower117的磁盘属性及属主，属性为crw-rw----，属主为grid.asmadmin，两个节点都要修改
shajx1、shajx2都要运行：
i=58
while [ $i -lt 90 ]
do
chown grid.asmadmin /dev/rhdiskpower$i
((i=i+1))
done

i=102
while [ $i -lt 118 ]
do
chown grid.asmadmin /dev/rhdiskpower$i
((i=i+1))
done

4、登陆ASM实例确认新加的盘都能认到，两个节点都要执行
sqlplus '/as sysasm'
col path format a40
set linesize 160 pagesize 200
select mount_status,header_status,mode_status,os_mb,state,path from v$asm_disk where header_status='CANDIDATE' order by path;
MOUNT_S HEADER_STATU MODE_ST OS_MB STATE PATH
------- ------------ ------- ---------- -------- ----------------------------------------
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower0
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower1
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower10
CLOSED CANDIDATE ONLINE 937 NORMAL /dev/rhdiskpower102
CLOSED CANDIDATE ONLINE 937 NORMAL /dev/rhdiskpower103
CLOSED CANDIDATE ONLINE 937 NORMAL /dev/rhdiskpower104
CLOSED CANDIDATE ONLINE 937 NORMAL /dev/rhdiskpower105
CLOSED CANDIDATE ONLINE 937 NORMAL /dev/rhdiskpower106
CLOSED CANDIDATE ONLINE 937 NORMAL /dev/rhdiskpower107
CLOSED CANDIDATE ONLINE 937 NORMAL /dev/rhdiskpower108
CLOSED CANDIDATE ONLINE 937 NORMAL /dev/rhdiskpower109
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower11
CLOSED CANDIDATE ONLINE 937 NORMAL /dev/rhdiskpower110
CLOSED CANDIDATE ONLINE 937 NORMAL /dev/rhdiskpower111
CLOSED CANDIDATE ONLINE 937 NORMAL /dev/rhdiskpower112
CLOSED CANDIDATE ONLINE 937 NORMAL /dev/rhdiskpower113
CLOSED CANDIDATE ONLINE 937 NORMAL /dev/rhdiskpower114
CLOSED CANDIDATE ONLINE 937 NORMAL /dev/rhdiskpower115
CLOSED CANDIDATE ONLINE 937 NORMAL /dev/rhdiskpower116
CLOSED CANDIDATE ONLINE 937 NORMAL /dev/rhdiskpower117
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower2
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower3
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower4
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower5
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower59
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower6
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower60
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower61
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower62
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower63
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower64
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower65
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower66
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower67
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower68
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower69
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower7
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower70
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower71
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower72
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower73
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower74
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower75
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower76
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower77
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower78
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower79
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower8
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower80
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower81
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower82
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower83
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower84
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower85
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower86
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower87
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower88
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower89
CLOSED CANDIDATE ONLINE 46477 NORMAL /dev/rhdiskpower9

5、挑首尾两块盘作读测试及kfed测试，使用oracle或者grid用户执行：
dd if=/dev/rhdiskpower58 of=/dev/null count=100 bs=8192
100+0 records in.
100+0 records out.

dd if=/dev/rhdiskpower89 of=/dev/null count=100 bs=8192
100+0 records in.
100+0 records out.

kfed read /dev/rhdiskpower58
kfed read /dev/rhdiskpower89

kfed的输出如下：
kfbh.endian: 0 ; 0x000: 0x00
kfbh.hard: 0 ; 0x001: 0x00
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt: 0 ; 0x003: 0x00
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 0 ; 0x008: file=0
kfbh.check: 0 ; 0x00c: 0x00000000
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000

MOS里关于KFBTYP_INVALID的描述：If the disk is not a FORMER disk, then kfed on new disks will return: kfbh.type: 0 ; 0x002: KFBTYP_INVALID and that is expected

6、修改hdiskpowerX的reserve_policy属性，两节点都要执行
i=58
while [ $i -lt 90 ]
do
chdev -l hdiskpower${i} -a reserve_policy=no_reserve
(( i=i+1 ))
done

i=102
while [ $i -lt 118 ]
do
chdev -l hdiskpower${i} -a reserve_policy=no_reserve
(( i=i+1 ))
done

***验证reserve_policy是否已经成功
i=58
while [ $i -lt 90 ]
do
echo "hdiskpower${i} " `lsattr -El hdiskpower${i} |grep reserve_policy | awk '{print $2}'`
(( i=i+1 ))
done

i=102
while [ $i -lt 118 ]
do
echo "hdiskpower${i} " `lsattr -El hdiskpower${i} |grep reserve_policy | awk '{print $2}'`
(( i=i+1 ))
done

7、选一块新的盘在上面创建一个Diskgroup
create diskgroup dgnew1 external redundancy disk '/dev/rhdiskpower58';

SQL> SELECT NAME,TOTAL_MB,usable_file_mb,state FROM V$ASM_DISKGROUP where name='DGNEW1';

NAME TOTAL_MB USABLE_FILE_MB STATE
------------------------------ ---------- -------------- -----------
DGNEW1 46477 46427 MOUNTED <---损耗主要体现在Allocation Table、PST、Disk Directory等各种directory

diskgroup里所有的metadata见下：

PST
Allocation Table
Free Space Table
Disk Directory (file number 2)
File Directory (file number 1)
ACD (file number 3)
Alias Directory (file number 6)
Attribute Directory (file number 9)
User and Group Directory
Used Space Directory (file number 8)
How many partners
Continuing Operations Directory (file number 4)
Template Directory (file number 5)
Volume Directory
staleness_directory_and_registry

8、检查crs里新建diskgroup的状态
grid@qc740701a:/home/grid>crsctl status res ora.DGNEW1.dg -t
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DGNEW1.dg
OFFLINE OFFLINE qc740701a <---第二个节点上需要mount
ONLINE ONLINE qc740702a

因为是RAC环境，在第二个节点上也把这个diskgroup mount起来
su - grid
srvctl start diskgroup -g dgnew1 -n qc740701a

crsctl status res ora.DGNEW1.dg -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DGNEW1.dg
ONLINE ONLINE qc740701a
ONLINE ONLINE qc740702a

9、如果在第8步里，两个节点都能mount起DG，那么将这个diskgroup从N-1个节点都dismount，只留最后一个节点用于drop
qc740702a: dismount
srvctl stop diskgroup -g dgnew1 -n qc740702a

qc740701a: drop
drop diskgroup dgnew1

10、至此测试完毕，新认的盘已可以使用

//////////////////
// 第二部分：迁移ASM spfile, voting disk，ocr所在的DG至SATA盘
//////////////////
1、ASM spfile, voting disk 存放在SYSDG1、SYSDG2中，SYSDG3里暂时没有文件
ASMCMD> spget
+SYSDG1/shajx-cluster/asmparameterfile/registry.253.854489087 <---spfile
ASMCMD> quit

grid@qc740702a:/home/grid>ocrcheck -config <---ocr file
Oracle Cluster Registry configuration is :
Device/File Name : +sysdg1
Device/File Name : +sysdg2

grid@qc740702a:/home/grid>crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 46c6fdafabb74fa1bf2987f4d9a846b5 (/dev/rhdiskpower46) [SYSDG2] <---voting disk
Located 1 voting disk(s).

2、将hdiskpower102~hdiskpower117添加到SYSDG1、SYSDG2、SYSDG3
SYSDG1:
alter diskgroup sysdg1 add disk '/dev/rhdiskpower102','/dev/rhdiskpower103','/dev/rhdiskpower104','/dev/rhdiskpower105' drop disk SYSDG1_0000,SYSDG1_0001 rebalance power 500;

SYSDG2:
alter diskgroup sysdg2 add disk '/dev/rhdiskpower106','/dev/rhdiskpower107','/dev/rhdiskpower108','/dev/rhdiskpower109' drop disk SYSDG2_0000,SYSDG2_0001 rebalance power 500;

SYSDG3:
alter diskgroup sysdg3 add disk '/dev/rhdiskpower110','/dev/rhdiskpower111','/dev/rhdiskpower112','/dev/rhdiskpower113' drop disk SYSDG3_0000,SYSDG3_0001 rebalance power 500;

3、将hdiskpower58~hdiskpower73添加到AJXDG1
AJXDG1: 通过shell脚本获取SQL脚本
i=58
while [ $i -lt 74 ]
do
echo "alter diskgroup AJXDG1 add disk '/dev/rhdiskpower${i}' rebalance power 0;";
(( i=i+1 ))
done

得到如下脚本直接运行
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower58' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower59' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower60' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower61' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower62' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower63' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower64' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower65' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower66' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower67' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower68' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower69' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower70' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower71' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower72' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower73' rebalance power 0;

执行如下SQL
select 'alter diskgroup '||d.name||' drop disk '||s.name ||' rebalance power 0;' from v$asm_disk_stat s,v$asm_diskgroup_stat d where s.group_number=d.group_number and d.name in ('AJXDG1');

得到drop old disk的cmd
alter diskgroup AJXDG1 drop disk AJXDG1_0000 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0001 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0002 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0003 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0004 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0005 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0006 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0007 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0008 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0009 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0010 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0011 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0012 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0013 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0014 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0015 rebalance power 0;

---3.1 观察rebalance前的磁盘状态
col path format a30
col header_status format a10
col mode_status format a20
set linesize 130 pagesize 130
select s.name,s.state,s.total_mb,s.free_mb,path,create_date from v$asm_disk_stat s,v$asm_diskgroup_stat d where s.group_number=d.group_number and d.name in ('AJXDG1');

NAME STATE TOTAL_MB FREE_MB PATH CREATE_DA
------------------------------ -------- ---------- ---------- ------------------------------ ---------
AJXDG1_0000 DROPPING 46477 21880 /dev/rhdiskpower12 02-AUG-14
AJXDG1_0001 DROPPING 46477 21885 /dev/rhdiskpower13 02-AUG-14
AJXDG1_0002 DROPPING 46477 21883 /dev/rhdiskpower14 02-AUG-14
AJXDG1_0003 DROPPING 46477 21878 /dev/rhdiskpower15 02-AUG-14
AJXDG1_0004 DROPPING 46477 21883 /dev/rhdiskpower16 02-AUG-14
AJXDG1_0005 DROPPING 46477 21874 /dev/rhdiskpower17 02-AUG-14
AJXDG1_0006 DROPPING 46477 21886 /dev/rhdiskpower18 02-AUG-14
AJXDG1_0007 DROPPING 46477 21879 /dev/rhdiskpower19 02-AUG-14
AJXDG1_0008 DROPPING 46477 21879 /dev/rhdiskpower20 02-AUG-14
AJXDG1_0009 DROPPING 46477 21885 /dev/rhdiskpower21 02-AUG-14
AJXDG1_0010 DROPPING 46477 21887 /dev/rhdiskpower22 02-AUG-14
AJXDG1_0011 DROPPING 46477 21884 /dev/rhdiskpower23 02-AUG-14
AJXDG1_0012 DROPPING 46477 21881 /dev/rhdiskpower24 02-AUG-14
AJXDG1_0013 DROPPING 46477 21880 /dev/rhdiskpower25 02-AUG-14
AJXDG1_0014 DROPPING 46477 21885 /dev/rhdiskpower26 02-AUG-14
AJXDG1_0015 DROPPING 46477 21877 /dev/rhdiskpower27 02-AUG-14
AJXDG1_0017 NORMAL 46477 46475 /dev/rhdiskpower59 21-APR-16
AJXDG1_0018 NORMAL 46477 46475 /dev/rhdiskpower60 21-APR-16
AJXDG1_0019 NORMAL 46477 46475 /dev/rhdiskpower61 21-APR-16
AJXDG1_0020 NORMAL 46477 46475 /dev/rhdiskpower62 21-APR-16
AJXDG1_0021 NORMAL 46477 46475 /dev/rhdiskpower63 21-APR-16
AJXDG1_0022 NORMAL 46477 46475 /dev/rhdiskpower64 21-APR-16
AJXDG1_0023 NORMAL 46477 46475 /dev/rhdiskpower65 21-APR-16
AJXDG1_0024 NORMAL 46477 46475 /dev/rhdiskpower66 21-APR-16
AJXDG1_0025 NORMAL 46477 46475 /dev/rhdiskpower67 21-APR-16
AJXDG1_0026 NORMAL 46477 46475 /dev/rhdiskpower68 21-APR-16
AJXDG1_0027 NORMAL 46477 46475 /dev/rhdiskpower69 21-APR-16
AJXDG1_0028 NORMAL 46477 46475 /dev/rhdiskpower70 21-APR-16
AJXDG1_0029 NORMAL 46477 46475 /dev/rhdiskpower71 21-APR-16
AJXDG1_0030 NORMAL 46477 46475 /dev/rhdiskpower72 21-APR-16
AJXDG1_0031 NORMAL 46477 46475 /dev/rhdiskpower73 21-APR-16
AJXDG1_0016 NORMAL 46477 46475 /dev/rhdiskpower58 21-APR-16

---3.2 发起rebalance，查看rebalance进度
alter diskgroup ajxdg1 rebalance power 500;

select * from v$asm_operation;
GROUP_NUMBER OPERA STAT POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES ERROR_CODE
------------ ----- ---- ---------- ---------- ---------- ---------- ---------- ----------- --------------------------------------------
1 REBAL RUN 500 500 46040 393494 15207 22 <---est_rate=15207表示每分钟正在被move的AU个数，AU_SIZE=1M时表示每分钟15GB/分钟，折合250MB/s

***调高优先级，观察同步速度是否加快

alter diskgroup ajxdg1 rebalance power 999;

select * from v$asm_operation;

GROUP_NUMBER OPERA STAT POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES ERROR_CODE
------------ ----- ---- ---------- ---------- ---------- ---------- ---------- ----------- --------------------------------------------
1 REBAL RUN 999 999 13462 236076 15482 14 <---改变了优先级，相当于将原来优先级为500的rebalance先停止然后以新的优先级发起所以EST_WORK值比一开始时小，刨掉了已经完成同步的AU

select * from v$asm_operation;

GROUP_NUMBER OPERA STAT POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES ERROR_CODE
------------ ----- ---- ---------- ---------- ---------- ---------- ---------- ----------- --------------------------------------------
1 REBAL RUN 999 999 13822 236076 15469 14

看出调整为999之后速率并未明显上升，可能是因为IO请求已经出现了排队
powermt display every=1
Symmetrix logical device count=118
==============================================================================
----- Host Bus Adapters --------- ------ I/O Paths ----- ------ Stats ------
### HW Path Summary Total Dead IO/Sec Q-IOs Errors
==============================================================================
0 fscsi0 optimal 118 0 117 1 0 <---Q-IOs!=0
1 fscsi2 optimal 118 0 132 2 0
2 fscsi4 optimal 118 0 127 2 0
3 fscsi6 optimal 118 0 124 1 0

---3.3 rebalance完成后的结果检查
col path format a30
col header_status format a10
col mode_status format a20
set linesize 130 pagesize 130
select s.name,s.state,s.total_mb,s.free_mb,path,create_date from v$asm_disk_stat s,v$asm_diskgroup_stat d where s.group_number=d.group_number and d.name in ('AJXDG1');

NAME STATE TOTAL_MB FREE_MB PATH CREATE_DA
------------------------------ -------- ---------- ---------- ------------------------------ ---------
AJXDG1_0017 NORMAL 46477 21882 /dev/rhdiskpower59 21-APR-16
AJXDG1_0018 NORMAL 46477 21882 /dev/rhdiskpower60 21-APR-16
AJXDG1_0019 NORMAL 46477 21882 /dev/rhdiskpower61 21-APR-16
AJXDG1_0020 NORMAL 46477 21881 /dev/rhdiskpower62 21-APR-16
AJXDG1_0021 NORMAL 46477 21881 /dev/rhdiskpower63 21-APR-16
AJXDG1_0022 NORMAL 46477 21882 /dev/rhdiskpower64 21-APR-16
AJXDG1_0023 NORMAL 46477 21882 /dev/rhdiskpower65 21-APR-16
AJXDG1_0024 NORMAL 46477 21881 /dev/rhdiskpower66 21-APR-16
AJXDG1_0025 NORMAL 46477 21882 /dev/rhdiskpower67 21-APR-16
AJXDG1_0026 NORMAL 46477 21882 /dev/rhdiskpower68 21-APR-16
AJXDG1_0027 NORMAL 46477 21882 /dev/rhdiskpower69 21-APR-16
AJXDG1_0028 NORMAL 46477 21882 /dev/rhdiskpower70 21-APR-16
AJXDG1_0029 NORMAL 46477 21882 /dev/rhdiskpower71 21-APR-16
AJXDG1_0030 NORMAL 46477 21881 /dev/rhdiskpower72 21-APR-16
AJXDG1_0031 NORMAL 46477 21880 /dev/rhdiskpower73 21-APR-16
AJXDG1_0016 NORMAL 46477 21882 /dev/rhdiskpower58 21-APR-16

4、将hdiskpower74~hdiskpower89添加到AJXDG2
AJXDG1: 通过shell脚本获取SQL脚本
i=74
while [ $i -lt 90 ]
do
echo "alter diskgroup AJXDG2 add disk '/dev/rhdiskpower${i}' rebalance power 0;";
(( i=i+1 ))
done

得到如下添加新盘的命令直接运行：
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower74' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower75' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower76' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower77' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower78' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower79' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower80' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower81' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower82' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower83' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower84' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower85' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower86' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower87' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower88' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower89' rebalance power 0;

执行如下SQL：
select 'alter diskgroup '||d.name||' drop disk '||s.name ||' rebalance power 0;' from v$asm_disk_stat s,v$asm_diskgroup_stat d where s.group_number=d.group_number and d.name in ('AJXDG2');

得到drop old disk的command
alter diskgroup AJXDG2 drop disk AJXDG2_0000 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0001 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0002 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0003 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0004 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0005 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0006 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0007 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0008 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0009 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0010 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0011 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0012 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0013 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0014 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0015 rebalance power 0;

---4.1 观察rebalance前的磁盘状态
col path format a30
col header_status format a10
col mode_status format a20
set linesize 130 pagesize 130
select s.name,s.state,s.total_mb,s.free_mb,path,create_date from v$asm_disk_stat s,v$asm_diskgroup_stat d where s.group_number=d.group_number and d.name in ('AJXDG2');
NAME STATE TOTAL_MB FREE_MB PATH CREATE_DA
------------------------------ -------- ---------- ---------- ------------------------------ ---------
AJXDG2_0000 DROPPING 46477 20092 /dev/rhdiskpower28 02-AUG-14
AJXDG2_0001 DROPPING 46477 20091 /dev/rhdiskpower29 02-AUG-14
AJXDG2_0002 DROPPING 46477 20088 /dev/rhdiskpower30 02-AUG-14
AJXDG2_0003 DROPPING 46477 20090 /dev/rhdiskpower31 02-AUG-14
AJXDG2_0004 DROPPING 46477 20092 /dev/rhdiskpower32 02-AUG-14
AJXDG2_0005 DROPPING 46477 20097 /dev/rhdiskpower33 02-AUG-14
AJXDG2_0006 DROPPING 46477 20092 /dev/rhdiskpower34 02-AUG-14
AJXDG2_0007 DROPPING 46477 20089 /dev/rhdiskpower35 02-AUG-14
AJXDG2_0008 DROPPING 46477 20092 /dev/rhdiskpower36 02-AUG-14
AJXDG2_0009 DROPPING 46477 20089 /dev/rhdiskpower37 02-AUG-14
AJXDG2_0010 DROPPING 46477 20089 /dev/rhdiskpower38 02-AUG-14
AJXDG2_0011 DROPPING 46477 20089 /dev/rhdiskpower39 02-AUG-14
AJXDG2_0012 DROPPING 46477 20091 /dev/rhdiskpower40 02-AUG-14
AJXDG2_0013 DROPPING 46477 20094 /dev/rhdiskpower41 02-AUG-14
AJXDG2_0014 DROPPING 46477 20089 /dev/rhdiskpower42 02-AUG-14
AJXDG2_0015 DROPPING 46477 20087 /dev/rhdiskpower43 02-AUG-14
AJXDG2_0016 NORMAL 46477 46475 /dev/rhdiskpower74 22-APR-16
AJXDG2_0028 NORMAL 46477 46475 /dev/rhdiskpower86 22-APR-16
AJXDG2_0029 NORMAL 46477 46475 /dev/rhdiskpower87 22-APR-16
AJXDG2_0030 NORMAL 46477 46475 /dev/rhdiskpower88 22-APR-16
AJXDG2_0031 NORMAL 46477 46475 /dev/rhdiskpower89 22-APR-16
AJXDG2_0024 NORMAL 46477 46475 /dev/rhdiskpower82 22-APR-16
AJXDG2_0025 NORMAL 46477 46475 /dev/rhdiskpower83 22-APR-16
AJXDG2_0026 NORMAL 46477 46475 /dev/rhdiskpower84 22-APR-16
AJXDG2_0027 NORMAL 46477 46475 /dev/rhdiskpower85 22-APR-16
AJXDG2_0021 NORMAL 46477 46475 /dev/rhdiskpower79 22-APR-16
AJXDG2_0022 NORMAL 46477 46475 /dev/rhdiskpower80 22-APR-16
AJXDG2_0023 NORMAL 46477 46475 /dev/rhdiskpower81 22-APR-16
AJXDG2_0017 NORMAL 46477 46475 /dev/rhdiskpower75 22-APR-16
AJXDG2_0018 NORMAL 46477 46475 /dev/rhdiskpower76 22-APR-16
AJXDG2_0019 NORMAL 46477 46475 /dev/rhdiskpower77 22-APR-16
AJXDG2_0020 NORMAL 46477 46475 /dev/rhdiskpower78 22-APR-16

---4.2 发起rebalance，查看rebalance进度
alter diskgroup ajxdg2 rebalance power 1011;

SQL> select * from v$asm_operation

GROUP_NUMBER OPERA STAT POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES ERROR_CODE
------------ ----- ---- ---------- ---------- ---------- ---------- ---------- ----------- --------------------------------------------
2 REBAL RUN 1011 1011 7834 422149 14768 28

---4.3 rebalance完成后的结果检查
col path format a30
col header_status format a10
col mode_status format a20
set linesize 130 pagesize 130
select s.name,s.state,s.total_mb,s.free_mb,path,create_date from v$asm_disk_stat s,v$asm_diskgroup_stat d where s.group_number=d.group_number and d.name in ('AJXDG2') order by s.name;
NAME STATE TOTAL_MB FREE_MB PATH CREATE_DA
------------------------------ -------- ---------- ---------- ------------------------------ ---------
AJXDG2_0016 NORMAL 46477 20090 /dev/rhdiskpower74 22-APR-16
AJXDG2_0017 NORMAL 46477 20091 /dev/rhdiskpower75 22-APR-16
AJXDG2_0018 NORMAL 46477 20091 /dev/rhdiskpower76 22-APR-16
AJXDG2_0019 NORMAL 46477 20091 /dev/rhdiskpower77 22-APR-16
AJXDG2_0020 NORMAL 46477 20091 /dev/rhdiskpower78 22-APR-16
AJXDG2_0021 NORMAL 46477 20090 /dev/rhdiskpower79 22-APR-16
AJXDG2_0022 NORMAL 46477 20091 /dev/rhdiskpower80 22-APR-16
AJXDG2_0023 NORMAL 46477 20091 /dev/rhdiskpower81 22-APR-16
AJXDG2_0024 NORMAL 46477 20090 /dev/rhdiskpower82 22-APR-16
AJXDG2_0025 NORMAL 46477 20091 /dev/rhdiskpower83 22-APR-16
AJXDG2_0026 NORMAL 46477 20091 /dev/rhdiskpower84 22-APR-16
AJXDG2_0027 NORMAL 46477 20091 /dev/rhdiskpower85 22-APR-16
AJXDG2_0028 NORMAL 46477 20090 /dev/rhdiskpower86 22-APR-16
AJXDG2_0029 NORMAL 46477 20090 /dev/rhdiskpower87 22-APR-16
AJXDG2_0030 NORMAL 46477 20091 /dev/rhdiskpower88 22-APR-16
AJXDG2_0031 NORMAL 46477 20091 /dev/rhdiskpower89 22-APR-16

5、(可选)，将原先的Votedisk从external redundancy的DG迁移到normal redundancy的DG
grid@qc740702a:/home/grid>crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE a48d1185b7be4f17bf06f6dd01e84d88 (/dev/rhdiskpower106) [SYSDG2] <---原来voting disk只有一块盘
Located 1 voting disk(s).

***创建normal redundancy的diskgroup
create diskgroup sysdg3 normal redundancy failgroup vdfg1 disk '/dev/rhdiskpower110','/dev/rhdiskpower111' failgroup vdfg2 disk '/dev/rhdiskpower112','/dev/rhdiskpower113' failgroup vdfg3 disk '/dev/rhdiskpower114','/dev/rhdiskpower115' attribute 'compatible.asm'='11.2.0.3','compatible.rdbms'='11.2.0.0.0' ;

***迁移到sysdg3
grid@qc740702a:/home/grid>crsctl replace votedisk +sysdg3
Successful addition of voting disk 72d81026f64d4f86bf7a7ec0845201db.
Successful addition of voting disk 2fa55993ec494fa8bf839661e2c43e89.
Successful addition of voting disk 1127e47a60084f8ebf6f2292143710c0.
Successful deletion of voting disk a48d1185b7be4f17bf06f6dd01e84d88.
Successfully replaced voting disk group with +sysdg3.
CRS-4266: Voting file(s) successfully replaced

grid@qc740702a:/home/grid>crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 72d81026f64d4f86bf7a7ec0845201db (/dev/rhdiskpower110) [SYSDG3] <---现在voting disk有三块了
2. ONLINE 2fa55993ec494fa8bf839661e2c43e89 (/dev/rhdiskpower112) [SYSDG3]
3. ONLINE 1127e47a60084f8ebf6f2292143710c0 (/dev/rhdiskpower114) [SYSDG3]
Located 3 voting disk(s).

正文到此结束