已经实施过2次Hadoop大数据平台的基础环境搭建,对于硬盘的RAID模式都不约而同选择了JBOD,毕竟大数据对于文件系统的管理本身就是基于分布式,由至少3份副本来保障数据可靠性,只是不做RAID后的SAS硬盘读写性能就有点惨不忍睹。这次Hadoop大数据平台工具选择的是Cloudera,大家感兴趣的化也可以浏览下它的官网。本篇文章主要介绍如何实现在不停机状态下在线更换JBOD硬盘。
利用StorCLI命令不停机在线更换JBOD硬盘
2016年12月31日 - 初稿
阅读原文 - https://wsgzao.github.io/post/jbod/
Cloudera - http://www.cloudera.com/
JBOD ( Just a Bunch Of Disks )不是标准的 RAID 等级,它通常用来表示一个没有控制软件提供协调控制的磁盘集合。 JBOD 将多个物理磁盘串联起来,提供一个巨大的逻辑磁盘。 JBOD 的数据存放机制是由第一块磁盘开始按顺序往后存储,当前磁盘存储空间用完后,再依次往后面的磁盘存储数据。 JBOD 存储性能完全等同于单块磁盘,而且也不提供数据安全保护。它只是简单提供一种扩展存储空间的机制, JBOD 可用存储容量等于所有成员磁盘的存储空间之和。目前 JBOD 常指磁盘柜,而不论其是否提供 RAID 功能。
RAID磁盘阵列配置和调优小结 - https://wsgzao.github.io/post/raid/
StorCLI_RefMan_revf
https://www.broadcom.com/support/download-search/?dk=StorCLIStorCLI (Command Line) Utility for Storage Management v1.07.07 on Linux
https://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5092950#查询硬件RAID卡信息 [root@localhost Linux]# cat /proc/scsi/scsi |more Attached devices: Host: scsi0 Channel: 02 Id: 00 Lun: 00 Vendor: IBM Model: ServeRAID M5210 Rev: 4.62 Type: Direct-Access ANSI SCSI revision: 05 #解压安装storcli unzip ibm_utl_sraidmr_storcli-1.07.07_linux_32-64.zip rpm -Uvh storcli-1.07.07-1.noarch.rpm #设置软连接 ln -s /opt/MegaRAID/storcli/storcli64 /bin/storcli ln -s /opt/MegaRAID/storcli/storcli64 /sbin/storcli #Raid数量及当前状态 [root@localhost Linux]# storcli show ctrlcount Status Code = 0 Status = Success Description = None Controller Count = 2 #查询虚拟磁盘大小,raid结构等 [root@localhost Linux]# storcli /c0 /v0 show Controller = 0 Status = Success Description = None Virtual Drives : ============== ----------------------------------------------------------- DG/VD TYPE State Access Consist Cache sCC Size Name ----------------------------------------------------------- 0/0 RAID1 Optl RW Yes RWTD - 278.465 GB ----------------------------------------------------------- Cac=CacheCade|Rec=Recovery|OfLn=OffLine|Pdgd=Partially Degraded|dgrd=Degraded Optl=Optimal|RO=Read Only|RW=Read Write|B=Blocked|Consist=Consistent| R=Read Ahead Always|NR=No Read Ahead|WB=WriteBack| AWB=Always WriteBack|WT=WriteThrough|C=Cached IO|D=Direct IO|sCC=Scheduled Check Consistency #可以看出此为RAID1,一个虚拟磁盘,大小为278.465 GB #查看control0的物理硬盘 [root@localhost Linux]# storcli /c0 /eall /sall show Controller = 0 Status = Success Description = Show Drive Information Succeeded. Drive Information : ================= ------------------------------------------------------------------------- EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp ------------------------------------------------------------------------- 252:16 8 Onln 0 278.465 GB SAS HDD N Y 512B ST300MP0035 X U 252:17 9 Onln 0 278.465 GB SAS HDD N Y 512B ST300MP0035 X U ------------------------------------------------------------------------- EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup DHS-Dedicated Hot Spare|UGood-Unconfigured Good|GHS-Global Hotspare UBad-Unconfigured Bad|Onln-Online|Offln-Offline|Intf-Interface Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info SeSz-Sector Size|Sp-Spun|U-Up|D-Down|T-Transition|F-Foreign UGUnsp-Unsupported|UGShld-UnConfigured shielded|HSPShld-Hotspare shielded CFShld-Configured shielded #可以看到总计有两颗硬盘,每一颗大小278.465GB,是否在线等 #查看control1的物理硬盘 [root@localhost Linux]# storcli /c1 /eall /sall show Controller = 1 Status = Success Description = Show Drive Information Succeeded. Drive Information : ================= ------------------------------------------------------------------------ EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp ------------------------------------------------------------------------ 8:0 23 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U 8:1 24 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U 8:2 22 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U 8:3 19 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U 8:4 13 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U 8:5 14 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U 8:6 18 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U 8:7 12 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U 8:8 10 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U 8:9 16 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U 8:10 11 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U 8:11 21 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U 8:12 15 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U 8:13 9 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U ------------------------------------------------------------------------ EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup DHS-Dedicated Hot Spare|UGood-Unconfigured Good|GHS-Global Hotspare UBad-Unconfigured Bad|Onln-Online|Offln-Offline|Intf-Interface Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info SeSz-Sector Size|Sp-Spun|U-Up|D-Down|T-Transition|F-Foreign UGUnsp-Unsupported|UGShld-UnConfigured shielded|HSPShld-Hotspare shielded CFShld-Configured shielded #可以看到总计有两颗硬盘,每一颗大小1.0810 TB,是否在线等 #设置硬盘状态 storcli /c1/e8/s0 show all storcli /c1/e8/s0 start locate storcli /c1/e8/s0 set offline storcli /c1/e8/s0 set online storcli /c1/e8/s0 set jbod
#查看StorCLI帮助信息 [root@localhost Linux]# storcli -h Storage Command Line Tool Ver 1.07.07 Nov 15, 2013 (c)Copyright 2013, LSI Corporation, All Rights Reserved. storcli -v storcli -h| -help| ? storcli -h| -help| ? legacy storcli show storcli show all storcli show ctrlcount storcli /cx add vd r[0|1|5|6|00|10|50|60] [Size=<VD1_Sz>,<VD2_Sz>,..|all] [name=<VDNAME1>,..] drives=e:s|e:s-x|e:s-x,y,e:s-x,y,z [PDperArray=x][SED] [pdcache=on|off|default][pi][DimmerSwitch(ds)=default|automatic(auto)| none|maximum(max)|MaximumWithoutCaching(maxnocache)] [wt|wb|awb][nora|ra] [direct|cached] [cachevd] [Strip=<8|16|32|64|128|256|512|1024>] [AfterVd=X] [Spares = [e:]s|[e:]s-x|[e:]s-x,y] [force][ExclusiveAccess] storcli /cx add vd each r0 [name=<VDNAME1>,..] [drives=e:s|e:s-x|e:s-x,y] [SED] [pdcache=on|off|default][pi] [DimmerSwitch(ds)=default|automatic(auto)| none|maximum(max)|MaximumWithoutCaching(maxnocache)] [wt|wb|awb] [nora|ra] [direct|cached] [Strip=<8|16|32|64|128|256|512|1024>] [ExclusiveAccess] storcli /cx add VD cachecade|nytrocache r[aid][0,1,10, 1EC] drives = [e:]s|[e:]s-x|[e:]s-x,y [WT| WB] [assignvds = 0,1,2] [BOOTVOLSIZE=x] storcli /cx/ex show all storcli /cx/ex show status storcli /cx/vx del [cachecade] [force] storcli /cx delete events storcli /cx show events [[type= <sincereboot| sinceshutdown| includedeleted| latest=x| ccincon vd=<0,1,...>] filter=<info|warning|critical|fatal>] file=<filepath> storcli /cx show eventloginfo storcli /cx show health [all] storcli /cx delete securitykey storcli /cx set securitykey=xxxxxxxx {passphrase=xxxx} {keyid=xxx} storcli /cx start Diag Duration=<Val> storcli /cx set securitykey keyid=xxx storcli /cx compare securitykey=xxxxxxxxxx storcli /cx set termlog[=on|off|offthisboot] storcli /cx show termlog [type=config|contents] storcli /cx delete termlog storcli /cx Compare factorysettings file=<filepath> storcli /cx Compare factorysettings raw file=<filepath> storcli /cx set securitykey=xxxxxxxx oldsecuritykey=xxxxxxxx {passphrase=xxxx} {keyid=xxx} storcli /cx/dx show storcli /cx/dall show cachecade storcli /cx/dx show all storcli /cx show freespace storcli /cx/fall show [all] [securityKey = xxx] storcli /cx/fall del|delete [securityKey = xxx] storcli /cx/fall import [preview] [securityKey = xxx] storcli /cx/vx set ssdcaching=on|off storcli /cx/dx set security=on storcli /cx/vx show expansion storcli /cx show fshinting storcli /cx/vx set fshinting=<value> storcli /cx/vx expand Size=<xx> [expandarray] storcli /cx[/ex]/sx show storcli /cx[/ex]/sx show all storcli /cx[/ex]/sx start rebuild storcli /cx[/ex]/sx stop rebuild storcli /cx[/ex]/sx pause rebuild storcli /cx[/ex]/sx resume rebuild storcli /cx[/ex]/sx show rebuild storcli /cx[/ex]/sx show health storcli /cx[/ex]/sx start copyback target=e:s storcli /cx[/ex]/sx stop copyback storcli /cx[/ex]/sx pause copyback storcli /cx[/ex]/sx resume copyback storcli /cx[/ex]/sx show copyback storcli /cx[/ex]/sx start initialization storcli /cx[/ex]/sx stop initialization storcli /cx[/ex]/sx show initialization storcli /cx[/ex]/sx start locate storcli /cx[/ex]/sx stop locate storcli /cx[/ex]/sx show securitykey keyid storcli /cx[/ex]/sx add hotsparedrive [DGs=<N|0,1,2...>] [enclaffinity] [nonrevertible] storcli /cx[/ex]/sx delete hotsparedrive storcli /cx[/ex]/sx spinup storcli /cx[/ex]/sx spindown storcli /cx[/ex]/sx set online storcli /cx[/ex]/sx set offline storcli /cx[/ex]/sx set missing storcli /cx[/ex]/sx set jbod storcli /cx[/ex]/sx set good [force] storcli /cx[/ex]/sx insert dg=A array=B row=C storcli /cx/vx set emulationType=0|1 storcli /cx/vx set wrcache=WT|WB|AWB storcli /cx/vx set rdcache=RA|NoRA storcli /cx/vx set iopolicy=Cached|Direct storcli /cx/vx set accesspolicy=RW|RO|Blocked|RmvBlkd storcli /cx/vx set pdcache=On|Off|Default storcli /cx/vx set name=<NameString> storcli /cx/vx set HostAccess=ExclusiveAccess|SharedAccess storcli /cx/vx set ds=Default|Auto|None|Max|MaxNoCache storcli /cx/vx set autobgi=On|Off storcli /cx/vx set pi=Off storcli /cx/vx show storcli /cx/vx show all storcli /cx/vx show init storcli /cx/vx show cc storcli /cx/vx show erase storcli /cx/vx show migrate storcli /cx/vx show bgi storcli /cx/vx show autobgi storcli /cx/vx show trim storcli /cx set consistencycheck|cc=[off|seq|conc] [delay=value] starttime=yyyy/mm/dd hh] [excludevd=x-y,z] storcli /cx show cc|consistencycheck storcli /cx show ocr storcli /cx set ocr=<on|off> storcli /cx/vx start init[Full][Force] storcli /cx/vx start erase [simple|normal|thorough|standard][patternA=<val>] [patternB=<val>] storcli /cx/vx start cc[Force] storcli /cx/vx start migrate type=raidx [option=add|remove drives=[e:]s|[e:]s-x|[e:]s-x,y] [Force] storcli /cx/vx stop init storcli /cx/vx stop erase storcli /cx/vx stop cc storcli /cx/vx stop bgi storcli /cx/vx pause cc storcli /cx/vx pause bgi storcli /cx/vx resume cc storcli /cx/vx resume bgi storcli /cx show storcli /cx show all storcli /cx show preservedcache storcli /cx/vx delete preservedcache[force] storcli /cx[/ex]/sx download src=<filepath> [satabridge] [mode= 5|7] storcli /cx/ex download src=<filepath> [forceActivate] storcli /cx[/ex]/sx secureerase [force] storcli /cx[/ex]/sx start erase [simple| normal| thorough | standard| threepass] [patternA=<val>] [patternB=<val>] storcli /cx[/ex]/sx stop erase storcli /cx[/ex]/sx show erase storcli /cx[/ex]/sx show rawdata pageaddr=<pageaddress in hex> file=<filename> storcli /cx[/ex]/sx set bootdrive=<on|off> storcli /cx/vx set bootdrive=<on|off> storcli /cx show bootdrive storcli /cx show bootwithpinnedcache storcli /cx set bootwithpinnedcache=<on|off> storcli /cx show activityforlocate storcli /cx set activityforlocate=<on|off> storcli /cx show copyback storcli /cx set copyback=<on|off> type=smartssd|smarthdd|all storcli /cx show jbod storcli /cx set jbod=<on|off> storcli /cx set autorebuild=<on|off> storcli /cx show autorebuild storcli /cx show cachebypass storcli /cx set cachebypass=<on|off> storcli /cx show usefdeonlyencrypt storcli /cx set usefdeonlyencrypt=<on|off> storcli /cx show prcorrectunconfiguredareas storcli /cx set prcorrectunconfiguredareas=<on|off> storcli /cx show batterywarning storcli /cx set batterywarning=<on|off> storcli /cx show abortcconerror storcli /cx set abortcconerror=<on|off> storcli /cx show ncq storcli /cx set ncq=<on|off> storcli /cx show maintainpdfailhistory storcli /cx set maintainpdfailhistory=<on|off> storcli /cx show restorehotspare storcli /cx set restorehotspare=<on|off> storcli /cx set bios [state=<on|off>] [Mode=<SOE|BE|HCOE|HSM>] [abs=<on|off>] storcli /cx show bios storcli /cx show alarm storcli /cx set alarm=<on|off|silence> storcli /cx show foreignautoimport storcli /cx set foreignautoimport=<on|off> storcli /cx show directpdmapping storcli /cx set directpdmapping=<on|off> storcli /cx show rebuildrate storcli /cx set rebuildrate=<value> storcli /cx show loadbalancemode storcli /cx set loadbalancemode=<on|off> storcli /cx show eghs storcli /cx set eghs [state=<on|off>] [eug=<on|off>] [smarter=<on|off>] storcli /cx show cacheflushint storcli /cx set cacheflushint=<value> storcli /cx show prrate storcli /cx set prrate=<value> storcli /cx show ccrate storcli /cx set ccrate=<value> storcli /cx show bgirate storcli /cx set bgirate =<value> storcli /cx show dpm storcli /cx set dpm =<on|off> storcli /cx show sgpioforce storcli /cx set sgpioforce =<on|off> storcli /cx set supportssdpatrolread =<on|off> storcli /cx show migraterate storcli /cx set migraterate=<value> storcli /cx show spinupdrivecount storcli /cx show wbsupport storcli /cx set spinupdrivecount=<value> storcli /cx show spinupdelay storcli /cx set spinupdelay=<value> storcli /cx show coercion storcli /cx set coercion=<value> storcli /cx show smartpollinterval storcli /cx set smartpollinterval=<value> storcli /cx show eccbucketsize storcli /cx set eccbucketsize=<value> storcli /cx show eccbucketleakrate storcli /cx set eccbucketleakrate=<value> storcli /cx show backplane storcli /cx set backplane mode=<value> expose=<on/off> storcli /cx show perfmode storcli /cx set perfmode=<value> [maxflushlines=<value> numiostoorder=<value>] storcli /cx show perfmodevalues storcli /cx show pi storcli /cx set pi [state=<on|off>] [import=<on|off>] storcli /cx show time storcli /cx set time=<yyyymmdd hh:mm:ss | systemtime> storcli /cx show ds storcli /cx set ds=OFF type=1|2|3|4 storcli /cx set ds=ON type=1|2 [properties] storcli /cx set ds=ON type=3|4 DefaultLdType=<val> [properties] storcli /cx set ds [properties] storcli /cx show safeid storcli /cx show rehostinfo storcli /cx show pci storcli /cx show ASO storcli /cx set aso key=<key value> preview storcli /cx set aso key=<key value> storcli /cx set aso transfertovault storcli /cx set aso rehostcomplete storcli /cx set aso deactivatetrialkey storcli /cx set factory defaults storcli /cx download file=<filepath> [fwtype=<val>] [ResetNow] [nosigchk] [noverchk] storcli /cx flush|flushcache storcli /cx/px show storcli /cx/px show all storcli /cx/px set linkspeed=0|1.5|3|6|12 storcli /cx/bbu show storcli /cx/bbu set SCAPVPD file=<input file path> VPDPage=<SCapVPDFixed> storcli /cx/bbu show all storcli /cx/bbu show status storcli /cx/bbu show properties storcli /cx/bbu show learn storcli /cx/bbu show gasgauge Offset=xxxx Numbytes=n storcli /cx/bbu start learn storcli /cx/bbu show modes storcli /cx/bbu set [ learnDelayInterval=<val> | bbuMode=<val> |learnStartTime=[DDD HH | off] | autolearnmode=<val> | powermode=sleep | writeaccess=sealed ] storcli /cx/cv show storcli /cx/cv show all storcli /cx/cv show status storcli /cx/cv show learn storcli /cx/cv start learn storcli /cx show securitykey keyid storcli /cx start patrolread storcli /cx stop patrolread storcli /cx pause patrolread storcli /cx resume patrolread storcli /cx show patrolRead storcli /cx show powermonitoringinfo storcli /cx set patrolread = {{on mode=<auto|manual> }|{off}} storcli /cx set patrolread [starttime=< yyyy/mm/dd hh>] [maxconcurrentpd =<value>] [includessds=<on|onlymixed|off>] [uncfgareas=on|off] storcli /cx set patrolread delay = <value> storcli /cx[/ex]/sx show diag paniclog [Query] | [ExtractSlot=x] | [EraseSlot=x] [file=filepath] storcli /cx[/ex]/sx show diag smartlog [file=filepath] storcli /cx[/ex]/sx show diag errorlog [file=filepath] storcli /cx del Nytrocache [force] storcli /cx show badblocks storcli /cx flasherase storcli /cx shutdown storcli /cx/mx set mode=<Internal | External | Auto> storcli /cx/mx show storcli /cx transform iMR storcli /cx restart storcli /cx/vx show BBMT storcli /cx/vx delete BBMT storcli /xd add type=CacheDev <devices=comma separated cache devices | ALL> [force] storcli /xd add type=VirtualDrive <devices=comma separated Virtual drives | ALL> storcli /xd delete type=<CacheDev | VirtualDrive> <devices=comma separated cache devices or Virtual drives as applicable | ALL> storcli /xd show type=<CacheDev | VirtualDrive> [state=<assigned|unassigned>] storcli /xd show storcli /xd show perfmon storcli /xd/wd<X|ALL> show storcli /xd/wd<X|ALL> show safeid storcli /xd set online [Vds=comma separated VD ids] storcli /xd/wdX apply key=value storcli /cx[/ex]/sx start format [thorough] storcli /cx show dequeuelog file=<filepath> Note:Use 'page=[x]'as the last option in all the commands to set the page break. X=lines per page. E.g. 'storcli help page=10'