NetApp存储方案及巡检命令
一、MCC概述
Clustered Metro Cluster(简称MCC)是Netapp Data Ontap提供的存储双活解决方案，当初的方案是把1个FAS/ V系列双控在数据中心之间拉远形成异地HA Pair，每站点只有单控制器节点，数据中心两站点之间通过额外的FC/VI集群适配器相连，数据中心间SAS磁盘框通过SAS转FC的FibreBridge相连。在500米以内、同一个机房采用直接光纤通道交换机连接；在500米以上(最远100km)采用光纤通道和DWDM交换机相连。
MetroCluster在此架构上也进行了演变。通过在站点A、B两个站点分别放置两套FAS/ V双控阵列，阵列A的A控和阵列B的A控，阵列A的B控和阵列B的B控分别形成集群，这样可以充分把A、B站点数据中心资源充分利用，同时对外提供存储服务；但阵列内的A、B不是集群。如果站点间形成集群Pair的任意一个控制器节点故障，故障站点的主机都需要远程访问远端控制器节点；如何站点间形成集群Pair的两个节点同时故障，就会发生业务中断。
Netapp Data Ontap8.3版本推出了4控双活解决方案，最远支持200公里距离，4控Metro Cluster方案首先由2个HA Pair组成2个本地集群，然后再从2个集群上做4节点集群。集群控制器之间内存日志通过存放在NVRAM里面，NVRAM对没有下盘的日志做了镜像，保证节点故障以后，HA Pair集群的Partner节点能够接管业务；或者站点故障以后，远端HA Pair集群能够接管业务。当日志到达一定水位或者发生系统操作刷盘时，下盘数据同步通过SyncMirror实现主从站点双写，从而确保一个站点磁盘故障以后，另外一个站点磁盘还能提供系统访问，实现站点故障切换，保证业务不中断。
MetroCluster使用两个不同地点的镜像和集群来保护数据，每个集群把数据和Storage Virtual Machine (SVM) 配置都镜像同步另一个集群。当某个站点发生灾难时，管理员可以激活远端SVM并在另一站点接管业务。此外，每个集群在本地节点均配置为HA Pair，从而提供了本地故障转移能力。
NetApp MetroCluster是以NetApp SyncMirror是配合Cluster_remote和控制器Cluster Failover的功能实现的。
-
Clustered Failover – 在主存储和容灾存储间提供高可用性失败恢复能力，故障接管的决策是由管理员通过单一命令行决定的。
-
SyncMirror – 为远端存储提供即时的数据拷贝，当故障接管时，数据可以仅通过远端的存储进行访问。
-
ClusterRemote – 提供管理机制用以判断灾难的发生并初始远端存储进行接管。
二、MCC巡检常用命令
1、系统健康状态检查
- cluster1::> system health status show
- Status
- ---------------
- ok
2、集群状态检查
- cluster1::> cluster show
- Node Health Eligibility
- --------------------- ------- ------------
- cluster1-01 true true
- cluster1-02 true true
- 2 entries were displayed.
3、集群统计状态检查
- cluster1::> cluster statistics show
- Counter Value Delta
- ---------------- ----------------- -------------
- CPU Busy: 0% -
- Operations:
- Total: 0 -
- NFS: 0 -
- CIFS: 0 -
- Data Network:
- Busy: 0% -
- Received: 5.78GB -
- Sent: 13.7GB -
- Cluster Network:
- Busy: 0% -
- Received: 967KB -
- Sent: 979KB -
- Storage Disk:
- Read: 6.38PB -
- Write: 6.26PB -
4、查看RAID组信息
- cluster1::> aggr show
- Aggregate Size Available Used% State #Vols Nodes RAID Status
- --------- -------- --------- ----- ------- ------ ---------------- ------------
- aggr0_A1 953.8GB 247.3GB 74% online 1 cluster1-01 raid4,
- mirrored,
- normal
- aggr0_A2 953.8GB 247.3GB 74% online 1 cluster1-02 raid4,
- mirrored,
- normal
- aggr_data_A1
- 68.93TB 16.04TB 77% online 32 cluster1-01 mixed_raid_
- type,
- mirrored,
- hybrid,
- normal
- aggr_data_A2
- 68.93TB 14.77TB 79% online 31 cluster1-02 mixed_raid_
- type,
- mirrored,
- hybrid,
- normal
- 4 entries were displayed.
5、查看节点信息
- cluster1::> node show
- Node Health Eligibility Uptime Model Owner Location
- --------- ------ ----------- ------------- ----------- -------- ---------------
- cluster1-01
- true true
- 369 days 19:12 FAS8040 gz_idc
- cluster1-02
- true true
- 369 days 19:23 FAS8040 gz_idc
- 2 entries were displayed.
6、查看版本信息
- cluster1::> version
- NetApp Release 8.3.2P9: Fri Jan 06 05:54:05 UTC 2017
7、查看序列号
- cluster1::> system license show
- Serial Number: 1-80-023992
- Owner: cluster1
- Package Type Description Expiration
- ----------------- ------- --------------------- --------------------
- Base license Cluster Base License -
- Serial Number: 1-81-0000000000000451515******
- Package Type Description Expiration
- ----------------- ------- --------------------- --------------------
- NFS license NFS License -
- iSCSI license iSCSI License -
- Serial Number: 1-81-0000000000000451515******
- Owner: cluster1-02
- Package Type Description Expiration
- ----------------- ------- --------------------- --------------------
- NFS license NFS License -
- iSCSI license iSCSI License -
- 5 entries were displayed.
8、查看子系统健康状态
- cluster1::> system health subsystem show
- Subsystem Health
- ----------------- ------------------
- SAS-connect ok
- Environment ok
- Memory ok
- Service-Processor ok
- Switch-Health ok
- CIFS-NDO ok
- Motherboard ok
- IO ok
- MetroCluster ok
- MetroCluster_Node ok
- FHM-Switch ok
- FHM-Bridge ok
- 12 entries were displayed.
9、查看MCC集群信息状态及节点信息状态
- cluster1::> metrocluster show
- Configuration: fabric
- Cluster Configuration State Mode
- ------------------------------ ---------------------- ------------------------
- Local: cluster1 configured normal
- Remote: cluster1_dr configured normal
- cluster1::> metrocluster node show
- DR Configuration DR
- Group Cluster Node State Mirroring Mode
- ----- ------- ------------------ -------------- --------- --------------------
- 1 cluster1
- cluster1-01 configured enabled normal
- cluster1-02 configured enabled normal
- cluster1_dr
- cluster1_dr-01 configured enabled normal
- cluster1_dr-02 configured enabled normal
- 4 entries were displayed.
10、查看控制器状态
- cluster1::> system controller show
- Controller Name System ID Serial Number Model Status
- ------------------------- ------------- ----------------- -------- -----------
- cluster1-01 536964819 451515****** FAS8040 ok
- cluster1-02 536961600 451515****** FAS8040 ok
- 2 entries were displayed.
11、查看故障硬盘
- cluster1::> storage disk show -broken
- There are no entries matching your query.
12、查看spare硬盘
- cluster1::> storage disk show -spare
- Original Owner: cluster1-01
- Checksum Compatibility: block
- Usable Physical
- Disk HA Shelf Bay Chan Pool Type RPM Size Size Owner
- --------------- ------------ ---- ------ ----- ------ -------- -------- --------
- 1.30.11 3a 30 11 A Pool0 SAS 10000 1.09TB 1.09TB cluster1-01
- 1.30.13 3a 30 13 A Pool0 SAS 10000 1.09TB 1.09TB cluster1-01
- 1.31.4 3a 31 4 A Pool0 SAS 10000 1.09TB 1.09TB cluster1-01
- 1.32.20 4b 32 20 B Pool0 SAS 10000 1.09TB 1.09TB cluster1-01
- 1.32.23 3a 32 23 A Pool0 SAS 10000 1.09TB 1.09TB cluster1-01
- 1.33.0 3a 33 0 A Pool0 SAS 10000 1.09TB 1.09TB cluster1-01
- 1.33.1 3a 33 1 A Pool0 SAS 10000 1.09TB 1.09TB cluster1-01
- 1.33.10 4b 33 10 B Pool0 SAS 10000 1.09TB 1.09TB cluster1-01
- 2.42.22 3a 42 22 A Pool1 SAS 10000 1.09TB 1.09TB cluster1-01
- 2.42.23 4b 42 23 B Pool1 SAS 10000 1.09TB 1.09TB cluster1-01
- 2.43.2 4b 43 2 B Pool1 SAS 10000 1.09TB 1.09TB cluster1-01
- 2.43.22 3b 43 22 A Pool1 SAS 10000 1.09TB 1.09TB cluster1-01
- 2.43.23 4b 43 23 B Pool1 SAS 10000 1.09TB 1.09TB cluster1-01
- 3.11.21 4b 11 21 B Pool0 SSD - 372.4GB 372.6GB cluster1-01
- 4.20.21 3a 20 21 A Pool1 SSD - 372.4GB 372.6GB cluster1-01
- 4.21.14 3a 21 14 A Pool1 SAS 10000 1.09TB 1.09TB cluster1-01
- Original Owner: cluster1-02
- Checksum Compatibility: block
- Usable Physical
- Disk HA Shelf Bay Chan Pool Type RPM Size Size Owner
- --------------- ------------ ---- ------ ----- ------ -------- -------- --------
- 2.44.23 3b 44 23 A Pool1 SAS 10000 1.09TB 1.09TB cluster1-02
- 3.12.21 4a 12 21 B Pool0 SSD - 372.4GB 372.6GB cluster1-02
- 4.23.21 3b 23 21 A Pool1 SSD - 372.4GB 372.6GB cluster1-02
- 5.60.23 3b 60 23 B Pool1 SAS 10000 1.09TB 1.09TB cluster1-02
- 20 entries were displayed.
13、查看SAS桥故障
- cluster1::> storage bridge show
- Is Monitor
- Bridge Symbolic Name Monitored Status Vendor Model Bridge WWN
- ------------------------ ------------- --------- ------- ------ --------------------- ----------------
- ATTO_10.0.15.17 BRIDGE_B_1
- true ok Atto FibreBridge 6500N 2000001086627bc0
- ATTO_10.0.15.18 BRIDGE_B_2
- true ok Atto FibreBridge 6500N 2000001086630f0e
- ATTO_10.0.15.19 BRIDGE_B_3
- true ok Atto FibreBridge 6500N 2000001086630edc
- ATTO_10.0.15.20 BRIDGE_B_4
- true ok Atto FibreBridge 6500N 2000001086630ed2
- ATTO_10.0.15.6 BRIDGE_A_1
- true ok Atto FibreBridge 6500N 2000001086630eb4
- ATTO_10.0.15.7 BRIDGE_A_2
- true ok Atto FibreBridge 6500N 2000001086630efa
- ATTO_10.0.15.8 BRIDGE_A_3
- true ok Atto FibreBridge 6500N 2000001086630f18
- ATTO_10.0.15.9 BRIDGE_A_4
- true ok Atto FibreBridge 6500N 2000001086630ef0
- ATTO_FibreBridge6500N_10 -
- false - Atto FibreBridge6500N 200000108663e514
- ATTO_FibreBridge6500N_11 -
- false - Atto FibreBridge6500N 200000108663e3f2
- ATTO_FibreBridge6500N_12 -
- false - Atto FibreBridge6500N 200000108663e488
- ATTO_FibreBridge6500N_13 -
- false - Atto FibreBridge6500N 20000010866114ec
- ATTO_FibreBridge6500N_14 -
- false - Atto FibreBridge6500N 2000001086627bc0
- ATTO_FibreBridge6500N_7 -
- false - Atto FibreBridge6500N 2000001086630e96
- ATTO_FibreBridge6500N_9 -
- false - Atto FibreBridge6500N 200000108663e4c4
- 15 entries were displayed.
14、查看纤交换机故障
- cluster1::> storage switch show
- Symbolic Is Monitor
- Switch Name Vendor Model Switch WWN Monitored Status
- --------------------- -------- ------- ----- ---------------- --------- -------
- Brocade_10.0.15.10
- SW_A_1
- Brocade Brocade6505
- 100050eb1a88327f true ok
- Brocade_10.0.15.11
- SW_A_2
- Brocade Brocade6505
- 100050eb1a881582 true ok
- Brocade_10.0.15.21
- SW_B_3
- Brocade Brocade6505
- 100050eb1a882f69 true ok
- Brocade_10.0.15.22
- SW_B_4
- Brocade Brocade6505
- 100050eb1a881522 true ok
- 4 entries were displayed.
15、查看failover状态
- cluster1::> storage failover show
- Takeover
- Node Partner Possible State Description
- -------------- -------------- -------- -------------------------------------
- cluster1-01 cluster1-02 true Connected to cluster1-02
- cluster1-02 cluster1-01 true Connected to cluster1-01
- 2 entries were displayed.
16、查看严重告警日志及错误告警日志
- cluster1::> event log show -severity critical
- There are no entries matching your query.
- cluster1::> event log show -severity error
- Time Node Severity Event
- ------------------- ---------------- ------------- ---------------------------
- 3/6/2018 02:28:30 cluster1-02 ERROR asup.post.drop: AutoSupport message (HA Group Notification from cluster1-02 (MANAGEMENT_LOG) INFO) for host (0) was not posted to NetApp. The system will drop the message.
- 3/6/2018 01:28:18 cluster1-02 ERROR asup.post.drop: AutoSupport message (HA Group Notification from cluster1-02 (PERFORMANCE DATA) INFO) for host (0) was not posted to NetApp. The system will drop the message.
- 3/6/2018 00:00:07 cluster1-02 ERROR mgmtgwd.certificate.expired: A digital certificate with Fully Qualified Domain Name (FQDN) cluster1, Serial Number 5589765F, Certificate Authority 'cluster1' and type server for Vserver cluster1 has expired.
- 3/6/2018 00:00:07 cluster1-02 ERROR mgmtgwd.certificate.expired: A digital certificate with Fully Qualified Domain Name (FQDN) UC_SVM2, Serial Number 55A03966, Certificate Authority 'SVM2' and type server for Vserver SVM2 has expired.
- 3/6/2018 00:00:07 cluster1-02 ERROR mgmtgwd.certificate.expired: A digital certificate with Fully Qualified Domain Name (FQDN) UC_SVM, Serial Number 559FFD76, Certificate Authority 'SVM' and type server for Vserver SVM has expired.
- 3/6/2018 00:00:07 cluster1-02 ERROR mgmtgwd.certificate.expired: A digital certificate with Fully Qualified Domain Name (FQDN) UCS_SVM_DR, Serial Number 545845C16E278, Certificate Authority 'SVM_DR' and type server for Vserver SVM_DR-mc has expired.
- 3/6/2018 00:00:07 cluster1-02 ERROR mgmtgwd.certificate.expired: A digital certificate with Fully Qualified Domain Name (FQDN) UCS_SVM2_DR, Serial Number 545845A7B01FA, Certificate Authority 'SVM2_DR' and type server for Vserver SVM2_DR-mc has expired.
- 7 entries were displayed.
17、查看某个聚合下的Volume状态信息
cluster1::> vol show -aggregate aggr_data_A1
18、查看Lun信息及Lun详细信息
- cluster1::> lun show
- cluster1::> lun show -v
19、查看map信息及map详情
- cluster1::> igroup show
- cluster1::> igroup show -v
20、查看Lun的map情况
- cluster1::> lun show -m
21、进入某一节点
- cluster1::> run -node cluster1-01
- Type 'exit' or 'Ctrl-D' to return to the CLI
- cluster1-01>
22、节点下查看spare disks
- cluster1-01> vol status -s
- Local spares
- Pool1 spare disks
- RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)
- --------- ------ ------------- ---- ---- ---- ----- -------------- --------------
- Spare disks for block checksum
- spare SW_B_3:6.126L41 3a 21 14 FC:A 1 SAS 10000 1142352/2339537408 1144641/2344225968 (not zeroed)
- spare SW_B_3:7.126L75 3a 42 22 FC:A 1 SAS 10000 1142352/2339537408 1144641/2344225968
- spare SW_B_3:7.126L101 3b 43 22 FC:A 1 SAS 10000 1142352/2339537408 1144641/2344225968
- spare SW_B_4:7.126L76 4b 42 23 FC:B 1 SAS 10000 1142352/2339537408 1144641/2344225968
- spare SW_B_4:7.126L29 4b 43 2 FC:B 1 SAS 10000 1142352/2339537408 1144641/2344225968
- spare SW_B_4:7.126L50 4b 43 23 FC:B 1 SAS 10000 1142352/2339537408 1144641/2344225968
- spare SW_B_3:6.126L22 3a 20 21 FC:A 1 SSD N/A 381304/780910592 381554/781422768
- Pool0 spare disks
- RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)
- --------- ------ ------------- ---- ---- ---- ----- -------------- --------------
- Spare disks for block checksum
- spare SW_A_1:7.126L12 3a 30 11 FC:A 0 SAS 10000 1142352/2339537408 1144641/2344225968
- spare SW_A_1:7.126L14 3a 30 13 FC:A 0 SAS 10000 1142352/2339537408 1144641/2344225968
- spare SW_A_1:7.126L31 3a 31 4 FC:A 0 SAS 10000 1142352/2339537408 1144641/2344225968
- spare SW_A_1:7.126L76 3a 32 23 FC:A 0 SAS 10000 1142352/2339537408 1144641/2344225968
- spare SW_A_1:7.126L79 3a 33 0 FC:A 0 SAS 10000 1142352/2339537408 1144641/2344225968
- spare SW_A_1:7.126L80 3a 33 1 FC:A 0 SAS 10000 1142352/2339537408 1144641/2344225968
- spare SW_A_2:7.126L73 4b 32 20 FC:B 0 SAS 10000 1142352/2339537408 1144641/2344225968
- spare SW_A_2:7.126L37 4b 33 10 FC:B 0 SAS 10000 1142352/2339537408 1144641/2344225968
- spare SW_A_2:6.126L74 4b 11 21 FC:B 0 SSD N/A 381304/780910592 381554/781422768
23、节点下查看fail disk
- cluster1-01> vol status -f
- Broken disks (empty)
24、显示没有ownership（归属权）的硬盘
- cluster1-01> disk show -n
- disk show : No unassigned disks
25、分配硬盘的归属(硬盘更换常用）
- cluster1-01> disk assign all
26、查看所有硬盘位置信息
- cluster1-01> storage show disk -p
版权声明：本文为cloudos原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。