NetApp 集群模式 Data ONTAP 8.3 网络管理

关于 Data ONTAP 8.2 7-mode的网络管理参考《 NetApp Data ONTAP 8.2 7-Mode 网络管理 – 存储人生 》。如果想了解 NetApp 集群模式 Data ONTAP 8.3 的新功能和架构，可以参考《 NetApp 集群模式Data ONTAP 8.3 新功能- 存储人生 》和《 NetApp 集群模式 Data ONTAP 8.3 架构 》。

要让客户能够高效，可靠的访问存储，网络管理一定是要考虑的。 Data ONTAP 8.3 网络管理的一些基本概念，大概和 Data ONTAP 8.2 7-mode差不多，我觉得关键的一点就是引入了 LIF 的概念，由于这个 LIF 可以在任何节点的物理接口迁移，带来了巨大的灵活性和可靠性。不知道 Data ONTAP 8.2 c-mode的网络管理是否也是这样。废话少说，我们直接切入正题。

网络拓扑图

我们再来看一下《 NetApp 集群模式 Data ONTAP 8.3 架构 》的那张4节点 Data ONTAP 8.3 集群的网络拓扑图，包括了三个不同的网络： Cluster Interconnect，Management Network，Data Network (Ethernet, FC, or Converged)。

对于集群之间的通信网络，如果是2个节点的话，可以直连而不需要经过交换机。一般来说2个交换机就可以提供冗余了。

Two cluster connections to each node are typically required for redundancy and improved cluster traffic flow. For the larger clusters that use higher-end platforms (FAS8040, FAS8060, and FAS8080) that are running clustered Data ONTAP 8.2.1, four cluster interconnects are the default. Optionally, a FAS8080 can be configured to use 6 cluster interconnect ports with expansion 10-gigabit Ethernet network interface cards (10- GbE NICs).

For proper configuration of the NetApp CN1601 and CN1610 switches, refer to the CN1601 and CN1610 Switch Setup and Configuration Guide.

网络接口分类

一共分为三层：物理层，虚拟层，逻辑层。

物理层

• Ethernet port : 1-Gb or 10-Gb Ethernet (10-GbE) ports that can be used in NFS, CIFS, and iSCSI environments
• FC port : 4-Gbps, 8-Gbps, or 16-Gbps port that can be used as a target in FC SAN environment. It can be configured as an initiator for disk shelves or tape drives.
• Unified Target Adapter (UTA) port : 10-GbE port that can be used in NFS, CIFS, iSCSI and FCoE environments
• Unified Target Adapter 2 (UTA2) port : Configured as either a 10-GbE Ethernet or 16-Gbps FC port – 10-Gb ports can be used in NFS, CIFS, iSCSI, and FCoE environments – 16-Gbps FC ports can be used as targets in FC SAN environments

虚拟层

• Interface group : An interface group implements link aggregation by providing a mechanism to group together multiple network interfaces (links) into one logical interface (aggregate). After an interface group is created, it is indistinguishable from a physical network interface.
• VLAN : Traffic from multiple VLAN s can traverse a link that interconnects two switches by using VLAN tagging.

逻辑层

Logical interfaces (LIFs):for clustered Data ONTAP only 。

物理层接口命名规则

和Data ONTAP 7-mode一样，参见《 NetApp Data ONTAP 8.2 7-Mode 网络管理 – 存储人生 》。还是比较有规律的，名字前面有e说明是以太网接口，没有就是FC接口。然后就是槽号(slot)，如果是0说明是背板自带。最后的字母就是递增的接口个数了。比如e0b就是指背板上的第二个以太网接口。3a是指第三个槽位的第一个FC口。e3a/3a指第三个槽位的第一个UTA口（以太网，FC双用)。

Ethernet ports are named e<location><letter>

• e0a is the first port on the controller’s motherboard
• e3a is a port on a card in slot 3

FC ports are named <location><letter>

• 0a is the first port on the controller’s motherboard
• 3a is a port on a card in slot 3

UTA ports have both an Ethernet name and an FC name e<location><letter>/<location><letter>

• e0e/0e is the first port on the controller’s motherboard
• e3a/3a is a port on a card in slot 3
• Show commands list only by FC label name (even in Ethernet mode)

下图就是FAS8040 and FAS8060的物理接口图：

Cluster ed Data ONTAP configuration:

• 4 x 10-GbE ports for cluster interconnects – Supported: two cluster interconnects (e0a and e0c) and two data (e0b and e0d) – Recommended: four cluster interconnects (switched clusters only)
• 4 x Unified Target Adapter 2 (UTA2) ports can be configured as either 10-GbE or 16-Gbps FC for data – Can only be used for data (not cluster interconnects) – Port pairs must be set the same:
• 4 x GbE ports for data
• 1 x management port (default for node-management network) – e0M runs at GbE – SP runs at 10/100
• 1 x private management port that is used as an alternate control path (ACP) for SAS shelves
• 1 x console port (can be configured for SP) – To toggle from serial console into SP, use Ctrl-G. – To toggle back, use Ctrl-D.

Data ONTAP 8.3 模拟器就只有4个物理接口，e0a,e0b为集群内部通讯接口，e0c为节点管理接口，e0d为真正的集群数据访问接口，均为以太网接口。如下所示：

网络管理的一些基本概念

Interface Groups

这个和Data ONTAP 7-mode基本一样，几乎所有NAS厂商都会提供这个功能，就是网络链路聚合，只是名字取得不同而已，可以参考《 NetApp Data ONTAP 8.2 7-Mode 网络管理 – 存储人生 》与《Isilon 集群网络设置》。

那么集群模式的Interface groups和7-mode相比有哪些不同呢？

• Interface groups must be named by using the syntax a<number><letter>.
• A port that is already a member of an interface group cannot be added to another interface group.
• Multimode load balancing methods: – mac: Network traffic is distributed on the basis of MAC addresses. – ip: Network traffic is distributed on the basis of IP addresses. – sequential: Network traffic is distributed as it is received. – port: Network traffic is distributed on the basis of the transport layer (TCP/UDP)ports.

For more information about load balancing, please refer to TR-4182: Ethernet Storage Best Practices for Cluster ed Data ONTAP Configurations.

Interface Group 在集群模式下注意事项

• Due to the limited capabilities of single mode, it is recommended that you do not use this type of interface group in clustered Data ONTAP
• To take advantage of all the performance and resiliency functionality of dynamic multimode ( LACP ), it is recommended when you use interface groups
  • Requires an LACP -enabled switch
  • All the interfaces in the group will be active, will share the same MAC address, and will use load balancing outbound traffic (not inbound)
  • A single host will not achieve larger bandwidth than any of the constituent connection (two 10-GbE ≠ 20GbE)
  • May not have any advantages for iSCSI hosts

For more information, refer to TR-4182: Ethernet Storage Best Practices for Clustered Data ONTAP Configurations.

VLAN

这个没啥好说的，和其他产品一样。

Ports, Interface Groups, and VLANs

Ports are the physical network adapters that are cabled to a switch or a virtualization. This switch or virtualization subdivides or groups the physical ports. Ports can be arranged in a hierarchy, with VLANs subdividing the interface groups. In 7-Mode, interface groups can be created from ports or other interface groups; the latter are called secondlevel interface groups. You can create a second-level interface group by using two multimode interface groups. Second-level interface groups enable you to provide a standby multimode interface group in case the primary multimode interface group fails. VLANs cannot be created from other VLANs, and interface groups cannot be created from VLANs. In clustered Data ONTAP, interface groups cannot be created from other interface groups, VLANs cannot be created from other VLANs, and interface groups cannot be created from VLANs. NOTE: You cannot create VLANs or interface groups on cluster ports on clustered Data ONTAP.

IPspaces

这个概念是Data ONTAP 8.3新出的，An IPspace defines a distinct IP address space in which virtual storage systems can participate. IP addresses that are defined for an IPspace are applicable only within that IPspace. A distinct routing table is maintained for each IPspace. No cross-IPspace traffic routing happens. Each IPspace has a unique loopback interface thatis assigned to it. The loopback traffic on each IPspace is completely isolated from the loopback traffic on other IPspaces 。

IPspaces到底有什么用呢？可以将IP进行逻辑分割。比如将NAS共享给不同的部门或公司，使用的是不同的相互隔离网络。如果都是公有IP，一般是不会有问题。假如都是私有IP，就可能有重复的IP，怎么办呢？有了IPspaces就好办了。

你看下图，每个公司一个IPspaces，由于是使用各自的网络路由表，两个公司都使用10.0.0.0网络没有问题。 另外我们也可以看出， SVM 是创建在IPspaces的，也就是说IPspaces包括 SVM ，另外还包括subnet和ports。每一个 SVM 拥有自己的数据卷和配置，管理起来也是独立的。我们稍后会详细介绍。

IPSPACES默认值

集群初始化的时候会自动创建2个IPspaces，且每一个IPspace都创建了一个SVM(Vfiler)，在CLI显示为Vserver。下面是Data ONTAP 8.3模拟器的IPspaces默认值。

labcluster2::> network ipspace show IPspace             Vserver List                  Broadcast Domains ------------------- ----------------------------- ---------------------------- Cluster                     Cluster                       Cluster Default                     labcluster2                   Default 2 entries were displayed.

“Default” IPspace

This IPspace is a container for ports, subnets, and SVMs that serve data. If your configuration does not need separate IPspaces for clients, all SVMs can be created in this IPspace. This IPspace also contains the cluster management and node management ports.

“Cluster” IPspace

This IPspace contains all cluster ports from all nodes in the cluster. It is created automatically when the cluster is created. It provides connectivity to the internal private cluster network. As additional nodes join the cluster, cluster ports from those nodes are added to the “Cluster” IPspace. 如果是一个节点那就不会有这个IPspace。

创建自定义IPspace

在了解了IPspace的原理之后，我们来创建2个自定义IPspaces，假如一个公司有2个大部门，一个是销售，一个是市场，那我们就各自创建一个IPspaces。

labcluster2::> network ipspace create -ipspace IPspace_Sales labcluster2::> network ipspace create -ipspace IPspace_Market labcluster2::> network ipspace show IPspace             Vserver List                  Broadcast Domains ------------------- ----------------------------- ---------------------------- Cluster  Cluster    Cluster Default  labcluster2                   Default IPspace_Market  IPspace_Market                - IPspace_Sales  IPspace_Sales                 - 4 entries were displayed. labcluster2::> network ipspace show -ipspace IPspace_Market      IPspace name: IPspace_Market             Ports: - Broadcast Domains: -          Vservers: IPspace_Market labcluster2::> network ipspace show -ipspace IPspace_Sales      IPspace name: IPspace_Sales             Ports: - Broadcast Domains: -          Vservers: IPspace_Sales 

2个和IPspace名字一样的系统SVM也自动创建了，我们可以看到Broadcast Domains还是空白的，我们稍后要手工创建，之后才能创建自定义SVM。

Broadcast Domains

Broadcast Domains又是个什么东东呢？Broadcast domains are commonly used when a system administrator wants to reserve specific ports for use by a certain client or group of clients. A broadcast domain should include ports from many nodes in the cluster to provide high availability for the connections to SVMs。

啥意思？还不明白，就是端口分配而已，用于SVM的数据访问或管理，只是取了个高大上的名字而已。 系统默认以及创建了2个Broadcast domains ，一个名叫Default，包括了节点管理和集群管理的端口。另外一个叫Cluster，包括了节点用于集群内部通讯的端口，二者都是在集群初始化时创建的。如下所示：

labcluster2::> network port broadcast-domain show IPspace Broadcast              Update Name    Domain Name    MTU  Port List                     Status Details ------- ----------- ------  ----------------------------- -------------- Cluster Cluster       1500  labcluster2-01:e0a            complete  labcluster2-01:e0b            complete  labcluster2-02:e0a            complete  labcluster2-02:e0b            complete Default Default       1500  labcluster2-01:e0c            complete  labcluster2-01:e0d            complete  labcluster2-02:e0c            complete  labcluster2-02:e0d            complete 

我们在之前创建2个自定义IPspaces，还没有Broadcast Domains，因此还不能创建SVM。但是端口只能存在一个Broadcast Domains。虚拟机默认就4个端口，都用完了，那我就自己给每个节点添加了2块网卡，注意要重启才能生效。

labcluster2::> network port show              Speed (Mbps) Node   Port      IPspace      Broadcast Domain Link   MTU    Admin/Oper ------ --------- ------------ ---------------- ----- ------- ------------ labcluster2-01  e0a Cluster      Cluster    up 1500  auto/1000  e0b Cluster      Cluster    up 1500  auto/1000  e0c Default      Default    up 1500  auto/1000  e0d Default      Default    up 1500  auto/1000  e0e Default      -    up 1500  auto/1000  e0f Default      -    up 1500  auto/1000 labcluster2-02  e0a Cluster      Cluster    up 1500  auto/1000  e0b Cluster      Cluster    up 1500  auto/1000  e0c Default      Default    up 1500  auto/1000  e0d Default      Default    up 1500  auto/1000  e0e Default      -    up 1500  auto/1000  e0f Default      -    up 1500  auto/1000 12 entries were displayed. 

创建自定义Broadcast Domains

有了富余的端口，我们就可以创建2个自定义Broadcast Domains了，一个给IPspaces Sales用，一个给IPspaces Market用。

labcluster2::> network port broadcast-domain create -broadcast-domain bcast_Sales -mtu 1500 -ipspace IPspace_Sales  -ports labcluster2-01:e0e,labcluster2-02:e0e labcluster2::> network port broadcast-domain create -broadcast-domain bcast_Market -mtu 1500 -ipspace IPspace_Market  -ports labcluster2-01:e0f,labcluster2-02:e0f labcluster2::> network port broadcast-domain show IPspace Broadcast              Update Name    Domain Name    MTU  Port List                     Status Details ------- ----------- ------  ----------------------------- -------------- Cluster Cluster       1500  labcluster2-01:e0a            complete  labcluster2-01:e0b            complete  labcluster2-02:e0a            complete  labcluster2-02:e0b            complete Default Default       1500  labcluster2-01:e0c            complete  labcluster2-01:e0d            complete  labcluster2-02:e0c            complete  labcluster2-02:e0d            complete IPspace_Market         bcast_Market  1500  labcluster2-01:e0f            complete  labcluster2-02:e0f            complete IPspace_Sales         bcast_Sales   1500  labcluster2-01:e0e            complete  labcluster2-02:e0e            complete 4 entries were displayed. 

我们再来看IPspaces信息：

labcluster2::> network ipspace show IPspace             Vserver List                  Broadcast Domains ------------------- ----------------------------- ---------------------------- Cluster                     Cluster                       Cluster Default                     labcluster2                   Default IPspace_Market                     IPspace_Market                bcast_Market IPspace_Sales                     IPspace_Sales                 bcast_Sales 4 entries were displayed.

Subnet

没啥复杂的，就是Broadcast Domains中的IP池，我觉得名字叫IP Pool更好理解一点。但这个IP池要来自同一个子网。每当你创建一个LIF时，系统就从IP池分配一个IP，当LIF删掉时，IP又退回到IP池中。

创建Subnet

我们来创建2个IP段完全相同的subnet，由于分别在不同的IPspaces，所以没有关系的。

labcluster2::> network subnet create -subnet-name subnet_Sales -broadcast-domain bcast_Sales  -ipspace IPspace_Sales -subnet 192.168.0.0/24 -gateway 192.168.0.130 -ip-ranges 192.168.0.30-192.168.0.35 labcluster2::> network subnet create -subnet-name subnet_Market -broadcast-domain bcast_Market  -ipspace IPspace_Market -subnet 192.168.0.0/24 -gateway 192.168.0.130 -ip-ranges 192.168.0.30-192.168.0.35 labcluster2::> network subnet show IPspace: IPspace_Market Subnet                     Broadcast                   Avail/ Name      Subnet           Domain    Gateway           Total   Ranges --------- ---------------- --------- --------------- --------- --------------- subnet_Market           192.168.0.0/24   bcast_Market            192.168.0.130      6/6    192.168.0.30-192.168.0.35 IPspace: IPspace_Sales Subnet                     Broadcast                   Avail/ Name      Subnet           Domain    Gateway           Total   Ranges --------- ---------------- --------- --------------- --------- --------------- subnet_Sales           192.168.0.0/24   bcast_Sales            192.168.0.130      6/6    192.168.0.30-192.168.0.35 2 entries were displayed. 

Logical Interfaces(LIF)

• An IP address or World Wide Port Name (WWPN) is associated with a LIF
• If subnets are configured (recommended), IP addresses are automatically assigned when a LIF is created
• If subnets are not configured, IP addresses must be manually assigned when LIF is created
• WWPNs are automatically assigned when an FC LIF is created
• One node-management LIF exists per node
• One cluster-management LIF exists per cluster
• Two* cluster LIFs exist per node
• Multiple data LIFs are allowed per port (Client-facing: NFS, CIFS, iSCSI,and FC access)
• For intercluster peering, intercluster LIFs must be created on each node

我们来看看系统默认的LIF。每个节点一个节点管理LIF，labcluster2-01_mgmt1和labcluster2-02_mgmt1。每个集群一个集群管理LIF，cluster_mgmt。还有集群内部通讯用的每个节点2个LIF。

labcluster2::> network interface show             Logical    Status     Network            Current       Current Is Vserver     Interface  Admin/Oper Address/Mask       Node          Port    Home ----------- ---------- ---------- ------------------ ------------- ------- ---- Cluster             labcluster2-01_clus1              up/up    192.168.10.12/24   labcluster2-01        e0a     true             labcluster2-01_clus2              up/up    192.168.10.13/24   labcluster2-01        e0b     true             labcluster2-02_clus1              up/up    192.168.10.10/24   labcluster2-02        e0a     true             labcluster2-02_clus2              up/up    192.168.10.11/24   labcluster2-02        e0b     true labcluster2             cluster_mgmt up/up    192.168.0.40/24    labcluster2-01        e0d     true             labcluster2-01_mgmt1              up/up    192.168.0.21/24    labcluster2-01        e0c     true             labcluster2-02_mgmt1              up/up    192.168.0.20/24    labcluster2-02        e0c     true 7 entries were displayed. 

多个数据LIF可以用在同一个Port上面，因此多个IP可能分配到同一个Port上。如果这个port有过载或其他故障，LIF可以在用户完全没有觉察的情况下迁移到其他节点的其它port上面。

创建数据LIF

语法：c1::> network interface create –vserver SVM_A-1 –lif SVM_A-1_lif2 –role data –data-protocol nfs –home-node c1-02 –home-port e0f –subnet-name subnet_A。 由于我还没有创建SVM，现在创建不了数据LIF，稍后再试。上面的–home-node 和–home-port 是什么意思呢？

• The home-node parameter is the node to which the LIF returns when the network interface revert command is run on the LIF.
• The home-port parameter is the port or interface group to which the LIF returns when the network interface revert command is run on the LIF.

Nondisruptive LIF Configuration

Clustered Data ONTAP 8 has always supported nondisruptive LIF configuration. The process of performing a failover has changed throughout the versions leading up to the 8.3 release. Here is a brief history:

• Data ONTAP 8.0: Failover rules (network interface failover) were the primary way to control failover based on port role and priority.
• Data ONTAP 8.1: Failover groups (network interface failover-groups) became theprimary method to control failover. Failover rules were deprecated.
• Data ONTAP 8.3: Failover groups and failover policies were changed to work with broadcast domains. There are fewer failover groups and more failover policies.

Conceptually, LIF failover is similar in the different versions of clustered Data ONTAP, but the configuration is very different. This lesson discusses only examples of clustered Data ONTAP 8.3. For more information about how to configure LIF failover in older versions of clustered Data ONTAP, refer to the Network Management Guide for the version of clustered Data ONTAP that you are configuring.

Network Load Balancing

这个好像Isilon的SmartConnect，参考《IsilonSmartConnect设置》，稍后要专文介绍。

• Clients can mount to an SVM by either: Specifying a LIF’s IP address or Specifying a host name (for multiple managed IP addresses)
• Load balancing dynamically evaluates the load on LIFs and either: Selects an appropriately loaded LIF or Moves a LIF to a less loaded port
• Load balancing types: DNS load balancing (NFS or CIFS); On-box (zoning based); Off-box (round robin); Automatic LIF rebalancing (NFSv3 only)

参考