Configuring peer-keepalive links

Distributed trunking uses UDP-based peer-keepalive messages to determine if an ISC link failure is at the link level or the peer has completely failed. The following operating rules must be followed to use peer-keepalive links:

  • An IP address must be configured for a peer-keepalive VLAN interface and the same IP address must be configured as a peer-keepalive destination on the peer DT switch.

  • There must be logical Layer 3 connectivity between the two IP addresses configured for the peer-keepalive VLAN interface.

  • Only peer-keepalive messages are sent over the peer-keepalive VLAN (Layer 3 link.) These messages indicate that the DT switch from which the message originates is up and running. No data or synchronization traffic is sent over the peer-keepalive VLAN.

  • STP cannot run on peer-keepalive links.

  • The peer-keepalive VLAN can have only one member port. If you attempt to assign a second member port to this VLAN, or if you attempt to configure a VLAN that has more than one member port as a peer-keepalive VLAN, this message displays:

    A keepalive VLAN can only have one member port.

  • A port cannot be a member of a regular VLAN and a peer-keepalive VLAN. An error message displays:

    A port cannot simultaneously be a member of a keepalive and a non-keepalive VLAN.

  • The DEFAULT VLAN cannot be a peer-keepalive VLAN. An error message displays:

    The default VLAN cannot be configured as a keepalive VLAN.


[NOTE: ]

NOTE: If you are upgrading your software from a version prior to K.15.05.xxxxx with a configuration that violates any of the above operating rules, the following message displays:

DT: Keepalive mis-configuration detected. Reconfigure the keepalive VLAN.

You must then manually correct the configuration.


DT switches have an operational role that depends on the system MAC address. The bridge with the lowest system MAC address acts as the DT primary device; the other device is the DT secondary device. These roles are used to determine which device forwards traffic when the ISC link is down.

ISC link failure with peer-keepalive

ISC link failure with peer-keepalive

Peer-keepalive messages are sent by both the DT switches as soon as the switches detect that the ISC link is down. Peer-keepalive message transmission (sending and receiving) is suspended until the peer-keepalive hold timer expires. When the hold timer expires, the DT switches begin sending peer-keepalive messages periodically while receiving peer-keepalive messages from the peer switch. If the DT switch fails to receive any peer-keepalive messages for the timeout period, it continues to forward traffic, assuming that the DT peer switch has completely failed.

Conversely, if the failure is because the ISC link went down and the secondary DT switch receives even one peer-keepalive message from the primary peer, the secondary switch disables all its DT ports. The primary switch always forwards the traffic on its DT ports even if it receives peer-keepalive messages from the secondary DT switch.

In both situations, if the ISC link or the DT switch becomes operational, both the DT peers sync the MAC addresses learned during the failover and continue to forward traffic normally. The peer-keepalive timers and operation is halted.

Maximum DT trunks and links supported

Maximum supported DT trunks and links shows the maximum number of DT trunks and DT links that are supported.

Maximum supported DT trunks and links

Description Max number
Maximum number of groups (DT trunks) in a DT switch (that is, maximum number of servers supported) 144
Maximum number of switches that can be aggregated 2
Maximum number of physical links that can be aggregated in a single switch from a server (that is, maximum number of ports that can be in a trunk connected to a single switch) 4

From the server perspective, this means that there could be a maximum total of 60 servers connected to two DT switches. Each server can have up to four physical links aggregated in a single switch, meaning that a single server could have a maximum of eight links (that is, four on each DT switch) in a DT trunk.

Forwarding traffic with distributed trunking and spanning tree

Refer to Distributed trunking with STP forwarding unicast, broadcast, and multicast traffic for the following discussion about forwarding traffic when spanning tree is enabled. In this example, it is assumed that traffic is sent from a host off switch B to a server, and from the server back to the host. STP can block any one of the upstream links; in this example, STP has blocked all the links except the I1 link connected to DT1.


[NOTE: ]

NOTE: STP is automatically disabled on the DT ports.


Distributed trunking with STP forwarding unicast, broadcast, and multicast traffic

Distributed trunking with STP forwarding unicast, broadcast, and multicast traffic

Forwarding unicast traffic

Refer to Unicast traffic flow across DT switches for the following discussion about forwarding traffic with switch-to-switch distributed trunking. Traffic from Host X or Y that is destined for Host F is always forwarded by Switch A over one of its standard 802.1AX trunk links to either Switch B or Switch C. When either Switch B or Switch C receives incoming traffic from Switch A, the traffic is directly forwarded to Switch F without traversing the ISC link.

Traffic from Host Y to Host D may go over the ISC if Switch A sends it to Switch C instead of sending it to Switch B.

Unicast traffic flow across DT switches

Unicast traffic flow across DT switches

Forwarding broadcast, multicast, and unknown traffic

In the example shown in Broadcast/multicast/unknown traffic flow access DT switches, multicast/broadcast/unknown traffic from Host X or Y is always forwarded by Switch A over one of its standard 802.3ad trunk links to either Switch B or C. Switch B or C forwards the traffic on all the links including the ISC port, but not on the port that the traffic was received on. The peer DT switch (B or C) that receives broadcast/multicast/unknown traffic over the ISC port does not forward the packets to any of the DT trunks; the packet is sent only over the non-DT ports. The one exception is if the DT trunk on the peer aggregation device is down, then traffic received over the ISC is forwarded to the corresponding DT trunk.

Broadcast/multicast/unknown traffic flow access DT switches

Broadcast/multicast/unknown traffic flow access DT switches

IP routing and distributed trunking

In switch-to-switch distributed trunking, the peer DT switches behave like independent Layer 3 devices with their own IP addresses in each active VLAN. If a DT switch receives a packet destined for the peer DT switch, it switches the packet through the ISC link. Interfaces on a VLAN using DT typically use a single default gateway pointing to only one of the DT switches in a DT pair.

The example in Layer 3 forwarding (IP unicast) in DT topology shows Layer 3 (IP unicast) forwarding in a DT topology. The packet is sent as follows:

  1. Switch A selects the link (using the trunk hash) to the DT pair. The packet is sent to the selected link DT_SW_1.

  2. When DT_SW_1 receives the packet, it determines, based on the MAC address, that the packet must be sent over the ISC link to DT_SW_2.

  3. When the packet arrives, DT_SW_2 performs a lookup and determines that the packet needs to be sent to Switch B.

Layer 3 forwarding (IP unicast) in DT topology

Layer 3 forwarding (IP unicast) in DT topology

Another example in Layer 3 forwarding (IP unicast) in DT topology shows Layer 3 (IP unicast) forwarding in a DT topology. The packet is sent as follows:

  1. Host 2 sends a packet to Switch C.

  2. Switch C performs a lookup in the routing table and determines that the default gateway IP address is 10.0.0.1.

  3. Layer 2 lookup determines that the outgoing interface is the DT port.

  4. Hashing determines that the trunk member chosen is DT_SW_2 and the packet is sent there.

  5. DT_SW_2 determines that the packet needs to be sent over the ISC link to DT_SW_1 based on the MAC address.

  6. DT_SW_1 performs a lookup and determines that the packet goes to Switch A.

The packet is only forwarded if the outgoing interface is not a DT port, or if the outgoing DT port does not have an active interface on the peer switch.

Layer 3 forwarding (IP unicast) in DT topology

Layer 3 forwarding (IP unicast) in DT topology

Distributed trunking restrictions

There are several restrictions with distributed trunking:

Beginning with software version K.15.07, the switch will not allow both Distributed Trunking and MAC-based mirroring to function simultaneously. The switch will respond as follows:

  • If the user attempts to configure both, an error message will appear.

  • When a switch is updated from older software to K.15.07, if the older config file has both Distributed Trunking and MAC-based mirroring, the switch will automatically remove the MAC-based mirroring lines from the config file, and will give an explanatory error message.

  • If a switch is running K.15.07 and an existing config file that has both Distributed Trunking and MAC-based mirroring is loaded onto the switch, the switch will automatically remove the MAC-based mirroring lines from the config file, and will give an explanatory error message.

  • All DT linked switches must be running the same software version.

  • The port trunk links should be configured manually (using manual LACP or manual trunks.) Dynamic linking across switches is not supported.

  • A distributed trunk can span a maximum of two switches.

  • A maximum total of 144 servers can be connected to two DT switches. Each server can have up to four physical links aggregated in a single switch, meaning that there can be a maximum of eight ports (four aggregated links for each DT switch) included in a DT trunk.

  • Only one ISC link is supported per switch, with a maximum of 60 DT trunks supported on the switch. The ISC link can be configured as a manual LACP trunk, non-protocol trunk, or as an individual link. Dynamic LACP trunks are not supported as ISCs.

  • An ISC port becomes a member of all VLANs that are configured on the switch. When a new VLAN is configured, the ISC ports become members of that VLAN.

  • Port trunk links can be done only on a maximum of two switches that are connected to a specific server.

  • Any VLAN that is in a distributed trunk must be configured on both switches. By default, the distributed trunk belongs to the default VLAN.

  • There can be eight links in a distributed trunk grouped across two switches, with a limit of four links per distributed trunking switch.

  • The limit of 144 manual trunks per switch includes distributed trunks as well.

  • ARP protection is not supported on the distributed trunks.

  • Dynamic IP Lockdown protection is not supported on the distributed trunks.

  • QinQ in mixed VLAN mode and distributed trunking are mutually exclusive.

  • Source Port Filter cannot be configured on an InterSwitch Connect (ISC) port.

  • Features not supported include:

    • SVLANs in mixed mode on DT or ISC links

    • Meshing

    • Multicast routing

    • IPv6 routing