Considerations for configuration a LACP bond

Standardized protocol is used to establish and maintain link-aggregation between systems. What sets LACP bonding mode apart from the other bonding modes supported by Linux is that the systems participating in LACP continuously exchange data (LACPDUs) with each other to determine which links within the bond should actively be used for the transmission and receipt of network traffic.

While the IEEE standard defines terminology specific to LACP, some vendors have chosen to use different terminology. In this section, LAG (link-aggregation group) and MLAG are used when describing link-aggregation on the switch side and bond is used when describing link-aggregation on the appliance side.

The following list is a non-comprehensive list of equivalent vendor-proprietary terms.

Interfaces in an LACP bond are either placed into an active or suspended state. In the active state an interface is allowed to collect and process incoming network traffic as well as transmit outgoing network traffic. In the suspended state an interface does not collect incoming network traffic or transmit any outgoing network traffic until the condition that caused the suspension is resolved.

For all bonded appliance interfaces to be considered active within a LACP bond, you must ensure the following:

A LAG cannot span multiple switches without the following:

Stacked switches, MLAG, and EVPN LAG multihoming allow multiple switches to function as a single logical switch from the perspective of LACP. When stacking, MLAG or EVPN LAG multihoming are deployed and the LAG is properly configured. The LACP bond on the appliance places all the bonded interfaces into an active state. If the appliance's bond spans across multiple switches without MLAG, EVPN LAG multihoming, or stacking configured on those switches, LACP on the appliance detects this and elects an interface or group of interfaces, which are members of the same LAG and connected to the same switch, to be active while placing all other bonded interfaces into a suspended state. Even if the LAGs on the switches are configured to use the same LAG ID, LACP on the appliance identifies that the LACPDUs were sent from multiple different switches.

Although Access Appliance is configured as a cluster, it is important to note that Red Hat Linux does not have a mechanism to perform MLAG. Therefore, each appliance must be treated as a unique end-system. For each bond created on the Access Appliance cluster, two LAGs must be created on the switch and each LAG must be assigned a different LAG ID.

For example, when you create a two-interface bond using the Access CLI, a bond interface is created on each node. The two switch ports which connect to node_A should be assigned to a LAG with a LAG ID of X and the two switch ports which connect to node_B should be assigned to a LAG with a LAG ID of Y. In this example, if all four switch ports are assigned the LAG ID of X, the switch or set of switches elects a set of switch ports connected to one of the appliance nodes and places it in an active state while other interfaces are placed into a suspended state. This leads to network connectivity issues as the suspended interfaces are still physically up on the appliance which allows VCS to assign an IP address to them. If this happens, when that node attempts to send traffic out the bond, the application sends the traffic down the network stack, and it is dropped by the bonding driver.

While LACP does not perform load balancing, it does have a mechanism for selecting an outgoing physical interface. This mechanism is responsible for calculating a hash based on selected attributes in the headers of the outgoing network traffic. LACP associates a hash with an interface and all traffic matching that hash egress the selected interface. It is important to note that you cannot control the outgoing interface, you can only influence the selection process. This mechanism is statically configured on each system and is not negotiated between the systems participating in LACP. In Linux this mechanism is called the xmit_hash_policy.

The Access Appliance supports the following configurations of the xmit_hash_policy:

During the planning stage, it is important to consider which xmit_hash_policy is most appropriate based on the expected usage of the bond. It is also important to include the network administrators in this planning as the xmit_hash_policy only controls the outgoing traffic on the appliance and has no effect on how the switch chooses an egress interface.