Equnix Metal
Loading...

Pure Layer 2 Modes

COMMAND LINE

Copy to clipboard

Pure Layer 2 Modes

Equinix Metal™ allows users to change the networking mode of a server from the default Layer 3 Bonded mode to modes that only use Layer 2 networking over VLANs. This means all access to the public Internet is lost, and the host can only be reached by the Serial Over SSH (SOS) console or from other servers attached to the same VLAN.

You can either have both interface in a bonded configuration or you can have both interfaces separated so that you can assign different VLANs to each interface.

Layer 2 Bonded Mode

Layer 2 Bonded Diagram

  • This option will preserve the LACP bond and allow you to add Layer 2 networks to the bonded interface.
  • This provides redundancy and HA, and an outage, maintenance, or reboot of one of the interfaces will not cause network interruptions.
  • On the server, all traffic has to be tagged, and need to create sub-interfaces for each VLAN assigned to the bond. Untagged packets and native VLANs are not supported.

Layer 2 Unbonded Mode

Layer 2 Unbonded Diagram

  • This option will destroy the LACP bond and allow you to place each interface into individual Layer 2 networks.
  • This does introduce a single point of failure on the upstream switch or the network interface. An outage, maintenance, or reboot of one of the ports will cause network interruptions.
  • On the server, you need to destroy the bond. If you have more than one VLAN assigned to a port, you can create sub-interfaces for each of them. Since the ports are not part of a bond in this mode, there is support for un-tagged packets and native VLANs.

Note: both of the Layer 2 networking configurations will permanently remove the server's IP management IP addresses. If the server is later converted back to Layer 3, new IP addresses will be assigned.

Converting to Layer 2 Bonded Mode

To use Bonded Layer 2 mode, you must first change your networking configuration to Layer 2. From the portal, navigate to the server's Network page, click Convert To Other Network Type and choose Layer 2.

In the Equinix Metal console, navigate to the server's Network page, click Convert To Other Network Type and choose Layer 2. Select the Bonded mode. Click Convert to Layer 2 to start the conversion.

Converting to Layer 2 Bonded mode panel

When it is finished, you will be back on the server's Network page.

In the Layer 2 section, click Add New VLAN, which will allow you to assign a VLAN to the bond0 port. Click Add to start the changes.

Adding a VLAN to bond0

In the API, you can convert the bond interface to Layer 2 and assign the VLAN by submitting a POST request to the /ports/{id}/convert/layer-2 endpoint.

You have to specify the port ID in the path, and it needs to be the UUID for that port as returned by the /devices/{id} endpoint.

The VLAN_ID is sent in the body of the request, and it can be either the VLAN's UUID as returned by the /projects/{id}/virtual-networks endpoint or the ID that is in the console.

curl -X POST -H "Content-Type: application/json" -H "X-Auth-Token: <API_TOKEN>" "https://api.equinix.com/metal/v1/ports/{id}/convert/layer-2" -d '{"vnid": "<VLAN_ID>"}'

Remember: once you have clicked Convert to Layer 2 in the console, or submitted the request to the API, the existing IP Addresses will be removed and the server will no longer be accessible through the Internet. You will still be able to use the SOS console to connect.

Server Configuration - Bonded

Once you have assigned a VLAN to the interface, you will need to configure the networking on the server's operating system to use the VLAN on bond0. Remember, you will have to use the SOS console to connect.

  1. Enable VLAN support.

    modprobe 8021q
    lsmod | grep 8021q
    echo "8021q" >> /etc/modules-load.d/networking.conf
    
  2. Add the VLAN to bond0. VLAN_ID should match the ID found on the console or the UUID from the API.

    ip link add link bond0 name bond0.<VLAN_ID> type vlan id <VLAN_ID>
    

    For example,

    ip link add link bond0 name bond0.1036 type vlan id 1036
    
  3. Add 192.168.100.1 IP address to the VLAN. IP addresses in the 192.168 range are recommended, as the 10.0.0.0/8 range is used internally by Equinix Metal. For example,

    ip addr add 192.168.100.1/24 brd 192.168.100.255 dev bond0.1036
    ip link set dev bond0.1036 up
    

    To make the changes permanent, configure /etc/network/interfaces with the IP address to the desired IP from your chosen block. For example,

    auto bond0.1036
    iface bond0.1036 inet static
    pre-up sleep 5
    address 192.168.100.1
    netmask 255.255.255.0
    vlan-raw-device bond0
    

    Note: The line pre-up sleep 5 helps to prevent conflicts on bond0 when the server boots.

  4. Ensure the IP address is configured.

    ip -d link show bond0.1036
    

You need to run through the same steps on all the servers that you are attaching to the VLAN, assigning a different IP address to each.

Attaching Multiple VLANs - Bonded

To assign multiple VLANs to an interface, navigate to the server's Network page. In the Layer 2 section, click Add New VLAN, which will allow you to assign additional VLANs to the bond0 port.

Click Add to start the changes. Note that if you assign multiple VLANs at once, they are added through an asynchronous batch process, which begins immediately, but may take some time to complete.

The procedure for assigning a single additional VLAN to an interface in the API is the same as assigning the first VLAN. Send a POST request to the /ports/{id}/assign endpoint.

You have to specify the port ID in the path, and it needs to be the UUID for that port as returned by the /devices/{id} endpoint.

The ID of the VLAN is sent in the body of the request, and it can be either the VLAN's UUID as returned by the /projects/{id}/virtual-networks endpoint or the VLAN ID that is in the console.

curl -X POST \
-H "Content-Type: application/json" \ 
-H "X-Auth-Token: <API_TOKEN> " \
"https://api.equinix.com/metal/v1/ports/{id}/assign" \
-d '{
    "vnid": "c4032b18-5494-451b-a779-a7d3c536bfd7"
    }'

It is also possible to assign VLANs to a port in bulk, as part of an asynchronous batch process. Send a POST request to the /ports/{id}/vlan-assignments/batches endpoint an array of VLAN assignments in the body of the request.

curl -X POST \
-H "Content-Type: application/json" \ 
-H "X-Auth-Token: <API_TOKEN> " \
"https://api.equinix.com/metal/v1/ports/{id}/vlan-assignments/batches" \
-d '{
    "vlan_assignments": [
        {
            "vlan": "string",
            "state": "assigned"
        },
        {
            "vlan": "string",
            "state": "assigned"
        },
    ]
}'

Once the VLAN (or VLANs) is assigned to the port, you repeat the process in Server Configuration - Bonded to create the additional tagged interfaces.

Converting to Layer 2 Unbonded Mode

In the Equinix Metal console, navigate to the server's Network page, click Convert To Other Network Type and choose Layer 2. Select the Unbonded mode. Click Convert to Layer 2 to start the conversion.

Converting to Layer 2 Unbonded mode panel

When it is finished, you will be back on the server's Network page.

To assign a VLAN to one of the ports, click Add New VLAN, and select either the eth0 or eth1 interface. Click Add to start the changes.

Adding a VLAN to eth0

In the API, converting to Layer 2 Unbonded mode is a two-step process. You have to first break the bond for all ports by sending a POST request to the /ports/{id}/disbond endpoint.

You have to specify the port ID of eth1 in the path, and it needs to be the UUID for that port as returned by the /devices/{id} endpoint.

Set the bulk_disable field to false in the body of the request.

curl -X POST -H "Content-Type: application/json" -H "X-Auth-Token: <API_TOKEN>" "https://api.equinix.com/metal/v1/ports/{id}/disbond" -d '{"bulk_disable": false}'

Then, you need to assign the VLAN to eth1. Send a POST request to the /ports/{id}/assign endpoint.

You have to specify the port ID of eth1, and it needs to be the UUID for that port as returned by the /devices/{id} endpoint.

The ID of the VLAN is sent in the body of the request, and it can be either the VLAN's UUID as returned by the /projects/{id}/virtual-networks endpoint or the vxlan ID that is in the console.

curl -X POST -H "Content-Type: application/json" -H "X-Auth-Token: <API_TOKEN>" "https://api.equinix.com/metal/v1/ports/{id}/assign" -d '{"vnid": "1173"}'

Server Configuration - Unbonded

The following configuration steps can be performed on both eth0 and eth1, depending on which ports you are assigning a VLAN to. The examples use eth1.

If only one VLAN is enabled on a port, packets are untagged. This means that the server's network configuration does not need to be VLAN-aware.

  1. Make sure eth1 has been removed from bond0:

    cat /sys/class/net/bond0/bonding/slaves
    

    If it hasn't been removed, remove it:

    echo "-eth1" > /sys/class/net/bond0/bonding/slaves
    

  2. Bring down the interface:

    sudo ifdown eth1
    
  3. Configure /etc/sysconfig/network-scripts/ifcfg-eth1 on each of the servers, changing the IPADDR field to the desired IP and network. Ensure the IP addresses are different on each server that you are attaching to the same VLAN. For example,

    DEVICE=eth1
    ONBOOT=yes
    HWADDR=e4:1d:2d:11:22:33
    IPADDR=192.168.1.2
    NETMASK=255.255.255.0
    NETWORK=192.168.1.0
    BOOTPROTO=none
    
  4. Bring up the interface:

    sudo ifup eth1
    
  1. Make sure eth1 has been removed from bond0:

    cat /sys/class/net/bond0/bonding/slaves
    

    If it hasn't been removed, remove it:

    echo "-eth1" > /sys/class/net/bond0/bonding/slaves
    

  2. Bring down the eth1 interface:

    sudo ifdown eth1
    
  3. Configure /etc/network/interfaces on each server, changing the IP address to the desired IP from your chosen block. For example,

    auto eth1
    iface eth1 inet static
        address 192.168.1.2
        netmask 255.255.255.0
    
  4. Bring up the interface:

    sudo ifup eth1
    

Attaching Multiple VLANs - Unbonded

To assign multiple VLANs to a port, navigate to the server's Network page. In the Layer 2 section, click Add New VLAN, which will allow you to assign more VLANs to eth0 or eth1.

Click Add to start the changes. Note that if you assign multiple VLANs at once, they are added through an asynchronous batch process, which begins immediately, but may take some time to complete.

The procedure for assigning a single additional VLAN to a port is the same as assigning the first VLAN. Send a POST request to the /ports/{id}/assign endpoint.

You have to specify the port ID in the path, and it needs to be the UUID for that port as returned by the /devices/{id} endpoint.

The ID of the VLAN is sent in the body of the request, and it can be either the VLAN's UUID as returned by the /projects/{id}/virtual-networks endpoint or the VLAN ID that is in the console.

curl -X POST \
-H "Content-Type: application/json" \ 
-H "X-Auth-Token: <API_TOKEN> " \
"https://api.equinix.com/metal/v1/ports/{id}/assign" \
-d '{
    "vnid": "c4032b18-5494-451b-a779-a7d3c536bfd7"
    }'

It is also possible to assign VLANs to a port in bulk, as part of an asynchronous batch process. Send a POST request to the /ports/{id}/vlan-assignments/batches endpoint an array of VLAN assignments in the body of the request.

curl -X POST \
-H "Content-Type: application/json" \ 
-H "X-Auth-Token: <API_TOKEN> " \
"https://api.equinix.com/metal/v1/ports/{id}/vlan-assignments/batches" \
-d '{
    "vlan_assignments": [
        {
            "vlan": "string",
            "state": "assigned"
        },
        {
            "vlan": "string",
            "state": "assigned"
        },
    ]
}'

In this scenario, IP packets that arrive at the server will have the VLAN ID populated. You will need to setup interfaces that will receive packets destined for each VLAN.

Note: If you need support for untagged packets, you can set a native VLAN for either port, since neither of them is part of a bond. More information is on the Setting a Native VLAN page.

  1. Install the prerequisites for VLANs:

    sudo modprobe 8021q
    sudo echo "8021q" >> /etc/modules
    
  2. Bring down eth1:

    ifdown eth1
    
  3. Configure /etc/sysconfig/network-scripts/ifcfg-eth1.1000 and /etc/sysconfig/network-scripts/ifcfg-eth1.1001 on your server. 1000 and 1001 should match the VLANs you've configured on the host in the portal. For example,

    DEVICE=eth1.1000
    BOOTPROTO=none
    ONBOOT=yes
    IPADDR=192.168.1.2
    PREFIX=24
    NETWORK=192.168.1.0
    VLAN=yes
    
  4. Restart networking:

    sudo ifup eth1.1000
    sudo ifup eth1.1001
    
  1. Install the prerequisites for VLANs:

    sudo apt-get install vlan
    sudo modprobe 8021q
    sudo echo "8021q" >> /etc/modules
    
  2. Bring down eth1:

    ifdown eth1
    

    Note: if you don't want eth1 to come up after a reboot be sure to comment out the eth1 configuration in your /etc/network/interfaces file.

  3. Add the new interface to /etc/network/interfaces. 1000 and 1001 should match the VLANs you've configured on the host in the portal. For example,

    auto eth1.1000
    iface eth1.1000 inet static
        address 192.168.100.1
        netmask 255.255.255.0
        vlan-raw-device eth1
    
    auto eth1.1001
    iface eth1.1001 inet static
        address 172.16.100.1
        netmask 255.255.255.0
        vlan-raw-device eth1
    
  4. Restart networking:

    sudo ifup eth1.1000
    sudo ifup eth1.1001
    

Example: Combining Hybrid and Layer 2 Modes

For this configuration you'll need two nodes, one in hybrid mode and one in Layer 2 networking mode, and one VLAN. The example assumes that the Hybrid mode server is in Hybrid Unbonded mode. The example also assumes that you have the server using exclusively Layer 2 in Layer 2 Unbonded mode, and assigns the VLAN to the eth1 interface.

  1. Attach the VLAN to the hybrid node's interface, eth1 in the case of a Hybrid Unbonded Mode.

  2. Attach that same VLAN to the server in Layer 2 mode. Remember, this node is in pure Layer 2 networking mode; there is no public connectivity to this server and you will have to use SOS to connect.

    If you get locked out, you can always change the networking mode back to Layer 3, or hybrid mode and SSH back in via the public IPv4 address.

While connected to SOS to the server in Layer 2 mode, edit the network interfaces file and remove all but the eth1 interface, which should be configured with its own private IP from whichever block you choose to use (e.g. 192.168.2.0/24). You'll also need to specify the gateway address as the hybrid node's IP address. This process varies by operating system.

  1. Tear down the bond0 interface:

    sudo ifdown bond0
    
  2. Configure /etc/sysconfig/network-scripts/ifcfg-eth1 with any free IP from the IPv4 private block used by eth1 on the hybrid node. Ensure that the netmask, network, and gateway details are correct.

    DEVICE=eth1
    ONBOOT=yes
    HWADDR=e4:1d:2d:11:22:32
    IPADDR=192.168.2.2
    NETMASK=255.255.255.0
    GATEWAY=192.168.2.1
    NETWORK=192.168.2.0
    BOOTPROTO=none
    
  3. Bring up eth1.

    sudo ifup eth1
    

    You can set the "ONBOOT" parameter for the rest of the network interfaces to "no" so they do not come up one reboots. bond0 will not be used, and eth0 will only be used if you choose to connect it to another VLAN (perhaps connected to other isolated node). In which case, it should be configured with its own IP accordingly.

  1. Tear down the bond0 interface:

    sudo ifdown bond0
    
  2. Configure /etc/network/interfaces with any free IP from the IPv4 private block used by eth1 on the hybrid node. Ensure that the netmask, network, and gateway details are correct.

    auto eth1
    iface eth1 inet static
        address 192.168.2.2
        netmask 255.255.255.0
        gateway 192.168.2.1
    
  3. Bring up eth1.

    sudo ifup eth1
    

    You can remove the other interfaces from this file. bond0 will not be used, but if you connect eth0 to another VLAN (perhaps connected to other isolated nodes) then configure it with its own IP, accordingly.

At this point your Hybrid mode server and your Layer 2 mode server node can talk to each other, but the Layer 2 server cannot reach the Internet. To give it Internet access you must configure IP masquerading on the Hybrid mode server.

  1. Make sure IP forwarding is enabled on the Hybrid mode server.

    sysctl net.ipv4.ip_forward=1
    
  2. Now add a new IP masquerade rule to the NAT table with iptables. We want this to route traffic from any of our private IPs through the Internet facing network interface on the Hybrid mode server, in this case, bond0.

    iptables -t nat -A POSTROUTING -s 192.168.2.0/24 -o bond0 -j MASQUERADE
    

Now your Layer 2 mode server should be able to ping outside the network.

ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp\_seq=1 ttl=120 time=1.85 ms
64 bytes from 8.8.8.8: icmp\_seq=2 ttl=120 time=1.93 ms
64 bytes from 8.8.8.8: icmp\_seq=3 ttl=120 time=1.87 ms
64 bytes from 8.8.8.8: icmp\_seq=4 ttl=120 time=1.86 ms
64 bytes from 8.8.8.8: icmp\_seq=5 ttl=120 time=1.81 ms

Converting Back to Layer 3

If you are in one of the Layer 2 modes and want to go back to Layer 3, you must first remove any attached VLANs.

To detach a VLAN in the console, navigate to the server's Network page. In the Layer 2 section, click Remove next to the VLAN you are detaching from the server. Confirm that you wish to remove it by clicking Yes.

Removing a VLAN

Note that detaching the VLAN from this server does NOT delete it from your project. The VLAN will continue to exist after detaching it from the server.

Then, to convert back to Layer 3, click Convert To Other Network Type, select Layer 3. Click Convert to Layer 3 to start the process.

Converting from Layer 2 to Layer 3

Frequently
asked questions

Slack

Still stuck?

Join us on Slack

Get support from our team and community members.

Frequently asked questions

What 3rd party Managed Services?

Neutra trust fund narwhal, palo santo bicycle rights live-edge stumptown health goth portland selfies kinfolk woke. ssss Echo park irony artisan vaporware typewriter. Next level pinterest drinking vinegar lyft mumblecore af small batch heirloom VHS. Fixie banh mi asymmetrical trust fund.

What 3rd party Managed Services?

Neutra trust fund narwhal, palo santo bicycle rights live-edge stumptown health goth portland selfies kinfolk woke. ssss Echo park irony artisan vaporware typewriter. Next level pinterest drinking vinegar lyft mumblecore af small batch heirloom VHS. Fixie banh mi asymmetrical trust fund.

What 3rd party Managed Services?

Neutra trust fund narwhal, palo santo bicycle rights live-edge stumptown health goth portland selfies kinfolk woke. ssss Echo park irony artisan vaporware typewriter. Next level pinterest drinking vinegar lyft mumblecore af small batch heirloom VHS. Fixie banh mi asymmetrical trust fund.

Copied code to clipboard