First-hop security concerns for multi-tenant clouds

I did some work recently assessing how Microsft’s Hypervisor (Hyper-V) addresses (or doesn’t address) some common network security based threats in a multi-tenant public cloud environment. I then set out to test whether there are controls in place to mitigate those threats. I would have liked to compare different Hypervisors and their capabilities, but I have yet to do that comparison. If you’re interested in testing independently, there are several good tools out there including nmap, yersinia, hping and scapy. Those four tools will allow you to test every scenario in this document. Some detail on the test case is provided in each section.

First, some basic definitions that I use in this post:

  • TOR switch: Top of rack switch
  • Switch: Generally references the network switch and not the virtual switch within the hypervisor
  • Tenant: Analogous to a guest operating system/VM
  • Host: The Hypervisor
  • Guest: A virtualized operating system running within the hypervisor

Architecture context

The two main control points to enforce security are the hypervisor (Hyper-V) and the L3 TOR switch. Additional technical context is needed to understand the potential threats and those details are provided below:

Hyper-V

A Hyper-V host likely most resembles the function of a switch with filtering capabilities. It generally does not play the role of a gateway device (router) and it is assumed it also provides capabilities that make it look less like a bridge. It is assumed that tenants share a common broadcast domain and thus multicast, Ethernet and IP broadcast packets will be visible to all tenants (guests) on the host. It is also assumed that similar traffic sourced by a guest/tenant would also be visible and reach the switch.

While the threat perspective concerned here is focused on protecting the switch, it is also possible that these attacks could be targeted at the host/Hyper-V itself.While the blast radius of a L3 TOR switch is far greater than a Hyper-V host, it is still possible that one tenant could disrupt services to many other tenants.

The scenario tested in this document was Hyper-V 2012 R2 in its default configuration (Windows Firewall completely disabled) and no VLANs, VLAN tagging or VLAN trunking configured unless noted otherwise. The ‘allow management operating system to share this network adapter’ configuration was not enabled during testing and no guests were configured for NVGRE encapsulation.

Network Switch

Generally switches will have three primary planes. First, the management plane. While this interface serves as the primary path for management via a dedicated management interface, depending on switch architecture, management features may also be accessible via “in-band” interfaces (gig/10g interface facing tenants/hosts). This makes it necessary to intentfully secure management services beyond just management network isolation. The second plane, the data plane, represents the main forwarding engine(s) and forwarding path for customer and service traffic. The third plane, the control plane, is where numerous functions exist such as routing protocol services, etc. It is assumed that the control plane and management plane will share the same computing resources in a commoditized ToR switch.

As the industry further commoditizes the L3 TOR switch, the processing capability/packet processing/general performance of the control/management plane will remain comparatively low in comparison to a devices total data plane forwarding capacity. Therefore, even small packet rates (sub-100,000 pps) hitting the control-plane will overwhelm the control plane. And, for the purposes of this post, it will be assumed that a control-plane issue will also impact the switches ability to forward traffic. Therefore, we must rigorously protect the control plane.

It is also important to discuss at least one vendor’s architecture to understand what traffic typically hits the control-plane. As an example, the Cisco Nexus 3K will send the following packets to the control-plane to be software processed: 1) Packets that require an ICMP response such as ICMP TTL exceeded, ICMP unreachable, MTU failures, etc. 2) Responses to packets directed at a “self-ip” local to the switch 3) Management functions such as SSH, AAA or SNMP 4) Routing protocols or control protocols (BGP, BFD, CDP, LLDP (true?), etc) 5) DHCP or DHCP relay packets and 6) ARP glean messages – the first time an ARP entry is populated on a switch.

It is assumed that specific control-plane punting behavior will vary from switch to switch. It is also assumed that switches will have multiple “self IPs” in many different networks and that generic blocking of known management network space (e.g: 10/8) is not a sufficient security control as the same services/protocols may exist on different IP addresses.

Risk management

Several approaches should be taken to minimize the threats documented in this post. First and foremost, control-plane protection mechanisms on the switch should be in place and enforced rigorously. The fewest amount of protocols and packets (ideally, but not realistic, zero in both cases) should be allowed to reach the control plane through the enforcement of switch control-plane protection technology. Second, we assume that control-plane protection mechanisms are not consistent vendor to vendor and the hypervisor must specifically filter traffic and prevent it from reaching the switch. Third, fuzzing/security testing on protocols that are accessible between tenants and the network must be fully executed on all switch platforms. A fourth approach, monitoring, can also be used to measure the amount of traffic hitting the control plane with alerting when excessive control-plane traffic is detected.

Threats

Threats are broken into three categories as follows:

  • “Information leakage” from the switch to a guest: Information advertised by a switch that a guest could see to gather information about a network. These are likely due to misconfiguration and/or caused by Hyper-V not properly managing its ‘broadcast domain’
  • L2 attacks from guests: Guests may be able to generate Ethernet or other L2 frames that require switch control-plane processing
  • L3 attacks from guests: Guests can also source L3 packets directed at any part of the infrastructure including IP address space not routable to the Internet

Information leakage threats

Routing protocol control messages

OSPF multicast messages

OSPF is a routing protocol common in many networks. OSPF uses a well-known multicast based Hello protocol to find adjacent neighbors. If a guest can form an OSPF adjacency then they could disrupt or intercept network forwarding.

Test plan: Enable OSPF on the northbound router and activate OSPF on the L3 interface directly connected to Hyper-V. Run tcpdump on a guest instance and watch for OSPF packets.

Expected result: Generally if Hyper-V is acting as a switch, then it should not forward multicast traffic to guests. This attack is also usually mitigated by running passive-interface default on the switch.

Test result: Hyper-V 2012 R2 guest operating systems will see OSPF multicast packets in default Hyper-V configuration. This means that if OSPF was running on an interface that a guest could likely also negotiate a neighbor relationship with the northbound switch.

L2 Control Messages

Link Layer Discovery Protocol (LLDP)

LLDP uses a well-known Ethernet MAC address and can be transmitted from the switch and can include a variety of information, but typically carries physical network adjacency information. An attacker could use this information to map a cloud provider’s network.

Test plan: Enable LLDP transmit/receive on the interface connected to Hyper-V and enable LLDP globally on the switch. Run tcpdump on a guest instance and watch for LLDP packets.

Expected result: Generally if Hyper-V is acting as a switch, then it should not forward LLDP traffic to guests. LLDP is enabled on every interface by default on a Cisco Nexus 3K when LLDP is enabled globally.

Test result: Hyper-V 2012 R2 guest operating systems do NOT appear to see LLDP frames in default Hyper-V configuration.

Note: UDLD is a similar Cisco proprietary protocol. Hyper-V 2012 R2 guest operating systems will see UDLD frames in default Hyper-V configuration. UDLD uses a well-known Ethernet MAC address.

Cisco Discovery Protocol (CDP)

CDP is similar to LLDP but is a Cisco proprietary protocol. CDP uses a well-known Ethernet MAC address and can be transmitted from the switch. CDP contains similar information to LLDP and has a similar threat scenario.

Test plan: Enable CDP on the interface connected to Hyper-V and enable LLDP globally on the switch. Run tcpdump on a guest instance and watch for CDP packets. Also oFn the guest, use the “Yersinia” toolkit to generate fake CDP messages towards the Switch.

Expected result: Generally if Hyper-V is acting as a switch, then it should not forward CDP traffic to guests.

Test result: Hyper-V 2012 R2 guest operating systems will see CDP frames in default Hyper-V configuration. A guest can also send frames that will be received by the switch and populate its CDP table.

Multicast

IGMP/multicast join messages forwarding multicast

IGMP is used to signal a systems intent to join a specific multicast group. An IGMP message is typically sent to the systems L3 gateway. The L3 gateway is typically a member of a PIM domain and will forward the multicast traffic for the group that was identified in the IGMP join. All hosts within the broadcast domain will likely see the multicast messages even if they did not specifically join the multicast group. These multicast messages may contain information that should never be observed by a guest.

Test plan: Enable IGMP and PIM on the L3 interface. Using iperf, source multicast messages as a sender and run tcpdump and iperf receiver on another guest and hypervisor in the same broadcast domain. Ping multicast addresses on one guest as well.

Expected result: Multicast messages will be seen within the broadcast domain

Test result: PIM Hello’s were seen on the guest. Multicast sourced on one guest was not visible on the other guest. ICMP destined at Multicast addresses was seen on both guests. This seems unusual. IGMP joins were also not seen between guests.

L2 attacks from guests

ARP storms

ARP storms can be caused by guests generating excessive amounts of ARP broadcast packets. These storms can impact network bandwidth as well as control-plane processing. ARP packets will be seen by every host in the Ethernet broadcast domain. Switches may have MAC address table limitations that when exhausted will cause the switch to forward all packets out all interfaces essentially acting as a hub.

Test plan: Run the “macof” tool from the dsniff distribution on a Linux server. The tool will generate a large volume of ARP requests.

Expected result: With MAC spoofing prevention disabled, the switch should not be flooded with ARP packets.

Test result: Switch was protected by MAC spoofing prevention.

Link aggregation control protocol (LACP)

LACP is a Layer 2 control protocol that can be used to automatically detect, configure and manage port aggregation. LACP negotiation may disrupt port state and cause forwarding

Test plan: Create a port-channel. On the interface connected to the Hyper-V host, put the interface into a port-channel with timers fast and LACP mode active. This should enable LACP packets to be transmitted. Very packets are transmitted using switch show commands. Run tcpdump on the guest and look for LACP packets.

Expected result: LACP packets should not be forwarded or seen by guest operating systems.

Test result: LACP packets were not observed.

VLAN Tag Hopping

A Hyper-V guest may add or one more VLAN tags which will allow it to participate or forward traffic into an unintended VLAN (such as management). This may provide the guest with access to infrastructure or networks that should not be accessible.

Test plan: Create multiple VLANs on a switch. Put the interface connected to the Hyper-V server into a switch-port mode trunk and allow all VLANs to be trunked. Configure Linux to perform 802.1q encapsulation and bring up multiple L3 interfaces. Ping the VLAN L3 addresses and look for a response.

Expected result: Hyper-V should not allow 802.1q encapsulated traffic to pass.

Test result: Hyper-V did not forward 802.1q encapsulated traffic.

STP BPDU

Spanning tree protocol is used to determine a loop free L2 forwarding topology. STP attacks can cause topology changes that may impact the forwarding path.

Test plan: Using Yersinia, generate STP BDPU flood packets – there are a variety of attack type options. Attempt all attack types.

Expected result: It is assumed Hyper-V would not filter STP frames.

Test result: No STP BPDU packets were observed on the destination switch. Ports did not block or become disabled.

MAC hijacking

MAC hijacking is when one guest impersonates the MAC address of another guest in order to perform a man in the middle attack or disrupt services. This may allow an attacker guest to intercept communication destined for another guest.

Test plan: Run the following commands to change the MAC address on a Linux server. Ifconfig eth0 down; ifconfig eth0 hw ether <spoofed mac address>; ifconfig eth0 up. Then ping your L3 gateway address and see if ping responses are received.

Expected result: MAC address spoofing should be disabled by default.

Test result: Hyper-V 2012 R2 guest operating systems can change their MAC address but Hyper-V will not forward those packets in its default configuration. The default configuration is to prevent MAC address spoofing.

Preventive Control: Hyper-V has a PowerShell cmdlet that prevents MAC address spoofing.

Set-VMNetworkAdapter  -MacAddressSpoofing Off

MAC spoofing/MAC overflow attack

MAC spoofing attacks are when a guest spoofs its MAC address in order to evade detection/traceability, impersonate another guest or to attack finite resources on a switch like the Cisco CAM table. Some switches may forward non-broadcast traffic out all interfaces if these resources are exhausted.

Test plan: Run the following commands to change the MAC address on a Linux server. Ifconfig eth0 down; ifconfig eth0 hw ether <spoofed mac address>; ifconfig eth0 up. Then ping your L3 gateway address and see if ping responses are received. Another command that can be used is dsniff’s “macof –i eth0” tool.

Expected result: MAC address spoofing should be disabled by default.

Test result: Hyper-V 2012 R2 guest operating systems can change their MAC address but Hyper-V will not forward those packets in its default configuration. The default configuration is to prevent MAC address spoofing.

Preventive Control: Hyper-V has a PowerShell cmdlet that prevents MAC address spoofing.

Set-VMNetworkAdapter  -MacAddressSpoofing Off

Ethernet broadcast/multicast

Ethernet broadcast/multicast traffic may require software processing and be punted to a software path. This could cause resource utilization issues on a switch.

Test plan: Using Scapy, generate packets with broadcast and multicast destination Ethernet addresses. Optionally add IP info. Ensure source and dest of packets w/ IP are not the IPs of either guest OS.

Expected result: It is expected that these packets will be seen on the broadcast domain from one guest to another.

Test result: Packets with both multicast and broadcast addresses were seen across guests in the same broadcast domain. Hyper-V did not isolate those packets.

Malformed or foreign L2 packets

Some unknown L2 packets may require software processing and be punted to a software path. This could cause resource utilization issues on a switch.

Test plan: Untested.

Expected result: It is assumed that Hyper-V does not perform any protocol anomaly detection and would forward the traffic.

Test result: Untested.

L3 attacks from guests

DHCP storms/pool exhaustion

A DHCP storm is used to drain a DHCP server of available IP addresses preventing allocation of IP addresses to new guests.

Test plan: Using the Yersinia attack toolkit, execute the DHCP type 1 attack for DISCOVER exhaustion.

Expected result: It is assumed that Hyper-V does not play a role in DHCP allocation or throttling of requests, but the DHCP itself server may have an abuse prevention control preventing this attack.

Test result: DHCP flood was generated and forwarded to DHCP server as expected. This has an impact on the L3 switch acting as a DHCP relay.

Rogue DHCP server attack

A rogue server is a DHCP server on a network which is not under the administrative control of the cloud provider A rogue DHCP server may respond to a DHCP message identifying itself as the default router allowing it to intercept communication from another guest.

Test plan: Using the Yersinia attack toolkit, execute the DHCP type 2 attack for acting as a rogue DHCP server.

Expected result: With DHCPGuard enabled, the server performing the DHCP discover should only receive a DHCP packet from the well-known DHCP server.

Test result: DHCP response was only received from well-known DHCP server.

Preventive Control: Hyper-V has a PowerShell cmdlet that prevents guests acting as rogue DHCP servers.  This can also be enabled within the v-switch manager. This feature is disabled by default.

Set-VMNetworkAdapter  -DhcpGuard -On

Router advertisement attacks

Router advertisements in IPv6 are used to announce the default gateway to IPv6 hosts. Rogue RAs can be used to intercept communication from another guest. This can also be enabled within the v-switch manager.

Test plan: Send a malicious router advertisement message on a v6 enabled network. The Linux daemon radvd can be used to generate the router advertisement.

Expected result: Hyper-V with RouterGuard enabled should prevent the malicious router advertisement from being forwarded.

Test result: Hyper-V behaved as expected with the RouterGuard feature on. ICMPv6 multicast packets were still observed but appeared to be truncated. Full RAs were seen with the feature off.

Preventive Control: Hyper-V has a PowerShell cmdlet that prevents RA packets from being sent by guests. This can also be enabled within the v-switch manager. This feature is disabled by default.

Set-VMNetworkAdapter  -RouterGuard on

IP spoofing

A guest can potentially source IP packets using IP addresses that weren’t assigned to it – also known as spoofing. There are several techniques to prevent this, but the most secure technique ensures only packets that have a valid, previously provisioned, MAC:IP binding are forwarded. The cable industry uses an approach called DHCP Lease Query to prevent this.

Test plan: From an Ubuntu server running as a guest on one Hyper-V host, run the command “nmap –e eth0 –S <source ip> <destination ip>” where destination IP is the address of another Ubuntu server running on second Hyper-V host.

Expected result: One would assume that Hyper-V has IP spoofing prevention mechanisms.

Test result: Spoofed traffic reached the destination server unexpectedly with the spoofed address preserved.

IP broadcast/multicast

IP broadcast/multicast traffic may require software processing and be punted to a software path. This could cause resource utilization issues on a switch.

Test plan: Using HPING, send TCP traffic to the local broadcast address. Send UDP traffic to a well-known multicast address. On another Hyper-V instance connected to the same VLAN, run tcpdump and observe traffic.

Expected result: It is assumed Hyper-V will forward and receive IP broadcast and multicast traffic.

Test result: Both broadcast and multicast traffic was observed on the second Hyper-V host.

Exposed services on switch

An Ethernet switch may expose IP services on non-management interfaces that can be attacked. The guest attacker may have more access to services than a typical Internet user due to their directly connected nature. Examples of services exposed range from DHCP to SSH.

Test plan: Perform nmap –p 1-65535 from guest instance on Hyper-V to L3 gateway next hop.

Expected result: Hyper-V generally is not responsible for protecting the upstream router’s control plane.

Test result: Control-plane services including management functions on switch are open to nmap scan. On the Nexus 3000, TCP/179 (BGP) and TCP/161 (SNMP) were open.

Topology exposure including management networks

Guests may have access to networks that are isolated within the cloud providers’ routing AS and not globally Internet reachable. Therefore, typical approaches using Internet perimeter security are not effective.

Test plan: This is not testable in the lab.

Expected result: It is assumed that infrastructure including RFC1918 addressed infrastructure is accessible.

Test result: It is assumed that infrastructure including RFC1918 addressed infrastructure is accessible.

TTL=1 flooding

An attacker can send an IP packet with TTL that is punted to the control plane in order for the control plane to send an ICMP time exceeded message.  This attack is used to cause excessive utilization on the switch’s control-plane.

Test plan: Using hping on an attacking system that is a Hyper-V guest, send TTL=0 and TTL=1 packets to a Hyper-V guest on another host in the same VLAN and to a destination on a different VLAN. On the destination server, observe if packets are received. On the switch, observe TTL=1 counters.

Expected result: Hyper-V likely should not play a role in filtering packets with TTL=1. But, one may expect it to drop packets with TTL=0.

Test result: Packets with TTL 0 AND TTL 1 were received by the guest on the other Hyper-V host in the same VLAN. When a remote destination was selected, the switch dropped all packets. Specifically for the switch: TTL=1 packets matched control plane counters and were rate-limited. Packets with TTL=1 were counted as “Bad ttl” in “show ip traffic” and did not match control plane rate-limiters. Hyper-V did not drop packets with TTL=0 sourced by guests.

Leave a Reply


three + 5 =