My intimacy issues with NVGRE – a NVGRE config guide on Hyper-V 2012 R2…. that actually works

Ever work with a protocol and delve into its inner-workings more than you ever wanted, more than you ever thought you needed to? That’s how I feel about NVGRE after the past two weeks.

I guess that is reality when you’re dealing with a new feature, newish protocol, not well documented and that runs on top of an operating system also going under lots of innovation (Hyper-V).

Overview

Let me just say that most NVGRE implementation guides are based on Hyper-V 2012 syntax — and, to be blunt, they’re outdated and didn’t even work for me on 2012 let alone 2012 R2.

For example, none of these examples will work on 2012 R2.

http://luka.manojlovic.net/

http://gallery.technet.microsoft.com/scriptcenter/Simple-Hyper-V-Network-d3efb3b8/view/Discussions

http://hikmatkanaan.wordpress.com/2013/03/28/windows-2012-hyper-v-3-0-network-virtualization/

They also are based on a single interface server, don’t provide any context for what the commands MEAN, and, as such, are not very educational. My assumption is – most people implementing Hyper-V virtualization are doing so either a) in a lab to test and/or b) on a server that has multiple interfaces — perhaps a 1g management and a 10g forwarding plane interface.

So – I will break down the configuration switches with some level of detail which will allow you to implement NVGRE on a multi-interface server. I will note that… I was not able to get NVGRE working on a multi-interface box, but I suspect that was an implementation specific issue in my lab. I will detail why I believe that later.

I could not get NVGRE working on 2012 (non-R2). First, let me note some changes I observed from both testing on 2012 and 2012 R2. It will add context to some of the other guides noted above:

  • In 2012 R2, it is not necessary to enable the ms_netwnv binding on the Interface.
  • The way 2012 vs 2012 R2 treats interfaces appears to be different also – most notably, how it “de-duplicates” physical and logical v-switch interfaces into a single interface
  • The firewall on Hyper-V R2 seemed far more aggressive and filtered much more than it did on 2012. I think I may have even had to disable it to do Windows Updates.
  • NVGRE related changes are not persistent – you lose them on reboot. I think the assumption is that you use a virtual machine manager like SCCM. This exercise was entirely based on PowerShell cmdlets, however.
  • In multiple guides, you’ll see a reference to “$NIC.InterfaceIndex”. Neither on 2012 or 2012 R2 was adding that text necessary.

Because of the poor documentation, I did a lot of trial and error, so hopefully this guide saves other people time and effort. The actual configuration of NVGRE is pretty simple and straight-forward once you understand it all.

What is NVGRE?

I think it is important to re-enforce the technical foundation before working with any protocol. After all, while you may be just doing a POC of the protocol as a learning exercise, it does have practical value in modern data centers and cloud provider networks.

A good video on why overlay networks are necessary is here: http://demo.ipspace.net/get/2.2%20-%20Need%20for%20Overlay%20Virtual%20Networks.mp4 . This really focuses on the downsides and networking challenges that led to overlay networks and not how overlay networks enable new services and capabilities in the data center. On the upside, the benefits of overlay networks is the ability to bridge the public and private cloud, the ability to extend L2 networks across site diversity while leveraging a common L3 core, etc.

What to expect — Architecturally, a virtual network (v-net) is isolated. One V-net can’t talk to another V-net without a L3 gateway. A V-Net also can’t talk outside of the V-Net (the Internet) without a L3 gateway.

Here are some good secondary readings:

Implementation

 Step 1: Configure a V-Switch

Just as you would normally do – configure a Virtual Switch in Hyper-V Manager on both servers.  Name it something obvious as you’ll need to look it up later. I used an External switch type. No special settings need to be used for the V-Switch – the defaults work. Put your VMs in the V-Switch.

When you build a V-Switch, I believe the physical interface will disappear from the sconfig.cmd shell and be replaced with the V-Switch name.

Step 2: Get physical and v-switch adapter IfIndex

The remaining steps should be run from a PowerShell command prompt (run powershell from a cmd window).

Run the command – Get-NetAdapter. Record the ifIndex of both the V-Switch you created in Step 1 and the physical Ethernet interface that the V-Switch is attached to.

Step 3: Disable the firewall – or modify it in a way that meets your organization’s security policy

From a command prompt (you’ll need to exit powershell)- run the command “netsh firewall set opmode disable“. This will disable the firewall. This command is deprecated but will still work on 2012 R2.

Step 4: Assign an IP address to the V-Switch

From the sconfig cmd shell in Hyper-V (accessed via console or Remote Desktop) assign an IP address, gateway, etc to the V-Switch interface.

This is a really good time to make sure you can ping your gateway and the other Hyper-V hosts in your topology!

Step 5: Create lookup records

Lookup records need to be created for every VM in the V-net on every Hyper-V host that has a guest in the V-Net. The command to do this is below.

New-NetVirtualizationLookupRecord

The parameters are as follows:

  • CustomerAddress: The IP address of the VM guest
  • ProviderAddress: The IP address of the Hyper-V host V-Switch interface
  • VirtualSubnetID: Most examples use 6001 but this is a numerical value representing the unique network
  • MACAddress: The MAC addres of the VM guest as one string — example “00155D0AB600″. You can look this up in Hyper-V manager
  • Rule:  ”TranslationMethodEncap”
  • VMName: Optional description of the VM guest

The parameters can be issued all at once on the command line or you can run the command and it will prompt you for input.

example: New-NetVirtualizationLookupRecord -CustomerAddress “1.1.1.11″ -ProviderAddress “10.196.10.182″ -VirtualSubnetID “6001″ -MACAddress “00155D0AB600″ -Rule “TranslationMethodEncap” -VMName rimayber-ubuntu-nvgre-11

I think this command effectively just means that Hyper-V will present an ARP entry to your guest OS for the other hosts in the V-Net when an ARP lookup is performed.

Step 6: Add customer routes

The customer route has to be on each Hyper-V host. The command is as follows:

New-NetVirtualizationCustomerRoute

The parameters are as follows:

  • DestinationPrefix: The “customer prefix” – the network used within the V-net
  • NextHop: I used  ”0.0.0.0″ –  I assume this means it follows the default gateway route that was assigned in Step 4
  • RoutingDomainID: A unique identifier for the routing domain. Most examples use “{11111111-2222-3333-4444-000000000000}”
  • Metric: I used 255
  • VirtualSubnetID: Same numerical value used in Step 6 and Step 8

Example: New-NetVirtualizationCustomerRoute -DestinationPrefix “1.1.1.0/24″ -NextHop “0.0.0.0″ -RoutingDomainID “{11111111-2222-3333-4444-000000000000}” -Metric 255 -VirtualSubnetID 6001

Step 7: Add Provider configuration

The provider concept is the endpoint for encapsulation/decapsulation – it is the Hyper-V host where the guest OS/VM resides. To set up provider routing, use the following two commands once on every host with guests in the V-Net.

New-NetVirtualizationProviderAddress takes the following parameters:

  • InterfaceIndex: The IfIndex of the physical interface
  • ProviderAddress: The IP address of the Hyper-V host V-Switch interface
  • PrefixLength: The numerical subnet length value (e.g: 24 )

and

New-NetVirtualizationProviderRoute takes the following parameters:

  • InterfaceIndex: The IfIndex of the physical interface
  • DestinationPrefix: I used all zeros 0.0.0.0/0
  • NextHop: The L3 gateway of the V-Switch interface

Example commands:

New-NetVirtualizationProviderAddress -InterfaceIndex 14 -ProviderAddress “10.196.10.182″ -PrefixLength 25

New-NetVirtualizationProviderRoute -InterfaceIndex 14 -DestinationPrefix “0.0.0.0/0″ -NextHop “10.196.10.192″

Step 8: Put VM guest in virtual network

Putting an interface into a VirtualSubnet is rather easy. You issue one command to get the interface and pipe the output to another command to set the V-net.

Get-VMNetworkAdapter <VMName> | Set-VMNetworkAdapter -VirtualSubnetId <VirtualSubnetID used in steps above>

Run this for every VM instance you want in the V-Net. This is also the simplest way to move a guest OS to another V-Net in order to do some negative testing to ensure one V-Net can’t bridge to another.

Step 9: Test

If you’re using two Hyper-V hosts, then ensure your guest IP addresses are configured correctly, and you should be able to ping from guest to guest. Issuing an “arp -a” on each host is another way to ensure that at least remote guest ARP entries are visible (means your Customer Lookups are present).

Useful commands

If you can’t ping across – check and make sure there’s no firewall on the guest OS.

You can also verify the config on each Hyper-V host with the following commands:

  • Get-NetVirtualizationLookupRecord
  • Get-NetVirtualizationProviderAddress
  • Get-NetVitrualizationProviderRoute
  • Get-NetVirtualizationCustomerRoute
  • Pipe commands through a “| fl” command to get additional detail or overcome command window limitations

On the Get-NetVirtualizationLookupRecord – pay special attention to making sure your MAC addresses are correct. You can verify MACs in Hyper-V manager or by running ifconfig in Linux.

Caveats

The primary issue I struggled with was getting NVGRE to work across a secondary server interface. In my setup, I had a 1g NIC and a 10G NIC. The 10G NIC was totally isolated in its own topology isolated from the Internet and connected to a Cisco Nexus 3064. I was able to get NVGRE working on hosts with two interfaces (the 1G interface), however, it never worked on the isolated 10G interface. I suspect that this is possibly because that interface was “degraded” (Microsoft lingo evidenced in Get-NetAdapter | fl” — which I think is just a word for not being able to reach the Internet.

Leave a Reply


× 6 = eighteen