Wednesday, October 2, 2013

Openstack Grizzly with Nicira NVP Plugin Provider Router Use Case with NAT

This post will review a common use-case for Openstack Networking and show step by step how to achieve the requirements for this use case using Openstack Networking in the Grizzly release with the Nicira Network Virtualization plugin for Openstack.

The specific use-case we will review is a Tenant network with a Provider Logical Router to reach the "External" network for access to the internet and corporate networks. The configuration of both Openstack Grizzly and Nicira Network Virtualization Platform or NVP is assumed to be existing and configured correctly. If there is enough interest I will cover these setups in a future post.

Starting Point

For the purposes of this example we are starting with a clean installation / configuration of NVP 3.1 components including an NVP Gateway Appliance and the NVP plugin for Openstack Networking installed and configured. Detailed instructions for this can be found in the Openstack Networking Administration Guide here
The Openstack Grizzly configuration is a simple single Openstack Controller on Ubuntu 12.04 running Nova, Keystone, Quantum, Glance, and Cinder. A second Ubuntu 12.04 system is used for KVM and nova-compute. Openvswitch is installed on both the Openstack Controller and the KVM Hypervisor.
At this point we are able to provision Virtual Machines, Networks(through Quantum) and provide DHCP services through quantum-dhcp-agent running on the Controller node.

NVP Plugin L3 Configuration

The first step is to edit the Quantum NVP Plugin Configuration and add the UUID of the NVP Layer 3 Gateway Service. You can obtain the UUID from the NVP Manager UI.

root@ubu-osctl02:/home/nicira# nano /etc/quantum/plugins/nicira/nvp.ini

Openstack Quantum Tenant Network Creation

  1. Source the credentials for user1 in the development Project.
  2. Create an internal network in the development project
  3. Create a subnet for the internal network in the development project.

Openstack Quantum External Network Creation

  1. Source the credentials for admin.
  2. Create an External network that corresponds to your Physical network
  3. Create a subnet for the External Network

Openstack Quantum Router Creation

  1. Create a Router called corp-external
  2. Create the External Interface for the dev-external router to connect it to the "shared-external" network created earlier.
  3. Create the Internal Interface for the dev-external router to connect it to the "dev-internal" network created earlier. 


Thursday, August 1, 2013

VMworld San Francisco 2013

I will be presenting two Network Virtualization sessions in San Francisco and aside from all of the crazy schedules and customer work leading up to VMworld I am really starting to feel the buzz surrounding the NSX launch scheduled for this year.

My first session is NET5516, titled "An Introduction to Network Virtualization" with Eric Lopez.  In this session, we will provide an instruction course on VMware NSX for Multi-Hypervisor. The first half of the session will establish the necessary foundation for what Network Virtualization is and how it is achieved as well as the architecture and system components that comprise VMware NSX for Multi-Hypervisor.  Once we have layed the foundation and established some common terminology, we will use Whiteboard technology to interactively design a Logical Network Virtualization Architecture to meet a set of business requirements.  I am really looking forward to this session as there is not a lot of information out there on NSX or even Network Virtualization in general.

My second session is NET5790, titled "Operational Best Practices for NSX in VMware Environments" with Ray Budavari.  This session we will provide a brief overview of the NSX vSphere architecture and components in order to provide a foundation for understanding the operational impact of these technologies. We will then focus on the day-to-day tasks such as automation, backup and recovery, logging, monitoring, and troubleshooting. These tasks are described in the context of core features such as VDS, VXLAN, NSX Controller, NSX Manager, Distributed Routing, Firewall and NSX Edge.  I am also excited about this session because we are planning to leverage demonstrations to provide examples of real world scenarios and that is always a crowd pleaser if you can pull it off

Tuesday, October 16, 2012

Four pillars of Network Virtualization

Outside of finishing up some of my existing projects and VMworld responsibilities, I have been focused on the newly acquired Nicira's Network Virtualization Platform or NVP. Whenever I examine a new or disruptive technology such as Cloud or Network Virtualization, I like to put myself in the role of a customer. Basically, before I invest a lot of time figuring out the "How" I like to try to understand the "Why" as in "Why is this technology relevant or beneficial to my customers?"

Usually as I begin to understand the "How" of a particular technology, I am able to piece together the "Why". In configuring and understanding NVP, I started to notice parallels to messaging used in the early days of VMware to explain to customers the "Why" of virtualization.  Does anyone remember this slide ?
Sometimes referred to as the four pillars or principles of virtualization, this slide embodies the highest level benefits of core virtualization. Many other specific benefits like simplified recovery or vMotion are possible as a result of these underlying capabilities.

I think these four fundamental benefits can also be realized and are relevant in the context of network virtualization leveraging Nicira NVP. Lets go through them and see if you agree.
  • Partitioning - Enables an underlying Layer 3 physical or transport network to be carved up into isolated Layer 2 segments or logical networks for individual tenants or application tiers.
  • Isolation - Provides separate Layer 2 Broadcast domains in logical space otherwise known as logical networks which are completely separate from other logical or physical networks.
  • Encapsulation - A distributed Flow table on each transport node can encapsulate a complex network configuration for a VM from L2 to L4. In traditional networking architecture, this configuration or state would normally exist across multiple networking devices including switches, routers, and firewalls, Encapsulation could also refer to the tunneling mechanism used by the solution but this is only one piece of the NVP solution.
  • Hardware Independence - The abstraction of the physical or transport network and the ability to provide the same functionality at the logical layer eliminates the need for much of the functionality previously provided by specific vendors at the Physical network layer. Of course you are dependent on the underlying physical or transport network so you still need to provide a resilient and performant Layer 3 Networking fabric but you are less dependent on specific vendors or features.
While there may be other aspects of network virtualization which are important like centralized management, I think the four above could be considered the minimum. I will explain the components and fundamental architecture of the Nicira Network Virtualization Platform in an upcoming post to answer the "How". 

Thursday, October 11, 2012

VMworld TV spot in Barcelona

I had an excellent and very productive trip to Barcelona for VMworld Europe. In between sessions I recorded a quick VMworld TV spot to discuss Nicira network virtualization with my colleagues Kamau and Chris. . . I will update with more experiences from VMworld Europe when I recover from the jet lag and sleep deprivation.

Thursday, October 4, 2012

vCloud Director API Extensions Part 3 of 3 - VCO

In the previous two posts in this series I introduced how to Register our API Extension in vCloud Director and How to setup our RabbitMQ AMQP server to handle or move the messages generated by our Extension Service.  In this post, we will look at the last piece of the puzzle, which is taking the relevant data from the AMQP message and actually doing the work of our "faulttolerance" Extension Service.  Relatively speaking this is the hard part, which is why it makes so much sense to use vCenter Orchestrator for this step.  Because you need to communicate with a system that is external to vCloud Director but maintain correlation to objects in vCloud Director VCO is the perfect tool for the job.

Using VCO as the extension service

The benefits of using VCO to provide the external service for our vCloud Director API Extension are:
  1. VCO has an AMQP plugin that provides simple workflow driven configuration options and built in message consumption and publishing.
  2. VCO has many VMware and third party plugins that you can leverage to provide a multitude of Extension Services.
  3. VCO has a graphical based IDE and intuitive interface, which lower sthe learning curve for developing against external API’s.
  4. VCO IDE leverages javascript as its scripting language, which can natively handle and parse JSON, which is the format of our AMQP message bodies.
For this example, I used the vCenter Orchestrator Virtual Appliance version 5.1 and then installed the vCloud Director 5.1 and AMQP Plug-ins.  I have provided access to the actual FT API Extensions workflow package file, which contains the VCO FT API Extension workflow itself.  


Step 1 – Configure the VCO AMQP plugin

In order to process messages from an AMQP queue we need to add the AMQP broker to the VCO inventory and then subscribe to the queue that we have created in RabbitMQ to receive messages for our Extension Service.  We could also have created the queues and the bindings between the AMQP exchange and queue from within VCO using the workflows that ship with the AMQP plug-in.  In this example, we have already done that using the Management plugin for RabbitMQ.
  1. In the workflows tab from the Design or Run context navigate to the LibraryAMQPConfiguration folder and run the “Add a broker” workflow.
    1. Enter a Name for the AMQP server.
    2. Enter the connection information for the AMQP server or in this case RabbitMQ server.
      1. Host= hostname or IP of AMQP server
      2. Port= 5672
      3. Virtual host= /
      4. Use SSL= No
      5. User name=guest and Password=*****
  2. In the workflows tab from the Design or Run context navigate to the LibraryAMQPConfiguration folder and run the “Subscribe to queues” workflow.
    1. Enter a Name for the subscription.
      1. Name=”ftqueue API Extension”
  3. Enter the connection information for the AMQP or in this case RabbitMQ server.
    1. Broker=Navigate to the AMQP broker we just added in the previous Post.
    2. Queues=(Type the name of the ftqueue queue with correct capitalization)

Step 2 - The FT API Extension VCO Workflow

If you have completed the steps in the first two posts in vCloud Director and RabbitMQ you can now just download and import the FT API Extensions workflow into vCenter Orchestrator and move onto Step 3 below to Create a VCO subscription policy. If you are interested in the code and logic that was used to create the workflow and what is happening within each Workflow element I will discuss that in painstaking detail in this seciton.

When designing the workflow I had an idea in mind that I wanted to enable Fault Tolerance on a vApp based on a Service Link registered for our API Extension.  Logically I worked backwards from this objective looking at the enable and disable FT workflows that ship with VCO to determine which parameters I would need to grab from the AMQP message being sent from the API Extension.  The workflow is the piece of the solution that will actually do the work. It is designed as follows.
Workflow Input
A single Input Parameter will be used and comes from the AMQP subscription policy we created so there will be no User Input required. All parameters needed such as the VCD host, target vApp, and whether FT should be enabled or disabled will come from the AMQP message body and headers. 

Workflow Schema
Below we see the “FT API Extension” workflow schema. Lets walk through each element to understand what is going on behind the scenes. 

Workflow Element 1 - Process Message (Scriptable Task)
Input=(AMQP:Subscription) subscription
Output=(vCloud:vApp)vApp, (Boolean)ftstate
This is where the parsing of the message components takes place with all of the parameters and information we need being taken from the message payload or message body, which we have already said is JSON. The picture below shows a message from our vCloud Director API Extension Service that is in the ftqueue using the RabbitMQ Management plugin.
Remember the goal is to ultimately hand off an Array of “VC:VirtualMachine” objects and a boolean value that represents whether the user wants to Enable or Disable Fault Tolerance.  We need these two objects because we are using Enable FT and Disable FT as nested workflow elements and these are required input parameters for these workflows. Running the Payload through a JSON validator shows the layout a little more clearly so we can identify what fields we need to hand off to the next workflow elements. 
        "method": "POST",
        "id": "85ea26af-f32a-42fb-b51a-fad0e90e811d",
        "scheme": "https",
        "protocol": "HTTP/1.1",
        "headers": {
            "Authorization": "Basic YWRtaW5Ac3lzdGVtOnZtd2FyZQ==",
            "Host": "",
            "Content-Length": "0",
            "x-vcloud-authorization": "NyJ+CcY18pUx2MlsCAO7CxgvqOQmfs8llOZR1pK5S2w=",
            "User-Agent": "Apache-HttpClient/4.2 (java 1.5)",
            "Connection": "keep-alive",
            "Accept": "application/*+xml;version=5.1"
        "statusCode": 0,
        "queryString": null,
        "localPort": 443,
        "localAddr": "",
        "remoteAddr": "",
        "remotePort": 58029,
        "request": false,
        "body": "SGVsbG8gQVBJIQ==",
        "requestUri": "/api/vApp/vapp-ca90bfdd-93f7-4b24-a8ca-14bf5ba27fd1/ft/off"
        "parameters": null,
        "user": "urn:vcloud:user:241b78de-a35a-46e6-8d7c-4afc86693b71",
        "org": "urn:vcloud:org:a93c9db9-7471-3192-8d09-a8f7eeda85f9",
        "rights": [
……(Some rights removed to conserve space)

The actual Javascript from the Scripting Tab of the element is below.

var props = subscription.retrieveLastOnMessageTrigger();
var Message = props.get('body');  
var headers = props.get('headers'); 
var properties = props.get('properties'); 
// From api-messages
var amqpMessage = eval(Message);
var httpMessage = amqpMessage[0];
var httpSecurityContext = amqpMessage[1];
var context = amqpMessage[2];
var vcdHostUri = "https://" + httpMessage.headers.Host;
var hRef = httpMessage.requestUri;
var cleanUri = hRef.substring(0,51);
var vcdHost = VclHostManager.getHostByUrl(vcdHostUri + ":443","admin","system");
var vApphRef = (vcdHostUri + cleanUri);
System.log("Object Type amqpmessage: " + System.getObjectType(amqpMessage));
System.log("AMQP Message Body: " + Message + "\n");
System.log("Message Method: " + httpMessage.method);
System.log("Message ID: " +;
System.log("Message Protocol: " + httpMessage.protocol);
System.log("Request URI: " + httpMessage.requestUri);
System.log("Message Authorization: " + httpMessage.headers.Authorization);
System.log("localPort: " + httpMessage.localPort);
System.log("JSON Host: " + httpMessage.headers.Host);
System.log("Scheme: " + httpMessage.scheme);
System.log("Is Request?: " + httpMessage.request);
System.log("VCD Host Uri: " + vcdHostUri);
System.log("vApp only hRef: " + vApphRef);
System.log("Full Extension API Call hRef: " + hRef);
System.log("Clean 20 character URI: " + cleanUri);
System.log("VCL Derived VCD Host: " + vcdHost);
//Original Code
if (vApphRef == null || vApphRef == ""){throw "cannot proceed without vApphref";}
if (hRef == null || hRef == ""){throw "cannot proceed without href";}
var urnId = generateUrnId(vApphRef);
var vApp = vcdHost.getEntityById(VclFinderType.VAPP,urnId);
if (vApp == null){
            System.error("Failed to retrieve vApp by ID");
            throw "Failed to retrieve vApp by Href: "+vApphRef;
var ftState = getFtState(hRef);
System.log("Derived vApp Name: " +;
System.log("Derived FT State Request: " + ftState);
/* URL Processing functions */
function generateUrnId(hRef){
            var tmpStr1 = hRef.split("/")[5];
            tmpStr1 = "urn:vcloud:vapp:" + tmpStr1.substring(5,(tmpStr1.length));
            return tmpStr1;  
function getFtState(href) {
  var hrefComponents = href.split("/");
  System.log("hRef Components Split " + hrefComponents);
  var mode = hrefComponents[hrefComponents.length - 1];
  System.log("FT Mode: " + mode); 
  if (mode == "on") {
    return true;
  } else if (mode == "off") {
    return false;
  } else {
    System.log("Invalid state request");
    return false;

While I am logging many of the fields and message properties, the bare minimum information I need to grab from this message is
1)     "Host": "",
a.     I need the VCD host to be able to leverage the VCD plug-in, which will be used to load a (vCloud:Vapp) object using the vApp component of the Uri below.  We will also use the VCD plugin in subsequent elements within the workflow.
2)     "requestUri": "/api/vApp/vapp-ca90bfdd-93f7-4b24-a8ca-14bf5ba27fd1/ft/off"
a.     I need the requestUri because it contains the href Identifier of the vApp
b.     I also need the requestUri because it contains the specific call that will tell our workflow whether the user wants to Turn Fault Tolerance ON or OFF.

Workflow Element 2 – getVmsFromVApp (Action element)


This is a simple Action element that is included with the vCloud Director plugin and will basically take a vApp object in vCloud Director and return an Array of the Virtual Machines that comprise that vApp.  We could have done this using Javascript but wherever possible I wanted to use existing logic in the plug-ins to simplify development.
Workflow Element 3 – Change VM Objects (Scriptable Task)
Remember we are working backwards from what we are trying to achieve and I know that FT is a vSphere only feature and is not supported in vCloud Director. Because of this I will need to make API calls against vSphere to enable Fault Tolerance. We will translate the vCloud Virtual Machines into vCenter Virtual Machines using a for loop and using the moRef for each of the vCloud Virtual Machines as the query parameter in the getAllVirtualMachines method.
var vcVms = new Array();
System.log("Vms passed in: " + vms.length);
for (var i in vms) {
    System.log("vCloud VM: " + vms[i].name);
                  var moref = vms[i].getVMVimRef().moRef;
     var XPath = "xpath:id='"+ moref +"'";
      var VMs  = VcPlugin.getAllVirtualMachines(null, XPath);
      var vm = VMs[0];
      var vmName =;
      System.log("Name =" + vmName);
vcVms[i] = vm;
                  System.log("VMs in array: " + vcVms.length);

Workflow Element 4 – Decision (Decision Element)


This is a simple decision element which will evaluate the boolean value ftState which we defined in Element 1 by evaluating the requestUri.  The decision is if:
            Fton = True = proceed to “Enable Fault Tolerance” foreach loop
            Ftoff = False = proceed to “Disable FT” Foreach loop
Workflow Elements 5 & 6– Foreach elements with nested workflows
Input=(Array/VC:VirtualMachine)vcVms, (boolean)ftState
vCenter Orchestrator introduced a new workflow element in 5.1 which is the Foreach element. This element is very powerful and replaces what was previously a series of elements that were needed to iterate through an array and act on a single object in the array.  VCO 5.1 now lets you do this with a single element. We have an Array of VC:VirtualMachine objects called vcVms and our embedded FT workflows expect a single VC:VirtualMachine object. We can fix this discrepancy with the Foreach loop. When you drag the Foreach element onto the Schema page of your workflow you will be asked to choose a workflow.
We can then edit our Foreach element we just added to map the element inputs.  

We need to provide three Inputs,
  1. host is NOT mandatory and we can let DRS choose the host for the secondary machine but we should set this to Null
  2. turnOn should be mapped to ftState boolean workflow parameter which was an output of our first workflow element and was derived from the requestUri.
  3. vm should be bound to vcVms but vm is a single object and vcVms is an Array of objects so this will not work until we check the Bind as iterator checkbox. This will tell VCO to run the “Enable FT” workflow on EACH object in the vcVms array.

Step 3 - Create a VCO subscription policy

I will briefly cover how to take the Policy Template that comes with the AMQP plug-in and apply it to instantiate a configurable version, which will monitor our “ftqueue” that we created in RabbitMQ and kick off a workflow when a message is received in this queue.  As with most things VCO, Christophe has already created an excellent post on the AMQP plug-in and even using a Policy Template that you can read here.

  1. From the Administer context, under the “Policy Templates” Tab navigate to LibraryAMQP folder and Right Click the Subscription Template and select “Apply Policy…”. Provide a name for your policy such as “Subscription ftqueue Policy” and select the AMQP subscription you created in Step 1, which in this example is ”ftqueue API Extension”.   
  2. From the Run context, you now have a “Subscription ftqueue Policy” you can edit. Right click and Edit the policy.
    1. On the General Tab change the Startup to “On server startup, start the policy”  
    2. On the Scripting Tab, highlight the OnMessage, and click the Search button to browse for the “FT API Extension” workflow that you imported in Step 2.
    3. Now set the Source parameter (I always miss this) by clicking on the “not set” link and setting it to “self”. When you are finished it should look like this. 

Save and close the policy and then start the Subscription ftqueue Policy by clicking the green Start button.  


If you have followed all three posts in the series, we now have the following setup
  1. A registered extension service in vCloud Director named “faulttolerance”
  2. A RabbitMQ AMQP server accepting the AMQP messages from our "faulttolerance" Extension Service in the “availability” exchange and moving them to the “ftqueue” Queue.
  3. A VCO policy that will run in the background when the VCO server starts and monitor the AMQP subscription object we created that points to the “ftqueue” on our RabbitMQ server.  This policy will start the “FT API Extension” workflow and pass to it a single input parameter of type AMQP:Subscription. 
At this point you are probably waiting to actually test the solution and make an API call against vCloud Director and actually invoke Fault Tolerance on the Virtual Machines in the vApp.

Step 4 – Test

In order to test our solution we need to call our faulttolernace Extension Service via the vCloud Director API. This service is only exposed through the API so I will use the RESTClient 2.5 but you could also use Curl or whatever you prefer.
  1. Start the RESTClient 
    1. C:\Users\Administrator> java –jar restclient-ui-2.5-jar-with-dependencies.jar
  2. Configure SSL settings. Because this is a lab and we are NOT using CA issued certificates I made the following changes. 
  3. Configure Auth Settings 

  4. Add the following http request header 
    1. Key=Accept
    2. Value= application/*+xml;version=5.1 
  5. Obtain an Authorization Token
    1. POST
  6. If you receive an HTTP Response of “200 OK” in the HTTP Response section then you have authenticated and been granted an authorization token. 
  7. Copy and Paste this Token by Right Clicking the HTTP Header in the Response labeled “x-vcloud-authorization” and selecting Copy Selected Header(s). Add this header to the HTTP Request Headers section by copying and pasting. When complete it should look like this with a different Value obviously.  
  8. Using the RESTClient, we now are now ready to iterate through vCloud Director objects in a RESTful manner as follows to obtain the URI for a vApp we want to execute the our API Extension methods on.
    1. GET 
      1. Returns all Organizations in the vCloud in the HTTP Response section under Body tab. Copy the href value without quotes for an Organization you wish to use and paste it into the URL field.
    2. GET
      1. Returns all Organization VDC’s in the target Organization in the HTTP Response section under Body tab. Copy the href value without quotes for the Organization VDC where your target vApp resides and paste it into the URL field.
    3. GET
      1. Returns all vApp’s in the target Organization VDC in the HTTP Response section under Body tab. Copy the href value without quotes for your target vApp and paste it into the URL field.
    4.  GET  
      1. Returns all the properties of the vApp. You can now see the Service Link and href for our extension service that will Turn ON Fault Tolerance. Copy the href value without quotes for your “fton” and paste it into the URL field. 
    5. POST  
      1. Calls the “fton” method we defined for our Extension Service, which executes a workflow run of the “FT API Extensions” workflow below.  

A look at the VCO system shows a successful run of our FT API Extensions workflow and the System.log entries for the run.

A look at the vCenter system mapped to vCloud Director shows that VCO is iteratively enabling Fault Tolerance on the two Virtual Machines which comprise this vApp.
NOTE: I have not had time to implement a Response back to vCloud Director yet with the status of the API call such as success or failure and an error code.  When I do that I will need to grab the message properties for reply_to and replyToExchange to tell VCO and AMQP to deliver the reply to the correct  exchange.

 Here is another link to download the FT API Extensions workflow for your reference.