Synchronizing NSX security tags with vSphere tags using AWS Lambda

This post will walk-through how to synchronize VMware vSphere Tags with NSX security tags via vRealize Log Insight Cloud and AWS Lambda Python scripts.

Thanks to William Lam, Gilles Chekroun and Guillaume Vibert for their help with this project.

If there’s one thing I’ve learned in my career is to listen to customers. It’s too easy when you’re in Technical Pre-Sales to fall in love with the sound of your own voice and not really pay attention to what the customer actually wants or needs (I’ve seen it – many times).

So when one of my favourite customers asked me for a feature that I knew wasn’t coming any time soon, I thought I would have a go at building it myself. But first, let’s review a couple of technical concepts.

vSphere Tags and NSX Security tags

VCenter has leveraged the concept of Tags for a long time. VMware administrators can apply a ‘tag’ (essentially some metadata) to a vCenter object (most typically a virtual machine). It’s used for various use cases, such as troubleshooting, compliance, auditing, automation.

This is how you would create them on vCenter:

On the other hand, we also have NSX Security Tags. NSX Security Tags are tags (metadata) you can apply to a VM and create a policy group based upon that tag. Instead of creating a rule based upon IP addresses (which have no business meaning), we can create security rules based upon these tags and create a security policy based upon business logic instead of meaningless IP addresses.

I covered the topic extensively in a blog post and in the VMC Networking and Security book.

And this is how you would create them on the VMware Cloud on AWS Networking and Security console:

Are NSX tags and vCenter tags the same?

A common question from customers is: are these vSphere tags and these NSX security tags the same?

The answer is no. VCenter and the NSX Manager don’t exchange this information, which is a shame as synchronization of these tags would enable users to greatly simplify their security policy.

And this is exactly what my customer requested:

That feature isn’t on the near term roadmap as far as I am aware so I thought I’d create some kind of scripts that would regularly check tags and synchronise them. But an internal Slack conversation with William Lam helped me come up with a much better model:

The key word here was Event. We want the Attach/Detag Tag vCenter event to trigger a Attach/Detag Tag in NSX.

So I came up with the following, which I dubbed vTagSync:

In short, this is what vTagSync does:

1. Tag assigned to a VM generates an vCenter event
2. Event is logged on Log Intelligence (aka vRealize Log Insight Cloud) as an alert
3. vRealize Log Insight Cloud forwards the alert to AWS API Gateway
4. API Gateway triggers a AWS Lambda function
5. AWS Lambda executes a Python-based function to apply the NSX-T tag over APIs

Let’s review these steps in more details:

Step 1 – Tag assigned/removed from VM on vCenter generates an Event

As William described in his blog post, whenever a user assigns or removes a tag from a VM in vCenter, an event is raised. For VMware Cloud on AWS customers, all vCenters events are exported into our Logging-As-A-Service platform (Log Intelligence, also known as vRealize Log Insight Cloud – vRLIC).

As you can below, the event includes the string “attached tag” or “detached tag” – this will become important in the next step.

Step 2 – Qualify the tag attachment/detachment Event as an Alert

There are thousands of logs and events pushed into vRLIC and we need to make sure this particular event is picked up and forwarded on to AWS API Gateway. The first step is to create an alert (Alerts >> Alert Definitions) that look for text including “tached tag“, capturing both “attached tag” and “detached tag” events.

Once I have created this alert, every time this specific event occurs, an alert is raised:

Step 3 – Alert triggers notification via Webhook

The next stage is to create a Webhook and forward the alert to AWS API Gateway. In vRLIC, just create a new Webhook, specify the Destination URL to API Gateway (see in Step 4) and make sure you include the $additionalInfo value below in “Workload Payload”. This will make sure we forward the content of the alert out to the API Gateway.

Step 4 – Webhook notification hits API endpoint

Again here, I leverage another of William’s posts . We set up AWS API Gateway to essentially trigger a Lambda function when we receive a packet to the API Gateway.

I created an AWS Lambda function called SETNSXTag and added API Gateway as a trigger to the function.

When creating the API Gateway Method, I selected “Lambda Function” as the Integration type and selected “Use Lambda Proxy integration“. This means that the content of the API request ($additionalInfo from the previous step) will be proxied in the ‘event’ of the AWS Lambda handler function and specify the name of your function as shown in the screenshot below

Step 5 – AWS Lambda Function executed

Once AWS API Gateway received the packet from vRealize Log Insight Cloud, the AWS Lambda function SetNSXTag is executed.

The function was written in Python. Gilles Chekroun gave me a hand with some of the functions and my bro Guillaume gave me a hand with some of the Python parsing.

I wrote a detailed blog post on Python parsing to explain how we took that JSON string in the original alert and extract the relevant information. But in essence, we extract the name of the tag and the name of the VM it’s applied to and use that in the API Call we make to the VMware Cloud on AWS NSX Manager.

You can download the code here or find it below:

# import modules

import json
import requests                         # need this for Get/Post/Delete
import configparser                     # parsing config file

# customer settings. The customer needs to specify which ORG, SDDC and REFRESH TOKEN will be used.
config = configparser.ConfigParser()
config.read("./config.ini")
strProdURL      = config.get("vmcConfig", "strProdURL")
strCSPProdURL   = config.get("vmcConfig", "strCSPProdURL")
Refresh_Token   = config.get("vmcConfig", "refresh_Token")
ORG_ID          = config.get("vmcConfig", "org_id")
SDDC_ID         = config.get("vmcConfig", "sddc_id")

# Note that the refresh token has, by default, a 6-month lifespan. A token without expiration can also be created.

def lambda_handler(event, context):
    print("Received event: " + str(event))
    string_event = str(event)
    if string_event.find('attached tag') == -1:
        print(string_event.find('attached tag'))
        tag_name = ""
        indexToObject = string_event.find("from object")
        # tagged_VM below is the extract 'VM name' from the Log Intelligence event. We know that the name of the tagged VM is 12 characters after the beginning of 'from object' and finishes before ']'.
        tagged_VM = string_event[indexToObject + 12 : string_event.find("]", indexToObject)]
    else:
        print(string_event.find('attached tag'))
        # tag_name below is the extract 'tag' from the Log Intelligence event. We look for the string 'attached tag ' and look ahead 13 characters ahead to where the name of the tag is.
        tag_name = string_event[string_event.find('attached tag') + 13 : (string_event.find('to object') - 1)]
        indexToObject = string_event.find("to object")
        # tagged_VM below is the extract 'VM name' from the Log Intelligence event. We know that the name of the tagged VM is 10 characters after the beginning of 'to object' and finishes before ']'.
        tagged_VM = string_event[indexToObject + 10 : string_event.find("]", indexToObject)]
    print("The tag " + tag_name + " will be applied to " + tagged_VM + ".")
    # Following section requests a auth token from the refresh token via API and extracts it from the API JSON response.
    params = {'refresh_token': Refresh_Token}
    headers = {'Content-Type': 'application/json'}
    response = requests.post('https://console.cloud.vmware.com/csp/gateway/am/api/auth/api-tokens/authorize', params=params, headers=headers)
    jsonResponse = response.json()
    access_token = jsonResponse['access_token']
    # Following section pulls the VM external_ID (also called VM InstanceUUID) based upon the name of the VM.
    myHeader = {'csp-auth-token': access_token}
    myURL = "{}/vmc/api/orgs/{}/sddcs/{}".format(strProdURL, ORG_ID, SDDC_ID)
    response = requests.get(myURL, headers=myHeader)
    json_response = response.json()
    proxy_url = json_response['resource_config']['nsx_api_public_endpoint_url']
    VMlist_url = (proxy_url + "/policy/api/v1/infra/realized-state/virtual-machines?enforcement_point_path=/infra/deployment-zones/default/enforcement-points/vmc-enforcementpoint")
    response = requests.get(VMlist_url, headers=myHeader)
    response_dictionary = response.json()
    extracted_dictionary = response_dictionary['results']
    # Below, we're extracting the Python dictionary for the specific VM and then we extract the external_ID/ Instance UUID from the dictionary.
    extracted_VM = next(item for item in extracted_dictionary if item["display_name"] == tagged_VM)
    extracted_VM_external_id = extracted_VM['external_id']
    # Finally, we're applying the NSX tag, using the External ID and tags.
    headers = {"Content-Type": "application/json","Accept": "application/json",'csp-auth-token': access_token}
    apply_tag_URL = (proxy_url + "/policy/api/v1/infra/realized-state/enforcement-points/vmc-enforcementpoint/virtual-machines?action=update_tags")
    json_data = {
    "virtual_machine_id": extracted_VM_external_id,
    "tags":[
      {
         "scope":"",
         "tag": tag_name
      }
   ]
   }
    response = requests.post(apply_tag_URL, json = json_data, params={'action': 'update_tags'}, headers=headers)
    print(tag_name)
    print(tagged_VM)
    print(apply_tag_URL)
    print(json_data)
    return apply_tag_URL, json_data, response.status_code

You also need in the same folder as your Lambda function a file called config.ini with your VMware Cloud on AWS SDDC variables.

[vmcConfig]
strProdURL      = https://vmc.vmware.com
strCSPProdURL   = https://console.cloud.vmware.com
refresh_Token   = XXXXXXXXX
org_id          = YYYYYYYYY
sddc_id		= ZZZZZZZZZ

Step 6 – NSX Tag is applied

As you can see below, the tag is applied to my VM:

When the tag is detached from the VM in vCenter, you can see how, after a few seconds and refreshing my screen a couple of times, the NSX security tag is also removed from my Security console.

I don’t use VMware Cloud on AWS – how can I do this on-prem?

Just use Log Insight and achieve the same results – I haven’t tried it myself but the same logic should be applicable.

What next?

If you think this feature is useful, please drop a comment below or a Tweet. I am looking at submitting it as a VMware Fling if there’s interest in such a feature.

And if you like the concept of Event-As-A-Service, make sure you check out the recording of William and Michael’s session at VMworld.

Many thanks for reading.