I recently came across a requirement to make some scale tests on my VMware environment. I had to validate whether a limit was a hard limit (literally, the system prevents you from going above) or a soft limit (you can go above but it’s not recommended and you’re no longer supported).
Given that the limit was 10,000, it would have taken me a while to do it manually on the user console… Thankfully, I could just use some automation to run the tests. I picked Terraform but yes, arguably I could have done the same with Ansible, Python or PowerShell (just pick whatever tool you’re most comfortable with).
The first option to create a vast amount of resources with Terraform is to use the count meta-argument:
variable "counter" { default = 5 }
resource "nsxt_policy_group" "groupscale" {
count = var.counter
display_name = "group-scale.${count.index}"
description = "Terraform provisioned Group"
domain = "cgw"
criteria {
ipaddress_expression {
ip_addresses = ["192.168.30.${count.index}/32"]
}
}
}
This is as easy as it gets – the code above will create 5 nsxt_policy_group resources and the name of each group and its IP address range will be based on the index as Terraform loops through the creation of each entity:
nvibert-a01:terraform-scale-test nicolasvibert$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# nsxt_policy_group.groupscale[0] will be created
+ resource "nsxt_policy_group" "groupscale" {
+ description = "Terraform provisioned Group"
+ display_name = "group-scale.0"
+ domain = "cgw"
+ id = (known after apply)
+ nsx_id = (known after apply)
+ path = (known after apply)
+ revision = (known after apply)
+ criteria {
+ ipaddress_expression {
+ ip_addresses = [
+ "192.168.30.0/32",
]
}
}
}
# nsxt_policy_group.groupscale[1] will be created
+ resource "nsxt_policy_group" "groupscale" {
+ description = "Terraform provisioned Group"
+ display_name = "group-scale.1"
+ domain = "cgw"
+ id = (known after apply)
+ nsx_id = (known after apply)
+ path = (known after apply)
+ revision = (known after apply)
+ criteria {
+ ipaddress_expression {
+ ip_addresses = [
+ "192.168.30.1/32",
]
}
}
}
# nsxt_policy_group.groupscale[2] will be created
+ resource "nsxt_policy_group" "groupscale" {
+ description = "Terraform provisioned Group"
+ display_name = "group-scale.2"
+ domain = "cgw"
+ id = (known after apply)
+ nsx_id = (known after apply)
+ path = (known after apply)
+ revision = (known after apply)
+ criteria {
+ ipaddress_expression {
+ ip_addresses = [
+ "192.168.30.2/32",
]
}
}
}
# nsxt_policy_group.groupscale[3] will be created
+ resource "nsxt_policy_group" "groupscale" {
+ description = "Terraform provisioned Group"
+ display_name = "group-scale.3"
+ domain = "cgw"
+ id = (known after apply)
+ nsx_id = (known after apply)
+ path = (known after apply)
+ revision = (known after apply)
+ criteria {
+ ipaddress_expression {
+ ip_addresses = [
+ "192.168.30.3/32",
]
}
}
}
# nsxt_policy_group.groupscale[4] will be created
+ resource "nsxt_policy_group" "groupscale" {
+ description = "Terraform provisioned Group"
+ display_name = "group-scale.4"
+ domain = "cgw"
+ id = (known after apply)
+ nsx_id = (known after apply)
+ path = (known after apply)
+ revision = (known after apply)
+ criteria {
+ ipaddress_expression {
+ ip_addresses = [
+ "192.168.30.4/32",
]
}
}
}
Plan: 5 to add, 0 to change, 0 to destroy.
------------------------------------------------------------------------
Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.
nvibert-a01:terraform-scale-test nicolasvibert$ terraform apply -auto-approve
nsxt_policy_group.groupscale[3]: Creating...
nsxt_policy_group.groupscale[4]: Creating...
nsxt_policy_group.groupscale[2]: Creating...
nsxt_policy_group.groupscale[1]: Creating...
nsxt_policy_group.groupscale[0]: Creating...
nsxt_policy_group.groupscale[2]: Creation complete after 1s [id=9a03e72d-2704-4119-aad7-d214c412fffe]
nsxt_policy_group.groupscale[4]: Creation complete after 2s [id=028c193a-a19f-4aeb-9384-556f0d31626f]
nsxt_policy_group.groupscale[3]: Creation complete after 2s [id=78fa5b74-cb5f-4582-a01d-49e4ebb35993]
nsxt_policy_group.groupscale[1]: Creation complete after 2s [id=05639648-34bd-46f6-b2d9-6232d34bddb1]
nsxt_policy_group.groupscale[0]: Creation complete after 2s [id=d2e35922-93fc-418c-a328-28e7afcc6e42]
Apply complete! Resources: 5 added, 0 changed, 0 destroyed.
That works a treat and obviously if I want to scale up to 1,000 groups, I can just change the counter value to “1,000”. I get this from my “terraform plan”…
Plan: 1000 to add, 0 to change, 0 to destroy.
…when I set my configuration file to this:
variable "counters" {default = 1000 }
resource "nsxt_policy_group" "ManyGroup" {
count = var.counters
display_name = "Group_.${count.index}"
description = "Terraform provisioned Group"
domain = "cgw"
criteria {
condition {
key = "Tag"
member_type = "VirtualMachine"
operator = "EQUALS"
value = "Tag_.${count.index}"
}
}
But what if you have nested blocks? By nested blocks, I refer to network_interface, disk, clone in a standard Terraform for vSphere configuration.
What if you want to check, for example, how many network NICs you can add to your VMs?
resource "vsphere_virtual_machine" "vm_terraform_from_cl_2" {
name = "vm_terraform_from_cl_2"
resource_pool_id = data.vsphere_resource_pool.pool.id
datastore_id = data.vsphere_datastore.datastore.id
folder = "Workloads"
num_cpus = 2
memory = 1024
guest_id = "other3xLinux64Guest"
network_interface {
network_id = data.vsphere_network.network13.id
}
disk {
label = "disk0"
size = 20
thin_provisioned = true
}
clone {
template_uuid = data.vsphere_content_library_item.library_item_photon.id
customize {
linux_options {
host_name = "Photon"
domain = "vmc.local"
}
network_interface {
ipv4_address = cidrhost(var.subnet1, 201)
ipv4_netmask = 24
}
ipv4_gateway = cidrhost(var.subnet1, 1)
}
}
}
You cannot apply “count” to nested blocks. Instead, we’re going to use a new-ish feature (came with 0.12) of Terraform that enables you to create multiple nested blocks dynamically.
Dynamic and for_each
The example below is for the NSX-T provider. To create distributed firewall rules, you need to create a policy (also referred to as a ‘section’) and create rules within the policy.
The config below creates 2 security rules (“Blue2Red” and “Red2Blue”) within the “Colors” policy.
resource "nsxt_policy_security_policy" "Colors" {
display_name = "Colors"
description = "Terraform provisioned Security Policy"
category = "Application"
domain = "cgw"
locked = false
stateful = true
tcp_strict = false
rule {
display_name = "Blue2Red"
source_groups = [
nsxt_policy_group.Blue_VMs.path]
destination_groups = [
nsxt_policy_group.Red_VMs.path]
action = "DROP"
services = ["/infra/services/ICMP-ALL"]
logged = true
}
rule {
display_name = "Red2Blue"
source_groups = [
nsxt_policy_group.Red_VMs.path]
destination_groups = [
nsxt_policy_group.Blue_VMs.path]
action = "DROP"
services = ["/infra/services/ICMP-ALL"]
logged = true
}
}
You can see that a rule is built around the same content, with the format being:
rule {
display_name = "name"
source_groups = [group_path]
destination_groups = [group_path]
action = "action"
services = [service_path]
logged = true
}
An elegant way of creating many rules would be to use the dynamic and for_each commands:
resource "nsxt_policy_security_policy" "nvibert-scale" {
display_name = "Scale-Security-Policy"
description = "Terraform provisioned Security Policy"
category = "Application"
domain = "cgw"
locked = false
stateful = true
tcp_strict = false
dynamic "rule" {
for_each = range(1,20)
content {
display_name = "rule_name_${rule.value}"
source_groups = []
destination_groups = [nsxt_policy_group.Red_VMsNico.path]
action = "DROP"
services = ["/infra/services/ICMP-ALL"]
logged = true
sequence_number = rule.value
}
}
}
As you can see above, the rule is now dynamic and every rule I create will be based on the content defined in – well, content.
We’re using the range function to specify how many rules we are going to create within the policy. Be aware that range(1,20) starts at 1 and finishes at 19.
The display_name and the sequence_number are automatically updated as we iterate through the loop. Note how I refer to “rule.value” – when you using for_each, you have to refer to the name of the dynamic resource (here, it’s called rule) and then use the keyword value to get its value.
# nsxt_policy_security_policy.nvibert-scale:
resource "nsxt_policy_security_policy" "nvibert-scale" {
category = "Application"
description = "Terraform provisioned Security Policy"
display_name = "Scale-Security-Policy"
domain = "cgw"
id = "d081f7be-99bf-4c27-8004-9d656ae9a9d4"
locked = false
nsx_id = "d081f7be-99bf-4c27-8004-9d656ae9a9d4"
path = "/infra/domains/cgw/security-policies/d081f7be-99bf-4c27-8004-9d656ae9a9d4"
revision = 0
scope = []
sequence_number = 0
stateful = true
tcp_strict = false
rule {
action = "DROP"
destination_groups = [
"/infra/domains/cgw/groups/ed921a6b-8a3b-434a-b55c-aafaeb8b782b",
]
destinations_excluded = false
direction = "IN_OUT"
disabled = false
display_name = "rule_name_1"
ip_version = "IPV4_IPV6"
logged = true
nsx_id = "fc0dd4d2-5239-4860-9842-9a5a50be2b06"
profiles = []
revision = 0
rule_id = 0
scope = []
sequence_number = 0
services = [
"/infra/services/ICMP-ALL",
]
source_groups = []
sources_excluded = false
}
///////////////// cut for brevity ////////////////////
rule {
action = "DROP"
destination_groups = [
"/infra/domains/cgw/groups/ed921a6b-8a3b-434a-b55c-aafaeb8b782b",
]
destinations_excluded = false
direction = "IN_OUT"
disabled = false
display_name = "rule_name_19"
ip_version = "IPV4_IPV6"
logged = true
nsx_id = "18defb8b-f3d5-4f3f-aae9-5abf32e306ef"
profiles = []
revision = 0
rule_id = 0
scope = []
sequence_number = 18
services = [
"/infra/services/ICMP-ALL",
]
source_groups = []
sources_excluded = false
}
}
The for_each and dynamic commands can be used for other things than just scaling. You might have a list of values you want to apply. In this case, you can refer to a variable. In this instance, my list is manually set to [100,200,300,400,500] but you can imagine that you have generated via other means.
variable "groups" {
type = list(number)
description = "list of ingress ports"
default = [100, 200, 300, 400, 500]
}
resource "nsxt_policy_security_policy" "nvibert-scale" {
display_name = "Scale-Security-Policy"
description = "Terraform provisioned Security Policy"
category = "Application"
domain = "cgw"
dynamic "rule" {
for_each = var.groups
content {
display_name = "rule_name_${rule.value}"
source_groups = []
destination_groups = [nsxt_policy_group.Red_VMsNico.path]
action = "DROP"
services = ["/infra/services/ICMP-ALL"]
sequence_number = rule.value
}
}
}
This configuration would create the following firewall rules:
# nsxt_policy_security_policy.nvibert-scale will be created
+ resource "nsxt_policy_security_policy" "nvibert-scale" {
+ category = "Application"
+ description = "Terraform provisioned Security Policy"
+ display_name = "Scale-Security-Policy"
+ domain = "cgw"
+ id = (known after apply)
+ locked = false
+ nsx_id = (known after apply)
+ path = (known after apply)
+ revision = (known after apply)
+ sequence_number = 0
+ stateful = true
+ tcp_strict = (known after apply)
+ rule {
+ action = "DROP"
+ destination_groups = (known after apply)
+ destinations_excluded = false
+ direction = "IN_OUT"
+ disabled = false
+ display_name = "rule_name_100"
+ ip_version = "IPV4_IPV6"
+ logged = false
+ nsx_id = (known after apply)
+ revision = (known after apply)
+ rule_id = (known after apply)
+ sequence_number = 100
+ services = [
+ "/infra/services/ICMP-ALL",
]
+ sources_excluded = false
}
+ rule {
+ action = "DROP"
+ destination_groups = (known after apply)
+ destinations_excluded = false
+ direction = "IN_OUT"
+ disabled = false
+ display_name = "rule_name_200"
+ ip_version = "IPV4_IPV6"
+ logged = false
+ nsx_id = (known after apply)
+ revision = (known after apply)
+ rule_id = (known after apply)
+ sequence_number = 200
+ services = [
+ "/infra/services/ICMP-ALL",
]
+ sources_excluded = false
}
+ rule {
+ action = "DROP"
+ destination_groups = (known after apply)
+ destinations_excluded = false
+ direction = "IN_OUT"
+ disabled = false
+ display_name = "rule_name_300"
+ ip_version = "IPV4_IPV6"
+ logged = false
+ nsx_id = (known after apply)
+ revision = (known after apply)
+ rule_id = (known after apply)
+ sequence_number = 300
+ services = [
+ "/infra/services/ICMP-ALL",
]
+ sources_excluded = false
}
+ rule {
+ action = "DROP"
+ destination_groups = (known after apply)
+ destinations_excluded = false
+ direction = "IN_OUT"
+ disabled = false
+ display_name = "rule_name_400"
+ ip_version = "IPV4_IPV6"
+ logged = false
+ nsx_id = (known after apply)
+ revision = (known after apply)
+ rule_id = (known after apply)
+ sequence_number = 400
+ services = [
+ "/infra/services/ICMP-ALL",
]
+ sources_excluded = false
}
+ rule {
+ action = "DROP"
+ destination_groups = (known after apply)
+ destinations_excluded = false
+ direction = "IN_OUT"
+ disabled = false
+ display_name = "rule_name_500"
+ ip_version = "IPV4_IPV6"
+ logged = false
+ nsx_id = (known after apply)
+ revision = (known after apply)
+ rule_id = (known after apply)
+ sequence_number = 500
+ services = [
+ "/infra/services/ICMP-ALL",
]
+ sources_excluded = false
}
}
While this might seem like simple commands, I didn’t find that many explanations of them. Before I conclude, these commands are not just to create a lot of resources and to test scaling but to create multiple resources based on a list of variables.
Thanks for reading.
PS: Thanks to Gilles for his help with this post!
Additional reading resources and examples:
nicovibert.com 
One thought on “Scale Testing with the Terraform count, for_each and dynamic arguments”