Terraform Registry and Provider Requirements

This blog post would have saved me hours (yes, literally hours) of work if a feature had been documented a bit more clearly… I hope it will help readers save some time too!

Let’s review first the challenges we sometimes come across with Terraform.

One of the first actions before you deploy resources with Terraform is to initialize Terraform with “terraform init“. When you run a “terraform init” command, a number of things happened, as documented in the official documentation.

During init, Terraform checks the configuration of configuration files (.tf files – main.tf in most of my examples) for direct and indirect references to providers and attempts to load the required plugins. If, for example, your file includes “provider aws“, Terraform will deduce it has to download the Terraform AWS provider before it tries to deploy AWS resources.

For providers distributed by HashiCorp, init will automatically download from the Terraform Registry and install plugins if necessary.

But what if you want to use an experimental provider? Or want to leverage one that is not officially provided by HashiCorp? How can you specify the provider you require?

By using “provider requirements” and the “required_providers” keyword!

This feature came in with Terraform 0.13 and was super helpful in a couple of situations.

Providers on the public registry

The first example is when I recently tested a provider my good friend Antoine Deleporte built by himself (Antoine is seriously smart).

This is to configure the VMware SD-WAN platform (formerly known as VeloCloud).

Antoine published it on the official Terraform registry:

I’ve been a fan of VeloCloud and of SD-WAN in general and I thought I’d try out his provider.

As it’s homemade and not published by Hashicorp, we need to tell Terraform where to find it. This is when we have to use the required_provider command:

terraform {
  required_providers {
    velocloud = {
      source = "adeleporte/velocloud"
    }
  }
}

provider "velocloud" {
  vco   = var.vco
  token = var.token
}

By default, when you define the source as field1/field2, Terraform will look at the user field1 and the provider field2 on the official Terraform Registry. For “adeleporte/velocloud“, Terraform will check the Terraform Registry for a “velocloud” provider registered under the “adeleporte” username.

This short command worked as I had hoped and so did Antoine’s experimental provider.

I was able to create a VMware SD-WAN edge as you can see in the brief demo below.

In-house providers

While the above was great for a provider that is publicly available, there are times when providers are still private or cannot be published on the registry yet. A recent example was when one of my customers required a minor improvement on the Terraform for NSX-T Policy Provider.

Our developers implemented the change and asked me to test the prototype.

I downloaded the code from GitHub, compiled it (I explained how to do this here) but ran into some issues as I couldn’t find a way to force Terraform to use the one provided from GitHub – instead, it kept downloading the official one on the registry instead of using the prototype I had downloaded and compiled from GitHub.

It turned out the “required_providers” could also work for in-house providers and for those located on my hard drive.

First, I had to move the compiled provider (always named something like terraform-provider-something) to the right folder – it has to be in the user plugins directory, located at ~/.terraform.d/plugins on most operating systems and %APPDATA%\terraform.d\plugins on Windows.

I create a folder (the path of the folder is important here) and move the provider to the newly created folder. Note the 3.2:

nicolasvibert$ mkdir -p ~/.terraform.d/plugins/vmc.nico/edu/nsxt/3.2/darwin_amd64
nicolasvibert$ mv terraform-provider-nsxt ~/.terraform.d/plugins/vmc.nico/edu/nsxt/3.2/darwin_amd64

My main.tf looks like this:

nicolasvibert$ more main.tf
terraform {
  required_providers {
    nsxt = {
      versions = ["3.2"]
      source = "vmc.nico/edu/nsxt"
    }

  }
}


provider "nsxt" {
  host                 = var.host
  vmc_token            = var.vmc_token
  allow_unverified_ssl = true
  enforcement_point    = "vmc-enforcementpoint"
}

As you can see above, I can specify the version (I arbitrarily chose “3.2”) and chose a source of “vmc.nico/edu/nsxt” to tell Terraform where to look for the provider.

The source format requires 3 fields – in the official docs, they use the following terminology:

terraform.example.com/examplecorp/ourcloud"

Terraform.example.com does not need to be resolvable (vmc.nico is what I used): it can be just the name of the directory.

“examplecorp” is again just a placeholder (I chose ‘edu’).

Finally, you can see the value of 3.2 in “versions” – it also refers to the folder where to find the provider. That’s to tell Terraform which version of the in-house provider I am looking to install.

When I run “terraform init” with the logs on, this is what happens:

nicolasvibert$ TF_LOG=trace terraform init
2020/11/23 19:59:38 [INFO] Terraform version: 0.13.0  
2020/11/23 19:59:38 [INFO] Go runtime version: go1.14.7

2020/11/23 19:59:38 [DEBUG] will search for provider plugins in /Users/nicolasvibert/.terraform.d/plugins
2020/11/23 19:59:38 [TRACE] getproviders.SearchLocalDirectory: found vmc.nico/edu/nsxt v3.2.0 for darwin_amd64 at /Users/nicolasvibert/.terraform.d/plugins/vmc.nico/edu/nsxt/3.2/darwin_amd64

Initializing the backend...

2020/11/23 19:59:38 [TRACE] providercache.fillMetaCache: scanning directory .terraform/plugins

2020/11/23 20:30:37 [TRACE] providercache.Dir.InstallPackage: installing vmc.nico/edu/nsxt v3.2.0 from /Users/nicolasvibert/.terraform.d/plugins/vmc.nico/edu/nsxt/3.2/darwin_amd64
Initializing provider plugins...

- Finding latest version of vmc.nico/edu/nsxt...
- Installing vmc.nico/edu/nsxt v3.2.0...
2020/11/23 20:30:37 [TRACE] providercache.fillMetaCache: scanning directory .terraform/plugins
2020/11/23 20:30:37 [TRACE] getproviders.SearchLocalDirectory: found vmc.nico/edu/nsxt v3.2.0 for darwin_amd64 at .terraform/plugins/vmc.nico/edu/nsxt/3.2.0/darwin_amd64
2020/11/23 20:30:37 [TRACE] providercache.fillMetaCache: including .terraform/plugins/vmc.nico/edu/nsxt/3.2.0/darwin_amd64 as a candidate package for vmc.nico/edu/nsxt 3.2.0
- Installed vmc.nico/edu/nsxt v3.2.0 (unauthenticated)
2020/11/23 20:30:37 [TRACE] providercache.fillMetaCache: using cached result from previous scan of .terraform/plugins

The following providers do not have any version constraints in configuration,
so the latest version was installed.


* vmc.nico/edu/nsxt: version = "~> 3.2.0"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

And a terraform version confirms that we are using the version required:

nicolasvibert$ terraform version

Your version of Terraform is out of date! The latest version
is 0.13.5. You can update by downloading from https://www.terraform.io/downloads.html
Terraform v0.13.0
+ provider vmc.nico/edu/nsxt v3.2.0

I hope this was helpful. Thanks for reading.