First look into… HCP Packer

Last week, HashiCorp announced the public beta for HCP Packer, a managed service for Packer.

Packer might not be as well-known as some of the other HashiCorp tools but it’s universally used to build golden images and templates and it quietly powers a lot of what happens in the public cloud: on Azure….

Azure's new Image Builder service is built on Packer! Looks like a great service. 😄Also their demo video shows Image Builder to Terraform, neat! 😉 https://t.co/Yy54U2wdh6 pic.twitter.com/7W1ncf1bU5
— Mitchell Hashimoto (@mitchellh) October 1, 2018

….but also on AWS:

Did you know the official Amazon EKS AMI is built by Packer? That’s cool by itself. What is cooler is that @awscloud open sourced the Packer configuration and scripts so you can tweak and build your own: https://t.co/5Q6Eli43Yl Thanks @arungupta
— Mitchell Hashimoto (@mitchellh) July 10, 2018

The Education team at HashiCorp made some excellent tutorials in time for the HCP Packer launch.

In this post, I’m going to go through the first couple of HCP Packer tutorials, provide some additional context when I need it and occasionally digress or divert from the tutorial.

Let’s get started and jump on HCP (HashiCorp Cloud Platform) and go to the HCP menu and onto HCP Packer.

Create your registry:

Create the Service Principal in Access control (IAM) and store the created client and client secret as environmental variables.

Clone the Git repo, jump into the folder and let’s look at the Packer HCL file we’ve got:

packer {
  required_plugins {
    amazon = {
      version = ">= 1.0.1"
      source  = "github.com/hashicorp/amazon"
    }
  }
}

source "amazon-ebs" "basic-example-east" {
  region = "us-east-2"
  source_ami_filter {
    filters = {
      virtualization-type = "hvm"
      name                = "ubuntu/images/*ubuntu-xenial-16.04-amd64-server-*"
      root-device-type    = "ebs"
    }
    owners      = ["099720109477"]
    most_recent = true
  }
  instance_type  = "t2.small"
  ssh_username   = "ubuntu"
  ssh_agent_auth = false
  ami_name       = "packer_AWS_{{timestamp}}"
}

source "amazon-ebs" "basic-example-west" {
  region = "us-west-2"
  source_ami_filter {
    filters = {
      virtualization-type = "hvm"
      name                = "ubuntu/images/*ubuntu-xenial-16.04-amd64-server-*"
      root-device-type    = "ebs"
    }
    owners      = ["099720109477"]
    most_recent = true
  }
  instance_type  = "t2.small"
  ssh_username   = "ubuntu"
  ssh_agent_auth = false
  ami_name       = "packer_AWS_{{timestamp}}"
}

The tutorial understandably skips some of the Packer configuration (it’s covered in other tutorials) but I think it’s worth walking through what we are doing.

Packer is a tool to build template images but Packer itself needs to build on top of an existing OS base image.

How does Packer find an existing image? Just like Terraform use datasource, Packer use source to look for something that already exists. Like Terraform, we can use filters to find a particular object.

With the config above, we use source_ami_filter to look for an Ubuntu image from the owner 099720109477. This happens to be the account Id of Canonical, the publisher of Ubuntu.

Quick sidenote: Packer recently added a AMI datasource function, just like Terraform does. It would probably be the recommended way to pull the AMI ID and will feel familiar to the Terraform user.

So an alternative to the code above would be:

data "amazon-ami" "basic-example" {
    filters = {
        virtualization-type = "hvm"
        name = "ubuntu/images/*ubuntu-xenial-16.04-amd64-server-*"
        root-device-type = "ebs"
    }
    owners = ["099720109477"]
    most_recent = true
}

You can refer elsewhere to the AMI ID as data.amazon-ami.basic-example.id.

Let’s resume the walkthrough. As mentioned above, we can use source or data to get a list of all the Canonical images published:

You can see there are thousands of images published by Canonical and there are hundreds of images that meet my criteria defined in my filter. The most_recent = true parameter would select the most recent Ubuntu image that meets the criteria defined in my filter.

When I run packer init . and packer build ubuntu-xenial.pkr.hcl, it would respectively initialize packer and build the images. We do this for two different regions, as AMIs are region-dependent in AWS. The two images are actually being built in parallel. You can see how useful it would be if you want to update instances across multiple regions.

There’s lot of stuff that happens above. Let’s drill down a bit and focus on what happened in Ohio: I will just focus on the output in blue on the screenshot above.

What’s going on there? The logs are pretty clear but let’s drill down a bit:

Found Image ID: ami-05803413c51f242b7

This is the AMI I was talking about: the most recent Ubuntu image that matches the filter criteria.

It uses the SSH communicator and SSH into the EC2 instance using a temporary keypair and security group to provision an instance. Typically, this is when we would modify the base image and run a provisioner or some kind of script on the base image. Here, we don’t do anything and simply use the base image for our golden image.

Eventually, Packer creates a new AMI in AWS:

The name of the AMI is packer_AWS_1634822775 – remember that we added the timestamp variable to it (if I revert the Unix timestamp to a real date, you can see when I wrote this blog):

What we also do is publishing some metadata to the HCP Packer Registry. My HCP Packer bucket is now populated:

And I’ve got some metadata about my image registered to HCP:

And I can see the AMIs created by Packer:

HCP Packer does not actually store any of these images – it stores information about them and it makes managing golden images easier (at least, it gives you some tracking history and it makes it easy for Terraform for example to refer to images as we will see shortly).

When I finally deployed an instance from the AMI and it was the version I expected it to be.

% ssh -i "NicoVibertOhio.pem" ubuntu@ec2-A-B-C-D.us-east-2.compute.amazonaws.com
Welcome to Ubuntu 16.04.7 LTS (GNU/Linux 4.4.0-1128-aws x86_64)

Let’s divert from the tutorial for a minute.

It turns out that Ubuntu 16.04 is no longer supported so we need to move to something a bit more recent like 20.04.

The next natural step here is to automate the process when a CVE is detected on an image or when a new base image is released. Perhaps using a GitLab runner like Mark explains in his post? Or perhaps with Amazon Inspector and Lambda like in this post?

So first, we need to update our Packer file and pick a more recent build:

    filters = {
      virtualization-type = "hvm"
      name                = "ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-*"
      root-device-type    = "ebs"
    }

I also updated the labels to be a bit more descriptive:

  hcp_packer_registry {
    bucket_name = "learn-packer-ubuntu"
    description = <<EOT
Our Ubuntu-based Golden Image
    EOT
    labels = {
      "ubuntu-version" = "20.04",
    }
  }

When I run packer build again, it worked perfectly again, it selected the AMI 20.04 below as a base image:

Packer requires a unique identifier for each build so we’ll use the hash of the most recent Git commit (remember – this is infra-as-code so Git is expected to be in the mix).

So we need to commit the file first before running Packer again. Note that the commit 5b6fde7.

bash-3.2$ git add ubuntu-xenial.pkr.hcl
bash-3.2$ git commit -m "Updated AMI to 20.04"
[main 5b6fde7] Updated AMI to 20.04
 1 file changed, 4 insertions(+), 5 deletions(-)
bash-3.2$ packer build ubuntu-xenial.pkr.hcl
amazon-ebs.basic-example-east: output will be in this color.
amazon-ebs.basic-example-west: output will be in this color.

==> amazon-ebs.basic-example-east: Publishing build details for amazon-ebs.basic-example-east to the HCP Packer registry
==> amazon-ebs.basic-example-west: Publishing build details for amazon-ebs.basic-example-west to the HCP Packer registry
==> amazon-ebs.basic-example-east: Prevalidating any provided VPC information
==> amazon-ebs.basic-example-east: Prevalidating AMI Name: packer_AWS_1634852290
==> amazon-ebs.basic-example-west: Prevalidating any provided VPC information
==> amazon-ebs.basic-example-west: Prevalidating AMI Name: packer_AWS_1634852290
    amazon-ebs.basic-example-east: Found Image ID: ami-0e5e17317f99b2932
==> amazon-ebs.basic-example-east: Creating temporary keypair: packer_6171ddc2-dccf-d9eb-3898-fce2ae2b7edf
==> amazon-ebs.basic-example-east: Creating temporary security group for this instance: packer_6171ddc5-95d8-c36f-f31d-93652e1b04c5
    amazon-ebs.basic-example-west: Found Image ID: ami-08f76d3bc7a0e55ce
==> amazon-ebs.basic-example-west: Creating temporary keypair: packer_6171ddc2-3a32-97aa-1b63-0d5392ff6e65
==> amazon-ebs.basic-example-east: Authorizing access to port 22 from [0.0.0.0/0] in the temporary security groups...
==> amazon-ebs.basic-example-west: Creating temporary security group for this instance: packer_6171ddc7-eaf5-e756-bc07-c50d6a846f90
==> amazon-ebs.basic-example-east: Launching a source AWS instance...
==> amazon-ebs.basic-example-east: Adding tags to source instance
    amazon-ebs.basic-example-east: Adding tag: "Name": "Packer Builder"
==> amazon-ebs.basic-example-west: Authorizing access to port 22 from [0.0.0.0/0] in the temporary security groups...
    amazon-ebs.basic-example-east: Instance ID: i-06cb83d9d43b88f5d
==> amazon-ebs.basic-example-east: Waiting for instance (i-06cb83d9d43b88f5d) to become ready...
==> amazon-ebs.basic-example-west: Launching a source AWS instance...
==> amazon-ebs.basic-example-west: Adding tags to source instance
    amazon-ebs.basic-example-west: Adding tag: "Name": "Packer Builder"
    amazon-ebs.basic-example-west: Instance ID: i-0f6a54f632d9ff8db
==> amazon-ebs.basic-example-west: Waiting for instance (i-0f6a54f632d9ff8db) to become ready...
==> amazon-ebs.basic-example-east: Using SSH communicator to connect: 18.117.161.224
==> amazon-ebs.basic-example-east: Waiting for SSH to become available...
==> amazon-ebs.basic-example-east: Connected to SSH!
==> amazon-ebs.basic-example-east: Stopping the source instance...
    amazon-ebs.basic-example-east: Stopping instance
==> amazon-ebs.basic-example-east: Waiting for the instance to stop...
==> amazon-ebs.basic-example-west: Using SSH communicator to connect: 52.43.84.2
==> amazon-ebs.basic-example-west: Waiting for SSH to become available...
==> amazon-ebs.basic-example-west: Connected to SSH!
==> amazon-ebs.basic-example-west: Stopping the source instance...
    amazon-ebs.basic-example-west: Stopping instance
==> amazon-ebs.basic-example-west: Waiting for the instance to stop...
==> amazon-ebs.basic-example-east: Creating AMI packer_AWS_1634852290 from instance i-06cb83d9d43b88f5d
    amazon-ebs.basic-example-east: AMI: ami-0fe15aef3fb965734
==> amazon-ebs.basic-example-east: Waiting for AMI to become ready...
==> amazon-ebs.basic-example-west: Creating AMI packer_AWS_1634852290 from instance i-0f6a54f632d9ff8db
    amazon-ebs.basic-example-west: AMI: ami-08ab6809bca2a93d7
==> amazon-ebs.basic-example-west: Waiting for AMI to become ready...
==> amazon-ebs.basic-example-east: Skipping Enable AMI deprecation...
==> amazon-ebs.basic-example-east: Terminating the source AWS instance...
==> amazon-ebs.basic-example-east: Cleaning up any extra volumes...
==> amazon-ebs.basic-example-east: No volumes to clean up, skipping
==> amazon-ebs.basic-example-east: Deleting temporary security group...
==> amazon-ebs.basic-example-east: Deleting temporary keypair...
==> amazon-ebs.basic-example-east: Running post-processor:
Build 'amazon-ebs.basic-example-east' finished after 3 minutes 13 seconds.
==> amazon-ebs.basic-example-west: Skipping Enable AMI deprecation...
==> amazon-ebs.basic-example-west: Terminating the source AWS instance...
==> amazon-ebs.basic-example-west: Cleaning up any extra volumes...
==> amazon-ebs.basic-example-west: No volumes to clean up, skipping
==> amazon-ebs.basic-example-west: Deleting temporary security group...
==> amazon-ebs.basic-example-west: Deleting temporary keypair...
==> amazon-ebs.basic-example-west: Running post-processor:
Build 'amazon-ebs.basic-example-west' finished after 3 minutes 53 seconds.

==> Wait completed after 3 minutes 53 seconds

==> Builds finished. The artifacts of successful builds are:
--> amazon-ebs.basic-example-east: AMIs were created:
us-east-2: ami-0fe15aef3fb965734

--> amazon-ebs.basic-example-east: Published metadata to HCP Packer registry packer/learn-packer-ubuntu/iterations/01FJJCMGFMB4010AE8PNA3FTEX
--> amazon-ebs.basic-example-west: AMIs were created:
us-west-2: ami-08ab6809bca2a93d7

--> amazon-ebs.basic-example-west: Published metadata to HCP Packer registry packer/learn-packer-ubuntu/iterations/01FJJCMGFMB4010AE8PNA3FTEX

I’ve now got two iterations of my Packer build. I associate the second one with my newly created channel.

We call this channel “production”, like in the second tutorial, and Terraform will refer to this channel. Terraform does not need to know of the multiple Packer iterations behind the scene – it just needs to know which channel it can deploy from (I feel like branch might have been a better word than channel but maybe the product team didn’t want users to get confused with Git branches).

So Terraform will find the AMI ID by looking up an iteration based on the name of the channel.

This is the important bit: you didn’t have to specify the AMI ID in the TF config below but you simply refer to the channel and iteration and TF will pull the desired AMI.

data "hcp_packer_iteration" "ubuntu" {
  bucket_name = "learn-packer-ubuntu"
  channel     = "production"
}

data "hcp_packer_image" "ubuntu_us_east_2" {
  bucket_name    = "learn-packer-ubuntu"
  cloud_provider = "aws"
  iteration_id   = data.hcp_packer_iteration.ubuntu.ulid
  region         = "us-east-2"
}

resource "aws_instance" "app_server" {
  ami           = data.hcp_packer_image.ubuntu_us_east_2.cloud_image_id
  instance_type = "t2.micro"
  tags = {
    Name = "Learn-HCP-Packer"
  }
}

If we carry on with the second HCP tutorial (here), it will take you through running Terraform locally to build an EC2 instance referencing this channel. I will divert slightly here and use Terraform Cloud.

I put the Terraform code in a GitHub repo (here), created a Terraform workspace in TF Cloud linked to that repo, added my environment variables to my workspace:

Finally, I ran a Terraform plan and boom, I got my EC2 instance based on my Packer-built AMI:

So when a new production image is created by Packer (because we’ve moved to a new base OS for example or because the initialization script has changed), the Terraform configuration is still valid and does not change. However some of the values – like the AMI ID – will be different as a new image has been built.

When I go back to the Packer UI, guess what the fingerprint is for this new iteration: 5b6fde7 (the Git commit hash from earlier).

So what we learned?

Packer is used extensively across the Cloud industry and can easily create your golden images across any clouds
HCP Packer provides a governance wrap-up around your image packaging process with a user interface to track your images, their location and their history
Using HCP Packer alongside Terraform gives you an elegant workflow and it should simplify how you refer to your validated images.

Now I think the value of HCP Packer will eventually reside more in API-driven architecture – such as:

vulnerability is detected on an image → API-driven process to check which iterations have the package or OS with the vulnerability → alert users using the vulnerable image
vulnerability is detected on an image → API-driven process to check which iterations have the package or OS with the vulnerability → automatically update the labels on HCP Packer with a “vulnerable” flag
new base image release → automatic build of test image → automatic update of AMI used by Terraform in image testing environment

In the next post, I look at the HCP Packer APIs and explore what can be done.

Thanks for reading.

First look into… HCP Packer

2 thoughts on “First look into… HCP Packer”

Leave a comment Cancel reply

Published by Nico

Share this:

2 thoughts on “First look into… HCP Packer”

Leave a comment Cancel reply

Published by Nico