AWS CLI – back to basics

I have been setting up Terraform with S3 as back-end and one of the blog posts I was consulting suggested using AWS CLI which I hadn’t been using for a couple of years.

So… time to go back to basics!

What is the AWS CLI ?

The AWS Command Line Interface is “a unified tool to manage your AWS services. With just one tool to download and configure, you can control multiple AWS services from the command line and automate them through scripts” (from there).

Obviously there are are many ways to manage and consume AWS resources and I tend to use the GUI the vast majority of the time but it’s good to be aware of alternatives.

AWS CLI Installation

I followed the instructions here and it was fairly straight-forward. I actually downloaded the new CLI version (version 2). It was announced fairly recently. You will see below I went back to using version 1 for a reason I will explain later.

Download the Zip (I was on the train which explains why it took 17 minutes to download 19.2M !), unzip it and install it and you’re good to move on to the next step.

bash-3.2$ curl "https://d1vvhvl2y92vvt.cloudfront.net/awscli-exe-macos.zip" -o "awscliv2.zip"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 19.2M  100 19.2M    0     0  19875      0  0:16:54  0:16:54 --:--:-- 13736
bash-3.2$ unzip awscliv2.zip
Archive:  awscliv2.zip
   creating: aws/
   creating: aws/dist/
  inflating: aws/install             
  inflating: aws/README.md           
   creating: aws/dist/_struct/
   creating: aws/dist/awscli/
   creating: aws/dist/botocore/
   creating: aws/dist/docutils/
   creating: aws/dist/include/
   creating: aws/dist/lib/
   creating: aws/dist/zlib/
 [.....]
bash-3.2$ sudo ./aws/install
Password:
You can now run: /usr/local/bin/aws2 --version

bash-3.2$ aws2 --version
aws-cli/2.0.0dev0 Python/3.7.4 Darwin/18.7.0 botocore/2.0.0dev0

Once you’ve got the AWS CLI installed, you will need to create a ‘programmatic’ user and its credentials to allow access from the AWS CLI to the AWS account you want to consume. Go the AWS Console and Identify and Access Management (IAM) and create a new User.

Add IAM user with Programmatic access
Add IAM user with Programmatic access

Attach the right permissions. You might not want your AWS CLI User account to have full access to all AWS services. In my case, I want to use EC2, S3 and DynamoDB from the CLI so I will attach these policies to my user.

Attach IAM policies
Attach IAM policies

Once created, I get an Access Key ID and a Secret Access Key.

IAM User with keys
IAM User with keys

Going back to my Terminal, you have to do the following to set your credentials. Use the Keys from the previous step. You can also specify the default region and output format. I select AWS Frankfurt and json. Use ‘aws configure’ if you’re using AWS CLI Version 1.

bash-3.2$ aws2 configure
AWS Access Key ID [None]: YYYYYYYYYY
AWS Secret Access Key [None]: XXXXXXXXXXXXX
Default region name [None]: eu-central-1
Default output format [None]: json

AWS Auto-Completer

A feature I haven’t seen yet with AWS CLI Version 2 is the ‘auto-complete’. Given how long some of the AWS CLI commands are, it’s quite painful to use a CLI without the ability to use ‘tab’ and for the syntax to be finished or suggested like you can see below.

The commands below activate the auto-completer.

bash-3.2$ ls /usr/local/aws/bin | grep aws_completer
aws_completer
bash-3.2$ 
bash-3.2$ complete -C '/usr/local/aws/bin/aws_completer' aws
bash-3.2$ aws g[tab]
gamelift            glacier             globalaccelerator   glue                greengrass          guardduty 

AWS CLI in action

The first thing I wanted to refresh my memory on was around creating an EC2 instance. To configure an instance, you always need a security group to secure it and a key pair to access it.

So first, we need to create the key pair and the security group using the CLI. It’s straight-forward.

bash-3.2$ aws2 ec2 create-key-pair --key-name MyAWS2CLIKeyPair --query 'KeyMaterial' --output text > MyAWS2CLIKeyPair.pem
bash-3.2$ 
bash-3.2$ aws2 ec2 create-security-group --group-name MyAWS2CLI-SG --description "My AWS Security Group" --vpc-id vpc-05e331f249df27b2d
{
    "GroupId": "sg-043e35f5171b20d0e"
}

I always like to see the GUI and the CLI side-by-side. Unsurprisingly, it works pretty smoothly:

Once I have created my key pair and my security group, I set out to create my EC2 instance.

I initially made a mistake when I picked a subnet and a security-group in different VPCs:

bash-3.2$ aws ec2 run-instances --image-id ami-00aa4671cbf840d82 --count 1 --instance-type t2.micro --key-name MyAWS2CLIKeyPair --security-group-ids sg-043e35f5171b20d0e --subnet-id subnet-fb2207b6

An error occurred (InvalidParameter) when calling the RunInstances operation: Security group sg-043e35f5171b20d0e and subnet subnet-fb2207b6 belong to different networks.

I rectified my error and my EC2 instance was launched.

bash-3.2$ aws ec2 run-instances --image-id ami-00aa4671cbf840d82 --count 1 --instance-type t2.micro --key-name MyAWS2CLIKeyPair --security-group-ids sg-043e35f5171b20d0e --subnet-id subnet-0b52935f5c35bcd89
{
    "Instances": [
        {
            "Monitoring": {
                "State": "disabled"
            }, 
            "PublicDnsName": "", 
            "StateReason": {
                "Message": "pending", 
                "Code": "pending"
            }, 
            "State": {
                "Code": 0, 
                "Name": "pending"
            }, 
            "EbsOptimized": false, 
            "LaunchTime": "2019-11-13T16:31:43.000Z", 
            "PrivateIpAddress": "172.16.6.16", 
            "ProductCodes": [], 
            "VpcId": "vpc-05e331f249df27b2d", 
            "CpuOptions": {
                "CoreCount": 1, 
                "ThreadsPerCore": 1
            }, 
            "StateTransitionReason": "", 
            "InstanceId": "i-0d5bfef9c2c2ff1b5", 
            "ImageId": "ami-00aa4671cbf840d82", 
            "PrivateDnsName": "ip-172-16-6-16.eu-central-1.compute.internal", 
            "KeyName": "MyAWS2CLIKeyPair", 
            "SecurityGroups": [
                {
                    "GroupName": "MyAWS2CLI-SG", 
                    "GroupId": "sg-043e35f5171b20d0e"
                }
            ], 
            "ClientToken": "", 
            "SubnetId": "subnet-0b52935f5c35bcd89", 
            "InstanceType": "t2.micro", 
            "CapacityReservationSpecification": {
                "CapacityReservationPreference": "open"
            }, 
            "NetworkInterfaces": [
                {
                    "Status": "in-use", 
                    "MacAddress": "02:60:47:58:55:42", 
                    "SourceDestCheck": true, 
                    "VpcId": "vpc-05e331f249df27b2d", 
                    "Description": "", 
                    "NetworkInterfaceId": "eni-0137efa06796ba554", 
                    "PrivateIpAddresses": [
                        {
                            "Primary": true, 
                            "PrivateIpAddress": "172.16.6.16"
                        }
                    ], 
                    "SubnetId": "subnet-0b52935f5c35bcd89", 
                    "Attachment": {
                        "Status": "attaching", 
                        "DeviceIndex": 0, 
                        "DeleteOnTermination": true, 
                        "AttachmentId": "eni-attach-06121023fee0b4c50", 
                        "AttachTime": "2019-11-13T16:31:43.000Z"
                    }, 
                    "Groups": [
                        {
                            "GroupName": "MyAWS2CLI-SG", 
                            "GroupId": "sg-043e35f5171b20d0e"
                        }
                    ], 
                    "Ipv6Addresses": [], 
                    "OwnerId": "XXXXXXXXXX", 
                    "PrivateIpAddress": "172.16.6.16"
                }
            ], 
            "SourceDestCheck": true, 
            "Placement": {
                "Tenancy": "default", 
                "GroupName": "", 
                "AvailabilityZone": "eu-central-1a"
            }, 
            "Hypervisor": "xen", 
            "BlockDeviceMappings": [], 
            "Architecture": "x86_64", 
            "RootDeviceType": "ebs", 
            "RootDeviceName": "/dev/xvda", 
            "VirtualizationType": "hvm", 
            "AmiLaunchIndex": 0
        }
    ], 
    "Groups": [], 
    "OwnerId": "XXXXXXXXXX"
}

Now that worked a treat but what I really wanted was to create a S3 bucket.

There are actually two ways to interact with S3 objects via the AWS CLI – the high-level commands (aws s3 like you can see below) but also the API Level commands (aws s3api). The first ones are very simple and let you create and remove buckets and objects and synchronize them between a local folder.

bash-3.2$ aws s3 mb s3://terraform-s3-backend-nvibert 

make_bucket: terraform-s3-backend-nvibert

bash-3.2$ aws s3 cp "Nico Vibert 1.JPG" s3://terraform-s3-backend-nvibert 
upload: ./Nico Vibert 1.JPG to s3://terraform-s3-backend-nvibert/Nico Vibert 1.JPG
bash-3.2$

By default, this bucket and this file was not publicly accessible.

If you want to do something more sophisticated, you need to use the API-level commands.

In my case, I want to be able to use S3 versioning so that I can keep track of my Terraform-managed resources and state and rollback if necessary.

aws s3api put-bucket-versioning --bucket terraform-s3-backend-nvibert --versioning-configuration Status=Enabled

In a future post, I will set up Terraform with a couple of different backends: S3 and HashiCorp Consul.

Thanks for reading!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s