I have been setting up Terraform with S3 as back-end and one of the blog posts I was consulting suggested using AWS CLI which I hadn’t been using for a couple of years.
So… time to go back to basics!
What is the AWS CLI ?
The AWS Command Line Interface is “a unified tool to manage your AWS services. With just one tool to download and configure, you can control multiple AWS services from the command line and automate them through scripts” (from there).
Obviously there are are many ways to manage and consume AWS resources and I tend to use the GUI the vast majority of the time but it’s good to be aware of alternatives.
AWS CLI Installation
I followed the instructions here and it was fairly straight-forward. I actually downloaded the new CLI version (version 2). It was announced fairly recently. You will see below I went back to using version 1 for a reason I will explain later.
Download the Zip (I was on the train which explains why it took 17 minutes to download 19.2M !), unzip it and install it and you’re good to move on to the next step.
bash-3.2$ curl "https://d1vvhvl2y92vvt.cloudfront.net/awscli-exe-macos.zip" -o "awscliv2.zip"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 19.2M 100 19.2M 0 0 19875 0 0:16:54 0:16:54 --:--:-- 13736
bash-3.2$ unzip awscliv2.zip
Archive: awscliv2.zip
creating: aws/
creating: aws/dist/
inflating: aws/install
inflating: aws/README.md
creating: aws/dist/_struct/
creating: aws/dist/awscli/
creating: aws/dist/botocore/
creating: aws/dist/docutils/
creating: aws/dist/include/
creating: aws/dist/lib/
creating: aws/dist/zlib/
[.....]
bash-3.2$ sudo ./aws/install
Password:
You can now run: /usr/local/bin/aws2 --version
bash-3.2$ aws2 --version
aws-cli/2.0.0dev0 Python/3.7.4 Darwin/18.7.0 botocore/2.0.0dev0
Once you’ve got the AWS CLI installed, you will need to create a ‘programmatic’ user and its credentials to allow access from the AWS CLI to the AWS account you want to consume. Go the AWS Console and Identify and Access Management (IAM) and create a new User.
Attach the right permissions. You might not want your AWS CLI User account to have full access to all AWS services. In my case, I want to use EC2, S3 and DynamoDB from the CLI so I will attach these policies to my user.
Once created, I get an Access Key ID and a Secret Access Key.
Going back to my Terminal, you have to do the following to set your credentials. Use the Keys from the previous step. You can also specify the default region and output format. I select AWS Frankfurt and json. Use ‘aws configure’ if you’re using AWS CLI Version 1.
bash-3.2$ aws2 configure
AWS Access Key ID [None]: YYYYYYYYYY
AWS Secret Access Key [None]: XXXXXXXXXXXXX
Default region name [None]: eu-central-1
Default output format [None]: json
AWS Auto-Completer
A feature I haven’t seen yet with AWS CLI Version 2 is the ‘auto-complete’. Given how long some of the AWS CLI commands are, it’s quite painful to use a CLI without the ability to use ‘tab’ and for the syntax to be finished or suggested like you can see below.
The commands below activate the auto-completer.
bash-3.2$ ls /usr/local/aws/bin | grep aws_completer
aws_completer
bash-3.2$
bash-3.2$ complete -C '/usr/local/aws/bin/aws_completer' aws
bash-3.2$ aws g[tab]
gamelift glacier globalaccelerator glue greengrass guardduty
AWS CLI in action
The first thing I wanted to refresh my memory on was around creating an EC2 instance. To configure an instance, you always need a security group to secure it and a key pair to access it.
So first, we need to create the key pair and the security group using the CLI. It’s straight-forward.
bash-3.2$ aws2 ec2 create-key-pair --key-name MyAWS2CLIKeyPair --query 'KeyMaterial' --output text > MyAWS2CLIKeyPair.pem
bash-3.2$
bash-3.2$ aws2 ec2 create-security-group --group-name MyAWS2CLI-SG --description "My AWS Security Group" --vpc-id vpc-05e331f249df27b2d
{
"GroupId": "sg-043e35f5171b20d0e"
}
I always like to see the GUI and the CLI side-by-side. Unsurprisingly, it works pretty smoothly:
Once I have created my key pair and my security group, I set out to create my EC2 instance.
I initially made a mistake when I picked a subnet and a security-group in different VPCs:
bash-3.2$ aws ec2 run-instances --image-id ami-00aa4671cbf840d82 --count 1 --instance-type t2.micro --key-name MyAWS2CLIKeyPair --security-group-ids sg-043e35f5171b20d0e --subnet-id subnet-fb2207b6
An error occurred (InvalidParameter) when calling the RunInstances operation: Security group sg-043e35f5171b20d0e and subnet subnet-fb2207b6 belong to different networks.
I rectified my error and my EC2 instance was launched.
bash-3.2$ aws ec2 run-instances --image-id ami-00aa4671cbf840d82 --count 1 --instance-type t2.micro --key-name MyAWS2CLIKeyPair --security-group-ids sg-043e35f5171b20d0e --subnet-id subnet-0b52935f5c35bcd89
{
"Instances": [
{
"Monitoring": {
"State": "disabled"
},
"PublicDnsName": "",
"StateReason": {
"Message": "pending",
"Code": "pending"
},
"State": {
"Code": 0,
"Name": "pending"
},
"EbsOptimized": false,
"LaunchTime": "2019-11-13T16:31:43.000Z",
"PrivateIpAddress": "172.16.6.16",
"ProductCodes": [],
"VpcId": "vpc-05e331f249df27b2d",
"CpuOptions": {
"CoreCount": 1,
"ThreadsPerCore": 1
},
"StateTransitionReason": "",
"InstanceId": "i-0d5bfef9c2c2ff1b5",
"ImageId": "ami-00aa4671cbf840d82",
"PrivateDnsName": "ip-172-16-6-16.eu-central-1.compute.internal",
"KeyName": "MyAWS2CLIKeyPair",
"SecurityGroups": [
{
"GroupName": "MyAWS2CLI-SG",
"GroupId": "sg-043e35f5171b20d0e"
}
],
"ClientToken": "",
"SubnetId": "subnet-0b52935f5c35bcd89",
"InstanceType": "t2.micro",
"CapacityReservationSpecification": {
"CapacityReservationPreference": "open"
},
"NetworkInterfaces": [
{
"Status": "in-use",
"MacAddress": "02:60:47:58:55:42",
"SourceDestCheck": true,
"VpcId": "vpc-05e331f249df27b2d",
"Description": "",
"NetworkInterfaceId": "eni-0137efa06796ba554",
"PrivateIpAddresses": [
{
"Primary": true,
"PrivateIpAddress": "172.16.6.16"
}
],
"SubnetId": "subnet-0b52935f5c35bcd89",
"Attachment": {
"Status": "attaching",
"DeviceIndex": 0,
"DeleteOnTermination": true,
"AttachmentId": "eni-attach-06121023fee0b4c50",
"AttachTime": "2019-11-13T16:31:43.000Z"
},
"Groups": [
{
"GroupName": "MyAWS2CLI-SG",
"GroupId": "sg-043e35f5171b20d0e"
}
],
"Ipv6Addresses": [],
"OwnerId": "XXXXXXXXXX",
"PrivateIpAddress": "172.16.6.16"
}
],
"SourceDestCheck": true,
"Placement": {
"Tenancy": "default",
"GroupName": "",
"AvailabilityZone": "eu-central-1a"
},
"Hypervisor": "xen",
"BlockDeviceMappings": [],
"Architecture": "x86_64",
"RootDeviceType": "ebs",
"RootDeviceName": "/dev/xvda",
"VirtualizationType": "hvm",
"AmiLaunchIndex": 0
}
],
"Groups": [],
"OwnerId": "XXXXXXXXXX"
}
Now that worked a treat but what I really wanted was to create a S3 bucket.
There are actually two ways to interact with S3 objects via the AWS CLI – the high-level commands (aws s3 like you can see below) but also the API Level commands (aws s3api). The first ones are very simple and let you create and remove buckets and objects and synchronize them between a local folder.
bash-3.2$ aws s3 mb s3://terraform-s3-backend-nvibert
make_bucket: terraform-s3-backend-nvibert
bash-3.2$ aws s3 cp "Nico Vibert 1.JPG" s3://terraform-s3-backend-nvibert
upload: ./Nico Vibert 1.JPG to s3://terraform-s3-backend-nvibert/Nico Vibert 1.JPG
bash-3.2$
By default, this bucket and this file was not publicly accessible.
If you want to do something more sophisticated, you need to use the API-level commands.
In my case, I want to be able to use S3 versioning so that I can keep track of my Terraform-managed resources and state and rollback if necessary.
aws s3api put-bucket-versioning --bucket terraform-s3-backend-nvibert --versioning-configuration Status=Enabled
In a future post, I will set up Terraform with a couple of different backends: S3 and HashiCorp Consul.
Thanks for reading!
nicovibert.com 