On Sep 28, 2021

Determining AWS IAM Policies According To Terraform And AWS CLI

Meir Gabay
Meir GabayDevOps Engineer

Let's write a practical blog post of how to implement this principle in the CI/CD realm.

I find myself mentioning the term Principle Of Least
Privilege
often, so I thought, "Let's write a practical blog post of how to implement this principle in the CI/CD realm".

In this blog post, I'll describe the process of granting the least privileges required to execute `aws s3 ls` and `terraform apply` by a CI/CD runner.

HIPAA

In case you're from health tech, this process might help you with qualifying
some of the HIPAA compliance requirements, see HIPAA's Minimum Necessary
Requirement
.

Scenario

Imagine this, you've created an IAM user `cicd-user`, with
AdministratorAcess and generated the access keys `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` for that user. You've created that user so that your CI/CD service, whatever it is, GitHub Actions, drone.io, Jenkins, etc.,
will be able to apply changes in your AWS account.

This is a common scenario that usually happens in small startup companies,
where the product and sales are far more important than meeting regulations and
securing the product. But why do people do that? Because every time a CI/CD
attempts to do something, figuring out which policies are required for the job is a nightmare.

"Let's provide the CI/CD service an admin permission, we'll deal with that later, we must focus on the product."

TIP: Listen to Avenged Sevenfold - Nightmare while reading this blog-post.

The Nightmare

A typical "Nightmare Situation" where you need to create an IAM policy for a
CI/CD service; here goes.

  1. Create an IAM user with no permissions and generate access keys for the CI/CD service.
  2. IAM user (CI/CD job or you) invokes some `aws` or `terraform` command.
  3. If the operation fails due to authorization (403) issues, then you'll get
    an error message that states which permission(s), usually it's singular, is
    required.
  4. Add the required permission(s) to the user's IAM policy.
  5. Start again from step 2 - invoke `aws` or `terraform` ...

NOTE: Sometimes, you'll get `Forbidden status code: 403`, which is quite
useless for finding out the required IAM permissions.

There's A Tool For That

Here comes iamlive by Ian Mckay.

Generate an IAM policy from AWS calls using client-side monitoring (CSM) or embedded proxy

I'll focus on the Proxy Mode, which supports running `iamlive` as a Docker container. No worries, we'll get to that.

Proxy Mode? What Do you Mean?

The way `iamlive` works is pretty simple. `iamlive` runs in the background in
proxy mode and serves `0.0.0.0:10080`, which allows access from "any IP", but
only we have access to this process, so we're good. In a separated terminal,
you set `HTTP_PROXY`, `HTTPS_PROXY`, and `AWS_CA_BUNDLE` according to iamlive.

Running iamlive in Docker


Build The Image Locally

For full visibility, since we're talking about credentials, let's build the
Docker image locally. Copy-paste the following `Dockerfile`:

Dockerfile

1```
2ARG GO_VERSION=1.16.3
3ARG REPO_NAME=""
4ARG APP_NAME="iamlive"
5ARG APP_PATH="/go/src/iamlive"
6
7
8
9# Dev
10FROM golang:${GO_VERSION}-alpine AS dev
11RUN apk add --update git
12ARG APP_NAME
13ARG APP_PATH
14ENV APP_NAME="${APP_NAME}" \
15APP_PATH="${APP_PATH}" \
16GOOS="linux"
17WORKDIR "${APP_PATH}"
18COPY . "${APP_PATH}"
19ENTRYPOINT ["sh"]
20
21
22
23# Build
24FROM dev as build
25RUN go install
26ENTRYPOINT [ "sh" ]
27
28
29
30# App
31FROM alpine:3.12 AS app
32RUN apk --update upgrade && \
33apk add --update ca-certificates && \
34update-ca-certificates
35WORKDIR "/app/"
36COPY --from=build "/go/bin/iamlive" ./iamlive
37RUN addgroup -S "appgroup" && adduser -S "appuser" -G "appgroup" && \
38chown "appuser:appgroup" "./iamlive"
39
40
41
42USER "appuser"
43EXPOSE 10080
44ENTRYPOINT ["./iamlive"]
45CMD ""
46```
47
48
49
50

We need the source code of iamlive to build the Docker image, so let's `git clone` it and then build the Docker image. Though before building the image, let's add a
.dockerignore file to copy only the required files to build the `iamlive` application.

dockerignore

1**
2!LICENSE
3!service/
4!vendor/
5!.gon-*.json
6!map.json
7!iam_definition.json
8!*.go
9!*.mod
10!*.sum

Now let's get the source code and build the Docker image.

1# Get source code
2git clone https://github.com/iann0036/iamlive.git
3cd iamlive
4
5
6
7# Build the Docker image from source code and tag it
8docker build -t iamlive-test .

Run iamlive-test Docker Container

We need to proxy all of `aws` and `terraform` requests via `iamlive-test`, and then iamlive will be able to generate the relevant IAM permissions according to the invoked request. Pretty awesome, right?

Zooming in on some of the arguments

  1. 1. `-p 80:10080` and `-p 443:10080` - Maps the ports 80 and 443 in the Host to the Container port 10080
  2. `--bind-addr 0.0.0.0:10080` - iamlive listens on port 10080, from any IP address
  3. `--force-wildcard-resource` - Makes it easier to iterate over missing permissions
  4. `--output-file "/app/iamlive.log"` - Save the generated permissions to a file upon kill -HUP 1.

1docker run \
2-p 80:10080 \
3-p 443:10080 \
4--name iamlive-test \
5-it iamlive-test \
6--mode proxy \
7--bind-addr 0.0.0.0:10080 \
8--force-wildcard-resource \
9--output-file "/app/iamlive.log"
10# Runs in the background ...
11# Average Memory Usage: 88MB

Using The Proxy

First, I recommend that you create a fresh new IAM user with no permissions at
all
, let's name that user `dummy-user`. Doing so will ease getting the minimum
required permissions (all of them).

The fact that the `iamlive-test` container is running means nothing to `aws` and `terraform`. To configure both CLIs to use this proxy server, open a new terminal window and execute the below commands.

1export AWS_ACCESS_KEY_ID="AKIA_DUMMY_USER_ACCESS_KEY_ID"
2export AWS_SECRET_ACCESS_KEY="DUMMY_USER_SECRET_ACCESS_KEY"
3
4
5export HTTP_PROXY=http://127.0.0.1:80 \
6HTTPS_PROXY=http://127.0.0.1:443 \
7AWS_CA_BUNDLE="${HOME}/.iamlive/ca.pem"

Say what? From where did this AWS_CA_BUNDLE come from? Well, this environment variable instructs tools that use the AWS SDK to trust the provided Certificate Authority Certificate, in our case, it's `ca.pem`.

The `ca.pem` file is generated by `iamlive` for each execution of the `iamlive-test` container. We need to copy the `ca.pem` file from the container to our machine (Host).

1docker cp iamlive-test:/home/appuser/.iamlive/ ~/

The environment variables HTTP_PROXY and HTTPS_PROXY are telling AWS's SDK to forward traffic via a proxy server (iamlive-test container).

Generating IAM Policies With Relevant Permissions

So far, we've got two terminal windows. In the first terminal, we have the
`iamlive-test` container running, and it probably looks like it's doing nothing,
but it's running, trust me. In the second terminal, we exported the relevant
environment variables and copied `ca.pem` from the `iamlive-test` docker container to our machine (Host).

Executing A Command With AWS CLI

In the second terminal, we'll execute commands with `aws` and `terraform`. After the command execution is completed, we'll inspect the logs of the first
terminal, which runs the `iamlive-test` container.

1aws s3 ls
2
3# Output
4# An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied

Really? All I need is the `ListBuckets` permission? The real permission is called
`s3:ListAllMyBuckets`, and I know that because the logs of `iamlive-test` look like this.

json

1{
2"Version": "2012-10-17",
3"Statement": [
4{
5"Effect": "Allow",
6"Action": [
7"s3:ListAllMyBuckets"
8],
9"Resource": "*"
10}
11]
12}

Executing A Command With Terraform CLI

Before we proceed, it's important to mention that `terraform init` cannot be
proxied via `iamlive-test` since it attempts to access registry.terraform.io,
and it's not covered by `iamlive`. So first, unset the proxy settings, and then
execute `terraform init`.

This is what it looks like when you attempt to execute `terraform init` with the
proxy settings (environment variables) on.

1terraform init
2
3Error: Failed to query available provider packages
4
5Could not retrieve the list of available versions for provider
6hashicorp/aws: could not connect to registry.terraform.io: Failed to
7request discovery document: Get
8"https://registry.terraform.io/.well-known/terraform.json": x509:
9certificate signed by unknown authority
10
11
12

For testing purposes, create a new directory and add the following `main.tf`
file.

1variable "region" {
2type = string
3default = "eu-west-1"
4}
5
6provider "aws" {
7region = var.region
8}
9
10resource "aws_s3_bucket" "app" {}

Unset proxy related environment variables and then execute terraform init

1unset HTTP_PROXY HTTPS_PROXY AWS_CA_BUNDLE
2
3mkdir terraform-iamlive
4cd terraform-iamlive
5vim main.tf # copy-paste the above main.tf file
6
7terraform init
8# Terraform has been successfully initialized!

Finally, we can execute `terraform apply` and see which permissions are required
for the task.

1export AWS_ACCESS_KEY_ID="AKIA_DUMMY_USER_ACCESS_KEY_ID"
2export AWS_SECRET_ACCESS_KEY="DUMMY_USER_SECRET_ACCESS_KEY"
3
4export HTTP_PROXY=http://127.0.0.1:80 \
5HTTPS_PROXY=http://127.0.0.1:443 \
6AWS_CA_BUNDLE="${HOME}/.iamlive/ca.pem"
7
8# In terraform-iamlive dir
9terraform apply
10
11# Output
12# Error: error reading S3 Bucket (terraform-20210422212704452600000001): Forbidden: Forbidden
13# │ status code: 403, request id: A25SVTBABN0B3DSH, host id: /c1b5TsnsBE23AaDDHJQ34yLAYdrR7y3kvu2lqEX7VvstffawROKWwcPYfxNjleeluZPg9nucKY=

No idea what that means; let's check `iamlive-test` container logs to see if
`iamlive` knows which permissions are required.

json

1{
2"Version": "2012-10-17",
3"Statement": [
4{
5"Effect": "Allow",
6"Action": [
7"sts:GetCallerIdentity",
8"ec2:DescribeAccountAttributes",
9"s3:ListBucket"
10],
11"Resource": "*"
12}
13]
14}

Sometimes, you won't get the full list of required permissions. To overcome
that, add the given IAM policy and invoke `terraform apply` again to see which
permissions are missing.

Also, you might want to limit "`*`" to specific resources or patterns, but still,
it's better than the current nightmare.

Here's the output after adding the above IAM policy to my "dummy-user".

1{
2"Version": "2012-10-17",
3"Statement": [
4{
5"Effect": "Allow",
6"Action": [
7"sts:GetCallerIdentity",
8"ec2:DescribeAccountAttributes",
9"s3:ListBucket",
10"s3:GetBucketAcl"
11],
12"Resource": "*"
13}
14]
15}

A new permission was added - `s3GetBucketAcl`. We need to iterate this process a few times, but each time it's a simple copy-paste of the generated permissions to the existing IAM policy in AWS Console.

This is the final result, after eight (8) iterations. If you're about to deploy
a large stack with multiple resources, you'll have to iterate more than a few
times.

1{
2"Version": "2012-10-17",
3"Statement": [
4{
5"Effect": "Allow",
6"Action": [
7"sts:GetCallerIdentity",
8"ec2:DescribeAccountAttributes",
9"s3:ListBucket",
10"s3:GetBucketAcl",
11"s3:GetBucketCORS",
12"s3:GetBucketWebsite",
13"s3:GetBucketVersioning",
14"s3:GetAccelerateConfiguration",
15"s3:GetBucketRequestPayment",
16"s3:GetBucketLogging",
17"s3:GetLifecycleConfiguration",
18"s3:GetReplicationConfiguration",
19"s3:GetEncryptionConfiguration",
20"s3:GetBucketObjectLockConfiguration",
21"s3:GetBucketTagging",
22"s3:CreateBucket"
23],
24"Resource": "*"
25}
26]
27}

IMPORTANT: Remember, limit "`*`" to specific resources or patterns.

NOTE: After adding the policy to an IAM user, it took a few retries to get the
updated IAM policy from `iamlive-test`. Just keep on adding permissions until a
successful attempt.

Stop And Start iamlive-test

The beauty of omitting the flag `--rm` in the `docker run` command is that
`iamlive-test` will **not be removed** when it stops. This is important! We want to keep the same `ca.pem` in the next execution of `iamlive-test` instead of
re-copying `ca.pem` from `iamlive-test` to our machine (Host).

1# Hit CTRL+C To stop the container
2
3docker start -i iamlive-test
4# Keep it running in the background

Get The Lastest Generated IAM Policy

We can send a SIGHUP signal to the `iamlive-test` container, which instructs
`iamlive` to dump its latest output to the file `iamlive.log`. I piped the output
through jq to beautify it.

1docker exec iamlive-test kill -HUP 1 && \
2docker exec iamlive-test cat /app/iamlive.log | jq

Stop Using The Proxy

To avoid using a proxy server for AWS
SDK
operations, unset the relevant environment variables, or restart your terminal
window.

1unset HTTP_PROXY HTTPS_PROXY AWS_CA_BUNDLE

Alternatives

CloudTrail

I've tried getting the required permissions by investigating AWS
CloudTrail logs
, and again, it was a nightmare. I just wanted a simple way, with the least overhead, to generate permissions easily for my CI/CD services.

IAM Policy Simulator

In case you don't know, AWS provides the free service IAM Policy
Simulator
, which is great for testing and debugging IAM policies in your AWS account. Then again, different tools for different purposes.

IAM Access Analyzer

Fresh from the oven, AWS extended the capabilities of IAM Access
Analyzer
, posted on Apr 19, 2021. Though its capabilities are still limited and do not
cover all AWS services. It's still nice to see that AWS is improving the
capabilities to ease writing least privilege IAM policies.

Final Thoughts

I'll probably write some Bash script that invokes `terraform apply` and when the error code equals `403`, adds the output of `iamlive.log` to the relevant IAM policy in AWS. That might make it friendlier than it is now; it's annoying to
copy-paste.

Got a better way to achieve the same thing? Have some thoughts about this
process? Let's discuss it; feel free to comment below!

Cover photo by Dim Hou on Unsplash

    Get the IAM Pulse Check Newsletter

    We send out a periodic newsletter full of tips & tricks, contributions from the community, commentary on the industry, relevant social posts, and more.

    Checkout past issues for a sampling of the goods.