In this article, we'll explore how we can use an AWS IAM user and grant it permissions inside a k8s cluster on AWS EKS. I'll assume you have an EKS cluster already created.

In AWS EKS, the IAM user used to create AWS EKS is the default k8s admin user and we can also declare other new users/roles via aws-auth ConfigMap.

Adding new users in AWS EKS has two parts

1. AWS IAM
2. RBAC (Role-Based Access Management on Kubernetes)

In this article, we will add two users

1. One with administrator permission for management - k8sadmin
2. One to manage only deployments/replicasets/pods to a particular namespace

We can grant IAM users K8s permissions without attaching any AWS permission. Here's how.

Create the AWS IAM K8s Admin User

This IAM user will not need AWS console access, but programmatic access will be necessary. No need to set any permissions for AWS account and tags. Let's create the k8sadmin user and save the credentials(AWS access and secrets key) and IAM arn, we'll need them soon.

Use this AWS user guide for help on creating this user.

Now we need to

1. Edit the aws-auth configmap.

1 $ kubectl edit configmap aws-auth -n kube-system

2. Map the IAM user k8sadmin arn to a pre-defined systems:masters group. That group gives admin rights on the k8s cluster to the new IAM user.

1   data:
2     mapRoles: |
3       - groups:
4         - system:bootstrappers
5         - system:nodes
6         rolearn: arn:aws:iam::<accountid>:role/<node-role>
7         username: system:node:{{EC2PrivateDNSName}}
8     mapUsers: |
9       - userarn: arn:aws:iam::<accountid>:user/k8sadmin
10         username: k8sadmin
11         groups:
12           - system:masters
13   kind: ConfigMap
14   metadata:
15     creationTimestamp: "2021-07-08T09:47:15Z"
16     name: aws-auth
17     namespace: kube-system
18     resourceVersion: "2315"
19     selfLink: /api/v1/namespaces/kube-system/configmaps/aws-auth
20     uid: 12c84ad916b6-5df31bad-a165-11e9-90f3

3. To test new IAM user run the aws configure --profile k8sadmin command and configure the IAM user credentials for the key and secret key.

4. Export the environmental varaible AWS_DEFAULT_PROFILE to the new IAM user with the following command: export AWS_DEFAULT_PROFILE=k8sadmin

5. Check the IAM user in use with the aws sts get-caller-identity command:

1  $ aws sts get-caller-identity
2    {
3             "UserId": "AIDAVF3YR75UYCMUXIBIA",
4             "Account": "<accountid>",
5             "Arn": "arn:aws:iam::<accountid>:user/k8sadmin"
6     }

6. Validate the k8sadmin user has admin rights with the following kubectl commands. If all the kubectl commands return real data, our k8s is using the new user.

1$ kubectl get nodes
2 NAME                                          STATUS   ROLES    AGE    VERSION
3 ip-**-**-*-***.ap-south-1.compute.internal   Ready    <none>   99d    v1.18.9-eks-d1db3c
4 ip-**-**-*-***.ap-south-1.compute.internal   Ready    <none>   113d   v1.18.9-eks-d1db3c
5$ kubectl -n kube-system get pod
6 NAME                       READY   STATUS    RESTARTS   AGE
7 aws-node-q6xgv             1/1     Running   0          51m
8 aws-node-h9qw4             1/1     Running   0          51m
9 coredns-8f33c5c4b9-5j9ck   1/1     Running   0          21m
10 coredns-9f21c6c4b9-sl9t5   1/1     Running   0          21m
11 kube-proxy-ffrkb           1/1     Running   0          51m
12 kube-proxy-tnr6r           1/1     Running   0          51m

Now we have an IAM user with no permissions in our AWS account, but with admin rights in our Kubernetes cluster

Create Deployment Only User

Our administrative rights are ready to go, but we need to establish our user which only has deployment permissions in the development namespace. We will use our k8sadmin administrative user to create this limited user.

Let's start by creating the namespace.

1$ kubectl create namespace development

We will use the reference from the below article to create the k8s role and role-binding.

Bitmani Configure RBAC in your Kubernetes Cluster

1. Create the role for managing deployments in a new file called role-deployment-manager.yaml. In this role YAML, we are adding the rules that allow a user to execute several operations(get/list/watch/create/update/patch/delete) on Deployments, Pods, and ReplicaSets.

1kind: Role
2  apiVersion: rbac.authorization.k8s.io/v1beta1
3  metadata:
4    namespace: development
5    name: development-manager
6  rules:
7  - apiGroups: ["", "extensions", "apps"]
8    resources: ["deployments", "replicasets", "pods"]
9    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

2. Create the role in the above YAML

1$ kubectl apply -f role-deployment-manager.yaml
2  role.rbac.authorization.k8s.io/deployment-manager created

3. Bind the role to the developer user by creating this file: rolebinding-deployment-manager.yaml:

1 kind: RoleBinding
2  apiVersion: rbac.authorization.k8s.io/v1beta1
3  metadata:
4    name: development-manager-binding
5    namespace: development
6  subjects:
7  - kind: User
8    name: developer
9    apiGroup: ""
10  roleRef:
11    kind: Role
12    name: development-manager
13    apiGroup: ""

4. Create the above role-binding with kubectl:

1$ kubectl apply -f rolebinding-deployment-manager.yaml
2 rolebinding.rbac.authorization.k8s.io/deployment-manager-binding created

Now Let's create an IAM user named developer in the AWS console (as above, only programmatic access, no AWS account permissions, no tags, save credentials, and IAM arn)

After creating developer IAM, edit the aws-auth configmap (kubectl edit configmap aws-auth -n kube-system) and add a new user mapping for the developer IAM user, but this time mapped to the development-manager role.

1  data:
2    mapRoles: |
3      - groups:
4        - system:bootstrappers
5        - system:nodes
6        rolearn: arn:aws:iam::<accountid>:role/<node-role>
7        username: system:node:{{EC2PrivateDNSName}}
8    mapUsers: |
9      - userarn: arn:aws:iam::<accountid>:user/k8sadmin
10        username: k8sadmin
11        groups:
12          - system:masters
13      - userarn: arn:aws:iam::<accountid>:user/developer
14        username: developer
15        groups:
16          - development-manager
17  kind: ConfigMap
18  metadata:
19    creationTimestamp: "2021-07-08T09:47:15Z"
20    name: aws-auth
21    namespace: kube-system
22    resourceVersion: "2315"
23    selfLink: /api/v1/namespaces/kube-system/configmaps/aws-auth
24    uid: 12c84ad916b6-5df31bad-a165-11e9-90f3

To test the developer IAM user permission, we need to create the aws profile for the developer IAM user and configure the AWS key and secrets key and export the new profile.

1$ aws configure - profile developer
2$ export AWS_DEFAULT_PROFILE="developer"

Now let's test out the developer k8s user permissions by trying to do something we shouldn't be able to

1$ kubectl get node
2Error from server (Forbidden): nodes is forbidden: User "developer" cannot list resource "nodes" in API group "" at the cluster scope

And we're blocked - perfect, this is the expected output.

Now lets do something we should be able to do, deploy Nginx in the development namespace.

1$ kubectl apply -f https://k8s.io/examples/application/deployment.yaml -n development
2deployment.apps/nginx-deployment created

Summary

In this article, we covered how to create an AWS IAM user with no AWS permissions. We then granted that IAM user administrative permissions within our AWS EKS cluster, and connected together with the IAM access key and secret.

We also created a limited deployment user which our Dev team (or CI/CD!) can use to deploy their application. We tested to make sure this user didn't have any additional rights.