In this article, we'll explore how we can use an AWS IAM user and grant it permissions inside a k8s cluster on AWS EKS. I'll assume you have an EKS cluster already created.

In AWS EKS, the IAM user used to create AWS EKS is the default k8s admin user and we can also declare other new users/roles via aws-auth ConfigMap.

Adding new users in AWS EKS has two parts

1. AWS IAM
2. RBAC (Role-Based Access Management on Kubernetes)

In this article, we will add two users

1. One with administrator permission for management - k8sadmin
2. One to manage only deployments/replicasets/pods to a particular namespace

We can grant IAM users K8s permissions without attaching any AWS permission. Here's how.

Create the AWS IAM K8s Admin User

This IAM user will not need AWS console access, but programmatic access will be necessary. No need to set any permissions for AWS account and tags. Let's create the k8sadmin user and save the credentials(AWS access and secrets key) and IAM arn, we'll need them soon.

Use this AWS user guide for help on creating this user.

Now we need to

1. Edit the aws-auth configmap.

    $ kubectl edit configmap aws-auth -n kube-system
   

2. Map the IAM user k8sadmin arn to a pre-defined systems:masters group. That group gives admin rights on the k8s cluster to the new IAM user.

      data:
        mapRoles: |
          - groups:
            - system:bootstrappers
            - system:nodes
            rolearn: arn:aws:iam::<accountid>:role/<node-role>
            username: system:node:{{EC2PrivateDNSName}}
        mapUsers: |
          - userarn: arn:aws:iam::<accountid>:user/k8sadmin
            username: k8sadmin
            groups:
              - system:masters
      kind: ConfigMap
      metadata:
        creationTimestamp: "2021-07-08T09:47:15Z"
        name: aws-auth
        namespace: kube-system
        resourceVersion: "2315"
        selfLink: /api/v1/namespaces/kube-system/configmaps/aws-auth
        uid: 12c84ad916b6-5df31bad-a165-11e9-90f3
   

3. To test new IAM user run the aws configure --profile k8sadmin command and configure the IAM user credentials for the key and secret key.

4. Export the environmental varaible AWS_DEFAULT_PROFILE to the new IAM user with the following command: export AWS_DEFAULT_PROFILE=k8sadmin

5. Check the IAM user in use with the aws sts get-caller-identity command:

     $ aws sts get-caller-identity
       {
                "UserId": "AIDAVF3YR75UYCMUXIBIA",
                "Account": "<accountid>",
                "Arn": "arn:aws:iam::<accountid>:user/k8sadmin"
        }
   

6. Validate the k8sadmin user has admin rights with the following kubectl commands. If all the kubectl commands return real data, our k8s is using the new user.

   $ kubectl get nodes
    NAME                                          STATUS   ROLES    AGE    VERSION
    ip-**-**-*-***.ap-south-1.compute.internal   Ready    <none>   99d    v1.18.9-eks-d1db3c
    ip-**-**-*-***.ap-south-1.compute.internal   Ready    <none>   113d   v1.18.9-eks-d1db3c
   $ kubectl -n kube-system get pod
    NAME                       READY   STATUS    RESTARTS   AGE
    aws-node-q6xgv             1/1     Running   0          51m
    aws-node-h9qw4             1/1     Running   0          51m
    coredns-8f33c5c4b9-5j9ck   1/1     Running   0          21m
    coredns-9f21c6c4b9-sl9t5   1/1     Running   0          21m
    kube-proxy-ffrkb           1/1     Running   0          51m
    kube-proxy-tnr6r           1/1     Running   0          51m
   

Now we have an IAM user with no permissions in our AWS account, but with admin rights in our Kubernetes cluster

Create Deployment Only User

Our administrative rights are ready to go, but we need to establish our user which only has deployment permissions in the development namespace. We will use our k8sadmin administrative user to create this limited user.

Let's start by creating the namespace.

$ kubectl create namespace development

We will use the reference from the below article to create the k8s role and role-binding.

Bitmani Configure RBAC in your Kubernetes Cluster

1. Create the role for managing deployments in a new file called role-deployment-manager.yaml. In this role YAML, we are adding the rules that allow a user to execute several operations(get/list/watch/create/update/patch/delete) on Deployments, Pods, and ReplicaSets.

  kind: Role
  apiVersion: rbac.authorization.k8s.io/v1beta1
  metadata:
    namespace: development
    name: development-manager
  rules:
  - apiGroups: ["", "extensions", "apps"]
    resources: ["deployments", "replicasets", "pods"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

2. Create the role in the above YAML

$ kubectl apply -f role-deployment-manager.yaml
  role.rbac.authorization.k8s.io/deployment-manager created

3. Bind the role to the developer user by creating this file: rolebinding-deployment-manager.yaml:

  kind: RoleBinding
  apiVersion: rbac.authorization.k8s.io/v1beta1
  metadata:
    name: development-manager-binding
    namespace: development
  subjects:
  - kind: User
    name: developer
    apiGroup: ""
  roleRef:
    kind: Role
    name: development-manager
    apiGroup: ""

4. Create the above role-binding with kubectl:

$ kubectl apply -f rolebinding-deployment-manager.yaml
 rolebinding.rbac.authorization.k8s.io/deployment-manager-binding created

Now Let's create an IAM user named developer in the AWS console (as above, only programmatic access, no AWS account permissions, no tags, save credentials, and IAM arn)

After creating developer IAM, edit the aws-auth configmap (kubectl edit configmap aws-auth -n kube-system) and add a new user mapping for the developer IAM user, but this time mapped to the development-manager role.

  data:
    mapRoles: |
      - groups:
        - system:bootstrappers
        - system:nodes
        rolearn: arn:aws:iam::<accountid>:role/<node-role>
        username: system:node:{{EC2PrivateDNSName}}
    mapUsers: |
      - userarn: arn:aws:iam::<accountid>:user/k8sadmin
        username: k8sadmin
        groups:
          - system:masters
      - userarn: arn:aws:iam::<accountid>:user/developer
        username: developer
        groups:
          - development-manager
  kind: ConfigMap
  metadata:
    creationTimestamp: "2021-07-08T09:47:15Z"
    name: aws-auth
    namespace: kube-system
    resourceVersion: "2315"
    selfLink: /api/v1/namespaces/kube-system/configmaps/aws-auth
    uid: 12c84ad916b6-5df31bad-a165-11e9-90f3

To test the developer IAM user permission, we need to create the aws profile for the developer IAM user and configure the AWS key and secrets key and export the new profile.

$ aws configure - profile developer
$ export AWS_DEFAULT_PROFILE="developer"

Now let's test out the developer k8s user permissions by trying to do something we shouldn't be able to

$ kubectl get node
Error from server (Forbidden): nodes is forbidden: User "developer" cannot list resource "nodes" in API group "" at the cluster scope

And we're blocked - perfect, this is the expected output.

Now lets do something we should be able to do, deploy Nginx in the development namespace.

$ kubectl apply -f https://k8s.io/examples/application/deployment.yaml -n development
deployment.apps/nginx-deployment created

Summary

In this article, we covered how to create an AWS IAM user with no AWS permissions. We then granted that IAM user administrative permissions within our AWS EKS cluster, and connected together with the IAM access key and secret.

We also created a limited deployment user which our Dev team (or CI/CD!) can use to deploy their application. We tested to make sure this user didn't have any additional rights.