Security is paramount in cloud environments and an IAM service can help, but there are some limits to know and to manage.
Security is paramount in cloud environments and IAM (Identity and Access Management) service helps on any cloud provider (you can find IAM services on Azure, AWS and GCP for example).
On Google Cloud, the IAM service uses OAuth and OpenID protocols. It allows for authentication and authorization of an account (user account or service account).
The authorization is performed only for Google Cloud components; you can’t add your custom authorization/permissions for your own app in their IAM service.
The authentication part uses OAuth protocols to generate a credential. You can use your own credential (user account, with interactive authentication in a browser) or a technical credential (service account).
Service account credentials are automatically loaded on the Google Cloud environment (I'll explain how later). However, you can also generate a service account key file to use this credential anywhere - and it’s an ugly practice especially to keep the secret…secret!
Ugly service account key files
Service account key files are useful in some cases, but they are also ugly when badly used.
First, think about what is it: a file. A simple file.
A file is the most common object in computing. You can copy it, send it by email, commit it into a Git repository. Sometimes this repository is public and you receive an email from Google Cloud that informs you about the leak of your secrets file.
In my company, we had this issue occur twice with bad actors that silently created VMs with bitcoin miner. We fixed it quickly and we limited the cost. Luckily, our dev projects did not experience more dramatic issues (confidential data exfiltration for example). I will release another article to present how we tackled the problem to increase our security.
Service account key files can be shared between developers, who may sometimes be more focused on the features’ development than on keeping this secret file safe. There can also be external developers who work for a while in your company and then go elsewhere with the key file still in their computer.
So, if the users aren’t aware of the confidentiality of this key file, you quickly loose control of it. That leads to the second problem: Google recommends to regularly rotate the service account key file, at least every 90 days.
How do you perform this key rotation is you don’t control the key files?
Because of this, service account key files are a nightmare to manage. They are useful in specific cases, when the Application Default Credential isn’t enough to solve your authentication. In all other cases…
To avoid any leaks of secrets, never have stored secrets
Secrets never stored
Based on this, IAM service allows to never have to store secrets, like service account key file, and to work seamlessly in your local environment and on Google Cloud.
Therefore, the principle is to use the environment context to be authenticated by IAM service. This strategy is called ADC (Application Default Credential) and the Google Cloud client libraries support this authentication mode in several languages. The library retrieves, according with the environment, the credential, and uses it as the default credential in the application.
In Python, for example, it’s the google-auth library that allows you to do this.
import google.auth credentials, project_id = google.auth.default()
You never mention your environment or an account, you let the library do its job. It also works for component libraries (like Google Storage): default constructor() or keyword defaultCredential are used in that case.
ADC on local environment
You can use your user credentials, the same that you use in the Google Cloud console.
- To use your user account credential in the gcloud command line, get your credential like this
gcloud auth login
- To use your user credential in your code and use the ADC, configure your local environment with this command
gcloud auth application-default login
In both cases, you will have to go in your browser, to select your Google account, if many. You might have to re-authenticate yourselves, maybe with 2 factors mechanism. And finally authorize the use of your account with gcloud SDK.
Only a refresh token is stored locally, never your authentication credential login/password or secrets.
ADC on Google Cloud Environment
All Google Cloud services have access to Metadata server. This internal server provides informations on the environment, including the service account used for the service. And thus, the Google Cloud client library can detect this server and get credentials through this Metadata server.
Some products allow you to customize the service account that you want to use in the service, such as Cloud Functions and Cloud Run. Others may not when, it’s a default service account which is used. But, in any case, a service account is loaded and usable.
Limits of ADC
ADC works well when you use your user credential or when you run your code on Google Cloud products.
What happens when you have to connect your on-prem environment to Google Cloud?
Your CI/CD (gitlab CI or Github Action for example)?
Or other applications hosted on other Cloud Provider?
There is no magic solution, you need a credential file to be authenticated, with a secret (a private key) store in it: the service account key file. The service account key files are mainly designed and useful in this context.
Other ADC limitations
However, even with local user account credentials or service account credentials on Google Cloud environment, there are still 2 main limitations;
The inability to add scope to the App Engine default service account on App Engine
The inability to generate a signed identity token to reach private Cloud Run and Cloud Functions with a local user credential.
For these 2 cases, service account impersonation is preferred (to avoid key file generation), but it’s not always the safest solution.
1. App Engine limitation
App Engine was the first product of Google Cloud and is more than 12 years old! It allows you to deploy a set of (micro)services to serve a web application. However, as an older product, there are some legacies that limit you in the future.
App Engine has 2 limitations:
- All services on App Engine have the same default service account, and you can’t customize it (on App Engine or per service). That’s a concern because all your services on App Engine will have the same level of permission, and it breaks the least privilege principle.
- App Engine default service account can’t be scoped. More exactly, the scope of the credential allows you to reach all Google Cloud services, not more. In my company use case, we are migrating App Maker apps (the service stops in January 2021) to App Engine. The existing App Maker apps use a lot of GSuite documents (Sheet especially). To access to Sheet API, you need to scope your credential with https://www.googleapis.com/auth/spreadsheets scope. And you can’t.
The best way is to use service account impersonation (and thus to avoid the service account key file). In other words, you won’t use directly the ADC to access to the service, but you will use them to generate a credential on behalf of another service account. With IAM service, you can manage which service account can be impersonated or not.
You use the same ADC (the App Engine default service account) to impersonate all the service accounts that you want. That means, “service1” in App Engine can impersonate a “service account1” thanks to the App Engine default credential, and “service1” can also impersonate the “service account2” (initially created for the “service2”). And at the end, if all services can access all “impersonate-able” service accounts, that breaks the least privilege principle.
The Cloud Functions default service account is also the App Engine default service account. If you don’t customize your Cloud Functions identity during the deployment, your function has automatically the same permissions as App Engine services.
Additional issue: Python and Java Google Auth library natively include impersonation methods. It’s not the case for the other languages.
When you use impersonation on App Engine, you can also use it with your user credential in your local environment. Therefore, your code on App Engine and on your local is the same and the code that you test locally is exactly the same as this one that will be run on App Engine.
You can also generate service account key file per service and load them with your code.
It’s useless to store key files inside Secret Manager and to retrieve them, at runtime, with the App Engine default service account credential, because we go back to the #1 tradeoffs. And in this case, impersonation is a far better solution.
In my company, we sadly use service account key file for App Engine critical services only, not for all.
If you are deploying on another component other than App Engine, like Cloud Functions and Cloud Run, you can use ADC. However, you will have scope issue with your personal user account in your local environment.
To solve this, scope your credential when you create it by defining the scope like this
gcloud auth application-default login \ --scopes='[https://www.googleapis.com/auth/spreadsheets](https://www.googleapis.com/auth/spreadsheets)',\ '[https://www.googleapis.com/auth/cloud-platform](https://www.googleapis.com/auth/cloud-platform)'
2. Private Cloud Functions and Cloud Run limitation
The second issue is when your local code tries to call a service on Cloud Functions or on Cloud Run deployed in private mode. This mode implies the caller to present an signed identity token.
To call the privately deployed services with a simple curl, you can do this
curl -H "Authorization: Bearer \ $(gcloud auth print-identity-token)" https://service.url.run.app
Gcloud SDK has the capability to generate a signed identity token with your user credential. However, the Google Auth client libraries do not implement this feature for user account credential to reach the deployed service directly from your app code (for example a service being developed locally that call another service deployed privately on Google Cloud).
Impersonation is also an option here. But, I don’t like this solution because you have to perform a hook in your code.
If "I'm using user account credential" then "impersonate a service account" else "use ADC"
And thus, your code doesn’t run exactly in the same way in local and in the Cloud. Therefore, you could have issues on Google Cloud that you haven't detected locally. It’s also not really safe (a bug point-of-view, not at security point-of-view).
Of course, you could impersonate service account in any case, but that increases your app complexity (and thus decreases the maintainability).
The latest option is to use service account key file, only for local development, and with only the role
**run.invoker** to limit the impact in case of leak.
What to do?
IAM service offers a lot of possibilities, and also lot of opportunities to do the wrong things with security.
One solution can work for any use cases: generate a service account key file every time. It’s also the worst and the most dangerous solution.
Security implies tradeoffs and to understand what we want to achieve. It’s not always easy and automatic. There isn’t a unique solution.
The best starting point is to think ADC for all the use cases except for 3 situations:
- Your workload/application runs outside Google Cloud (API, website backend, CI/CD,…)
- You want to customize the service account per service on App Engine
- You want to invoke locally, with your user account, a private Cloud Run or Cloud functions from your code (and not from gcloud CLI)
This is true even if some examples, tutorials, even on Google Cloud documentations present code samples with a service account key files!
Originally published on Google Cloud Community on Medium