Reducing Risk while Rotating Secrets

Dangers of Shared Credentials and Mitigation Strategies

Photo by regularguy.eth on Unsplash

You are in the middle of a busy sprint when an email arrives. An important SaaS service for your organization has been compromised, and recommends immediately rotating all secrets stored in their system.

Consider the following diagram, which displays a common way of storing secrets in a simple Monolithic Application:

Diagram of Secrets stored on a simple Monolith Application

For organizations with a small application footprint, secret rotation can be a relatively straightforward process: possibly as simple as resetting some database passwords and environment variables, then deploying.

Reset, update, and deploy after secrets are compromised for a simple Monolith Application

As the complexity of an organization’s application infrastructure grows, so does the difficulty of rolling keys. One common mistake that contributes to this difficulty is sharing credentials across services. It increases complexity of responding to a hack in much the same way as when an individual reuses the same password for multiple logins.

Reusing Credentials: A Danger for Users

Any password manager will advise against reusing credentials for multiple services. Since passwords are often difficult to remember, people are tempted to memorize a single password and use it for every service. For example:

In this situation, if Google is hacked and a bad actor knows my credentials, they can also log into my Facebook account.

In this case, I would need to go through the complex and time consuming process to change my password for every service that uses in order to secure my credentials.

For someone with dozens or hundreds of logins, this could mean hours of wasted time.

Since most application secrets are randomly generated, copied and pasted, stored in settings files, and used automatically, this is not a common problem in the world of software engineering.

No one is ever going to remember , leaving little incentive to re-use this password for any other purpose.

Shared Credentials: A Danger for Organizations

Let’s imagine that instead of a Monolith, we are deploying several different services. In this case, there are two services per environment:

1. Web API
2. Cron Worker Container

Additional there are two resources per environment:

1. SQL Database (accessed by both the Web API and Cron Worker Container)
2. External API Keys (accessed only by the Web API)

Diagram of Secrets stored on a system with two deployed repositories

In this case, there is only one database password for staging, and one for production. For sake of argument, let’s call them and

If CI/CD Secrets are compromised, hackers will know the password for our databases in staging and production.

This time though, it is not a simple fix. We cannot just reset the database password, set the new password in CI/CD Secret settings, and redeploy the Cron Worker Container because the same password is also being used to authenticate the Web API. As soon as we reset the database password, the Web API AND Cron Worker Containers will go down.

Instead, we must take the following actions:

Reset, update, and deploy after secrets are compromised for a system with two repositories

Despite the fact that only two secrets were exposed (as opposed to four secrets in the first example), we are required to update twice as many secret storage services (CI/CD and AppService), and deploy twice as many applications (Web API and Cron Worker Container)

As the complexity of infrastructure increases, this problem compounds until what used to be a simple task becomes tremendously complex, risky, and prone to downtime.

How can the risk of shared credentials be mitigated?

There are many strategies, but I will discuss two:

1. Unique Credentials by Secret Storage Service
2. Automated Secret Management

Unique Credentials by Secret Storage Service

If your organization uses the same credentials across services, you can achieve a quick win for resilience in the face of a compromised secret storage service by implementing unique credentials for each one.

For example, in the example above, if CI/CD used a different database credential than your AppService (maybe and , the recovery strategy would be much simpler and less risky.

Reset, update, and deploy after secrets are compromised for system with two repositories, but the organization uses unique credentials by secret storage service

Automated Secret Management

Eventually, most organizations will grow to the point where a purpose built solution for secret management makes the most sense.

Common Vault services include:

• Vault by HashiCorp: https://www.vaultproject.io/
• Azure Key Vault: https://learn.microsoft.com/en-us/azure/key-vault/general/overview
• AWS Secrets Manager: https://aws.amazon.com/secrets-manager/
• Google Secret Manager: https://cloud.google.com/secret-manager
• 1Password Secrets Automation: https://1password.com/product/secrets/

Common features of key vaults include:

• Centralized secret management
• Access to secrets from applications via API, CLI, or other automated methodologies
• Automated credential rotation
• Integrations with common hosting services and tools (Azure, AWS, Google, Kubernetes, etc)

While the process varies by tool, the response to a security breach at GitHub for an organization that leverages a secret management vault could be this simple:

Rolling secrets using a managed vault

Some secret management vaults can even automatically trigger an update for the database credential:

Rolling secrets using a managed vault with auto-update

Summary

In today’s cybersecurity environment, it is critical for an engineering team to be able to quickly rotate secrets used by application infrastructure.

A good first step towards this goal is to eliminate shared secrets across services. While it is easy to set up a single database login to be used by all applications, it creates unacceptable risk, especially as an organization’s application footprint grows.

Eventually, the cost of manual secret management can grow to the point where the benefits of a centralized secret vault service cannot be ignored.

---

Originally published on 2023-01-20 on Medium.