Reducing Risk while Rotating Secrets

Nathan Clement · September 8, 2026 · 6 min read

Dangers of Shared Credentials and Mitigation Strategies

Photo by regularguy.eth on Unsplash

You are in the middle of a busy sprint when an email arrives. An important SaaS service for your organization has been compromised, and recommends immediately rotating all secrets stored in their system.

Consider the following diagram, which displays a common way of storing secrets in a simple Monolithic Application:

Diagram of Monolith Application

Diagram of Secrets stored on a simple Monolith Application

For organizations with a small application footprint, secret rotation can be a relatively straightforward process: possibly as simple as resetting some database passwords and environment variables, then deploying.

Diagram showing how to reset, update, and deploy settings after an AppService’s settings are compromised

Reset, update, and deploy after secrets are compromised for a simple Monolith Application

As the complexity of an organization’s application infrastructure grows, so does the difficulty of rolling keys. One common mistake that contributes to this difficulty is sharing credentials across services. It increases complexity of responding to a hack in much the same way as when an individual reuses the same password for multiple logins.

Reusing Credentials: A Danger for Users

Any password manager will advise against reusing credentials for multiple services. Since passwords are often difficult to remember, people are tempted to memorize a single password and use it for every service. For example:

// List of credentials
const Google = {username: 'me@example.com', password: 'mypassword123'};
const Facebook = {username: 'me@example.com', password: 'mypassword123'};

In this situation, if Google is hacked and a bad actor knows my credentials, they can also log into my Facebook account.

In this case, I would need to go through the complex and time consuming process to change my password for every service that uses mypassword123 in order to secure my credentials.

For someone with dozens or hundreds of logins, this could mean hours of wasted time.

Since most application secrets are randomly generated, copied and pasted, stored in settings files, and used automatically, this is not a common problem in the world of software engineering.

// .env file
DB_PASSWORD=example_dHbaCWtP7ENdVx_bqy@y-@jYNjb2
EXTERNAL_API_KEY=example_fRQ@ZMocfYT4r2u@gdV2v3P*TA-A

No one is ever going to remember example_dHbaCWtP7ENdVx_bqy@y-@jYNjb2, leaving little incentive to re-use this password for any other purpose.

Shared Credentials: A Danger for Organizations

Let’s imagine that instead of a Monolith, we are deploying several different services. In this case, there are two services per environment:

  1. Web API
  2. Cron Worker Container

Additional there are two resources per environment:

  1. SQL Database (accessed by both the Web API and Cron Worker Container)
  2. External API Keys (accessed only by the Web API)
Diagram showing two repositories and how their secrets are stored across environments. One repository is the Web API, the other is a Cron Worker Container.

Diagram of Secrets stored on a system with two deployed repositories

In this case, there is only one database password for staging, and one for production. For sake of argument, let’s call them db_pw_staging and db_pw_production

If CI/CD Secrets are compromised, hackers will know the password for our databases in staging and production.

This time though, it is not a simple fix. We cannot just reset the database password, set the new password in CI/CD Secret settings, and redeploy the Cron Worker Container because the same password is also being used to authenticate the Web API. As soon as we reset the database password, the Web API AND Cron Worker Containers will go down.

Instead, we must take the following actions:

Diagram showing how to reset, update, and deploy credentials in a multi-repository environment if CI/CD Secrets are Compromised

Reset, update, and deploy after secrets are compromised for a system with two repositories

Despite the fact that only two secrets were exposed (as opposed to four secrets in the first example), we are required to update twice as many secret storage services (CI/CD and AppService), and deploy twice as many applications (Web API and Cron Worker Container)

As the complexity of infrastructure increases, this problem compounds until what used to be a simple task becomes tremendously complex, risky, and prone to downtime.

How can the risk of shared credentials be mitigated?

There are many strategies, but I will discuss two:

  1. Unique Credentials by Secret Storage Service
  2. Automated Secret Management

Unique Credentials by Secret Storage Service

If your organization uses the same credentials across services, you can achieve a quick win for resilience in the face of a compromised secret storage service by implementing unique credentials for each one.

For example, in the example above, if CI/CD used a different database credential than your AppService (maybe db_password_cicd_sandbox and db_password_cicd_production, the recovery strategy would be much simpler and less risky.

Diagram showing how to reset, update, and deploy changes after CI/CD Secrets are compromised, but this time the organization is using different credentials for each service.

Reset, update, and deploy after secrets are compromised for system with two repositories, but the organization uses unique credentials by secret storage service

Automated Secret Management

Eventually, most organizations will grow to the point where a purpose built solution for secret management makes the most sense.

Common Vault services include:

Common features of key vaults include:

  • Centralized secret management
  • Access to secrets from applications via API, CLI, or other automated methodologies
  • Automated credential rotation
  • Integrations with common hosting services and tools (Azure, AWS, Google, Kubernetes, etc)

While the process varies by tool, the response to a security breach at GitHub for an organization that leverages a secret management vault could be this simple:

Diagram showing how to reset and update secrets using a secret vault management system.

Rolling secrets using a managed vault

Some secret management vaults can even automatically trigger an update for the database credential:

Diagram showing how to update credentials after CI/CD Secrets are compromised if the Password Management Service has Auto-Secret updates.

Rolling secrets using a managed vault with auto-update

Summary

In today’s cybersecurity environment, it is critical for an engineering team to be able to quickly rotate secrets used by application infrastructure.

A good first step towards this goal is to eliminate shared secrets across services. While it is easy to set up a single database login to be used by all applications, it creates unacceptable risk, especially as an organization’s application footprint grows.

Eventually, the cost of manual secret management can grow to the point where the benefits of a centralized secret vault service cannot be ignored.

Originally published on 2023-01-20 on Medium.