Secret Management - Don't Let API Keys Wander Around
As the number of services increases, managing passwords and API keys becomes a massive security risk. How we keep secrets safe.
One of the fastest ways to ruin a developer's career is to accidentally push a
.envfile containing a Database Password to public GitHub. In Microservices, this risk is multiplied by 10.
The Problem: 10 services, 100 secrets
Initially, we kept sensitive information (Database URL, OpenAI API Key, JWT Secret...) in CI/CD environment variables or local .env files.
But as the system grew:
- Who is authorized to see the Production database password?
- How do you rotate a secret without redeploying all 10 services?
- How do we ensure new devs don't accidentally read important keys from the real system?
1. The Golden Rule: Never Commit Secrets
Even in human private repos, committing secrets into code is a sin. Code can leak, and git history will preserve that secret forever.
2. The Solution: Centralized Secret Manager
We switched to a centralized secret management service. For small teams, there are 2 good choices:
- Cloud Native: AWS Secrets Manager or Google Secret Manager. Very easy to use if you are already on their Cloud.
- HashiCorp Vault: Extremely powerful but a bit difficult to operate for small teams.
How it works:
When a service starts, instead of reading from a .env file, it calls an API to the Secret Manager to retrieve information. Or better, we use a "Sidecar service" to inject these secrets into the application's memory.
3. Secret Rotation Strategy
What happens if an API Key is exposed? Normally, you'd have to change the key at the provider, then fix it in 10 services. With a Secret Manager, you only change it in one place. Services will automatically receive the new key on the next restart (or even in real-time if coded carefully).
Lessons Learned for Small Teams
- Clear Permissions (RBAC): Only necessary services should have the right to access their corresponding secrets. The Payment Service doesn't need to know the Shipping Service's secrets.
- Client-side Encryption (if needed): For extremely sensitive information, we encrypt it with another layer before pushing it to the Secret Manager.
- Local Development: Use tools like
infisicalor simply a.env.examplefile so devs can fill in their own private keys when working locally.
Conclusion
Security is not a "feature"; it is the foundation. Good secret management helps your team expand the system more confidently and sleep better knowing that the company's most important information is properly protected.
Series • Part 17 of 20