Secrets Management

Secrets are credentials or values that grant access to systems and must remain confidential. They include API keys, database passwords, private TLS certificates, OAuth client secrets, SSH private keys, service account JSON keys, and encryption keys. What distinguishes a secret from ordinary configuration is consequence, if a hostname leaks, nothing is immediately compromised, if a database password leaks, an attacker has direct access to the database.

The lifecycle of a secret spans creation, distribution to authorised consumers, use in running systems, rotation to a new value, and revocation of the old value. Each stage in this lifecycle is an opportunity for a mistake. Secrets are often created with excessive validity periods, distributed to more systems than necessary, never rotated, and never revoked even when the consuming service is decommissioned.

Long-lived secrets are particularly dangerous because the window of exposure is long. A secret that was stolen six months ago but never rotated remains usable by the attacker today. Short-lived secrets that expire in minutes or hours dramatically reduce this window. Even if stolen, they expire before they can be meaningfully exploited. This is why ephemeral, dynamically generated credentials are the gold standard.

Many organisations underestimate how many secrets they have. A single application can accumulate dozens. Credentials for each external API it calls, credentials for each database, credentials for each message queue, credentials for each monitoring and logging system, TLS certificates for each hostname it serves, and encryption keys for each data category. Discovering and inventorying all of these is the first step toward managing them effectively.

A secrets store (vault) is a dedicated system for storing, managing, and distributing secrets to applications that need them. Rather than embedding secrets in code or config files, applications authenticate to the vault at startup or on demand and retrieve the specific secrets they are authorized to access. The vault is the single source of truth, making rotation and revocation simple. Update the vault, and all applications pick up the change.

HashiCorp Vault is the most widely deployed open-source secrets store. It supports multiple authentication backends (Kubernetes service accounts, AWS IAM, LDAP, certificate-based) and multiple secrets backends (key-value, database, PKI, cloud provider credentials). Its dynamic secrets capability generates unique, short-lived credentials for each request, eliminating the static shared password problem entirely.

Cloud-native secrets stores (AWS Secrets Manager, Azure Key Vault, GCP Secret Manager) integrate seamlessly with their respective cloud platforms. They support automatic rotation through cloud service integration (for example, rotating an RDS password and updating the application automatically), fine-grained IAM-based access control, and detailed audit logs through the cloud provider's logging service. Teams that already use a cloud platform should consider the cloud-native store first before deploying a self-managed Vault.

The bootstrap problem is the main challenge in adopting any secrets store. The application still needs a credential to authenticate to the vault in the first place. The correct solution is to use a platform-native authentication mechanism rather than a stored secret. In Kubernetes, pods authenticate to Vault using the Kubernetes service account token, which is issued by the cluster itself. In AWS, EC2 instances authenticate using the instance's IAM role, which requires no stored credential.

Secret rotation is the process of replacing a secret with a new value on a defined schedule or in response to a security event. The goal is to limit the damage from credential theft by ensuring stolen credentials expire before (or quickly after) they are exploited. Rotation also limits the window of exposure from silent breaches where a secret was stolen but the theft was not immediately detected.

Automated rotation is the only practical approach at scale. Manual rotation requires coordinating across teams, updating every consuming system simultaneously, verifying nothing broke, and then retiring the old credential. These steps are error-prone and interrupt service if not done in exactly the right order. Automated rotation as implemented in AWS Secrets Manager and Vault handles orchestration. It creates a new credential, updates consuming systems, verifies success, and then removes the old credential.

The blue/green rotation pattern prevents downtime during credential changes. The new credential is created alongside the old one, applications are updated to use the new credential while the old one remains valid, once all applications have switched (confirmed by verifying no traffic uses the old credential), the old credential is revoked. This overlapping validity period eliminates the race condition where an application request fails because the old credential was revoked before the new one was distributed.

Emergency rotation, triggered by a confirmed or suspected breach, follows a different approach. The old credential must be revoked immediately, accepting any service disruption, because leaving it active while the replacement is deployed gives the attacker continued access. Emergency rotation procedures should be documented and tested before they are needed. A rotation that takes two hours to execute in a drill is not fast enough for a real breach.

Secret injection is the process of providing secrets to running applications without hardcoding them in source code or configuration files. Applications should never contain secret values at build time, secrets should be injected at deployment time or fetched at runtime from a vault. This separation ensures that the artifact (container image, compiled binary) is identical across environments and that secrets never appear in version control or build logs.

Environment variable injection is the most common pattern. The deployment system retrieves secrets from a vault and sets them as environment variables in the process before it starts. Container orchestrators like Kubernetes can retrieve secrets through the Secrets Store CSI Driver and expose them as environment variables or mounted files. The application reads the secret from the environment, never knowing or caring where it came from.

File-based injection has advantages over environment variables in some scenarios. Environment variables are visible to all child processes and can appear in process listings and crash dumps. Files mounted with restrictive permissions (readable only by the application user) are more controlled. The Secrets Store CSI Driver for Kubernetes syncs secrets from external vaults (HashiCorp Vault, AWS Secrets Manager, Azure Key Vault) and mounts them directly into pods as files.

A common mistake is storing secrets in Kubernetes native Secret objects without additional protection. By default, Kubernetes Secrets are base64-encoded (not encrypted) in etcd. Anyone with read access to the Kubernetes API or direct access to etcd can retrieve them. Enabling etcd encryption at rest helps, but the preferred approach is to use the Secrets Store CSI Driver or a Vault agent sidecar to retrieve secrets directly from an external vault, so they never persist in the Kubernetes API at all.

Secrets auditing is the practice of logging and monitoring every interaction with secrets. Who accessed a secret, when, from which system, what they did, and whether the access was authorized. Audit logs are the primary mechanism for detecting unauthorized access, meeting compliance requirements, and conducting post-incident investigations. Without audit logs, it is impossible to know whether a secret was accessed before it was rotated.

HashiCorp Vault, AWS Secrets Manager, and similar tools generate detailed audit events for every secret retrieval, creation, rotation, and deletion. These logs should be shipped to a centralized SIEM or log aggregation system where they can be correlated with other security events. A service account that was compromised and used to read secrets will appear in the vault audit log, correlating that with network logs showing data leaving the environment provides a complete picture of the breach.

Effective auditing requires alerting on suspicious patterns. A service that never accesses a particular secret suddenly requesting it is suspicious. Access from an unexpected source IP or region is suspicious. Bulk retrieval of many secrets in a short time suggests reconnaissance or data exfiltration. These anomalies can indicate a compromised service account or an insider threat. Automated detection rules in the SIEM surface these patterns without requiring manual log review.

Audit trails are also required for compliance. Regulations such as PCI DSS require demonstration that access to sensitive credentials is logged and that those logs are protected from tampering. Shipping vault audit logs to an immutable log store (S3 with Object Lock, a WORM-compliant log management system, or a managed SIEM) ensures logs can be trusted as evidence and cannot be deleted or modified by an attacker who has compromised other systems.

Secrets appear in two distinct contexts in modern software delivery. CI/CD pipelines (where they are needed during build, test, and deployment) and runtime environments (where they are consumed by long-running services). Each context has different security requirements and different risk profiles, and they should use different credentials.

CI/CD secrets are particularly sensitive because the CI/CD system has privileged access to production environments and artifact registries. A compromised CI secret can allow an attacker to deploy arbitrary code to production. The preferred approach is to avoid storing long-lived CI secrets entirely by using OIDC federation. The CI job receives a short-lived token from the CI platform (GitHub Actions, GitLab CI) that it exchanges for temporary cloud credentials using a federated trust relationship. No secret is stored, the token expires within minutes.

Runtime secrets are consumed by long-running services and have different management requirements. They need to survive restarts and be refreshed automatically when rotated. Using a vault agent sidecar pattern, a separate process in the same pod retrieves secrets from the vault and keeps them fresh, writing new values to a shared volume. The application reads from the shared volume and does not need to interact with the vault directly.

A common mistake is using the same credentials for CI/CD and runtime. A deploy pipeline that builds, tests, and deploys should use deploy-scoped credentials (permission to push images and update deployments) that are completely separate from the production database credentials used by the running application. Mixing them means a compromised CI credential gives direct access to the production database. Separation of concerns here is a basic security property.