The Ultimate Guide to Cloud Native Secret Management

Secret management is the method of defending delicate info resembling credentials, keys, and certificates throughout their whole lifecycle.

This security matter is a real headache to clear up utterly, as and not using a pragmatic strategy it could lead to taking place a seemingly endless collection of rabbit holes as there isn’t a good one measurement matches all answer out there.

Adopting an answer across a whole organization is difficult – Many solutions remedy the issue for a single specific use case which leads to gaps when wanting at the greater picture, and others have a prohibitively excessive barrier to entry due to the amount of effort required across each engineering and course of perspectives. Fixing all secret administration points swiftly is probably not achievable, through which your first milestone ought to be to understand what a complete future state on your organization resembles. Give attention to the tackling low-hanging fruit initially (reminiscent of eradicating plaintext secrets from source management) then construct upon it and in the direction of a more complete answer.

Defending software secrets and techniques is the main target here, and whilst there’s some overlap with worker secret administration, it is beyond the scope of this submit. Just remember that each subjects exist and have barely totally different considerations.

The initial belief drawback

Your software or instance wants a mechanism of authenticating itself with its secret retailer / key management to either retrieve or decrypt its secrets and techniques, this is often in the form of one other secret which should then be managed out of band to secrets inside the key store.

Establishing initial trust together with your secret administration tooling is a standard drawback you will encounter. Some solutions supply extra artistic approaches for dealing with this, nevertheless it’s necessary to recognize that this drawback is all the time going to exist, and also you typically find yourself merely shifting this drawback down the stack.

With this in thoughts:

  • All the time factor the preliminary trust into the design of an answer: think about when and where initial trust wants to be established, and ensure there’s a good reply for handling secrets which exist out of band to your main secret retailer
  • All the time use identity-based access control (AWS IAM roles, Azure Managed Service Identities, and so on) the place potential – this typically helps with the preliminary belief drawback, and reduces the necessity for secrets and techniques when accessing different cloud assets
  • Having to retailer the preliminary belief secret outdoors of your main secret store shouldn’t be thought-about a deal breaker – 99% of your secrets will nonetheless be saved and managed within the main secret store
  • Establishing the initial trust in software situations is more durable than doing so on your automation servers and presents some fascinating challenges:
    • The variety of situations: you’re in all probability speaking about 10s of automation techniques vs 100s or 1000s of software situations
    • Software situations are naturally more exposed, whilst automation tooling is usually secured on an inner community

Secret management answer necessities

A complete secret management answer ought to meet these requirements:

  • Secrets aren’t saved in plain textual content at-rest
  • Secrets in-transit aren’t despatched in plain text
  • Encryption keys can’t be exported
  • Encryption keys might be rotated
  • Highly out there – does not impression the supply purposes which depend upon it
  • Secrets ought to be immutable via versioning (purposes ought to have the ability to reference specific secret versions)
  • Audited operations (E.g create, learn, update, delete)
  • Straightforward to use and properly understood otherwise you danger individuals reverting to lazy and insecure processes


So now we have now coated the challenges and requirements, lets transfer on to several types of tooling which is availabe. There are three classes which most secret management solutions fit in to:

  • File / SCM encryption: storing encrypted information source control
  • Automation and orchestration tooling: storing secrets and techniques in your pipeline or configuration administration tooling
  • Secret management service: storing secrets in a devoted software/service

For every class, I’ll explain the advantages and drawbacks, present some examples of tools, and display a typical workflow.

Tooling: File / SCM encryption options

Instruments in this class embrace:

Encrypting knowledge and storing it in git is a standard sample for low-cost secret management options. It solves the widespread drawback of plaintext secrets and techniques being saved in git – and subsequently on the workstation of any developer who has labored on that code base. The nature of these solutions means you open the door to all the standard benefits of storing something in source management (branching, diffs, pull requests, traceability, rollback, and so on).

The tooling on this category are all free open supply tasks, they usually sometimes comprise of an executable or script which is combined with a personal key to facilitate encryption and decryption operations. Aside from the important thing administration, there isn’t any infrastructure to provision or manage so the barrier to entry is low.

Key management within these instruments primarily depend on manually generated GPG or PGP keys which coupled with the truth that information need to be re-encrypted and committed back into git, makes rotation of keys troublesome, during which case rotation is more occur reactively slightly than pro-actively, which is essential to think about however could be positive depending on your necessities. There are additionally auditing issues due to the at-rest nature of this answer structure.

The means to encrypt selective elements of a file relatively than your complete file is a elementary differentiator between the out there tooling in this class, as having significant diffs when tracking modifications to information in supply control is a pleasant to have. Though, encrypting selective elements of a file may require storing metadata comparable to key references inside a file, which modifications the structure, so you may want to get just a little artistic when it comes to consuming the contents in downstream techniques.

Implementing this across a corporation is challenging as these usually are not centralized options, but adoption is straight ahead inside the context of a workforce. On the flip aspect, this can be a good fit implementations outdoors of a corporation reminiscent of an open supply challenge where a centralized answer is just not going to work.

For my part, Mozilla SOPS is the leader in this category

  • Key administration may be offloaded to AWS/Azure/GCP cloud KMS providers
    • Granular entry to this key may be managed by way of cloud IAM permissions
    • This improves auditing as you possibly can monitor decrypt operations on this key
  • Can partially encrypt information for significant diffs
  • Implements Shamir’s secret sharing which supplies flexibility for key management

Instance workflow: File / SCM encryption

This can be a typical workflow for this class of tooling. You can carry out decryption either on your automation tooling or on the goal software occasion

Summary: File / SCM encryption

Minimal infrastructure to manageRequires key administration
Secrets are model controlledAuditing issues
Aligns with GitOps workflowsRotation of keys
Low effort and priceNot a centralized answer

Tooling: Automation techniques

Instruments in this category embrace:

There is a vary of automation tools which reside between improvement groups and production methods, this primarily covers CI/CD techniques and configuration administration tooling.

Sadly, secret administration is usually a secondary concern for most automation techniques, and implementations here might have shortcomings in safety or functionality in comparison to devoted secret administration options, so ensure to assessment the answer capabilities towards the factors listed within the Secret administration answer requirements part. You in all probability have CI/CD tooling AND configuration administration tooling so leveraging the key administration capabilities of disparate orchestration tooling may end up in fragmentation, as exposing secrets outdoors of the context of jobs inside that system is often troublesome.

Organizations who’ve adopted multiple orchestration tools in a given area (totally different groups using totally different CI/CD tools) are going to find it more durable nonetheless to discover consistency when relying on secret management capabilities these methods.

Putting secrets and techniques into these methods is carried out by an operator via a CLI or UI portal, and the secrets and techniques are then saved within the server. Consuming these secrets is completed by means of an automation pipeline or job, which can sometimes have a framework for reworking a tokenized file by substituting the tokens with their desired values.

As this class of tooling is focussed on automation, integrating the secrets and techniques management right into a launch course of is trivial, and there are sometimes mechanisms for scoping secrets and techniques to specific environments or deployment targets which may simplify setting specific configuration.

Traceability of secrets right here is restricted, altering a secret is a straight forward operation however there isn’t often any mechanism for answering who made a change, when it was made, and why it was made.

Instance workflow: Automation techniques

Summary: Automation methods

Constructed-in automation means straightforward integration with the discharge processSafety and compliance may be restricted
Low barrier to entry as most organizations have this tooling alreadyFunctionality varies across tools in this area
Centralized answerPoor traceability
Multiple orchestration tools cause fragmentation
One other source of fact

Tooling: Secret service

Instruments on this category embrace:

  • Azure Key Vault
  • AWS Parameter Retailer
  • Vault (Hashicorp)
  • KeyWhiz (Sq.)
  • com

Storing secrets and techniques in a delegated net service is rising in popularity as it opens the door for taking a more dynamic strategy to secrets, but this area is less mature than the other classes. Entry to secrets is completed via an API which probably means entry to the system may be very granular and security could be tightly locked down.

There are a selection of instruments on this category – some hosted and others on-premise / self-hosted type solutions.

The commonest are in all probability the generic cloud provider providers reminiscent of AWS parameter store or Azure Key Vault. Azure Key Vault has limitations for assigning granular permissions, as access to a person Key Vault have to be granted on an all-or-nothing foundation, so you have got to provision a number of discrete Key Vaults to present isolation between purposes and environments which may improve complexity. Assuming your software situations are additionally operating in the same cloud supplier, then assigning identities to software situations will help with the preliminary belief.

There are additionally SaaS type hosted choices – EnvKey is an fascinating answer which is extra opinionated than the cloud supplier options. It treats purposes and environments as first-class residents which makes integration actually straight forward, but this does include a price due to the licensing model being per-user. Though that is in all probability the costliest answer, you will have to factor within the potential value savings in comparison to rolling your personal answer. Again, you’ve gotten the preliminary belief drawback the place you continue to need to handle the secret which grants you software entry to envkey.

There are self-hosted solutions corresponding to Hashicorp Vault, which is the de-facto in this class because it offers some neat features comparable to dynamic secrets and may also carry out encryption as a service to other purposes inside your surroundings. There’s flexibility in the implementation because the backend storage is abstracted to help numerous storage and database providers, nevertheless on the whole this answer requires a considerably more effort to implement and operate at scale as compared with the cloud/hosted providers.

For all tools on this category there’s flexibility when it comes to integration for consuming secrets: An software can retrieve the secrets at runtime by way of a direct integration, or a pipeline can retrieve secrets and inject them into a config file or setting variables.

Integrating an software instantly with a secret service places this dependency firmly in your crucial path, so it’s absolutely very important to plan for failure. In case your secret administration service goes down and it takes your software that’s unacceptable, as compared to a state of affairs where you’re retrieving secrets and techniques inside a CI/CD pipeline – being unable to deploy for half an hour might be a suitable impression.

You do want to think about possession and deployment models – having a centralized service means constant safety and compliance, while individual teams or departments managing their very own situations supplies more control and will mitigate the influence of a centralized system failing.

Traceability of modifications may be troublesome poor here, although secrets are often individually versioned you in all probability can’t reply questions comparable to who made a change, why it was changed, and so on that you would give you the option to reply with a git-based answer. Additionally it is troublesome to describe the state of the system as an entire – as particular person operators are often performed manually by operators this needs extra rigor to ensure alignment with release processes.

Example workflow: Secret service

There may seem like a easy workflow at a look, however the effort is usually heavy upfront within the design and implementation of the secret service. You additionally want to think about how the appliance integrates with the secret service – depending on how many languages and frameworks are used to write purposes then this can be non-trivial to guarantee constant approaches across numerous groups.

In the event you go down the direct software integration then remember the initial trust drawback becomes tougher, as you want to bootstrap all your software situations as compared to only bootstrapping your automation servers.

Robust security measuresThe excessive effort is a barrier to adoption
High-quality-grained entry controlCan impression software availability
Dynamic secrets and techniques (Hashicorp vault)Another source of fact

Poor traceability of modifications
Service could also be internet-facing (cloud / hosted)

However Kubernetes handles secrets and techniques?

To begin with – Adopting a k8s specific answer is ok in case your complete estate is operating k8s, in any other case, you may want to take a look at a more basic answer to keep away from fragmentation.

Kubernetes doesn’t truly drastically change the strategy when it comes to secret management, whilst it does open the door to some nifty solutions (native k8s secrets and techniques, helm-secrets, sealed-secrets) which only exist inside the k8s ecosystem, whenever you drill down, they still expertise the same challenges as other options.

The native k8s secret useful resource has major safety gaps in its present state, as secrets are saved and distributed across cluster nodes in plain textual content. There’s beta help for encrypting secrets, however then an external KMS supplier can also be required to additionally avoid storing the master key-encryption-keys in plain textual content. Ultimately, that is going to enhance over time but in its current state, it’s dangerous.

The initial belief drawback additionally requires a better degree of granularity to clear up in k8s (or any container orchestration platform), as to enable fine-grained entry to secrets and techniques on a per-application foundation your belief boundary is an individual pod quite than a whole machine (cluster node). Unfortunately, the options for assigning identities on a per-pod basis are immature (AWS, Azure) and you might find yourself having to resort to per-machine identities which suggests poor isolation between purposes. Factoring this into your cluster design could lead on to provisioning and managing further isolated clusters to meet security requirements.

When utilizing native k8s secrets you’ll need to think about how the secrets and techniques get into Kubernetes within the first place – You sometimes need to leverage tooling from the above categories to help this course of.

Taking all of that under consideration, there are some benefits to utilizing the native k8s secrets, resembling the convenience of use and simple integration with purposes, simply watch out for their limitations!

Score of different classes

Listed here are some scores for my part of key metrics for the totally different classes of tooling. There are exceptions within every categories, however on the entire this is where I really feel most tools inside a given category sit from a excessive degree.

Record of obtainable tooling

Here is an summary of all the secret administration tooling which I’m at present aware of. If there are any established tools lacking from this record, then please depart a remark.

There are a number of open source secret administration tasks which build upon AWS assets (KMS, DynamoDB, s3, and so on) and provide a CLI for CRUD operations. Whereas they’re quite neat tasks, for my part they don’t supply much over the native AWS parameter store service aside from the truth that they don’t seem to be constrained by the parameter retailer service limits. I have labelled these as  “AWS wrapper” in the desk above.

Wrapping all of it up

A comprehensive strategy will sometimes embrace a mixture of a number of options. You may use a git-based answer as the supply of fact, then use a pipeline script to put these right into a secret service the place the purposes consumes secrets from, which combines a well-known developer expertise with granular and audited entry to secrets and techniques.

Think about what secret management capabilities are required at particular person levels of your software improvement life cycle to build up an inventory of requirements, then you possibly can map these requirements to methods and determine tooling which then meets these wants.

Mapping methods and tooling to levels of your workflow would appear to be this:

(Obviously, you would not use all the above tools collectively, it’s simply to reveal the place certain tools may match into the larger picture)

Some ultimate ideas:

  • The initial trust (or bootstrapping) situation just isn’t distinctive to a specific software or strategy, factor it into your design
  • Think about your complete workflow from improvement to production
  • The totally different classes of tooling can complement one another
  • There isn’t a perfect answer, should you can implement a solution which is best than what you presently have, then go for it as its a step in the correct path!