Versioning and Deploying Secrets

I am curious to understand how others manage their secret and sensitive info in conjunction with Terraform.

Most of my use-cases with terraform are provisioning Infra (Usually AWS) and then Application resources that depend on the infra.

Examples of Secrets:

  • single-line strings
    • passwords
    • api-keys
    • tokens
  • multi-line strings
    • ascii-armored pem files
    • ascii license data
  • binary license data

I’ll explain the requirements I’m trying to fulfill and then currently how I achieve the success criteria.

Requirements:

  • Audit-able - As secrets change, the ability to identify that, who did it, when, and what the previous and changed values are are a critical requirement.
  • Shareable - Multiple TF operators need to be able to access the secret data.
  • Secure - Only TF operators and TF should be able to see the data.
  • Orchestrated - Multiple-TF Operators should not have to guess if the data they have is up to date. It should be apparent at plan-time at minimum.
  • Reference-able - Values should be securely reference-able within TF to provide to resources.
  • Avoid OutOfBand - Unless absolutely necessary, avoid using multiple processes that are unaware of eachother. E.g. boostrapping done by System-A and a requirement for System-B.

My Implementation:

  • Secure - I use ansible-vault to secure a yaml formatted secrets (from here, known as secret-yaml) file with a symmetric passphrase (stored in a file)
  • Audit-able - I include the secret-yaml in the SCM project with the terraform-code and the passphrase-file is .git-ignored. I also consider setting up precommit hooks to confirm the secret-yaml is encrypted.
    • This provides insight into who, what when and if sufficient commit messages provided, why.
  • Shareable - Since the secret-yaml is in SCM, its accessible to other TF Operators.
  • Reference-able - I create an external datasource using a python script to decrypt and parse the secret data. Its then available as datasource result value.
  • Orchestrated - Since the data is in the tfstate and part of datasources being referenced, the plan changes when the data changes. If operator sees an unexpected change, they can confirm they’re on head of proper branch or check with other operators to see if they have un-merged changes. But if all standard process is followed, no guessing is required to identify if you have latest data.

Weak Spots:

  • The state isn’t encrypted except through S3 backend KMS. Its a single point of failure so if that key is breached, all secret-data is exposed.
  • Initial creation of symmetric passphrase file is small but out of band and must remember to gitignore.
  • feel free to add more here. Looking to improve the process while maintaining the requirements.

Regarding examples of secrets, these are good, though we should also call out the different ways secrets are consumed. Especially when dealing with third-party software, the configuration mechanisms vary. Sometimes environment variables suffice, sometimes configuration files are required. Other times, with in-house software, they might directly interface with something like HashiCorp Vault or the AWS Secrets Manager (ASM).

What I like about your current approach is that it provides a consistent way CRUD secrets that works well with your CI/CD process. It’s also a well-understood mechanism of encrypting secrets with a public key. It also provides audit trails.

What I like less is that when the secrets change, there’s no real oversight as part of the PR process of what specifically changed (E.g. was it Datadata Integration Key or the backend database password?). Someone who updates the file must (a) decrypt it using the private key (b) ensure that decrypted copy doesn’t leak © not bungle the format of the file or accidentally edit other secrets. And aside from using lots of different encryption keys, there’s no easy way to scope access to the secrets.

So what’s our suggestion?

Hrm… I’ve never seen a solution yet that I like 100%. Also, we have very few, if any secrets we need to inject into terraform during runtime because most things that need secrets are either (a) deployed with helm and helmfile (b) rely on IAM roles for temporary AWS credentials. The direction we’ve been leaning towards when we do have secrets for terraform is to store those secrets using the aws-cli or chamber in AWS SSM Parameter store. Then from there, we use the terraform SSM parameter store data resource to read those values and pass them where we need. The standard caveat applies: terraform state necessarily has plain-text secrets that get stored in an S3 bucket, encrypted with a KMS key and secured with an IAM policy. What I like about this approach is we’re relying on first-class resources supported both by AWS and terraform - no escape hatches required. What I don’t like about this approach is the manual process of updating secrets (though that’s pretty hard to get around in any solution)

The problem we’re more actively concerned with solving is how to get secrets into Kubernetes. If this is interesting, I can go in to further depth. For that, our most recent attempts include using the aws-secrets-operator takes care of provisioning secrets inside of kubernetes. Then on the git-side of things, we just refer to a GUID of a secret stored in ASM.

Also, we discuss some of this in last week’s office hours. Check it out! https://cloudposse.com/devops/office-hours/ (skip forward to 46:40).

1 Like

What I like less is that when the secrets change, there’s no real oversight as part of the PR process of what specifically changed (E.g. was it Datadata Integration Key or the backend database password?).

This is definitely an issue and makes PR’s and review a pain. The data is all there but it definitely doesn’t lend itself as observable or transparent. I’ve considered switching from ansible-vault to sops for this reason, as sops will only encrypt the value. But that limits the types of secrets you manage and to your earlier point, how they’re ingested.

Someone who updates the file must (a) decrypt it using the private key (b) ensure that decrypted copy doesn’t leak © not bungle the format of the file or accidentally edit other secrets. And aside from using lots of different encryption keys, there’s no easy way to scope access to the secrets.

These are all great points that I seem to get amnesia on from working in a vacuum or in a small group that has the pattern down. It can definitely be very error prone.

The consistent issues that bother me are:

  • where is the source of truth for my secret data before I provision it
  • how does one wrap that in a lifecycle-management process like we do for everything else.
  • how to do it in the most agnostic way possible.

I :heart: HC Vault. Its hard to get consensus and appetite for it based on the learning curve depending what and how you’re implementing it. I do like the fuse-fs like approach where secrets are content to files in a filesystem and each individuals access to Vault predicates their level of authorization. Maybe I’ll dig in more there.

Thanks for sharing your perspective, I really appreciate it :slight_smile: I would love to hear more about your Kubernetes and Helm approaches too as I’m just learning Helm myself.

With my above implementation I’m currently adding my secrets to the secrets-yaml, and plugging them into k8s with tf kubernetes_secret resource. Then on the helm side I’m just referencing via valueFrom / secretKeyRef in the templates. No idea if this is a naive approach yet :smiley:

FWIW on my small team (< 5 engineers) we use mozilla’s sops with great success.

For ages we relied purely on the PGP feature, but recently switched to KMS and it works great. We also use it in ansible using a custom plugin.

I believe our technique meets all your requirements. You can only store json or yaml data, but we get around that by wrapping blobs (pems, etc) in json/yaml and then shoveling it where it needs to go. The terraform-sops-provider is a must.

HC Vault looks great, but would be too much of a burden for our size.

1 Like