Latest news about Bitcoin and all cryptocurrencies. Your daily crypto news habit.
At my current gig, weâve got 50+ services, ~6+ environments, and a rough count of 3.5k parameters used by services and humans across our environments. We used to use Vault, but after a year with it, we finally settled on using Parameter Store since weâre hosted on AWS. Parameter Store was a clear winner for us as it gives us all the security we want by leveraging IAM policies and KMS keys. And⊠I didnât have to run Vault myself.
If youâre considering migrating things or have already started using Parameter Store, hereâs a few tips from my experience with Param Store for managing secrets.
Use the CLI for searching for parameters
One really annoying bit is that I canât easily search for parameters across the whole Parameter Store. If youâve tried to do this using the UI, its a bit of pain between searching by regex and/or path. For example, I canât search for all parameters starting with âdb_â i.e. *db_*
a useful workaroundâââdo a quick search* of parameter store. this requires getting all the parametersâââbut lets you apply plain regex across available paths which you canât quite do in the UI. *It is a little bit slow but can get you a quick preview of the secrets you have available
> aws ssm describe-parameters \ --output text \ | egrep '^PARAMETERS' \ | awk '{print $5}' \ | egrep $REGEX
/dev/myApp/foo /dev/myApp/bar ...
Thereâs probably a better way of doing this.
keep your db connection parameters together
Try to avoid scattering your db credentials among different keys in Parameter Store, and keep them together. Its easier to audit which users/roles have accessed the credentials since theres only one place you can get them from.
/$env/databases/$appDb/host = mydb.somewhereinaws.com /user = read-only /password = 'a good password' /port = 5432 /dbname = 'orders' /scheme = 'postgres'
connection_string = $scheme://$user:$password@$host:$port/$dbname
This would let you control who has access to which sets of databases via IAM policies. You can also script it to automatically log you into a database without you needing to see the password. Hereâs an example of how you might do it in bash
dbInfo=$(aws ssm get-parameters \ --names "/dev/dbs/$dbName/database" \ "/dev/dbs/$dbName/host" \ "/dev/dbs/$dbName/password" \ "/dev/dbs/$dbName/port" \ "/dev/dbs/$dbName/user" \ "/dev/dbs/$dbName/scheme" \ --region $REGION \ --with-decryption \ --query Parameters[*].Value \ --output text | tr "\t" " ")
database="${dbInfo[0]}"database_host="${dbInfo[1]}"database_pass="${dbInfo[2]}"database_port="${dbInfo[3]}"database_scheme="${dbInfo[4]}"database_user="${dbInfo[5]}"
if [ "$database_scheme" = "mysql" ]; then MYSQL_PWD=$database_pass mysql -h $database_host -P $listen_port -u $database_user $database_database else PGPASSWORD=$database_pass psql -h $database_host -p $listen_port -U $database_user -d $database_database fi
stick to the naming convention
This is the convention weâve used for our parameters which has scaled ok.
/$environment_name/databases/$database_name/{host,port,pass,user} /databags/$service_name/{all,my,server,creds} /other_sensitive_info/{foo,bar,baz}
We borrowed databags from our days using chef cookbooks & encrypted databags. It basically just means that thereâs a bunch of parameters under keys belonging to a service.
This lets us scope access in a few different dimensions via IAM policies. We can say that a developer should have access to database secrets in a dev environment, but only service secret in prod.
{ "Effect": "Allow", "Action": [ "ssm:GetParameters" ], "Resource": [ "arn:aws:ssm:us-east-2:xxxx:parameter/dev/databases/*", "arn:aws:ssm:us-east-2:xxxx:parameter/prod/databags/myService/*" ] }
you canât stuff big items into parameters
We used to use the certificate cookbook to install certs on our hosts which required us to keep our certificates stored in this format
{ "id": "mail", "cert": "-----BEGIN CERTIFICATE-----\nMail Certificate Here...", "key": "-----BEGIN PRIVATE KEY\nMail Private Key Here...", "chain": "-----BEGIN CERTIFICATE-----\nCA Root Chain Here..." }
But you canât stuff this as json into a parameter due to the size limitations of 4096 characters. This is more of a nuisance when youâre migrating secrets over, but its easy to workaround. It will just unfortunately require you to update some scripts that are expecting the previous format of the secret.
use labels/pointers to help you rotate passwords
Password rotation is an inherently hard problem, but one strategy could be to use pointers (or what they now call labels) to secrets. For example, lets say you want to rotate an encryption key, but you have multiple services relying on it.
you could define your structure like:
/dev/databags/my-key/current = 2 (index of actual secret) /1 = 'secret 2018' /2 = 'secret 2019'
Your applications would need to know to follow the pointer from current and read the intended secret. From there, youâd have to regenerate a config file and bounce your service.
Fortunately, You can do this natively now on AWS with labels on parameters. Rotating passwords can also be done using Lambda and Secrets Managerâââcheck out this walkthrough here
param store keeps versions of your parameters, but not if you delete them
The docs indicate that parameters are versioned, and they are versioned while they exist. But if you accidentally delete a parameter, the history is gone with it.
The consensus seems to be that you should export your parameter store database to S3 or Dynamodb but I havenât come across tools in this space yet.
some related tools that support param-store
secretly for exporting secrets into your environment, written in pythonâââhttps://github.com/energyhub/secretly
parameter-store-execâââsimilar to secretly, written in go.âââhttps://github.com/cultureamp/parameter-store-exec
confd for putting into config filesâââhttps://github.com/kelseyhightower/confd
as part of a consul-templateâââhttps://github.com/hellofresh/consul-template-plugin-ssm
chamber for managing secrets including parameter storeâââhttps://github.com/segmentio/chamber
Originally published at www.intricatecloud.io on October 20, 2018.
a few tips for storing secrets using aws parameter store was originally published in Hacker Noon on Medium, where people are continuing the conversation by highlighting and responding to this story.
Disclaimer
The views and opinions expressed in this article are solely those of the authors and do not reflect the views of Bitcoin Insider. Every investment and trading move involves risk - this is especially true for cryptocurrencies given their volatility. We strongly advise our readers to conduct their own research when making a decision.