Managing Rails application secrets with encrypted credentials
[Ruby on Rails is a great web application framework for startups to see their ideas evolve to products quickly. It’s for this reason that most products at Freshworks are built using Ruby on Rails. Moving from startup to scaleup means having to constantly evolve your applications so they can scale up to keep pace with your customers’ growth. In this new Freshworks Engineering series, Rails@Scale, we will talk about how we store and access our application’s secret configurations, keys and tokens.]
Freshservice and its microservices use Nginx and passenger-backed servers hosted on four data centers – US East (US), Europe Central (EUC), India (IND), and Australia (AU). Freshservice is a large application that interacts with a variety of third party services. These services include a lot of internal platforms, microservices and external integrations. To interact with these third party services securely, the app needs to maintain a list of configurations such as API keys, tokens, passwords, endpoints, and so on. These configurations must not be checked into source control in plain text to avoid security breaches. Configurations pushed to remote repositories will be present in commit logs even if the changes are reverted.
Configurations management through ejson
The application loads these configurations through YML files and assigns them to top-level constants for easy access. The configurations were encrypted and stored in ejson (Encrypted JSON) files and added to a repository used for orchestration and deployment. These encrypted configurations were then decrypted during runtime to generate the said YML files specific to the environment / POD. However, there were a few drawbacks to this approach:
- Our application containers run in a “read-only” mode. We had to make special adjustments to accommodate the config generation step during the deployment.
- Depending on another repository for storing these configurations made it difficult while adding new changes. Maintenance is also a problem.
- It is hard to view these configurations locally while editing since we store them in Encrypted JSON files.
- The encryption process was manual and there is always a risk of human error while pushing these secret configurations. The impact of this would also be huge as we can’t risk exposing production configurations.
Introducing Rails Credentials
Ruby on Rails has matured in recent times. Upgrades have fewer breaking changes, several new features like multi DB, encrypted attributes are built keeping large applications in mind, ones that require scale with security at its core. One such feature that focuses on the security of your applications is credentials, which was introduced in Rails 5.2. Credentials provides an easy way to encrypt and store configurations along with the application code itself. The configurations stored using credentials can also be accessed directly through the application using Rails.application.credentials[key]. The decryption is done internally by the framework itself.
Credentials are stored and accessed using two files:
- credentials.yml.enc – This is an encrypted file that stores all the credentials. It is a YAML file that is encrypted by using the master key. Since this is encrypted, it is safe to push this file to the remote repository.
- master.key – To decrypt the encrypted file, this file is used. It is impossible to read the encrypted file if this key is lost/modified. And, this should not be pushed to the remote repository.
Both these files are stored in the config folder. Reading and writing the credentials.yml.enc file can be done using a command that is provided by Rails.
EDITOR='code --wait' rails credentials:edit
“code” inside the command above can be replaced by other editors and the wait flag is required by editors which create a separate child process or it will be saved immediately. This command will decrypt and open the contents of credentials.yml.enc as a temporary file. This is a YAML file that will open up in the specified editor. Once it is saved and the file is closed, it is again encrypted with the latest contents. When the command is run for the first time, it generates a file like this:
# credentials.yml.enc
secret_key_base: SECRET_KEY_BASE
aws:
access_key_id: AWS_ACCESS_KEY_ID
secret_access_key: AWS_SECRET_ACCESS_KEY
We then use the credentials command to access the configs within the YMLs.
# aws.yml.erb access_key_id: <%= Rails.application.credentials.aws[:access_key_id] %> secret_access_key: <%= Rails.application.credentials.aws[:secret_access_key] %>
This was a nice feature to have, but it still couldn’t effectively be used with large applications. Larger applications need to be able to run across different environments and sometimes, from multiple points of delivery. Each environment (and PODs) would require to have different configurations. With the Rails 5.2 credentials feature, there was only a single file that would hold all your applications’ configurations. To support multiple environments, the YAML has to be modified to have the environment as a top-level key.
# credentials.yml.enc
development:
secret_key_base: SECRET_KEY_BASE_DEV
aws:
access_key_id: AWS_ACCESS_KEY_ID_DEV
secret_access_key: AWS_SECRET_ACCESS_KEY_DEV
production:
secret_key_base: SECRET_KEY_BASE_PROD
aws:
access_key_id: AWS_ACCESS_KEY_ID_PROD
secret_access_key: AWS_SECRET_ACCESS_KEY_PROD
Like the last example, this can also be accessed from the application in the following way:
Rails.application.credentials[Rails.env.to_sym].dig(:aws, :access_key_id)
However, as we add more configurations over time, the files become very large and it will be difficult to manage.
Multi-environment credentials
Rails 6 added built-in support for multi-environment credentials. This allowed us to create and manage files for each environment / POD.
rails credentials:edit --environment development
When the environment option is passed, config/credentials/#{environment}.yml.enc and config/credentials/#{env}.key are used as the credential files. If this was not present, the credentials.yml.enc file will be used.
The earlier example, when split into multiple files, becomes:
# config/credentials/development.yml.enc
secret_key_base: SECRET_KEY_BASE_DEV
aws:
access_key_id: AWS_ACCESS_KEY_ID_DEV
secret_access_key: AWS_SECRET_ACCESS_KEY_DEV
# config/credentials/production.yml.enc
secret_key_base: SECRET_KEY_BASE_PROD
aws:
access_key_id: AWS_ACCESS_KEY_ID_PROD
secret_access_key: AWS_SECRET_ACCESS_KEY_PROD
Now when we run `Rails.application.credentials.aws[:access_key_id]`, the value returned will depend on the environment the process is running on.
Along with this, Rails 6 also provides a way to customize the location of these credential files. To use a custom location, we can configure config.credentials.content_path and config.credentials.key_path. Content path and key_path locates the .yml.enc file and the .key file respectively.
# config/environment/development.rb config.credentials.content_path = ‘config/credentials/local.yml.enc’
This configuration tells Rails to look for the configurations in `config/credentials/local.yml.enc` instead of the default `config/credentials/development.yml.enc`
Deploying with credentials
Before starting the application, we have to ensure that the key is available for the credentials we have added. By setting `config.require_master_key = true` in the application or environment configuration, Rails will not be able to boot any processes unless it had the valid key.
In the local environment, the key is placed in the path configured in the key path or the default credentials key path, which is config/master.key or config/credentials/environment.key in case it is environment specific. But we won’t be pushing these keys to the remorse repository. Hence, it won’t be available in the production environment in the same location. To overcome this we can set an environment variable RAILS_MASTER_KEY in the production environment. This variable should hold the decryption key that was created and stored in the .key file. When this variable is set, the .key file is not required to decrypt the encrypted credentials.
More customization
As explained earlier, even in production, we have four different PODs. Not all secrets are the same for each of these PODs. By leveraging the ability to add custom credential files, we can set up different credentials for each POD. Credentials are flexible enough to support a great level of customization.
By using credentials, we can now effortlessly access secret configurations inside the application without worrying about encryption, decryption, unknowingly committing configurations to remote repositories, etc. Every new major version of Rails brings us a ton of features. It also provides the option to customize it based on our requirements to a great extent.
