Pulumi Overview #2 Pulumi: Configuration and secrets

Coming from Node.js and having used environment variables or even AWS Config for setting up project configuration, Pulumi’s config system is a breath of fresh air.

The configuration is split between the main project settings and a stack-specific configuration. In this article, we’ll look at how to manage stack configuration using the Pulumi CLI, we’ll go over different project-specific configuration options and handling secrets in Pulumi, including setting a custom secrets provider.

TOC

• Project Configuration
  • Strongly typed config
  • Other properties
• Stack Configuration
• Setting up stack configuration with the CLI
• Provider specific options
  • AWS Provider Config
  • Pulumi Provider Config
• Using configuration in code
• Using a custom secret provider
  • Setting AWS KMS as a secrets provider
• Encrypting with a passphrase
• Decrypting secrets
  • Decrypting secrets for use in the program
  • Decrypting secrets in the config file
  • Decrypting secrets in the programme outputs
• Persisting config to new stacks
• Conclusion

Project Configuration

The project configuration is stored in the Pulumi.yaml file. It contains the project name, description, runtime and language information. You can also use it to define a strongly typed configuration schema for your project or change the project structure to your liking.

name: pulumi-first-look
runtime: nodejs
description: A minimal AWS TypeScript Pulumi program
config:
  aws:region: us-east-1
  pulumi:tags:
    value:
      pulumi:template: ''

By default, Pulumi will use index.ts for the entry point of the program but you can change it by setting the main property in the Pulumi.yaml file.

+ main: ./src/index.ts
+ stackConfigDir: ./.config
name: pulumi-first-look
runtime: nodejs
description: A minimal AWS TypeScript Pulumi program
config:
  aws:region: us-east-1
  pulumi:tags:
    value:
      pulumi:template: ""

You can also change the directory where the stack configuration files are stored by setting the stackConfigDir property. This is useful when you have multiple stacks and you want to keep the configuration files separate as they could clutter up the project’s root directory.

Strongly typed config

As mentioned previously, we can also define a strongly typed schema using the project config file. This is done in the config property, but instead of assigning the property a single value, we add an indentation, that’s how yaml parses objects, and we have a few possible options here.

config:
  name:
    type: string
    description: company name
    secret: true
    default: default value
  regions:
    type: array
    items:
      - type: string

We can specify a type with the available scalar types – string, integer and boolean – additionally an array type is also available but more complex structures are not supported yet. If we use the array type, we can also specify the types for its items.

For me, the most enticing feature here is the secret property. The string and integer types are somewhat flexible because an integer value can be coerced into a string and vice versa. When adding a config value via the CLI it will be added as a string even if we enter an integer which will be automatically coerced into a number. I suppose that’s not a big issue but on the other hand the feature is supposed to add strong typing to the config schema. Coercing the values seems to be missing the point.

Then there’s the question of maintaining a schema in a yaml file for a large project. Will it be manageable or is it better to use something like zod for managing a valid configuration schema. That’s where the secret property comes in as it might not be worth the effort to try and replicate that behaviour.

Other properties

Here’s a quick run through of some other properties you can set in the Pulumi.yaml file that might not be obvious.

• template can be used for configuring your project as a template, similar to the existing templates that can be used with pulumi new command when creating a new project
• backend is used to specify a custom backend for the Pulumi CLI, by default this is the Pulumi Cloud SaaS offering
• plugins is for those of you that wish to create your own Pulumi plugins and can be used to link to local plugin binaries. It specifies which plugins should be automatically installed when the project is run. This property is not mandatory because Pulumi can often determine which plugins are needed based on the resources defined in your program. However, you can use this property to control the exact version of a plugin that your project uses, or to ensure that a plugin is installed even if it’s not directly referenced by any resources in your code.

You can find an exhaustive list of possible Project configuration options in the documentation .

Stack Configuration

A Pulumi programme can be split into stacks which is simply a logical separation of concerns. When creating a project, you can either name your own stack or Pulumi will use dev as the default stack name and create a Pulumi.dev.yaml file in the project root. This file contains the stack configuration and can be used to override the project configuration.

config:
  aws:region: us-east-2

In this example, the default project configuration is to deploy resources to the us-east-1 region, but the stack configuration overrides it to us-east-2. So there would probably be a Pulumi.prod.yaml file for the production stack and Pulumi.test.yaml for the test stack, but developers could also create their own ephemeral stacks for feature development without interfering with the main stacks and they could deploy them to a region closest to them.

Setting up stack configuration with the CLI

It is possible to manually add config values to the stack configuration file but it is recommended to use the Pulumi CLI to set them up.

pulumi config set aws:region us-west-2

You can also set a config value in a more complex structure like an object or an array.

pulumi config set --path 'custom.structure[0]' value

Last but not least, no config is complete without a secret. You can set a secret value using the --secret flag.

pulumi config set API_KEY somesecretvalue --secret

This will use a secret key managed by Pulumi to encrypt the values and store them securely in the config so that it can be committed to a remote repository.

Provider specific options

If you look into your config file, all properties are saved with the "<project_name>:<property_name>": <value> pattern. This is your config, and it will not be used by anyone else, but you can also define provider specific config and you most likely already have.

AWS Provider Config

When creating a new aws-typescript project, the CLI prompts you to choose a region and then saves it as aws:region. That’s how you can set any provider specific configuration option to customize the default provider. You can check its interface by going to the aws.Provider class constructor.

import * as aws from '@pulumi/aws';

const provider = new aws.Provider('custom-provider', { ...provider_props });

Pulumi Provider Config

There are also Pulumi specific options , for now there are only two.

Disable Default Provider

Use this option to disable default providers, this ensures that a provider will be explicitly created for each package that’s used e.g., aws, kubernetes, azure etc. This is a list, so you can specify each package individually.

config:
  pulumi:disable-default-providers:
    - aws
    - kubernetes
    - azure

Or, use a wild card * to disable them all

config:
    pulumi:disable-default-providers:
        - *

Tags

Second option is for tagging purposes, which uses stack tags to add stack metadata. The pulumi:tags config property is an equivalent of running

pulumi stack tag set <name> <value>

However, when adding it manually, it will not show-up in any of the config files. You can either check it in the Pulumi Cloud dashboard or by running

pulumi stack tag ls

Stack tags can be useful in larger organizations with multiple pulumi projects that need to be grouped together based on environment or function. After setting a stack tag you can use Pulumi Cloud for grouping stacks using those tags rather than the default project option

Using configuration in code

Pulumi provides a Config class that can be used to retrieve config values for use in the programme. To use the config, we need to create an instance of it and call one of the methods. There are two distinct categories of methods get* and require*. The get* methods return the value if it exists or undefined if it doesn’t whereas the require* methods throw an error if the value is not found.

import * as pulumi from '@pulumi/pulumi';

const config = new pulumi.Config();
const region = config.get('aws:region') || 'us-east-1';
const certificateArn = config.require('certificateArn');

The base methods get<T> and require<T> are generic and can be used to retrieve any value, defaulting to a string return type. However, there are also dedicated methods for retrieving specific types of values.

config.getNumber('');
config.requireNumber('');

config.getBoolean('');
config.requireBoolean('');

config.getObject('');
config.requireObject('');

Using a custom secret provider

Understandably, some if not all dev teams might want to use their own secret provider especially for production workloads. Luckily, Pulumi supports four popular platforms for encrypting secrets: AWS KMS, Azure Key Vault, GCP KMS and HashiCorp Vault.

Setting AWS KMS as a secrets provider

When using AWS, we can choose three different options for specifying the custom provider – key’s arn, alias or id – and change it at any time

When creating a new project

pulumi new aws-typescript --secrets-provider "awskms://{KEY_ID}?region=eu-central-1&context_project=first-look&context_environment=dev"

When initializing a new stack

pulumi stack init new-stack --secrets-provider="awskms:///arn:aws:kms:{REGION}:{ACCOUNT_ID}:key/{KEY_ID}?context_project=first-look&context_environment=new-stack"

Changing an existing stack’s provider

pulumi stack change-secrets-provider "awskms://alias/ExampleAlias?&context_project=first-look&context_environment=prod"

Naturally, should you want to change the provider, and you already have some encrypted secrets, the existing secrets will be re-encrypted with the new key and saved in your config file when changing the secret provider.

As you may have noticed, we can also add additional data to the encryption context which can be useful for auditing purposes. Context data will be included in the events and could be used with IAM policies as well.

Encrypting with a passphrase

Rather than using a cloud provider, we can also encrypt secrets with a passphrase which could be very useful for local development.

To do that, we can use the same commands as before but with passphrase as the provider. CLI will then prompt you to enter your passphrase.

pulumi stack init new-stack --secrets-provider="passphrase"

This will prompt you to enter your passphrase and confirm it.

The passphrase will be used to encrypt the secrets and will be required to decrypt them when running the program.

To avoid having to enter the passphrase manually all the time, you can set a variable PULUMI_CONFIG_PASSPHRASE in your environment.

export PULUMI_CONFIG_PASSPHRASE=somepassphrase

Decrypting secrets

There are three different situations when we might want/need to decrypt secrets. First, when we actually want to use the secret in our program, second, when we want to see what are the encrypted values in the config file, and third when there are secrets in the outputs of the pulumi programme e.g., database passwords.

Decrypting secrets for use in the program

To decrypt a secret in the program, we can use dedicated methods on the config object.

// index.ts
import * as pulumi from '@pulumi/pulumi';

const config = new pulumi.Config();
config.getSecret('API_KEY');
config.requireSecret('API_KEY');

Similarly to the regular get and require methods, there are getSecret and requireSecret methods for retrieving secret values, but also type specific methods.

config.getSecretNumber('');
config.requireSecretNumber('');

config.getSecretBoolean('');
config.requireSecretBoolean('');

config.getSecretObject('');
config.requireSecretObject('');

Decrypting secrets in the config file

To decrypt secrets using the Pulumi CLI, we can use the config command with the show-secrets flag.

pulumi config --show-secrets

Decrypting secrets in the programme outputs

Similarly to config, we can use the stack output command with the --show-secrets flag.

pulumi stack output --show-secrets

Persisting config to new stacks

Instead of copying over the existing config to a new stack, you can also use the --copy-config-from argument to use an existing stack’s config as a starting point

pulumi stack init qa --copy-config-from dev

One important detail to keep in mind is that this will not migrate a custom secrets provider, but it will default back to the Pulumi Cloud. To setup the secrets provider when creating a new stack you can use the --secrets-provider argument like we covered when setting kms as a custom provider .

Conclusion

Pulumi’s configuration system offers a fresh and efficient way to manage project and stack settings compared to traditional environment variables or AWS Config. It allows for detailed configurations via the Pulumi.yaml file, supports strongly typed schemas, handles secrets securely, and provides flexibility in setting custom secret providers and organizing stack configurations, all while simplifying management through the Pulumi CLI.

FAQ