Batch Spec Templating

Understand how to use templating to make your batch changes more powerful.

Certain fields in a batch spec YAML support templating to create even more powerful and performant batch changes. Templating in a batch spec uses the delimiters ${{ and }}. Inside the delimiters, template variables and template helper functions may be used to produce a text value.

Example batch spec with templating

Here is an excerpt of a batch spec that uses templating:

YAML
on: - repositoriesMatchingQuery: lang:go fmt.Sprintf("%d", :[v]) patterntype:structural -file:vendor steps: - run: comby -in-place 'fmt.Sprintf("%d", :[v])' 'strconv.Itoa(:[v])' ${{ join repository.search_result_paths " " }} # ^ templating starts here container: comby/comby - run: goimports -w ${{ join previous_step.modified_files " " }} # ^ templating starts here container: unibeautify/goimports

Before executing the first run command, repository.search_result_paths will be replaced with the relative-to-root-dir file paths of each search result yielded by repositoriesMatchingQuery. Using the template helper function join, an argument list of whitespace-separated values is constructed.

The final run value that will be executed will look similar to this:

YAML
run: comby -in-place 'fmt.Sprintf("%d", :[v])' 'strconv.Itoa(:[v])' cmd/src/main.go internal/fmt/fmt.go

The result is that comby only searches and replaces in those files instead of having to search through the complete repository.

Before the second step is executed, previous_step.modified_files will be replaced with the list of files that the previous comby step modified. It will look similar to this:

YAML
run: goimports -w cmd/src/main.go internal/fmt/fmt.go
See Templating Examples for more examples of how to use and leverage templating in batch specs.

Fields with template support

Templating is supported in the following fields:

Template variables

Template variables are the names that are defined and accessible when using templating syntax in a given context. Different variables are available depending on the context in which templating is used.

For example, in the context of steps, the template variable previous_step is available, but not in the context of changesetTemplate.

steps context

The following template variables are available in the fields under steps. They are evaluated before executing each entry in steps, except for the step.* variables, which only contain values after the step has executed.

Template VariableTypeDescription
batch_change.namestringThe name of the batch change, as set in the batch spec
batch_change.descriptionstringThe description of the batch change, as set in the batch spec
repository.search_result_pathslist of stringsUnique list of file paths relative to the repository root directory in which the search results of the on.repositoriesMatchingQuerys have been found. Empty list if a select:repo filter is used in the on.repositoriesMatchingQuery, or if only on.repository entries are specified
repository.branchstringThe target branch of the repository in which the step is being executed
repository.namestringFull name of the repository in which the step is being executed. For example, org_foo/repo_bar
previous_step.modified_fileslist of stringsList of files that the previous steps have modified. Empty list if no files have been modified
previous_step.added_fileslist of stringsList of files that the previous steps have added. Empty list if no files have been added
previous_step.deleted_fileslist of stringsList of files that the previous steps have deleted. Empty list if no files have been deleted
previous_step.stdoutstringThe complete output of the previous step on standard output
previous_step.stderrstringThe complete output of the previous step on standard error
step.modified_fileslist of stringsOnly in steps.outputs: List of files that the just-executed step has modified. Empty list if no files have been modified
step.added_fileslist of stringsOnly in steps.outputs: List of files that the just-executed step has added. Empty list if no files have been added
step.deleted_fileslist of stringsOnly in steps.outputs: List of files that the just-executed step has deleted. Empty list if no files have been deleted
step.stdoutstringOnly in steps.outputs: The complete output of the just-executed step on standard output
step.stderrstringOnly in steps.outputs: The complete output of the just-executed step on standard error
steps.modified_fileslist of stringsList of files modified by the steps. Empty list if no files have been modified
steps.added_fileslist of stringsList of files that have been added by the steps. Empty list if no files have been added
steps.deleted_fileslist of stringsList of files deleted by the steps. Empty list if no files have been deleted
steps.pathstringPath (relative to the root of the directory, no leading / or .) in which the steps have been executed. Empty if no workspaces have been used and the steps were executed in the root of the repository

changesetTemplate context

The following template variables are available in the fields under changesetTemplate. They are evaluated after the execution of all entries in steps.

Template VariableTypeDescription
batch_change.namestringThe name of the batch change, as set in the batch spec
batch_change.descriptionstringThe description of the batch change, as set in the batch spec
repository.search_result_pathslist of stringsUnique list of file paths relative to the repository root directory in which the search results of the on.repositoriesMatchingQuerys have been found. Empty list if a select:repo filter is used in the on.repositoriesMatchingQuery, or if only on.repository entries are specified
repository.branchstringThe target branch of the repository in which the step is being executed
repository.namestringFull name of the repository where the step is being executed. For example, org_foo/repo_bar
steps.modified_fileslist of stringsList of files that have been modified by the steps. Empty list if no files have been modified
steps.added_fileslist of stringsList of files that have been added by the steps. Empty list if no files have been added
steps.deleted_fileslist of stringsList of files deleted by the steps. Empty list if no files have been deleted
steps.pathstringPath (relative to the root of the directory, no leading / or .) in which the steps have been executed. Empty if no workspaces have been used and the steps were executed in the root of the repository
outputs.<name>depends on outputs.<name>.format, default: stringValue of an output set by steps. If the outputs.<name>.format is yaml or json and the value a data structure (i.e., array, object, ...), then subfields can be accessed too. See Templating Examples below
batch_change_linkstringOnly available in changesetTemplate.body
Link back to the batch change that produced the changeset on Sourcegraph. If omitted, the link will be automatically appended to the end of the body.
Requires Sourcegraph CLI 3.40.9 or later

Template helper functions

  • ${{ join repository.search_result_paths "\n" }} - Joins the list of strings given as the first argument with the separator as the last argument
  • ${{ join_if "---" "a" "b" "" "d" }} - Uses the first argument as a separator to join the remaining arguments, ignoring blank strings
  • ${{ replace "a/b/c/d" "/" "-" }} - Replaces occurrences of the second argument in the first one with the last one
  • ${{ split repository.name "/" }} - Splits the first argument into a list of strings at each occurrence of the last argument
  • ${{ matches repository.name "github.com/my-org/terra*" }} - Matches the first argument against the glob pattern in the second argument, returning true/false
  • ${{ "${{ repository.name }}" }} - Outputs the inner expression as a literal string, for example, to ignore the inner set of ${{ }}

The features of Go's text/template package are also available, including conditionals and loops, since it is the underlying templating engine.

Templating Examples

Pass the exact list of search result file paths to a command:

YAML
steps: - run: comby -in-place -config /tmp/go-sprintf.toml -f ${{ join repository.search_result_paths "," }} container: comby/comby files: /tmp/go-sprintf.toml: | [sprintf_to_strconv] match='fmt.Sprintf("%d", :[v])' rewrite='strconv.Itoa(:[v])'

Run a command for each search result file path:

YAML
steps: - run: | for file in "${{ join repository.search_result_paths " " }}"; do sed -i 's/mydockerhub-user/ci-dockerhub-user/g;' ${file} done container: alpine:3

Format and fix files after a previous step modified them:

YAML
steps: | - run: | | | find . -type f -name '*.go' -not -path "*/vendor/*" | \ | xargs sed -i 's/fmt.Println/log.Println/g' container: alpine:3 - run: goimports -w ${{ join previous_step.modified_files " " }} container: unibeautify/goimports

Use the steps.files combined with template variables to construct files inside the container:

YAML
steps: | - run: | | | cat /tmp/search-results | while read file; | do | ruplacer --subvert whitelist allowlist --go ${file} | | echo "nothing to replace"; | | ruplacer --subvert blacklist denylist --go ${file} | | echo "nothing to replace"; | done container: ruplacer files: /tmp/search-results: ${{ join repository.search_result_paths "\n" }}

Put information in environment variables, based on the output of previous step steps.env also

YAML
steps: - run: echo $LINTER_ERRROS >> linter_errors.txt container: alpine:3 env: LINTER_ERRORS: ${{ previous_step.stdout }}

If you need to escape the ${{ and }} delimiters you can simply render them as string literals:

YAML
steps: - run: cp /tmp/escaped.txt . container: alpine:3 files: /tmp/escaped.txt: ${{ "${{" }} ${{ "}}" }}

Accessing the outputs set by steps in subsequent steps and the changesetTemplate:

YAML
steps: - run: echo "Hello there!" container: alpine:3 outputs: myFriendlyMessage: value: "${{ step.stdout }}" - run: echo "We have access to the output here: ${{ outputs.myFriendlyMessage }}" container: alpine:3 outputs: stepTwoOutput: otherMessage: "here too: ${{ outputs.myFriendlyMessage }}" changesetTemplate: # [...] body: | The first step left us the following message: ${{ outputs.myFriendlyMessage }} The second step left this one: ${{ outputs.otherMessage }}

Using the steps.outputs.<name>.format field, it's possible to parse the value of output as JSON or YAML and access it as a data structure instead of just text:

YAML
steps: - run: cat .goreleaser.yml container: alpine:3 outputs: goreleaserConfig: value: "${{ step.stdout }}" # The step's output is parsed as YAML, making it accessible as a YAML # object in the other templating fields. format: yaml goreleaserConfigExists: # We can use the power of Go's text/template engine to dynamically produce complex values value: "exists: ${{ gt (len step.stderr) 0 }}" format: yaml changesetTemplate: # [...] # Since templating fields use Go's `text/template` and `goreleaserConfig` was # parsed as YAML we can iterate over every field: body: | This repository has a `gorelaserConfig`: ${{ outputs.goreleaserConfigExists.exists }}. The `goreleaser.yml` defines the following `before.hooks`: ${{ range $index, $hook := outputs.goreleaserConfig.before.hooks }} - `${{ $hook }}` ${{ end }}

Using the steps.if field to conditionally execute different steps in different repositories:

YAML
steps: # `if:` is true, step always executes. - if: true run: echo "name of repository is ${{ repository.name }}" >> message.txt container: alpine:3 # `if:` checks for repository name. Only runs in github.com/sourcegraph/automation-testing - if: ${{ eq repository.name "github.com/sourcegraph/automation-testing" }} run: echo "hello from automation-testing" >> message.txt container: alpine:3 # `if:` uses glob pattern to match repository name. - if: ${{ matches repository.name "*sourcegraph-testing*" }} run: echo "name contains sourcegraph-testing" >> message.txt container: alpine:3 # Checks for go.mod existence and saves to outputs - run: if [[ -f "go.mod" ]]; then echo "true"; else echo "false"; fi container: alpine:3 outputs: goModExists: value: ${{ step.stdout }} # `if:` uses the just-set `outputs.goModExists` value as condition - if: ${{ outputs.goModExists }} run: go fmt ./... container: golang # `if:` checks for path, in case steps are executed in workspace. - if: ${{ eq steps.path "sub/directory/in/repo" }} run: echo "hello workspace" >> workspace.txt container: golang

Combine the template helper functions with the helper functions built into Go's text/template library:

YAML
changesetTemplate: # [...] body: | The host of the repository: ${{ index (split repository.name "/") 0 }} The org of the repository: ${{ index (split repository.name "/") 1 }}

Render the batch change link at the beginning of the changeset body:

YAML
changesetTemplate: # [...] body: | ${{ batch_change_link }} This is the rest of my changeset description.
Previous
Batch Spec Reference