GoCD server plugin to keep pipelines and environments configuration in source-control in YAML. See this document to find out what are GoCD configuration repositories.
This is second GoCD configuration plugin, which enhances some of shortcomings of JSON configuration plugin
- Format is concise. Unlike JSON, there are no unnecessary quotations
", brackets{}and commas, - Tries to enforce correctness where possible. By using maps instead of lists and shorter object graphs.
- Allows to have multiple pipelines and environments in single file. But doesn't force it.
- Comments in configuration files - YAML supports comments, so you can explain why pipeline/environment it is configured like this.
- Shorter syntax for declaring single-job stages
- Very short syntax for declaring tasks with script executor plugin
Step 1: GoCD versions newer than 17.8.0 already have the plugin bundled. You don't need to install anything.
If you're using GoCD version older than 17.8.0, you need to install the plugin in the GoCD server. Download it from
the releases page and place it on the GoCD server in
plugins/external directory.
Step 2: Follow the GoCD documentation to add a new configuration repository.
You can use the example repository at: https://github.com/tomzo/gocd-yaml-config-example.git.
In your config repo (tomzo/gocd-yaml-config-example.git in this case), ensure that your GoCD yaml config is suffixed with .gocd.yaml. Any file ending in .gocd.yaml is picked up by the plugin. Give it a minute or so for the polling to happen. Once that happens, you should see your pipeline(s) on your dashboard.
More examples are in test resources.
#ci.gocd.yaml
format_version: 4
environments:
testing:
environment_variables:
DEPLOYMENT: testing
secure_variables:
ENV_PASSWORD: "s&Du#@$xsSa"
pipelines:
- example-deploy-testing
- build-testing
pipelines:
mypipe1: # definition of mypipe1 pipeline
group: mygroup # note that the group name can contain only of alphanumeric & underscore characters
display_order: 10
label_template: "${mygit[:8]}"
lock_behavior: none
parameters: # list of parameters that can be configured for a pipeline
param1: value1
materials:
mygit: # this is the name of material, the name can contain only of alphanumeric & underscore characters
# keyword git says about type of material and url at once
git: http://my.example.org/mygit.git
branch: ci
myupstream: # this name does not matter, but there should be no 2 materials with the same name
# type is optional here, material type is implied based on presence of pipeline and stage fields
# type: dependency
pipeline: pipe2
stage: test
stages: # list of stages in order
- build: # name of stage
clean_workspace: true
jobs:
csharp: # name of the job
resources:
- net45
artifacts:
- build:
source: bin/
destination: build
- test:
source: tests/
destination: test-reports/
- test:
source: coverage.xml
tabs:
report: test-reports/index.html
tasks: # ordered list of tasks to execute in job csharp
- fetch:
pipeline: pipe2
stage: build
job: test
source: test-bin/
destination: bin/
- exec: # indicates type of task
command: make
arguments:
- "VERBOSE=true"
# shorthand for script-executor plugin
- script: ./build.sh ciBy default GoCD configuration files should end with .gocd.yaml or .gocd.yml.
You can set a custom file pattern per configuration repository using the GoCD configuration UI or in the config XML like this:
<config-repos>
<config-repo pluginId="yaml.config.plugin" id="repo1">
<git url="https://github.com/tomzo/gocd-yaml-config-example.git" />
<configuration>
<property>
<key>file_pattern</key>
<value>pipeline.gocd.yaml</value>
</property>
</configuration>
</config-repo>
</config-repos>- If you have questions on usage, please ask them on the gitter chat room dedicated for configrepo-plugins
- If you think there is a bug, or you have an idea for a feature and you are not sure if it's plugin's or GoCD fault/responsibity, please ask on the chat first too.
Please note this brief overview of what is done by the plugin:
- parsing yaml into json when GoCD server asks for it. And this is done by the GoCD server:
- complex logic merging multiple config repo sources and XML
- validation of pipelines/stages/jobs/tasks domain
- any UI rendering
Since 0.8.2 plugin is an executable and supports basic syntax checking.
You need to download the jar from releases page and place it somewhere convenient.
For example /usr/lib/gocd-yaml-plugin/yaml-config-plugin.jar.
Then to validate your gocd.yaml file run something like:
java -jar /usr/lib/gocd-yaml-plugin/yaml-config-plugin.jar syntax ci.gocd.yaml
This is a product in development, so its command syntax is not stable and there are no distributed binaries yet.
The gocd tool is built in golang so you will need to familiarize yourself with how to set up your go workspace.
Build the gocd binary:
go get github.com/gocd-contrib/gocd-cli
cd ${GOPATH:-~/go}/src/github.com/gocd-contrib/gocd-cli
./build.shFollow the steps on https://github.com/gocd-contrib/gocd-cli to install the plugin jar to the correct place.
Then:
./gocd configrepo syntax -i yaml.config.plugin /path/to/your-pipeline.gocd.yamlIDE is a bash script, a cli wrapper around docker to help with running development tasks in docker.
You can install the ide script so that it is available on the PATH with:
sudo bash -c "`curl -L https://raw.githubusercontent.com/ai-traders/ide/master/install.sh`"
Add Idefile in your project with following content
IDE_DOCKER_IMAGE=tomzo/gocd-yaml-ide:<plugin-version>
To validate files run:
ide gocd-yaml syntax ci.gocd.yaml
Personally, I recommend the following project structure:
gocd/directory for all your GoCD configuration files.gocd/Idefilefile pointing which docker image can be used to validate configuration.
Then when working with gocd pipelines config, you can run from the root of your project
cd gocd
ide # will open interactive shell
watch gocd-yaml syntax ci.gocd.yaml
docker run -ti --rm --volume $(pwd):/ide/work tomzo/gocd-yaml-ide:<plugin-version> bash
Then you have an interactive shell as above.
See official GoCD XML configuration reference for details about each element. Below is a reference of format supported by this plugin. Feel free to improve it!
- Format version
- Environment
- Environment variables
- Parameters
- Pipeline
- Stage
- Job
- Tasks
- Materials
- Secure variables
- YAML Aliases
Please note that it is now recommended to declare format_version in each gocd.yaml file, consistent across all your files.
Supports format_version value of 4. In this version, support has been added to control the display order of pipelines.
This server version also supports format_version of 3 and 2. Using a newer format_version includes all the behavior of the previous versions too.
format_version: 4
pipelines:
...
environments:Supports format_version value of 3. In this version fetch artifact format was changed to include artifact_origin.
This server version also supports format_version of 2. Using a newer format_version includes all the behavior of the previous versions too.
format_version: 3
pipelines:
...
environments:Supports format_version value of 2. In this version pipeline locking behavior was changed.
format_version: 2
pipelines:
...
environments:Supports format_version value of 1. This is the initial version.
format_version: 1
pipelines:
...
environments:A minimal pipeline configuration must contain:
- pipeline name - as a key in hash
- materials - a hash
- stages - a list or a
template- as a key in hash
mypipe:
group: mygroup
materials:
mygit:
git: http://example.com/mygit.git
stages:
- build:
jobs:
build:
tasks:
- exec:
command: makeAll elements available on a pipeline object are:
grouplabel_template- locking
- tracking_tool or
mingle - timer
- environment_variables
- parameters
secure_variables- materials
- stages
template
pipe2:
group: group1
label_template: "foo-1.0-${COUNT}"
lock_behavior: none
tracking_tool:
link: "http://your-trackingtool/yourproject/${ID}"
regex: "evo-(\\d+)"
timer:
spec: "0 15 10 * * ? *"
environment_variables:
DEPLOYMENT: testing
secure_variables:
ENV_PASSWORD: "s&Du#@$xsSa"
materials:
...
stages:
...mypipe:
group: group1
label_template: "foo-1.0-${COUNT}"
lock_behavior: none
parameters:
param1: value
materials:
mygit:
git: http://example.com/mygit.git
template: template1Please note:
- Pipeline declares a group to which it belongs
When format_version is 4 (see above), the order of display of pipelines on the GoCD dashboard can be influenced by setting the display_order property.
- This is an integer property and the pipelines in a pipeline group will be ordered by this value.
- The default value for this property is
-1. - Pipelines defined in GoCD's config XML will also default to -1.
- If multiple pipelines have the same
display_ordervalue, their order relative to each other will be indeterminate.
mypipeline1:
group: group1
display_order: 10
mypipeline2:
group: group1
display_order: -10In the above example, since both pipelines are in the same group, pipeline2 will be shown ahead of pipeline1. If any pipelines are defined in the GoCD config XML, then they will appear in between these two pipelines.
tracking_tool:
link: "http://your-trackingtool/yourproject/${ID}"
regex: "evo-(\\d+)"timer:
spec: "0 15 10 * * ? *"
only_on_changes: yesSee XML reference for more information.
For GoCD >= 17.12 and format_version: 2 and above:
lock_behavior: noneWhere lock_behavior is defined as in GoCD documentation can be one of:
lockOnFailure- same aslocking: trueunlockWhenFinishednone- samelocking: false
For GoCD < 17.12 and format_version: 1:
locking: on
Where locking is a boolean.
A minimal stage must contain jobs: element or tasks: in single-job stage case.
build:
jobs:
firstJob:
...
secondJob:
...A custom stage:
test:
fetch_materials: yes
keep_artifacts: yes
clean_workspace: yes
approval:
type: manual
roles:
- manager
users:
- john
environment_variables:
TEST_NUM: 1
secure_variables:
PASSWORD: "!@ESsdD323#sdu"
jobs:
one:
...
two:
...Stage can have approval,
which is success by default. There are 2 ways to declare approval:
approval: manualIf you need to set associated users or roles:
approval:
type: manual
roles:
- manager
users:
- johnJob is a hash starting with jobs name:
test:
timeout: 5
run_instances: 7
environment_variables:
LD_LIBRARY_PATH: .
tabs:
test: results.xml
resources:
- linux
artifacts:
- test:
source: src
destination: dest
- build:
source: bin
- external:
id: docker-release-candidate
store_id: dockerhub
configuration:
options:
Image: gocd/gocd-demo
Tag: v${GO_PIPELINE_LABEL}
secure_options:
some_secure_property: "!@ESsdD323#sdu"
properties:
perf:
source: test.xml
xpath: "substring-before(//report/data/all/coverage[starts-with(@type,\u0027class\u0027)]/@value, \u0027%\u0027)"
tasks:
...Note: timeout is added since 0.2.0 version of yaml plugin
Job configuration may define elastic agents profile id, as such:
elastic_profile_id: "docker.unit-test"It MUST NOT be specified along with resources.
Available in GoCD server since v16.12.0, yaml plugin 0.4.0.
There are 3 types of artifacts recognized by GoCD. Build and Test artifacts are stored on the GoCD server.
The source and the destination of the artifact that should be stored on the GoCD server must be specified.
- build:
source: bin
destination: binaries- test:
source: reports
destination: test-reportsArtifacts of type external are stored in an artifact store outside of GoCD.
The external artifact store's configuration must be created in the main GoCD config. Support for external artifact store config to be checked in as yaml is not available.
The external artifact store is referenced by the store_id. The build specific artifact details that the artifact plugin needs to publish the artifact is provided as configuration.
- external:
id: docker-release-candidate
store_id: dockerhub
configuration:
options:
Image: gocd/gocd-demo
Tag: v${GO_PIPELINE_LABEL}
secure_options:
some_secure_property: "!@ESsdD323#sdu"Part of job object can be number of job to runs:
run_instances: 7Or to run on all agents:
run_instances: allTabs are a hash with <tab-name>: <path> pairs.
Path should exist in GoCD servers artifacts.
tabs:
tests: test-reports/index.html
gauge: functional-reports/index.htmlJob can have properties, declared as a hash:
properties:
cov1: # this is the name of property
source: test.xml
xpath: "substring-before(//report/data/all/coverage[starts-with(@type,\u0027class\u0027)]/@value, \u0027%\u0027)"
performance.ind1.mbps:
source: PerfTestReport.xml
xpath: "//PerformanceSuiteReport/WriteOnly/MBps"A common use case is that stage has only one job. This plugin provides a shorthand
to declared such stages - just omit the jobs: and job name from configuration tree.
You can then declare job and stage options on the same (stage) level:
stages:
- build:
approval: manual
resources:
- cpp
tasks:
- exec:
command: makeAbove configuration declares build stage with build job which executes make task.
Minimal configuration of a git pipeline material:
- material name is
mygit - git repository url is
http://example.com/mygit.git
mygit:
git: http://example.com/mygit.gitAbove can be also written more explicitly:
mygit:
type: git
url: http://example.com/mygit.gitMore customized git material is possible:
gitMaterial1:
git: "http://my.git.repository.com"
branch: feature12
blacklist:
- externals/**/*.*
- tools/**/*.*
destination: dir1
auto_update: false
shallow_clone: trueSince GoCD >= 16.7.0 whitelist is also supported,
you can specify whitelist instead of blacklist, as such
gitMaterial1:
git: "git@my.git.repository.com"
branch: "feature12"
whitelist:
- src/**/*.*For details about each option, see GoCD XML reference
svnMaterial1:
svn: "http://svn"
username: "user1"
password: "pass1"
check_externals: true
blacklist:
- tools
- lib
destination: destDir1
auto_update: falsehgMaterial1:
hg: repos/myhg
blacklist:
- externals
- tools
destination: dir1
auto_update: falsep4Material1:
p4: "host.domain.com:12345"
username: johndoe
password: pass
use_tickets: false
view: |
//depot/external... //ws/external...
//depot/tools... //ws/external...
blacklist:
- externals
- tools
auto_update: falseInstead of password you can use encrypted_password.
TODO: - not supported by yaml plugin yet
myPluggableGit:
scm: someScmGitRepositoryId
destination: destinationDir
blacklist:
- dir1
- dir2Since GoCD >= 19.2.0 defining new pluggable materials that are not defined
in the GoCD server is supported.
myPluggableGit:
plugin_configuration:
id: plugin_Id
version: 1 #plugin version
options:
url: git@github.com:tomzo/gocd-yaml-config-plugin.git
destination: destinationDir
blacklist:
- dir1
- dir2
This is a convenience for shorter and more consistent material declaration. When configuration repository is the same as one of pipeline materials, then you usually need to repeat definitions in XML and in JSON, for example:
materials:
foo:
git: "https://github.com/tomzo/gocd-json-config-example.git",
branch: ciAnd in server XML:
<config-repos>
<config-repo pluginId="yaml.config.plugin" id="repo1">
<git url="https://github.com/tomzo/gocd-json-config-example.git" branch="ci" />
</config-repo>
</config-repos>Notice that url and branch is repeated. This is inconvenient in case when you move repository, because it requires 2 updates, in code and in server XML.
Using configrepo material type, above repetition can be avoided,
last example can be refactored into:
materials:
foo:
type: configrepoServer interprets configrepo material in this way:
Clone the material configuration of the repository we are parsing as is in XML and replace name, destination and filters (whitelist/blacklist), then use the modified clone in place of
configrepomaterial.
To add a dependency on another pipeline stage:
mydependency:
pipeline: upstream-pipeline-1
stage: testNote: mydependency is the name of material - it must be unique
myapt:
package: apt-repo-idEvery task is a hash starting with its type.
Type can be exec, ant, nant, rake, fetch, plugin or script.
<type>:
<task-prop1>: <prop1-value>Optionally any task can have run_if and on_cancel.
run_ifis a string. Valid values arepassed,failed,anyon_cancelis a task object. Same rules apply as to tasks described on this page. See rake example.
exec:
run_if: any
working_directory: dir
command: make
arguments:
- -j3
- docs
- installant:
build_file: mybuild.xml
target: compile
run_if: passednant:
run_if: passed
working_directory: "script/build/123"
build_file: FilePath
target: Build
nant_path: NantExeA minimal rake task with default values is very short
rake:A complete rake example:
rake:
run_if: passed
working_directory: sample-project
build_file: SomeRakefile
target: build
on_cancel:
rake:
working_directory: sample-project
build_file: CancelRakefile
target: cancelfetch:
run_if: any
artifact_origin: gocd # default value
pipeline: pipe2
stage: upstream_stage
job: upstream_job
is_file: yes
source: result
destination: testartifact_originif not specified then, defaults togocd, which is consistent with previousformat_version1 and 2.pipelineif not specified, then defaults to current pipeline.
fetch:
run_if: any
artifact_origin: external
pipeline: pipe2
stage: upstream_stage
job: upstream_job
artifact_id: upstream_artifact_id
configuration:
options:
DestOnAgent: foo
secure_options:
some_secure_property: "!@ESsdD323#sdu"plugin:
options:
ConverterType: jsunit
secure_options:
password: "ssd#%fFS*!Esx"
run_if: failed
configuration:
id: xunit.converter.task.plugin
version: 1Because script-executor plugin requires quite a lot of boiler-plate configuration there is a shorthand for defining tasks with it:
script: ./build.sh compileAbove is equivalent of
plugin:
options:
script: ./build.sh compile
configuration:
id: script-executor
version: 1You can declare multi-line scripts too:
script: |
./build.sh compile
make testAbove executes a 2-line script:
./build.sh compile
make testIf you want to execute a single long line script, but break it into multiple lines in YAML,
you can do this with > as such:
script: >
./build.sh compile &&
make testAbove executes a single line script:
./build.sh compile && make testNOTE: The agents should be a guid, which is currently impossible to get for user
testing:
environment_variables:
DEPLOYMENT: testing
secure_variables:
ENV_PASSWORD: "s&Du#@$xsSa"
pipelines:
- example-deploy-testing
- build-testing
agents:
- 123Environment variables can be declared in Environments, Pipelines, Stages and Jobs.
In YAML you have 2 keywords for secure (encrypted) variables and standard variables.
environment_variables:
DEPLOYMENT: testing
FOO: bar
secure_variables:
# this value is encrypted by Go's private key (Note in 16.7.0 there is no easy way to obtain such value yet)
MY_PASSWORD: "s&Du#@$xsSa"Parameters can be declared at the pipeline level.
parameters:
param1: value1
production: noFor versions of GoCD >= 17.1:
See the encryption API.
For versions of GoCD <= 16.12:
There is no easy way to generate obtain encrypted value from GoServer, alternatively you can login into go-server and execute the following command to generate encrypted value
sudo -u go bash -c 'echo -n 'YOUR-INPUT' | openssl enc -des-cbc -a -iv 0 -K $(cat /etc/go/cipher)'Among all configuration elements there are boolean values, which can be defined using any of the keywords below (as in yaml specs):
- true -
y|Y|yes|Yes|YES|true|True|TRUE|on|On|ON - false -
n|N|no|No|NO|false|False|FALSE|off|Off|OFF
YAML Aliases (specification) are supported and provide a way to avoid duplication.
Aliases can be defined anywhere in the configuration as long as they are valid configuration elements.
- exec:
command: make
arguments:
- clean
- &verbose_arg "VERBOSE=true" # define the alias
- exec:
command: make
arguments:
- world
- *verbose_arg # use the aliasThere is also a dedicated top-level common section which allows you to have all aliases in one place and where you don't need to worry about correct placement within the configuration.
format_version: 4
common:
verbose_arg: &verbose_arg "VERBOSE=true"
build_tasks: &build_tasks
- exec:
command: make
arguments:
- clean
- exec:
command: make
arguments:
- world
pipelines:
pipe1:
stages:
- build:
jobs:
build:
tasks: *build_tasks
test:
tasks:
- *build_tasks # task list aliases can also be mixed with additional tasks in the same job
- exec:
command: make
arguments:
- testSimilarly stages can be re-used:
format_version: 4
common:
build_stages: &build_stages
- stage1:
jobs:
# ...
- stage2:
jobs:
# ...
pipelines:
pipeline_one:
stages: *build_stages
pipeline_two:
stages:
- *build_stages
- stage3:
jobs:
# ...Run all tests and create a ready to use jar
./gradlew test jarWe use semantic versioning.
If you are submitting a new feature then please run a major version bump by
./tasks.sh set_version 0.X.0
If you are submitting a fix, then do not change any versions as patch bump is made right after each release.
There are examples of yaml partials and their resulting json to be sent to GoCD server. If something is not working right we can always add a new case covering exact yaml that user has and json that we expect on server side.