Earlier this year, we open-sourced the Jenkins Config-Driven Pipeline Plugin to share some work we did internally to make it easy to provide a flexible pipeline setup for engineers. Our main goal was to allow engineers to easily onboard themselves onto Jenkins without having to know Jenkinsfile or write a single line of Groovy (unless, of course, they want to contribute or use Groovy in their pipeline).
At Salesforce, we internally use Jenkins to power our continuous integration (CI) workflows, and we have a lot of pipelines. Many of these pipelines generally follow the same pattern for an overall pipeline but have variations in how some of the steps run for different stages (using Jenkins Pipeline terminology). Additionally, there are some stages and steps that we want to run in a certain way for security and compliance. We have folks who prefer to use languages other than Java, such as Go, Python, Ruby, etc⌠We felt that they shouldnât necessarily have to learn Jenkinsfile, Groovy, and Jenkins Pipeline in general in order to have productive CI workflows. The config-driven pipeline approach allows us the ability to centrally roll out updates and improvements to hundreds or thousands of pipelines at a time since thousands of pipelines are using a small handful of shared Jenkinsfiles.
We love configuration as code and pipelines as code for many reasons. Reuse, history, and repeatability are fantastic benefits to using things like the Jenkins Configuration as Code plugin, Job DSL plugin, and Jenkins Pipelines. This plugin provides the ability for you to set up a different Project Recognizer which will look for the configuration file name that you select. This allows pipelines to reuse Jenkinsfiles across branches, repositories, and organizations. It also provides a branching model for trying out new templates and configurations. You can name the template Jenkinsfiles however youâd like and you can use any compatible Jenkinsfile syntax (scripted or declarative).
The config file format is flexible besides the one requirement of setting the pipeline_template to a path within the Jenkinsfiles repository. You would just set pipeline_template and any other configuration data using the configuration format that you prefer (YAML, JSON, Java Properties, and others are currently supported).
Example Config-Driven Pipeline
In this example, the .myci.yml file is the config file that weâll use to configure values. When a build is triggered for this pipeline, the pipeline reads pipeline_template and executes examples/dynamic/Jenkinsfile in this case. The contents of .myci.yml get passed to examples/dynamic/Jenkinsfile in the form of an environment variable ${env.PIPELINE_CONFIG}. examples/dynamic/Jenkinsfile then uses readYaml from the Pipeline Utility Steps plugin to parse the PIPELINE_CONFIG in YAML format. The values are then used to define the behavior and/or provide data for the pipeline. The Pipeline Utility Steps plugin supports multiple config file formats.
Configuring the Config-Driven Pipeline Plugin
In this example, weâve decided to have a Multi-Branch Pipeline look for .myci.yml in any repo and it will utilize https://github.com/justinharringa/jenkinsfiles to look up the pipeline_template which would be specified in .myci.yml. You can see a sample dynamic config file in justinharringa/jenkins-config-driven-pipeline-plugin-example repo which will invoke the dynamic Jenkinsfile in the jenkinsfiles repo. There are different .myci.yml configurations in different branches and PRs within the repo to demonstrate that you can try out different configurations.
As a result of this configuration, tworepositories show up in Jenkins for this organization and youâd notice that each branch and PR with .myci.yml shows up for that particular repository.
We hope that this will be useful for other Jenkins users who would like to share Jenkinsfiles across multiple repositories or organizations. It has served us well for quite some time and it has been a great way to easily maintain pipelines while also adding new features to thousands of pipelines. Please feel free to file any issues in the Jenkins issue tracker or, better yet, send us a pull request!
