526互联

Using JobDSL and Jenkinsfiles to fully automate Jenkins job management

发布时间 2023-08-09 17:26:19作者: lightsong

Using JobDSL and Jenkinsfiles to fully automate Jenkins job management

非常贴切的一个比喻

JobDSL 做Job管理的框架， 负责Job基本参数配置

Pipeline 做Job逻辑内容的容器，决定Job执行逻辑。

By using JobDSL and Jenkinsfiles we can fully automate all aspects of Jenkins job management. The two components play the following roles

  • JobDSL: The skeletal framework of a job that “registers” it in the Jenkins UI but does not contain any core job logic
  • Jenkinsfiles: Contains all the project specific steps to accomplish the project goals such as building or deploying code

Sandwhich DSL goodness

It does not have to be JobDSL vs. Jenkinsfiles but, rather, JobDSL and Jenkinsfiles.

 

JobDSLPermalink

The code statements in JobDSL map directly to what is in the Jenkins UI as shown below

Yummy JobDSL FRED_AI

JenkinsfilesPermalink

The “meat” of the job, that is, the core logic and all logic specific to the project in question is contained in the Jenkinsfile. Quite often this is just called “Jenkinsfile” or “Jenkinsfile.groovy” (if you want to clue in your editor/IDE) but it can be called anything.

Below are some Jenkinsfile snippets showing, roughly, how the correspond to what you might see in the Jenkins UI.

Yummy Jenkinsfiles

 

Jenkinsfile

https://www.jenkins.io/doc/book/pipeline/jenkinsfile/

Jenkins创造了Pipeline类型的Job，

并给其配备了代码的方式来描述Job的逻辑， Jenkinsfile

使得 Job本身 可以被源码控制， 存到git库中，部署的时候再拿出来，部署到Jenkins服务器上。

 

As discussed in the Defining a Pipeline in SCM, a Jenkinsfile is a text file that contains the definition of a Jenkins Pipeline and is checked into source control. Consider the following Pipeline which implements a basic three-stage continuous delivery pipeline.

pipeline {
    agent any

    stages {
        stage('Build') {
            steps {
                echo 'Building..'
            }
        }
        stage('Test') {
            steps {
                echo 'Testing..'
            }
        }
        stage('Deploy') {
            steps {
                echo 'Deploying....'
            }
        }
    }
}

 

 

Job DSL

https://plugins.jenkins.io/job-dsl/

jobDSL，跟jenkinfile类似，也是使用编程的方式来管理Job

Jenkins is a wonderful system for managing builds, and people love using its UI to configure jobs. Unfortunately, as the number of jobs grows, maintaining them becomes tedious, and the paradigm of using a UI falls apart. Additionally, the common pattern in this situation is to copy jobs to create new ones, these "children" have a habit of diverging from their original "template" and consequently it becomes difficult to maintain consistency between these jobs.

The Job DSL plugin attempts to solve this problem by allowing jobs to be defined in a programmatic form in a human readable file. Writing such a file is feasible without being a Jenkins expert as the configuration from the web UI translates intuitively into code.

 

Jenkinsfile更加适合创建复杂的自动化过程。

JobDSL可以创建任意类型的Job，包括Pipeline类型的Job，但是实现负责的逻辑，并不擅长。

The Pipeline plugins support implementing and integrating continuous delivery pipelines via the Pipeline DSL. While it is possible to use Job DSL to create complex pipelines using freestyle jobs in combination with many plugins from the Jenkins ecosystem, creating and maintaining these pipeline, including generating jobs for individual SCM branches and possibly running steps in parallel to improve build performance, poses a significant challenge. Jenkins Pipeline is often the better choice for creating complex automated processes. Job DSL can be used to create Pipeline and Multibranch Pipeline jobs. Do not confuse Job DSL with Pipeline DSL, both have their own syntax and scope of application.

 

JobDSL 教程

https://www.digitalocean.com/community/tutorials/how-to-automate-jenkins-job-configuration-using-job-dsl

 

JobDSL API参考

https://dev.astrotech.io/jenkins/index.html#job-dsl

https://jenkinsci.github.io/job-dsl-plugin/#

 

Jenkinsfile弊端一点

https://www.jenkins.io/doc/book/pipeline/syntax/#parameters

Jenkinsfile也是面向Job的全部内容， 包括Job的一些参数配置，

例如 用户输入界面的参数。

 

此Jenkinsfile内容更新后， 并发发布到Jenkins对应的Job后，

参数的变更并不会立刻生效，需要Job运行一次后，才能生效。

但是Job一旦运行， 其对应的 业务逻辑代码也 执行的一次，

这不是部署工作希望看到的。

 

可以使用一些tricky的方法规避此问题， 例如部署的build带特定的参数， 在stage中使用when语句做条件判断。

解法实在ugly。

 

pipeline {
    agent any
    parameters {
        string(name: 'PERSON', defaultValue: 'Mr Jenkins', description: 'Who should I say hello to?')

        text(name: 'BIOGRAPHY', defaultValue: '', description: 'Enter some information about the person')

        booleanParam(name: 'TOGGLE', defaultValue: true, description: 'Toggle this value')

        choice(name: 'CHOICE', choices: ['One', 'Two', 'Three'], description: 'Pick something')

        password(name: 'PASSWORD', defaultValue: 'SECRET', description: 'Enter a password')
    }
    stages {
        stage('Example') {
            steps {
                echo "Hello ${params.PERSON}"

                echo "Biography: ${params.BIOGRAPHY}"

                echo "Toggle: ${params.TOGGLE}"

                echo "Choice: ${params.CHOICE}"

                echo "Password: ${params.PASSWORD}"
            }
        }
    }
}

 

 

JobDSL+PipelineJob 解法

https://github.com/matschaffer/apm-pipeline-library/blob/0a5e76259ac31b7df048c9920fdc1261556748f5/.ci/jobdsl-generator.groovy

构建一个pipeline， 调用job dsl语句，

运行job dsl定义文件， 生成pipeline job。

这样运行此pipeline，可以做配置修改，并更新pipeline脚本， 但是并不运行pipeline脚本。

pipeline {
  agent {label 'master'}
  environment {
    REPO = 'apm-pipeline-library'
    BASE_DIR = "src/github.com/elastic/${env.REPO}"
    NOTIFY_TO = credentials('notify-to')
    PIPELINE_LOG_LEVEL = 'DEBUG'
    LANG = "C.UTF-8"
    LC_ALL = "C.UTF-8"
    SLACK_CHANNEL = '#observablt-bots'
    GITHUB_CHECK = 'true'
    BRANCH_NAME = "${params.branch_specifier}"
  }
  options {
    timeout(time: 1, unit: 'HOURS')
    buildDiscarder(logRotator(numToKeepStr: '5', artifactNumToKeepStr: '5'))
    timestamps()
    ansiColor('xterm')
    disableResume()
    durabilityHint('PERFORMANCE_OPTIMIZED')
  }
  parameters {
    string(name: 'branch_specifier', defaultValue: 'main', description: 'the Git branch specifier to scan.')
  }
  stages {
    stage('Checkout jobs'){
      steps {
        deleteDir()
        gitCheckout(basedir: "${BASE_DIR}")
      }
    }
    stage('Unit test'){
      when {
        expression {
          return false
        }
      }
      steps {
        dir("${BASE_DIR}/.ci/jobDSL"){
          sh(label: 'Run tests', script: './gradlew clean test --stacktrace')
        }
      }
      post {
        always {
          junit(allowEmptyResults: true,
            testDataPublishers: [
              [$class: 'AttachmentPublisher']
            ],
            testResults:"${BASE_DIR}/.ci/jobDSL/build/test-results/test/TEST-*.xml"
          )
        }
      }
    }
    stage('Generate Jobs') {
      steps {
        jobDsl(
          ignoreExisting: true,
          failOnMissingPlugin: true,
          failOnSeedCollision: true,
          removedConfigFilesAction: 'DELETE',
          removedJobAction: 'DELETE',
          removedViewAction: 'DELETE',
          sandbox: true,
          targets: "${BASE_DIR}/.ci/jobDSL/jobs/apm-ci/folder.groovy",
          unstableOnDeprecation: true
        )
        jobDsl(
          ignoreExisting: true,
          failOnMissingPlugin: true,
          failOnSeedCollision: true,
          removedConfigFilesAction: 'DELETE',
          removedJobAction: 'DELETE',
          removedViewAction: 'DELETE',
          sandbox: true,
          targets: "${BASE_DIR}/.ci/jobDSL/jobs/apm-ci/**/*.groovy",
          unstableOnDeprecation: true
        )
      }
    }
  }
}