526互联

kubesphere 自定义jenkins的agent镜像和maven仓库

发布时间 2023-04-15 15:18:16作者: 蒲公英PGY

kubesphere 自定义jenkins的agent镜像和maven仓库

修改maven仓库为本地

maven的cm配置【ks-devops-agent】在【kubesphere-devops-worker的namespace】下，修改节点data.MavenSetting下的内容为本地maven的配置，例如：

kind: ConfigMap
apiVersion: v1
metadata:
  name: ks-devops-agent
  namespace: kubesphere-devops-worker
  labels:
    app.kubernetes.io/managed-by: Helm
  annotations:
    meta.helm.sh/release-name: devops
    meta.helm.sh/release-namespace: kubesphere-devops-system
data:
  MavenSetting: |
    <?xml version="1.0" encoding="UTF-8"?>

    <settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">

      <localRepository>/var/jenkins_home/.m2/maven-repository</localRepository>
      <servers>
        <server>
          <username>admin</username>
          <password>password</password>
          <id>central</id>
        </server>
        <server>
          <username>admin</username>
          <password>password</password>
          <id>snapshots</id>
        </server>
      </servers>

      <profiles>
        <profile>
          <repositories>
            <repository>
              <snapshots>
                <enabled>false</enabled>
              </snapshots>
              <id>central</id>
              <name>libs-release</name>
              <url>http://127.0.0.1:8081/artifactory/libs-release</url>
            </repository>
            <repository>
              <snapshots />
              <id>snapshots</id>
              <name>libs-snapshot</name>
              <url>http://127.0.0.1:8081/artifactory/libs-snapshot</url>
            </repository>
          </repositories>
          <pluginRepositories>
            <pluginRepository>
              <snapshots>
                <enabled>false</enabled>
              </snapshots>
              <id>central</id>
              <name>libs-release</name>
              <url>http://127.0.0.1:8081/artifactory/libs-release</url>
            </pluginRepository>
            <pluginRepository>
              <snapshots />
              <id>snapshots</id>
              <name>libs-snapshot</name>
              <url>http://127.0.0.1:8081/artifactory/libs-snapshot</url>
            </pluginRepository>
          </pluginRepositories>
          <id>artifactory</id>
        </profile>
        <profile>
            <id>sonar</id>
            <activation>
                <activeByDefault>true</activeByDefault>
            </activation>
            <properties>
                <sonar.host.url>
                  http://127.0.0.1:8000
                </sonar.host.url>
            </properties>
        </profile>
      </profiles>
      <activeProfiles>
        <activeProfile>artifactory</activeProfile>
      </activeProfiles>
    </settings>

自定义agent模板

agent模板cm配置【jenkins-casc-config】在【kubesphere-devops-system】的namespace下，在节点data.jenkins_user.yml下添加自定义的agent模板配置，kubesphere已内置了一些模板(base、nodejs、maven、go、python)等，根据需要选择是否使用。每个模板需要有“jnlp”jenkins的agent来实现与jenkins通信，添加自己的模板例如"cus-maven"：

注意：新增或修改后需要等待2分钟左右才能生效使用

kind: ConfigMap
apiVersion: v1
metadata:
  name: jenkins-casc-config
  namespace: kubesphere-devops-system
  labels:
    app.kubernetes.io/managed-by: Helm
  annotations:
    devops.kubesphere.io/jenkins-config-customized: 'true'
    devops.kubesphere.io/jenkins-config-formula: custom
    meta.helm.sh/release-name: devops
    meta.helm.sh/release-namespace: kubesphere-devops-system
data:
  jenkins.yaml:
      ·············· # 默认配置不修改
  jenkins_user.yaml: |
    jenkins:
      mode: EXCLUSIVE
      numExecutors: 0
      scmCheckoutRetryCount: 2
      disableRememberMe: true

      clouds:
        - kubernetes:
            name: "kubernetes"
            serverUrl: "https://kubernetes.default"
            skipTlsVerify: true
            namespace: "kubesphere-devops-worker"
            credentialsId: "k8s-service-account"
            jenkinsUrl: "http://devops-jenkins.kubesphere-devops-system:80"
            jenkinsTunnel: "devops-jenkins-agent.kubesphere-devops-system:50000"
            containerCapStr: "2"
            connectTimeout: "60"
            readTimeout: "60"
            maxRequestsPerHostStr: "32"
            templates:
              - name: "cus-maven"  # 参照默认的maven配置修改
                namespace: "kubesphere-devops-worker"
                label: "cus-maven"
                nodeUsageMode: "EXCLUSIVE"
                idleMinutes: 0
                containers:
                - name: "cus-maven"
                  image: "myresgter.com/public/maven:v1.1"
                  command: "cat"
                  args: ""
                  ttyEnabled: true
                  privileged: true
                  resourceRequestCpu: "100m"
                  resourceLimitCpu: "4000m"
                  resourceRequestMemory: "100Mi"
                  resourceLimitMemory: "8192Mi"
                - name: "jnlp"
                  image: "jenkins/inbound-agent:4.10-2"
                  args: "^${computer.jnlpmac} ^${computer.name}"
                  resourceRequestCpu: "50m"
                  resourceLimitCpu: "500m"
                  resourceRequestMemory: "400Mi"
                  resourceLimitMemory: "1536Mi"
                workspaceVolume:
                  emptyDirWorkspaceVolume:
                    memory: false
                volumes:   # 根据自己的构建添加需要的存储
                - hostPathVolume:
                    hostPath: "/var/data/jenkins_sonar_cache"
                    mountPath: "/root/.sonar/cache"
                yaml: |
                  spec:
                    affinity:
                      nodeAffinity:
                        preferredDuringSchedulingIgnoredDuringExecution:
                        - weight: 1
                          preference:
                            matchExpressions:
                            - key: node-role.kubernetes.io/worker
                              operator: In
                              values:
                              - ci
                    tolerations:
                    - key: "node.kubernetes.io/ci"
                      operator: "Exists"
                      effect: "NoSchedule"
                    - key: "node.kubernetes.io/ci"
                      operator: "Exists"
                      effect: "PreferNoSchedule"
                    containers:
                    - name: "cus-maven"
                      resources:
                        requests:
                          ephemeral-storage: "1Gi"
                        limits:
                          ephemeral-storage: "10Gi"
                      volumeMounts:
                      - name: config-volume  #挂载maven配置文件
                        mountPath: /opt/apache-maven-3.5.3/conf/settings.xml
                        subPath: settings.xml
                      - name: maven-cache   #挂载maven缓存目录
                        mountPath: "/var/jenkins_home/.m2"
                      - name: dockerfile-storage
                        mountPath: /workspace
                    volumes:
                      - name: dockerfile-storage  # 使用kaniko 在容器中构建镜像需要存储的dockerfile和资源
                        persistentVolumeClaim:
                          claimName: jenkins-kaniko
                      - name: maven-cache
                        persistentVolumeClaim:
                          claimName: maven-cache
                      - name: config-volume
                        configMap:
                          name: ks-devops-agent
                          items:
                          - key: MavenSetting
                            path: settings.xml
                    securityContext:
                      fsGroup: 1000

              - name: "maven"  # 默认的maven配置
                namespace: "kubesphere-devops-worker"
                label: "maven"
                nodeUsageMode: "EXCLUSIVE"
                idleMinutes: 0
                containers:
                - name: "maven"
                  image: "kubesphere/builder-maven:v3.2.0-podman"
                  command: "cat"
                  args: ""
                  ttyEnabled: true
                  privileged: true
                  resourceRequestCpu: "100m"
                  resourceLimitCpu: "4000m"
                  resourceRequestMemory: "100Mi"
                  resourceLimitMemory: "8192Mi"
                - name: "jnlp"
                  image: "jenkins/inbound-agent:4.10-2"
                  args: "^${computer.jnlpmac} ^${computer.name}"
                  resourceRequestCpu: "50m"
                  resourceLimitCpu: "500m"
                  resourceRequestMemory: "400Mi"
                  resourceLimitMemory: "1536Mi"
                workspaceVolume:
                  emptyDirWorkspaceVolume:
                    memory: false
                volumes:
                - hostPathVolume:
                    hostPath: "/var/run/docker.sock"
                    mountPath: "/var/run/docker.sock"
                - hostPathVolume:
                    hostPath: "/var/data/jenkins_maven_cache"
                    mountPath: "/root/.m2"
                - hostPathVolume:
                    hostPath: "/var/data/jenkins_sonar_cache"
                    mountPath: "/root/.sonar/cache"
                yaml: |
                  spec:
                    affinity:
                      nodeAffinity:
                        preferredDuringSchedulingIgnoredDuringExecution:
                        - weight: 1
                          preference:
                            matchExpressions:
                            - key: node-role.kubernetes.io/worker
                              operator: In
                              values:
                              - ci
                    tolerations:
                    - key: "node.kubernetes.io/ci"
                      operator: "Exists"
                      effect: "NoSchedule"
                    - key: "node.kubernetes.io/ci"
                      operator: "Exists"
                      effect: "PreferNoSchedule"
                    containers:
                    - name: "maven"
                      resources:
                        requests:
                          ephemeral-storage: "1Gi"
                        limits:
                          ephemeral-storage: "10Gi"
                      volumeMounts:
                      - name: config-volume
                        mountPath: /opt/apache-maven-3.5.3/conf/settings.xml
                        subPath: settings.xml
                    volumes:
                      - name: config-volume
                        configMap:
                          name: ks-devops-agent
                          items:
                          - key: MavenSetting
                            path: settings.xml
                    securityContext:
                      fsGroup: 1000

              ················· # 默认配置