一般构建的流程如下:
1) 在 GitLab 中创建对应的项目;
2) 配置 Jenkins 集成 Kubernetes 集群,后期 Jenkins 的 Slave 将为在 Kubernetes 中动态创建的 Slave;
3) Jenkins 创建对应的任务(Job),集成该项目的 Git 地址和 Kubernetes 集群;
4) 开发者将代码提交到 GitLab;
5) 如有配置钩子,推送(Push)代码会自动触发 Jenkins 构建,如没有配置钩子,需要手动构建;
6) Jenkins 控制 Kubernetes(使用的是 Kubernetes 插件)创建 Jenkins Slave(Pod 形式);
7) Jenkins Slave 根据流水线(Pipeline)定义的步骤执行构建;
8) 通过 Dockerfile 生成镜像;
9) 将镜像提送(Push)到私有 Harbor(或者其它的镜像仓库);
10) Jenkins 再次控制 Kubernetes 进行最新的镜像部署;
11) 流水线结束删除 Jenkins Slave。
一.创建Jenkinsfile流水线(java)
1.通过blue ocean设计框架
我们可以通过blue ocean设计框架,通过导出的jenkinsfile来添加需要的内容,以节省时间:
2.定义jenkinsfile
在java项目代码仓库添加一个jenkinsfile文件:
注意文件名称为:Jenkinsfile(J是大写)
复制上步骤导出的框架和内容:
pipeline { agent { kubernetes { cloud 'study-kubernetes' slaveConnectTimeout 1200 workspaceVolume hostPathWorkspaceVolume(hostPath: "/opt/workspace", readOnly: false) yaml ''' apiVersion: v1 kind: Pod spec: containers: - args: [\'$(JENKINS_SECRET)\', \'$(JENKINS_NAME)\'] image: 'registry.cn-beijing.aliyuncs.com/citools/jnlp:alpine' name: jnlp imagePullPolicy: IfNotPresent volumeMounts: - mountPath: "/etc/localtime" name: "localtime" readOnly: false - command: - "cat" env: - name: "LANGUAGE" value: "en_US:en" - name: "LC_ALL" value: "en_US.UTF-8" - name: "LANG" value: "en_US.UTF-8" image: "registry.cn-beijing.aliyuncs.com/citools/maven:3.5.3" imagePullPolicy: "IfNotPresent" name: "build" tty: true volumeMounts: - mountPath: "/etc/localtime" name: "localtime" - mountPath: "/root/.m2/" name: "cachedir" readOnly: false - command: - "cat" env: - name: "LANGUAGE" value: "en_US:en" - name: "LC_ALL" value: "en_US.UTF-8" - name: "LANG" value: "en_US.UTF-8" image: "registry.cn-beijing.aliyuncs.com/citools/kubectl:self-1.17" imagePullPolicy: "IfNotPresent" name: "kubectl" tty: true volumeMounts: - mountPath: "/etc/localtime" name: "localtime" readOnly: false - command: - "cat" env: - name: "LANGUAGE" value: "en_US:en" - name: "LC_ALL" value: "en_US.UTF-8" - name: "LANG" value: "en_US.UTF-8" image: "registry.cn-beijing.aliyuncs.com/citools/docker:19.03.9-git" imagePullPolicy: "IfNotPresent" name: "docker" tty: true volumeMounts: - mountPath: "/etc/localtime" name: "localtime" readOnly: false - mountPath: "/var/run/docker.sock" name: "dockersock" readOnly: false restartPolicy: "Never" nodeSelector: build: "true" securityContext: {} volumes: - hostPath: path: "/var/run/docker.sock" name: "dockersock" - hostPath: path: "/usr/share/zoneinfo/Asia/Shanghai" name: "localtime" - name: "cachedir" hostPath: path: "/opt/m2" ''' } } stages { stage('Pulling Code') { parallel { stage('Pulling Code by Jenkins') { when { expression { env.gitlabBranch == null } } steps { git(changelog: true, poll: true, url: 'git@192.168.0.111:root/spring-boot-project.git', branch: "${BRANCH}", credentialsId: 'gitlab key') script { COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim() TAG = BUILD_TAG + '-' + COMMIT_ID println "Current branch is ${BRANCH}, Commit ID is ${COMMIT_ID}, Image TAG is ${TAG}" } } } stage('Pulling Code by trigger') { when { expression { env.gitlabBranch != null } } steps { git(url: 'git@192.168.0.111:root/spring-boot-project.git', branch: env.gitlabBranch, changelog: true,poll: true, credentialsId: 'gitlab key') script { COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim() TAG = BUILD_TAG + '-' + COMMIT_ID println "Current branch is ${env.gitlabBranch}, Commit ID is ${COMMIT_ID}, Image TAG is ${TAG}" } } } } } stage('Building') { steps { container(name: 'build') { sh """ mvn clean install -DskipTests ls target/* """ } } } stage('Docker build for creating image') { environment { HARBOR_USER = credentials('HARBOR_ACCOUNT') } steps { container(name: 'docker') { sh """ echo ${HARBOR_USER_USR} ${HARBOR_USER_PSW} ${TAG} docker build -t ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} . docker login -u ${HARBOR_USER_USR} -p ${HARBOR_USER_PSW} ${HARBOR_ADDRESS} docker push ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} """ } } } stage('Deploying to K8s') { environment { MY_KUBECONFIG = credentials('study-kubernetes') } steps { container(name: 'kubectl'){ sh """ /usr/local/bin/kubectl --kubeconfig $MY_KUBECONFIG set image deploy -l app=${IMAGE_NAME} ${IMAGE_NAME}=${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} -n $NAMESPACE """ } } } } environment { COMMIT_ID = "" HARBOR_ADDRESS = "192.168.0.110" REGISTRY_DIR = "kubernetes" IMAGE_NAME = "spring-boot-project" NAMESPACE = "kubernetes" TAG = "" } parameters { gitParameter(branch: '', branchFilter: 'origin/(.*)', defaultValue: '', description: 'Branch for build and deploy', name: 'BRANCH', quickFilterEnabled: false, selectedValue: 'NONE', sortMode: 'NONE', tagFilter: '*',type: 'PT_BRANCH') } }
3.jenkinsfile详解
(1).顶层agent
首先是顶层的Agent,定义的是Kubernetes 的Pod 作为Jenkins 的Slave:
agent { # 定义使用Kubernetes 作为agent kubernetes { # 选择的云为之前配置的名字 cloud 'kubernetes-study' slaveConnectTimeout 1200 # 将workspace 改成hostPath,因为该Slave 会固定节点创建,如果有存储可用,可以 改成PVC 的模式 workspaceVolume hostPathWorkspaceVolume(hostPath: "/opt/workspace", readOnly: false) yaml ''' apiVersion: v1 kind: Pod spec: containers: # jnlp 容器,和Jenkins 主节点通信 - args: [\'$(JENKINS_SECRET)\', \'$(JENKINS_NAME)\'] image: 'registry.cn-beijing.aliyuncs.com/citools/jnlp:alpine' name: jnlp imagePullPolicy: IfNotPresent volumeMounts: - mountPath: "/etc/localtime" name: "localtime" readOnly: false # build 容器,包含执行构建的命令,比如Java 的需要mvn 构建,就可以用一个maven 的 镜像 - command: - "cat" env: - name: "LANGUAGE" value: "en_US:en" - name: "LC_ALL" value: "en_US.UTF-8" - name: "LANG" value: "en_US.UTF-8" # 使用Maven 镜像,包含mvn 工具。NodeJS 可以用node 的镜像 image: "registry.cn-beijing.aliyuncs.com/citools/maven:3.5.3" imagePullPolicy: "IfNotPresent" # 容器的名字,流水线的stage 可以直接使用该名字 name: "build" tty: true volumeMounts: - mountPath: "/etc/localtime" name: "localtime" # Pod 单独创建了一个缓存的volume,将其挂载到了maven 插件的缓存目录,默认是 /root/.m2 - mountPath: "/root/.m2/" name: "cachedir" readOnly: false # 发版容器,因为最终是发版至Kubernetes 的,所以需要有一个kubectl 命令 - command: - "cat" env: - name: "LANGUAGE" value: "en_US:en" - name: "LC_ALL" value: "en_US.UTF-8" - name: "LANG" value: "en_US.UTF-8" # 镜像的版本可以替换为其它的版本,也可以不进行替换,因为只执行set 命令,所以版本 是兼容的 image: "registry.cn-beijing.aliyuncs.com/citools/kubectl:self-1.17" imagePullPolicy: "IfNotPresent" name: "kubectl" tty: true volumeMounts: - mountPath: "/etc/localtime" name: "localtime" readOnly: false # 用于生成镜像的容器,需要包含docker 命令 - command: - "cat" env: - name: "LANGUAGE" value: "en_US:en" - name: "LC_ALL" value: "en_US.UTF-8" - name: "LANG" value: "en_US.UTF-8" image: "registry.cn-beijing.aliyuncs.com/citools/docker:19.03.9- git" imagePullPolicy: "IfNotPresent" name: "docker" tty: true volumeMounts: - mountPath: "/etc/localtime" name: "localtime" readOnly: false # 由于容器没有启动docker 服务,所以将宿主机的docker 经常挂载至容器即可 - mountPath: "/var/run/docker.sock" name: "dockersock" readOnly: false restartPolicy: "Never" # 固定节点部署 nodeSelector: build: "true" securityContext: {} volumes: # Docker socket volume - hostPath: path: "/var/run/docker.sock" name: "dockersock" - hostPath: path: "/usr/share/zoneinfo/Asia/Shanghai" name: "localtime" # 缓存目录 - name: "cachedir" hostPath: path: "/opt/m2" ''' } 之后看一下Jenkinsfile 最后的环境变量和parameters 的配置: # 定义一些全局的环境变量 environment { COMMIT_ID = "" HARBOR_ADDRESS = "CHANGE_HERE_FOR_YOUR_HARBOR_URL" # Harbor 地址 REGISTRY_DIR = "kubernetes" # Harbor 的项目目录 IMAGE_NAME = "spring-boot-project" # 镜像的名称 NAMESPACE = "kubernetes" # 该应用在Kubernetes 中的命名空间 TAG = "" # 镜像的Tag,在此用BUILD_TAG+COMMIT_ID 组成 } parameters { # 之前讲过一些choice、input 类型的参数,本次使用的是GitParameter 插件 # 该字段会在Jenkins 页面生成一个选择分支的选项 gitParameter(branch: '', branchFilter: 'origin/(.*)', defaultValue: '', description: 'Branch for build and deploy', name: 'BRANCH', quickFilterEnabled: false, selectedValue: 'NONE', sortMode: 'NONE', tagFilter: '*', type: 'PT_BRANCH') }
(2).拉取代码判断
接下来是拉代码的 stage,这个 stage 是一个并行的 stage,因为考虑了该流水线是手动触
发还是触发:
stage('Pulling Code') { parallel { stage('Pulling Code by Jenkins') { when { expression { # 假如env.gitlabBranch 为空,则该流水线为手动触发,那么就会执行该 stage,如果不为空则会执行同级的另外一个stage env.gitlabBranch == null } } steps { # 这里使用的是git 插件拉取代码,BRANCH 变量取自于前面介绍的parameters 配置 # git@xxxxxx:root/spring-boot-project.git 代码地址 # credentialsId: 'gitlab-key',之前创建的拉取代码的key git(changelog: true, poll: true, url: 'git@xxxxxx:root/spring- boot-project.git', branch: "${BRANCH}", credentialsId: 'gitlab-key') script { # 定义一些变量用于生成镜像的Tag # 获取最近一次提交的Commit ID COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 -- pretty=format:'%h'").trim() # 赋值给TAG 变量,后面的docker build 可以取到该TAG 的值 TAG = BUILD_TAG + '-' + COMMIT_ID println "Current branch is ${BRANCH}, Commit ID is ${COMMIT_ID}, Image TAG is ${TAG}" } } } stage('Pulling Code by trigger') { when { expression { # 如果env.gitlabBranch 不为空,说明该流水线是通过webhook 触发,则此 时执行该stage,上述的stage 不再执行。此时BRANCH 变量为空 env.gitlabBranch != null } } steps { # 以下配置和上述一致,只是此时branch: env.gitlabBranch 取的值为 env.gitlabBranch git(url: 'git@xxxxxxxxxxx:root/spring-boot-project.git', branch: env.gitlabBranch, changelog: true, poll: true, credentialsId: 'gitlab-key') script { COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 -- pretty=format:'%h'").trim() TAG = BUILD_TAG + '-' + COMMIT_ID println "Current branch is ${env.gitlabBranch}, Commit ID is ${COMMIT_ID}, Image TAG is ${TAG}" } } } } }
(3).编译代码
代码拉下来后,就可以执行构建命令,由于本次实验是Java 示例,所以需要使用mvn 命令进行构建:
stage('Building') { steps { # 使用Pod 模板里面的build 容器进行构建 container(name: 'build') { sh """ # 编译命令, 需要根据自己项目的实际情况进行修改,可能会不一致 mvn clean install -DskipTests # 构建完成后,一般情况下会在target 目录下生成jar 包 ls target/* """ } } }
(4).生成镜像并推送到镜像仓库
生成编译产物后,需要根据该产物生成对应的镜像,此时可以使用Pod 模板的docker 容器:
stage('Docker build for creating image') { # 首先取出来HARBOR 的账号密码 environment { HARBOR_USER = credentials('HARBOR_ACCOUNT') } steps { # 指定使用docker 容器 container(name: 'docker') { sh """ # 执行build 命令,Dockerfile 会在下一小节创建,也是放在代码仓库,和 Jenkinsfile 同级 docker build -t ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} . # 登录Harbor,HARBOR_USER_USR 和HARBOR_USER_PSW 由上述environment 生 成 docker login -u ${HARBOR_USER_USR} -p ${HARBOR_USER_PSW} ${HARBOR_ADDRESS} # 将镜像推送至镜像仓库 docker push ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} """ } } }
(5).发版到kubernets集群
最后一步就是将该镜像发版至Kubernetes 集群中,此时使用的是包含kubectl 命令的容器:
stage('Deploying to K8s') { # 获取连接Kubernetes 集群证书 environment { MY_KUBECONFIG = credentials('study-k8s-kubeconfig') } steps { # 指定使用kubectl 容器 container(name: 'kubectl'){ sh """ # 直接set 更改Deployment 的镜像即可 /usr/local/bin/kubectl --kubeconfig $MY_KUBECONFIG set image deploy -l app=${IMAGE_NAME} ${IMAGE_NAME}=${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} -n $NAMESPACE """ } } } }
注意事项: 本次发版的命令为/usr/local/bin/kubectl --kubeconfig $MY_KUBECONFIG set image deploy -l app=${IMAGE_NAME} ${IMAGE_NAME}=${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} -n $NAMESPACE。 由于本示例的Deployment 名称、Label、容器名称均和项目名称一致,所以就统一了IMAGE_NAME,如果示例不是按照此规范命名,需要进行更改。
4.定义Dockerfile
在执行流水线过程时,需要将代码的编译产物做成镜像。Dockerfile 主要写的是如何生成公司业务的镜像。而本次示例是Java 项目,只需要把Jar 包放在有Jre 环境的镜像中,然后启动该Jar 包即可:
gitlab→选择项目→新建文件:Dockerfile
# 基础镜像可以按需修改,可以更改为公司自有镜像 FROM registry.cn-beijing.aliyuncs.com/dotbalo/jre:8u211-data # jar包名称改成实际的名称,本示例为spring-cloud-eureka-0.0.1-SNAPSHOT.jar COPY target/spring-cloud-eureka-0.0.1-SNAPSHOT.jar ./ # 启动Jar包 CMD java -jar spring-cloud-eureka-0.0.1-SNAPSHOT.jar
5.定义Kubernetes资源
本示例在 GitLab 创建的 Group 为 kubernetes,可以将其认为是一个项目,同一个项目可以部署至Kubernetes 集群中同一个Namespace 中,本示例为kubernetes 命名空间。由于使用的是私有仓库,因此也需要先配置拉取私有仓库镜像的密钥:
# kubectl create ns kubernetes namespace/kubernetes created # kubectl create secret docker-registry harborkey --docker-server=192.168.0.110 --docker-username=admin --docker-password=admin123 --docker-email=mail@kubeasy.com -n kubernetes secret/harborkey created
配置该应用的Deployment:
--- apiVersion: v1 kind: Service metadata: creationTimestamp: null labels: app: spring-boot-project name: spring-boot-project namespace: kubernetes spec: ports: # 端口按照实际情况进行修改 - name: web port: 8761 protocol: TCP targetPort: 8761 selector: app: spring-boot-project sessionAffinity: None type: ClusterIP status: loadBalancer: {} --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: creationTimestamp: null name: spring-boot-project namespace: kubernetes spec: rules: - host: spring-boot-project.test.com #设置对外访问域名,根据实际需求更改 http: paths: - backend: service: name: spring-boot-project port: number: 8761 path: / pathType: ImplementationSpecific status: loadBalancer: {} --- apiVersion: apps/v1 kind: Deployment metadata: creationTimestamp: null labels: app: spring-boot-project # Deployment 标签,和流水线的set -l 一致 name: spring-boot-project # Deployment 名称 namespace: kubernetes spec: replicas: 1 selector: matchLabels: app: spring-boot-project # Pod 的标签 strategy: rollingUpdate: maxSurge: 1 maxUnavailable: 0 type: RollingUpdate template: metadata: creationTimestamp: null labels: app: spring-boot-project # Pod 的标签 spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - spring-boot-project topologyKey: kubernetes.io/hostname weight: 100 containers: - env: - name: TZ value: Asia/Shanghai - name: LANG value: C.UTF-8 image: nginx # 此处使用的nginx作为原始的镜像,通过Jenkins构建并发版后,变成Java应用的镜像 imagePullPolicy: IfNotPresent lifecycle: {} livenessProbe: failureThreshold: 2 initialDelaySeconds: 30 periodSeconds: 10 successThreshold: 1 tcpSocket: port: 8761 # 端口号和健康检查按照实际情况进行修改 timeoutSeconds: 2 name: spring-boot-project # 容器的名称,需要和流水线set 命令的容器名称一 ports: - containerPort: 8761 # 端口号按照实际情况进行修改 name: web protocol: TCP readinessProbe: failureThreshold: 2 initialDelaySeconds: 30 periodSeconds: 10 successThreshold: 1 tcpSocket: port: 8761 # 端口号和健康检查按照实际情况进行修改 timeoutSeconds: 2 resources: # 资源请求按照实际情况修改 limits: cpu: 994m memory: 1170Mi requests: cpu: 10m memory: 55Mi dnsPolicy: ClusterFirst imagePullSecrets: - name: harborkey # Harbor仓库密钥,需要和上述创建的Secret 一致 restartPolicy: Always securityContext: {} serviceAccountName: default
创建该资源:
# kubectl create -f spring-boot-project.yaml service/spring-boot-project created ingress.networking.k8s.io/spring-boot-project created deployment.apps/spring-boot-project created
6.创建jenkins job任务
单击首页的创建任务(Job)选项(或者是New Item)并配置任务信息:
新建任务→选择流水线:
在新页面,点击Pipeline,输入Git 仓库地址、选择之前配置GitLab Key、分支为master,点击Save 即可:
创建完成后,点击 Build Now(由于 Jenkins 参数由 Jenkinsfile 生成,所以第一次执行流水线会失败):
选择分支,之后点击Build,然后点击Build History 的进度条即可看到构建日志:
构建日志上部分为创建Pod的日志,可以看到Pod为Agent指定的Pod:
可以看到BUILD SUCCESS 说明mvn 编译已经通过:
编译结束后,可以看到制作镜像的日志,并推送到仓库:
Finished:SUCCESS说明该流水线正常结束。接下来可以查看Deployment的镜像:
[root@k8s-master01 ~]# kubectl get deploy -n kubernetes spring-boot-project -oyaml | grep "image" f:imagePullPolicy: {} f:imagePullSecrets: f:image: {} image: 192.168.0.110/kubernetes/spring-boot-project:jenkins-spring-boot-project-7-039cf7e imagePullPolicy: IfNotPresent imagePullSecrets
如果配置了域名,可以通过域名访问(测试域名需要配置hosts):
二.自动化构建Vue/H5前端应用
自动化构建 Vue/H5 应用,其构建方式和自动化构建 Java 基本相同,重点是更改Deployment、Jenkinsfile 和Dockerfile 即可。
前端应用测试项目地址:https://gitee.com/dukuan/vue-project.git
1.定义Jenkinsfile
和上述java项目类似,只需要改部分内容即可:gitlab项目地址、nodejs环境镜像、部分标签。
pipeline { agent { kubernetes { cloud 'study-kubernetes' slaveConnectTimeout 1200 workspaceVolume hostPathWorkspaceVolume(hostPath: "/opt/workspace", readOnly: false) yaml ''' apiVersion: v1 kind: Pod spec: containers: - args: [\'$(JENKINS_SECRET)\', \'$(JENKINS_NAME)\'] image: 'registry.cn-beijing.aliyuncs.com/citools/jnlp:alpine' name: jnlp imagePullPolicy: IfNotPresent volumeMounts: - mountPath: "/etc/localtime" name: "localtime" readOnly: false - command: - "cat" env: - name: "LANGUAGE" value: "en_US:en" - name: "LC_ALL" value: "en_US.UTF-8" - name: "LANG" value: "en_US.UTF-8" image: "registry.cn-beijing.aliyuncs.com/citools/node:lts" # 构建容器改为具有NodeJS 环境的镜像,版本需要和公司项目一致 imagePullPolicy: "IfNotPresent" name: "build" tty: true volumeMounts: - mountPath: "/etc/localtime" name: "localtime" - mountPath: "/root/.m2/" # NodeJS 的缓存目录为node_modules,此处可以 不配置,因为workspace 采用的是hostPath,该目录会被缓存到创建Pod 节点的/opt/目录 name: "cachedir" readOnly: false - command: - "cat" env: - name: "LANGUAGE" value: "en_US:en" - name: "LC_ALL" value: "en_US.UTF-8" - name: "LANG" value: "en_US.UTF-8" image: "registry.cn-beijing.aliyuncs.com/citools/kubectl:self-1.17" imagePullPolicy: "IfNotPresent" name: "kubectl" tty: true volumeMounts: - mountPath: "/etc/localtime" name: "localtime" readOnly: false - command: - "cat" env: - name: "LANGUAGE" value: "en_US:en" - name: "LC_ALL" value: "en_US.UTF-8" - name: "LANG" value: "en_US.UTF-8" image: "registry.cn-beijing.aliyuncs.com/citools/docker:19.03.9-git" imagePullPolicy: "IfNotPresent" name: "docker" tty: true volumeMounts: - mountPath: "/etc/localtime" name: "localtime" readOnly: false - mountPath: "/var/run/docker.sock" name: "dockersock" readOnly: false restartPolicy: "Never" nodeSelector: build: "true" securityContext: {} volumes: - hostPath: path: "/var/run/docker.sock" name: "dockersock" - hostPath: path: "/usr/share/zoneinfo/Asia/Shanghai" name: "localtime" - name: "cachedir" hostPath: path: "/opt/m2" ''' } } stages { stage('Pulling Code') { parallel { stage('Pulling Code by Jenkins') { when { expression { env.gitlabBranch == null } } steps { git(changelog: true, poll: true, url: 'git@192.168.0.111:root/vue-project.git', branch: "${BRANCH}", credentialsId: 'gitlab key') # 此处的地址需要改为自己的Git 地址 script { COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim() TAG = BUILD_TAG + '-' + COMMIT_ID println "Current branch is ${BRANCH}, Commit ID is ${COMMIT_ID}, Image TAG is ${TAG}" } } } stage('Pulling Code by trigger') { when { expression { env.gitlabBranch != null } } steps { git(url: 'git@192.168.0.111:root/vue-project.git', branch: env.gitlabBranch, changelog: true,poll: true, credentialsId: 'gitlab key') # 此处的地址需要改为自己的Git 地址 script { COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim() TAG = BUILD_TAG + '-' + COMMIT_ID println "Current branch is ${env.gitlabBranch}, Commit ID is ${COMMIT_ID}, Image TAG is ${TAG}" } } } } } stage('Building') { steps { container(name: 'build') { sh """ # 此处为NodeJS/Vue 项目的构建命令,根据实际情况进行修改 npm install --registry=https://registry.npm.taobao.org npm run build """ } } } stage('Docker build for creating image') { environment { HARBOR_USER = credentials('HARBOR_ACCOUNT') } steps { container(name: 'docker') { sh """ echo ${HARBOR_USER_USR} ${HARBOR_USER_PSW} ${TAG} docker build -t ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} . docker login -u ${HARBOR_USER_USR} -p ${HARBOR_USER_PSW} ${HARBOR_ADDRESS} docker push ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} """ } } } stage('Deploying to K8s') { environment { MY_KUBECONFIG = credentials('study-kubernetes') } steps { container(name: 'kubectl'){ sh """ /usr/local/bin/kubectl --kubeconfig $MY_KUBECONFIG set image deploy -l app=${IMAGE_NAME} ${IMAGE_NAME}=${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} -n $NAMESPACE """ } } } } environment { COMMIT_ID = "" HARBOR_ADDRESS = "192.168.0.110" REGISTRY_DIR = "kubernetes" IMAGE_NAME = "vue-project" # 项目标签名字为vue-project NAMESPACE = "kubernetes" TAG = "" } parameters { gitParameter(branch: '', branchFilter: 'origin/(.*)', defaultValue: '', description: 'Branch for build and deploy', name: 'BRANCH', quickFilterEnabled: false, selectedValue: 'NONE', sortMode: 'NONE', tagFilter: '*',type: 'PT_BRANCH') } }
2.定义Dockerfile
FROM registry.cn-beijing.aliyuncs.com/dotbalo/nginx:1.15.12 COPY dist/* /usr/share/nginx/html/
3.定义Kubernetes资源
对于Kubernetes 的资源也是类似的,只需要更改资源名称和端口号即可:
--- apiVersion: v1 kind: Service metadata: creationTimestamp: null labels: app: vue-project name: vue-project namespace: kubernetes spec: ports: - name: web port: 80 protocol: TCP targetPort: 80 selector: app: vue-project sessionAffinity: None type: ClusterIP status: loadBalancer: {} --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: creationTimestamp: null name: vue-project namespace: kubernetes spec: rules: - host: vue-project.test.com http: paths: - backend: service: name: vue-project port: number: 80 path: / pathType: ImplementationSpecific --- apiVersion: apps/v1 kind: Deployment metadata: creationTimestamp: null labels: app: vue-project name: vue-project namespace: kubernetes spec: replicas: 1 selector: matchLabels: app: vue-project strategy: rollingUpdate: maxSurge: 1 maxUnavailable: 0 type: RollingUpdate template: metadata: creationTimestamp: null labels: app: vue-project spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - vue-project topologyKey: kubernetes.io/hostname weight: 100 containers: - env: - name: TZ value: Asia/Shanghai - name: LANG value: C.UTF-8 image: nginx imagePullPolicy: IfNotPresent lifecycle: {} livenessProbe: failureThreshold: 2 initialDelaySeconds: 30 periodSeconds: 10 successThreshold: 1 tcpSocket: port: 80 timeoutSeconds: 2 name: vue-project ports: - containerPort: 80 name: web protocol: TCP readinessProbe: failureThreshold: 2 initialDelaySeconds: 30 periodSeconds: 10 successThreshold: 1 tcpSocket: port: 80 timeoutSeconds: 2 resources: limits: cpu: 994m memory: 1170Mi requests: cpu: 10m memory: 55Mi dnsPolicy: ClusterFirst imagePullSecrets: - name: harborkey restartPolicy: Always securityContext: {} serviceAccountName: default
创建该资源:
# kubectl create -f vue-project.yaml service/vue-project created ingress.networking.k8s.io/vue-project created deployment.apps/vue-project created
4.创建Jenkins Job
创建Jenkins Job 和之前并无太大区别。首先创建Pipeline 类型的Job,名称为vue-project:
三.自动化构建Golang项目
上述演示了Java 和前端应用的自动化,接下来演示一下对于Golang 的自动化构建,本次示例的代码地址:https://gitee.com/dukuan/go-project.git。
1.定义Jenkinsfile
本次示例的Jenkinsfile 和之前的也无太大区别,需要的更改的位置是构建容器的镜像、缓存目录、Git 地址和项目名称
pipeline { agent { kubernetes { cloud 'study-kubernetes' slaveConnectTimeout 1200 workspaceVolume hostPathWorkspaceVolume(hostPath: "/opt/workspace", readOnly: false) yaml ''' apiVersion: v1 kind: Pod spec: containers: - args: [\'$(JENKINS_SECRET)\', \'$(JENKINS_NAME)\'] image: 'registry.cn-beijing.aliyuncs.com/citools/jnlp:alpine' name: jnlp imagePullPolicy: IfNotPresent volumeMounts: - mountPath: "/etc/localtime" name: "localtime" readOnly: false - command: - "cat" env: - name: "LANGUAGE" value: "en_US:en" - name: "LC_ALL" value: "en_US.UTF-8" - name: "LANG" value: "en_US.UTF-8" image: "registry.cn-beijing.aliyuncs.com/citools/golang:1.15" # 构建镜像改为golang,需要根据实际情况更改版本 imagePullPolicy: "IfNotPresent" name: "build" tty: true volumeMounts: - mountPath: "/etc/localtime" name: "localtime" - mountPath: "/go/pkg" #增加挂载缓存目录 name: "cachedir" readOnly: false - command: - "cat" env: - name: "LANGUAGE" value: "en_US:en" - name: "LC_ALL" value: "en_US.UTF-8" - name: "LANG" value: "en_US.UTF-8" image: "registry.cn-beijing.aliyuncs.com/citools/kubectl:self-1.17" imagePullPolicy: "IfNotPresent" name: "kubectl" tty: true volumeMounts: - mountPath: "/etc/localtime" name: "localtime" readOnly: false - command: - "cat" env: - name: "LANGUAGE" value: "en_US:en" - name: "LC_ALL" value: "en_US.UTF-8" - name: "LANG" value: "en_US.UTF-8" image: "registry.cn-beijing.aliyuncs.com/citools/docker:19.03.9-git" imagePullPolicy: "IfNotPresent" name: "docker" tty: true volumeMounts: - mountPath: "/etc/localtime" name: "localtime" readOnly: false - mountPath: "/var/run/docker.sock" name: "dockersock" readOnly: false restartPolicy: "Never" nodeSelector: build: "true" securityContext: {} volumes: - hostPath: path: "/var/run/docker.sock" name: "dockersock" - hostPath: path: "/usr/share/zoneinfo/Asia/Shanghai" name: "localtime" - name: "cachedir" # 缓存目录保存的位置 hostPath: path: "/opt/gopkg" ''' } } stages { stage('Pulling Code') { parallel { stage('Pulling Code by Jenkins') { when { expression { env.gitlabBranch == null } } steps { git(changelog: true, poll: true, url: 'git@192.168.0.111:root/go-project.git', branch: "${BRANCH}", credentialsId: 'gitlab key') script { COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim() TAG = BUILD_TAG + '-' + COMMIT_ID println "Current branch is ${BRANCH}, Commit ID is ${COMMIT_ID}, Image TAG is ${TAG}" } } } stage('Pulling Code by trigger') { when { expression { env.gitlabBranch != null } } steps { git(url: 'git@192.168.0.111:root/go-project.git', branch: env.gitlabBranch, changelog: true,poll: true, credentialsId: 'gitlab key') script { COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim() TAG = BUILD_TAG + '-' + COMMIT_ID println "Current branch is ${env.gitlabBranch}, Commit ID is ${COMMIT_ID}, Image TAG is ${TAG}" } } } } } stage('Building') { steps { container(name: 'build') { sh """ # 构建命令 export GO111MODULE=on go env -w GOPROXY=https://goproxy.cn,direct go build """ } } } stage('Docker build for creating image') { environment { HARBOR_USER = credentials('HARBOR_ACCOUNT') } steps { container(name: 'docker') { sh """ echo ${HARBOR_USER_USR} ${HARBOR_USER_PSW} ${TAG} docker build -t ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} . docker login -u ${HARBOR_USER_USR} -p ${HARBOR_USER_PSW} ${HARBOR_ADDRESS} docker push ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} """ } } } stage('Deploying to K8s') { environment { MY_KUBECONFIG = credentials('study-kubernetes') } steps { container(name: 'kubectl'){ sh """ /usr/local/bin/kubectl --kubeconfig $MY_KUBECONFIG set image deploy -l app=${IMAGE_NAME} ${IMAGE_NAME}=${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} -n $NAMESPACE """ } } } } environment { COMMIT_ID = "" HARBOR_ADDRESS = "192.168.0.110" REGISTRY_DIR = "kubernetes" IMAGE_NAME = "go-project" NAMESPACE = "kubernetes" TAG = "" } parameters { gitParameter(branch: '', branchFilter: 'origin/(.*)', defaultValue: '', description: 'Branch for build and deploy', name: 'BRANCH', quickFilterEnabled: false, selectedValue: 'NONE', sortMode: 'NONE', tagFilter: '*',type: 'PT_BRANCH') } }
2.定义Dockerfile
FROM registry.cn-beijing.aliyuncs.com/dotbalo/alpine-glibc:alpine-3.9 # 如果定义了单独的配置文件,可能需要拷贝到镜像中 COPY conf/ ./conf # 包名按照实际情况进行修改 COPY ./go-project ./ ENTRYPOINT [ "./go-project"] # 启动该应用
3.定义Kubernetes资源
--- apiVersion: v1 kind: Service metadata: creationTimestamp: null labels: app: go-project name: go-project namespace: kubernetes spec: ports: - name: web port: 8080 protocol: TCP targetPort: 8080 selector: app: go-project sessionAffinity: None type: ClusterIP status: loadBalancer: {} --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: creationTimestamp: null name: go-project namespace: kubernetes spec: rules: - host: go-project.test.com http: paths: - backend: service: name: go-project port: number: 8080 path: / pathType: ImplementationSpecific --- apiVersion: apps/v1 kind: Deployment metadata: creationTimestamp: null labels: app: go-project name: go-project namespace: kubernetes spec: replicas: 1 selector: matchLabels: app: go-project strategy: rollingUpdate: maxSurge: 1 maxUnavailable: 0 type: RollingUpdate template: metadata: creationTimestamp: null labels: app: go-project spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - go-project topologyKey: kubernetes.io/hostname weight: 100 containers: - env: - name: TZ value: Asia/Shanghai - name: LANG value: C.UTF-8 image: nginx imagePullPolicy: IfNotPresent lifecycle: {} livenessProbe: failureThreshold: 2 initialDelaySeconds: 30 periodSeconds: 10 successThreshold: 1 tcpSocket: port: 8080 timeoutSeconds: 2 name: go-project ports: - containerPort: 8080 name: web protocol: TCP readinessProbe: failureThreshold: 2 initialDelaySeconds: 30 periodSeconds: 10 successThreshold: 1 tcpSocket: port: 8080 timeoutSeconds: 2 resources: limits: cpu: 994m memory: 1170Mi requests: cpu: 10m memory: 55Mi dnsPolicy: ClusterFirst imagePullSecrets: - name: harborkey restartPolicy: Always securityContext: {} serviceAccountName: default
4.创建Jenkins Job
直接创建即可:
# kubectl create -f go-project.yaml service/go-project created ingress.networking.k8s.io/go-project created deployment.apps/go-project created
构建省略:
四.自动触发构建
之前的构建都是采用手动选择分支进行构建的,实际使用时,项目可能有很多,如果都是手动触发可能比较消耗人力。所以推荐可以按需配置自动触发,即提交代码后自动触发Jenkins 进行构建任务。
本次用Java 项目进行演示。首先找到Java 项目的Job,点击Configure:
之后选择构建触发器(Build Triggers),勾选Build when a change…,记录webhook URL:
之后点击高级(Advanced):
选择Allow all branches,如果不想任何分支都可以触发该流水线,可以选择Filter 进行条件匹配。之后点击Generate 生成Secret token,最后点击Save 即可。
接下来配置GitLab,首先点击Menu→Admin,选择Settings→Network,之后允许访问外部的请求:
保存后,找到Java 项目,点击Settings→WebHooks,在新页面输入Webhook 地址和token:
之后下方会添加一个新的Project Hooks,可以点击Test 进行Push 测试:
点击后,即可在Jenkins 页面看到任务被触发:
五.UAT和生产环境发版
UAT和生产环境最好跟测试环境分开,使用测试构建好的镜像经行发版:
1.如果生产环境用同样构建方式,那么生产构建的新镜像有可能和上次的不一样,新镜像没有经过测试。
2.使用测试环境测试过的镜像可以省去编译构建的时间。
流程简略:
创建一个新的Job,名字为go-project-uat,类型Pipeline:点击页面的This Project is parameterized(参数化构建):
选择参数类型为Image Tag Parameter(需要安装Image Tag 插件),之后定义Name 为变量的名称,Iamge Name 为Harbor 的目录和镜像名称:
点击Advance,输入仓库的地址,注意如果配置了证书,需要配置https:
之后可以先点击Save,然后测试能否获取到进行Tag。保存后点击Build with Parameters
之后可以直接添加Pipeline脚本,也可以在gitlab中用jenkinsfile来创建:
完整代码:
pipeline { agent { kubernetes { cloud 'study-kubernetes' slaveConnectTimeout 1200 yaml ''' apiVersion: v1 kind: Pod spec: containers: - args: [\'$(JENKINS_SECRET)\', \'$(JENKINS_NAME)\'] image: 'registry.cn-beijing.aliyuncs.com/citools/jnlp:alpine' name: jnlp imagePullPolicy: IfNotPresent - command: - "cat" env: - name: "LANGUAGE" value: "en_US:en" - name: "LC_ALL" value: "en_US.UTF-8" - name: "LANG" value: "en_US.UTF-8" image: "registry.cn-beijing.aliyuncs.com/citools/kubectl:self-1.17" imagePullPolicy: "IfNotPresent" name: "kubectl" tty: true restartPolicy: "Never" ''' } } stages { stage('Deploy') { environment { MY_KUBECONFIG = credentials('study-kubernetes') } steps { container(name: 'kubectl'){ sh """ echo ${IMAGE_TAG} kubectl --kubeconfig=${MY_KUBECONFIG} set image deployment -l app=${IMAGE_NAME} ${IMAGE_NAME}=${HARBOR_ADDRESS}/${IMAGE_TAG} -n ${NAMESPACE} kubectl --kubeconfig=${MY_KUBECONFIG} get po -l app=${IMAGE_NAME} -n ${NAMESPACE} -w """ } } } } environment { HARBOR_ADDRESS = "HARBOR_ADDRESS" NAMESPACE = "kubernetes" IMAGE_NAME = "spring-boot-project" TAG = "" } }
保存后,选择一个镜像,点击Build,即可看到是直接将镜像版本更新至Kubernetes,并无构建过程,可以省下很多时间。该流水
线也可以选择之前的版本进行回滚操作。