| 知乎专栏 |
https://gitlab.com/gitlab-examples
Gitlab(仓库) -> Gitlab Runner（持续集成/部署） -> Remote host（远程部署主机）
为远程服务器创建 www 用户，我们将使用该用户远程部署，远程启动程序。
[root@netkiller ~]# groupadd -g 80 www
[root@netkiller ~]# adduser -o --uid 80 --gid 80 -G wheel -c "Web Application" www
[root@netkiller ~]# id www
uid=80(www) gid=80(www) groups=80(www),10(wheel)
[root@netkiller ~]# PASSWORD=$(cat /dev/urandom | tr -dc [:alnum:] | head -c 32)
[root@netkiller ~]# echo www:${PASSWORD} | chpasswd
[root@netkiller ~]# echo "www password: ${PASSWORD}"
www password: 0Uz1heY9v9KJyRKbvTi0VlAzfEoFW9GH
mkdir -p /opt/netkiller.cn/www.netkiller.cn chown www:www -R /opt/netkiller.cn
进入项目设置界面，点击 Settings，再点击 CI / CD
 |
点击 Expand 按钮 展开 Runners
 |
这时可以看到 Set up a specific Runner manually, 后面会用到 http://192.168.1.96/ 和 zASzWwffenos6Jbbfsgu
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh" | sudo bash
dnf install gitlab-runner
cp /etc/gitlab-runner/config.toml{,.original}
systemctl enable gitlab-runner
使用 SSH 登录 Gitlab runner 服务器，运行 gitlab-runner register
[root@localhost ~]# gitlab-runner register
Runtime platform arch=amd64 os=linux pid=92925 revision=ac2a293c version=11.11.2
Running in system-mode.
Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
http://192.168.1.96/
Please enter the gitlab-ci token for this runner:
zASzWwffenos6Jbbfsgu
Please enter the gitlab-ci description for this runner:
[localhost.localdomain]:
Please enter the gitlab-ci tags for this runner (comma separated):
Registering runner... succeeded runner=zASzWwff
Please enter the executor: docker, docker-ssh, shell, ssh, docker-ssh+machine, parallels, virtualbox, docker+machine, kubernetes:
shell
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
返回 gitlab 查看注册状态
 |
[root@gitlab ~]# gitlab-runner register
Runtime platform arch=amd64 os=linux pid=1020084 revision=c1edb478 version=14.0.1
Running in system-mode.
Enter the GitLab instance URL (for example, https://gitlab.com/):
http://git.netkiller.cn/
Enter the registration token:
DyKdKyaJaq5KN-irgNGz
Enter a description for the runner:
[gitlab]:
Enter tags for the runner (comma-separated):
Registering runner... succeeded runner=DyKdKyaJ
Enter an executor: parallels, virtualbox, docker+machine, custom, docker, docker-ssh, shell, ssh, docker-ssh+machine, kubernetes:
shell
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
/etc/gitlab-runner/config.toml 配置文件
[root@gitlab ~]# cat /etc/gitlab-runner/config.toml
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "gitlab"
url = "http://git.netkiller.cn/"
token = "kVkzjDM74xZUN-aKbdPp"
executor = "shell"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
持续集成和部署运行在 gitlab-runner 用户下，切换到 gitlab-runner 用户
[root@gitlab ~]# su - gitlab-runner Last login: Mon Jul 19 19:01:37 CST 2021
生成 SSH 证书
[gitlab-runner@gitlab ~]$ ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/home/gitlab-runner/.ssh/id_rsa): Created directory '/home/gitlab-runner/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/gitlab-runner/.ssh/id_rsa. Your public key has been saved in /home/gitlab-runner/.ssh/id_rsa.pub. The key fingerprint is: SHA256:l90LYBeSF9l9JHXJUHeO+IyvscCziz4C8vFNpJoKEjo gitlab-runner@gitlab The key's randomart image is: +---[RSA 3072]----+ | ..o===B| | ..oo.**| | o.o . o| | .. = = | |. oS o + + | |... o . .o o . | |E o * o + . o | |.o + o o. + + | | .. oo.o.o | +----[SHA256]-----+ [gitlab-runner@gitlab ~]$
正常情况下，当我们链接一个 SSH 主机，会让我们输入 yes 确认继续链接。
[gitlab-runner@gitlab ~]$ ssh www@192.168.40.10 The authenticity of host '192.168.40.10 (192.168.40.10)' can't be established. ECDSA key fingerprint is SHA256:xmFF266MPdXhnlAljS+QWhQsw6jOw1sOwQXRr/PHi2w. Are you sure you want to continue connecting (yes/no/[fingerprint])?
配置 SSH
[gitlab-runner@gitlab ~]$ cat > ~/.ssh/config <<'EOF' Host * ServerAliveInterval=30 StrictHostKeyChecking no UserKnownHostsFile=/dev/null EOF chmod 600 -R ~/.ssh/config
授权远程执行 Shell
[gitlab-runner@gitlab ~]$ ssh-copy-id www@www.netkiller.cn
在构建过程中，我们需要备份数据库/同步数据库，下面安装了一些所需的工具
[root@localhost ~]# dnf install -y mysql
设置数据库备份账号和密码，这里偷懒使用了 root 账号，生产环境请创建专用的备份账号。
[root@localhost ~]# su - gitlab-runner Last login: Wed Sep 1 19:17:48 CST 2021 [gitlab-runner@localhost ~]$ vim ~/.my.cnf [gitlab-runner@localhost ~]$ cat ~/.my.cnf [mysql] user=root password=test [mysqldump] user=root password=test
测试数据库是否畅通
[gitlab-runner@localhost ~]$ mysql -h mysql.netkiller.cn Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 37602 Server version: 8.0.21 Source distribution Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql>
JRE：java-11-openjdk
JDK：java-11-openjdk-devel
[root@gitlab ~]# dnf install -y java-11-openjdk java-11-openjdk-devel [root@gitlab ~]# dnf install -y maven
修改 Maven 镜像路
[root@gitlab ~]# vim /etc/maven/settings.xml
<mirrors>
<mirror>
<id>aliyun</id>
<name>aliyun maven</name>
<url>http://maven.aliyun.com/nexus/content/groups/public/</url>
<mirrorOf>central</mirrorOf>
</mirror>
</mirrors>
如果需要安装最新版本 maven 使用下面脚本。
#!/bin/bash cd /usr/local/src/ wget https://mirrors.bfsu.edu.cn/apache/maven/maven-3/3.8.2/binaries/apache-maven-3.8.2-bin.tar.gz tar zxf apache-maven-3.8.2-bin.tar.gz mv apache-maven-3.8.2 /srv/ rm -f /srv/apache-maven ln -s /srv/apache-maven-3.8.2 /srv/apache-maven alternatives --install /usr/local/bin/mvn apache-maven-3.8.2 /srv/apache-maven-3.8.2/bin/mvn 0
[root@localhost src]# mvn -v Apache Maven 3.8.2 (ea98e05a04480131370aa0c110b8c54cf726c06f) Maven home: /srv/apache-maven-3.8.2 Java version: 17-ea, vendor: Red Hat, Inc., runtime: /usr/lib/jvm/java-17-openjdk-17.0.0.0.26-0.2.ea.el8.x86_64 Default locale: en_US, platform encoding: ANSI_X3.4-1968 OS name: "linux", version: "4.18.0-338.el8.x86_64", arch: "amd64", family: "unix"
apache-maven-3.8.2 配置
[root@localhost ~]# vim /srv/apache-maven/conf/settings.xml
<mirrors>
<!-- mirror
| Specifies a repository mirror site to use instead of a given repository. The repository that
| this mirror serves has an ID that matches the mirrorOf element of this mirror. IDs are used
| for inheritance and direct lookup purposes, and must be unique across the set of mirrors.
|
<mirror>
<id>mirrorId</id>
<mirrorOf>repositoryId</mirrorOf>
<name>Human Readable Name for this Mirror.</name>
<url>http://my.repository.com/repo/path</url>
</mirror>
-->
<mirror>
<id>maven-default-http-blocker</id>
<mirrorOf>external:http:*</mirrorOf>
<name>Pseudo repository to mirror external repositories initially using HTTP.</name>
<url>http://0.0.0.0/</url>
<blocked>true</blocked>
</mirror>
</mirrors>
apache-maven-3.8.2 默认会阻止其他镜像，需要会去掉 maven-default-http-blocker 配置
切换到 gitlab-runner 用户，随便运行一下 mvn 命令，这样就会产生 ～/.m2 文件夹
[root@gitlab ~]# su - gitlab-runner [gitlab-runner@gitlab ~]$ mvn -v
mvnd 是一个实验产品，用于替代 maven 编译速度比较快
cd /usr/local/src wget https://github.com/apache/maven-mvnd/releases/download/0.7.1/mvnd-0.7.1-linux-amd64.zip unzip mvnd-0.7.1-linux-amd64.zip mv mvnd-0.7.1-linux-amd64 /srv/mvnd-0.7.1 ln -s /srv/mvnd-0.7.1 /srv/mvnd alternatives --remove mvnd /usr/local/bin/mvnd alternatives --install /usr/local/bin/mvnd mvnd-0.7.1 /srv/mvnd-0.7.1/bin/mvnd 0
修改配置文件 mvnd.properties 制定 JAVA_HOME
[root@localhost cloud.netkiller.cn]# grep java.home /srv/mvnd/conf/mvnd.properties java.home=/usr/lib/jvm/java
[root@netkiller ~]# dnf install -y nodejs
安装 cnpm
[root@netkiller ~]# npm config set registry https://registry.npm.taobao.org [root@netkiller ~]# npm config get registry https://registry.npm.taobao.org/ [root@netkiller ~]# npm install -g cnpm
yarn
[root@netkiller ~]# curl -sL https://dl.yarnpkg.com/rpm/yarn.repo -o /etc/yum.repos.d/yarn.repo [root@netkiller ~]# dnf install -y yarn
yarn config set registry https://registry.npm.taobao.org
pm2 进程管理
[root@netkiller ~]# npm install -g pm2
设置 pm2 启动开启
[root@netkiller ~]# pm2 startup [root@netkiller ~]# pm2 save --force [root@netkiller ~]# systemctl enable pm2-root [root@netkiller ~]# systemctl start pm2-root [root@netkiller ~]# systemctl status pm2-root
[gitlab-runner@gitlab api.netkiller.cn]$ ssh www@192.168.40.10 "sudo ls" Warning: Permanently added '192.168.40.10' (ECDSA) to the list of known hosts. sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper
解决方案一
ssh -t www@www.netkiller.cn "echo <yourpassword> |sudo -S <yourcommand>"
解决方案二
cat > /etc/sudoers.d/www <<-EOF www ALL=(ALL) NOPASSWD: ALL EOF
tags 是给 Gitlab Runner 打个标签，我的用法是多次注册，例如 shell 执行器的标签是 shell, Docker 执行器的标签是 docker，这样便可以在.gitlab-ci.yml文件中来选择使用那个执行器来触发操作。
下面是 Shell 执行器
[root@localhost ~]# gitlab-runner register
Runtime platform arch=amd64 os=linux pid=268363 revision=58ba2b95 version=14.2.0
Running in system-mode.
Enter the GitLab instance URL (for example, https://gitlab.com/):
http://git.netkiller.cn/
Enter the registration token:
k_SsvMQV397gAMaP_q1v
Enter a description for the runner:
[localhost.localdomain]: development
Enter tags for the runner (comma-separated):
shell
Registering runner... succeeded runner=k_SsvMQV
Enter an executor: docker, docker-ssh, virtualbox, docker-ssh+machine, kubernetes, custom, parallels, shell, ssh, docker+machine:
shell
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
下面是 Docker 执行器
[root@localhost ~]# gitlab-runner register
Runtime platform arch=amd64 os=linux pid=268397 revision=58ba2b95 version=14.2.0
Running in system-mode.
Enter the GitLab instance URL (for example, https://gitlab.com/):
http://git.netkiller.cn/
Enter the registration token:
k_SsvMQV397gAMaP_q1v
Enter a description for the runner:
[localhost.localdomain]: development
Enter tags for the runner (comma-separated):
docker
Registering runner... succeeded runner=k_SsvMQV
Enter an executor: custom, docker-ssh, parallels, shell, ssh, docker-ssh+machine, docker, virtualbox, docker+machine, kubernetes:
docker
Enter the default Docker image (for example, ruby:2.6):
maven:latest
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
注册后的效果
 |
[root@localhost ~]# cat /etc/gitlab-runner/config.toml
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "development"
url = "http://git.netkiller.cn/"
token = "EztTBypKRW5ibtC5rs2h"
executor = "shell"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[[runners]]
name = "development"
url = "http://git.netkiller.cn/"
token = "51948sQbQsXGV-RxFMty"
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "maven:latest"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
gitlab-runner 用户需要 访问 /var/run/docker.sock 所以需要将 gitlab-runner 用户加入到 docker 组中。
[root@gitlab ~]# ll /var/run/docker.sock srw-rw---- 1 root docker 0 Nov 25 17:04 /var/run/docker.sock [root@gitlab ~]# id gitlab-runner uid=989(gitlab-runner) gid=984(gitlab-runner) groups=984(gitlab-runner) [root@gitlab ~]# usermod -aG docker gitlab-runner [root@gitlab ~]# id gitlab-runner uid=989(gitlab-runner) gid=984(gitlab-runner) groups=984(gitlab-runner),991(docker)
注册 Docker 执行器
[root@localhost ~]# gitlab-runner register
Runtime platform arch=amd64 os=linux pid=268397 revision=58ba2b95 version=14.2.0
Running in system-mode.
Enter the GitLab instance URL (for example, https://gitlab.com/):
http://git.netkiller.cn/
Enter the registration token:
k_SsvMQV397gAMaP_q1v
Enter a description for the runner:
[localhost.localdomain]: development
Enter tags for the runner (comma-separated):
docker
Registering runner... succeeded runner=k_SsvMQV
Enter an executor: custom, docker-ssh, parallels, shell, ssh, docker-ssh+machine, docker, virtualbox, docker+machine, kubernetes:
docker
Enter the default Docker image (for example, ruby:2.6):
maven:latest
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
配置缓存
[root@localhost ~]# cat /etc/gitlab-runner/config.toml
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "development"
url = "http://192.168.30.5/"
token = "EztTBypKRW5ibtC5rs2h"
executor = "shell"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[[runners]]
name = "development"
url = "http://192.168.30.5/"
token = "GP-ozvd6uw2nDxyRohZ-"
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "maven:latest"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache","/root/.m2"]
pull_policy = ["never"]
shm_size = 0
volumes = ["/cache","/root/.m2"] 将 Maven 仓库缓存
.gitlab-ci.yaml 编排脚本
cache:
untracked: true
stages:
- build
- test
- deploy
build-job:
image: maven:3.8.2-openjdk-17
stage: build
tags:
- docker
script:
- mvn clean package -Dmaven.test.skip=true
- ls target/*.jar
artifacts:
name: "$CI_PROJECT_NAME"
paths:
- target/*.jar
test-job:
image: maven:3.8.2-openjdk-17
stage: test
variables:
GIT_STRATEGY: none
tags:
- docker
script:
- mvn test
deploy-job:
stage: deploy
variables:
GIT_STRATEGY: none
HOST: 192.168.30.14
DOCKER_HOST: unix:///var/run/docker.sock mvn clean install docker:build
environment:
name: development
url: https://api.netkiller.cn
only:
- development
tags:
- shell
before_script:
- mvn docker:build -DpushImage
# - mvn docker:push
- rm -rf *.sql.gz
- mysqldump -hmysql.netkiller.cn test | gzip > test.$(date -u +%Y-%m-%d.%H:%M:%S).sql.gz
artifacts:
name: "$CI_PROJECT_NAME"
paths:
- ./*.sql.gz
script:
- scp src/main/docker/docker-compose.yaml www@$HOST:/opt/netkiller.cn/api.netkiller.cn/
- ssh www@$HOST "sudo docker-compose -f /opt/netkiller.cn/api.netkiller.cn/docker-compose.yaml up"
- ssh www@$HOST "sudo docker-compose -f /opt/netkiller.cn/api.netkiller.cn/docker-compose.yaml restart"
[root@agent-5 ~]# gitlab-runner register
Runtime platform arch=amd64 os=linux pid=1259091 revision=43b2dc3d version=15.4.0
Running in system-mode.
Enter the GitLab instance URL (for example, https://gitlab.com/):
https://gitlab.netkiller.cn/
Enter the registration token:
GR1348941WLrZvRebkiCocQgdGFwC
Enter a description for the runner:
[agent-5]: Kubernetes executor
Enter tags for the runner (comma-separated):
kubernetes
Enter optional maintenance note for the runner:
Registering runner... succeeded runner=GR1348941WLrZvReb
Enter an executor: docker-ssh, shell, docker+machine, docker-ssh+machine, kubernetes, custom, docker, parallels, ssh, virtualbox:
kubernetes
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
Configuration (with the authentication token) was saved in "/etc/gitlab-runner/config.toml"
/etc/gitlab-runner/config.toml
[root@agent-5 ~]# cat /etc/gitlab-runner/config.toml
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "Kubernetes executor"
url = "https://gitlab.netkiller.cn/"
id = 3
token = "5J6amB15rYWie_zGscFC"
token_obtained_at = 2022-10-19T07:16:07Z
token_expires_at = 0001-01-01T00:00:00Z
executor = "kubernetes"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.kubernetes]
host = ""
bearer_token_overwrite_allowed = false
image = ""
namespace = ""
namespace_overwrite_allowed = ""
pod_labels_overwrite_allowed = ""
service_account_overwrite_allowed = ""
pod_annotations_overwrite_allowed = ""
[runners.kubernetes.affinity]
[runners.kubernetes.pod_security_context]
[runners.kubernetes.init_permissions_container_security_context]
[runners.kubernetes.init_permissions_container_security_context.capabilities]
[runners.kubernetes.build_container_security_context]
[runners.kubernetes.build_container_security_context.capabilities]
[runners.kubernetes.helper_container_security_context]
[runners.kubernetes.helper_container_security_context.capabilities]
[runners.kubernetes.service_container_security_context]
[runners.kubernetes.service_container_security_context.capabilities]
[runners.kubernetes.volumes]
[runners.kubernetes.dns_config]
[runners.kubernetes.container_lifecycle]
将CA证书复制给 gitlab-runner
mkdir /etc/ssl/kubernetes [root@master ~]# scp /var/lib/rancher/k3s/server/tls/server-ca.crt root@agent-5:/etc/ssl/kubernetes/ca.crt
或者用下面命令查看 CA 证书，然后保存到 /etc/ssl/kubernetes/ca.crt 文件
[gitlab-runner@agent-5 ~]$ kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 -d
-----BEGIN CERTIFICATE-----
MIIBeDCCAR2gAwIBAgIBADAKBggqhkjOPQQDAjAjMSEwHwYDVQQDDBhrM3Mtc2Vy
dmVyLWNhQDE2NjI2MTYwOTMwHhcNMjIwOTA4MDU0ODEzWhcNMzIwOTA1MDU0ODEz
WjAjMSEwHwYDVQQDDBhrM3Mtc2VydmVyLWNhQDE2NjI2MTYwOTMwWTATBgcqhkjO
PQIBBggqhkjOPQMBBwNCAARGT8u7K3jFNKGid7qMWSYUMuv+kYQzvk5RQHFyEXA6
zNnGd0PBpDvsKpZGjkIwJnla0v98nFzsK6hp9eEDIVw3o0IwQDAOBgNVHQ8BAf8E
BAMCAqQwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQU7IRDLJoNv7bF8wkkJ4yQ
FA6gTBowCgYIKoZIzj0EAwIDSQAwRgIhAKMiz13pxq+IIXZfZT5R+Lh+pDoX2H1u
AskoLxoAutCPAiEA4ubxiK1DqjatxGb1/ovMLd4pfcPeAvg1AIokwhFhueU=
-----END CERTIFICATE-----
[root@master ~]# kubectl create serviceaccount secrets
serviceaccount/secrets created
[root@master ~]# kubectl create token secrets
[root@master ~]# cat role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: secrets
namespace: default
rules:
- apiGroups: ["*"]
resources: ["pods"]
verbs: ["list", "get", "watch", "create", "delete"]
- apiGroups: ["*"]
resources: ["pods/exec"]
verbs: ["create"]
- apiGroups: ["*"]
resources: ["pods/log"]
verbs: ["get"]
- apiGroups: ["*"]
resources: ["pods/attach"]
verbs: ["list", "get", "create", "delete", "update"]
- apiGroups: ["*"]
resources: ["secrets"]
verbs: ["list", "get", "create", "delete", "update"]
- apiGroups: ["*"]
resources: ["configmaps"]
verbs: ["list", "get", "create", "delete", "update"]
[root@master ~]# kubectl create rolebinding gitlab-runner-binding --role=secrets --serviceaccount=default:secrets
[[runners]]
name = "Kubernetes executor"
url = "https://gitlab.netkiller.cn/"
id = 3
token = "5J6amB15rYWie_zGscFC"
token_obtained_at = 2022-10-19T07:16:07Z
token_expires_at = 0001-01-01T00:00:00Z
executor = "kubernetes"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.kubernetes]
host = "https://k8s.netkiller.cn:6443"
ca_file = "/etc/ssl/kubernetes/ca.crt"
tls_verify = true
bearer_token_overwrite_allowed = true
image = ""
namespace = ""
namespace_overwrite_allowed = ""
pod_labels_overwrite_allowed = ""
service_account_overwrite_allowed = ""
pod_annotations_overwrite_allowed = ""
[runners.kubernetes.affinity]
[runners.kubernetes.pod_security_context]
[runners.kubernetes.init_permissions_container_security_context]
[runners.kubernetes.init_permissions_container_security_context.capabilities]
[runners.kubernetes.build_container_security_context]
[runners.kubernetes.build_container_security_context.capabilities]
[runners.kubernetes.helper_container_security_context]
[runners.kubernetes.helper_container_security_context.capabilities]
[runners.kubernetes.service_container_security_context]
[runners.kubernetes.service_container_security_context.capabilities]
[runners.kubernetes.volumes]
[runners.kubernetes.dns_config]
[runners.kubernetes.container_lifecycle]
.gitlab-ci.yml
cache:
# untracked: true
paths:
- target/
#variables:
# KUBERNETES_SERVICE_ACCOUNT: secrets
# KUBERNETES_BEARER_TOKEN: eyJhbGciOiJSUzI1NiIsImtpZCI6IktCOHRvYlZOLXFPRmEyb1JWdlQxSzBvN0tvZF9HNFBGRnlraDR5UU1jakkifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiLCJrM3MiXSwiZXhwIjoxNjY2MTc0Mzk1LCJpYXQiOjE2NjYxNzA3OTUsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJkZWZhdWx0Iiwic2VydmljZWFjY291bnQiOnsibmFtZSI6InNlY3JldHMiLCJ1aWQiOiIxMTM5NjVlMy1iZGVkLTQ5NGEtOGMyNS0zYjU3OTFmMTIzZjEifX0sIm5iZiI6MTY2NjE3MDc5NSwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmRlZmF1bHQ6c2VjcmV0cyJ9.KvTpp7vplWIBWZFKYK-zPFk0NJhoiMHHiE-0Bj3qmFvHaxF2D3A8YrsEfQxSIJ1p8J5IZSPYiAX_BqoNi4-fziFsJIIbza0bcPj-RWLoBY2Nz0gyzw93r2to71WS90OSxryWRMMmqr8rCFO0ewtdOPzC8-PQ1uSIblJ3gM2EbN7b9VfHJssrog7UMWcT0GuiNM27AzwxYmEvkioeTmQaCzLNUGpyFnu1wg0e7mHzHMxPwSwMiOUFEE7SKKFpyZT8ZLc8ZgEfZKMXK2FwCTpoktBr_h7u-2zpNK4x9Dwl2aqkMBNZL-QVNpaXXnA0K20PAVjK5-x7IkiELFFq_CSezg
stages:
- build
# - deploy
build-job:
stage: build
image: maven:latest
tags:
# - docker
- kubernetes
# before_script:
# after_script:
script:
- mvn -T 1C -Dmaven.test.skip=true package
artifacts:
name: "$CI_PROJECT_NAME"
paths:
- target/*.jar
[root@master ~]# kubectl create namespace gitlab
namespace/gitlab-runner created
[root@master ~]# kubectl create serviceaccount gitlab-runner -n gitlab
serviceaccount/secrets created
[root@master ~]# kubectl create token gitlab-runner -n gitlab
[root@master ~]# cat role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: gitlab-runner
namespace: gitlab
rules:
- apiGroups: ["*"]
resources: ["pods"]
verbs: ["list", "get", "watch", "create", "delete"]
- apiGroups: ["*"]
resources: ["pods/exec"]
verbs: ["create"]
- apiGroups: ["*"]
resources: ["pods/log"]
verbs: ["get"]
- apiGroups: ["*"]
resources: ["pods/attach"]
verbs: ["list", "get", "create", "delete", "update"]
- apiGroups: ["*"]
resources: ["secrets"]
verbs: ["list", "get", "create", "delete", "update"]
- apiGroups: ["*"]
resources: ["configmaps"]
verbs: ["list", "get", "create", "delete", "update"]
[root@master ~]# kubectl apply -f role.yaml
role.rbac.authorization.k8s.io/gitlab created
[root@master ~]# kubectl create rolebinding gitlab-runner --namespace=gitlab --role=gitlab-runner --serviceaccount=gitlab:gitlab-runner
kubectl create -n gitlab -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
name: gitlab-runner-token
annotations:
kubernetes.io/service-account.name: gitlab-runner
type: kubernetes.io/service-account-token
EOF
kubectl get secret gitlab-runner-token -o jsonpath='{.data.token}' -n gitlab | base64 -d
账号和Token创建完毕之后，使用下面命令检查
kubectl get namespace gitlab
kubectl get serviceaccounts gitlab-runner -n gitlab
kubectl get role gitlab-runner -n gitlab
kubectl get rolebinding gitlab-runner -n gitlab
kubectl get secrets gitlab-runner-token -n gitlab
kubectl get secret gitlab-runner-token -o jsonpath='{.data.token}' -n gitlab | base64 -d
创建 PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: driver.longhorn.io
volume.kubernetes.io/storage-provisioner: driver.longhorn.io
creationTimestamp: "2022-10-19T12:46:33Z"
finalizers:
- kubernetes.io/pvc-protection
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:spec:
f:accessModes: {}
f:resources:
f:requests:
.: {}
f:storage: {}
f:storageClassName: {}
f:volumeMode: {}
manager: kube-explorer
operation: Update
time: "2022-10-19T12:46:33Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:pv.kubernetes.io/bind-completed: {}
f:pv.kubernetes.io/bound-by-controller: {}
f:volume.beta.kubernetes.io/storage-provisioner: {}
f:volume.kubernetes.io/storage-provisioner: {}
f:spec:
f:volumeName: {}
manager: k3s
operation: Update
time: "2022-10-19T12:46:35Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:status:
f:accessModes: {}
f:capacity:
.: {}
f:storage: {}
f:phase: {}
manager: k3s
operation: Update
subresource: status
time: "2022-10-19T12:46:35Z"
name: maven
namespace: gitlab
resourceVersion: "4846862"
uid: ab0dc609-ff07-438e-b537-7e6182cec008
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: longhorn
volumeMode: Filesystem
volumeName: pvc-ab0dc609-ff07-438e-b537-7e6182cec008
status:
accessModes:
- ReadWriteOnce
capacity:
storage: 10Gi
phase: Bound
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: driver.longhorn.io
volume.kubernetes.io/storage-provisioner: driver.longhorn.io
creationTimestamp: "2022-10-19T12:48:39Z"
finalizers:
- kubernetes.io/pvc-protection
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:spec:
f:accessModes: {}
f:resources:
f:requests:
.: {}
f:storage: {}
f:storageClassName: {}
f:volumeMode: {}
manager: kube-explorer
operation: Update
time: "2022-10-19T12:48:39Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:pv.kubernetes.io/bind-completed: {}
f:pv.kubernetes.io/bound-by-controller: {}
f:volume.beta.kubernetes.io/storage-provisioner: {}
f:volume.kubernetes.io/storage-provisioner: {}
f:spec:
f:volumeName: {}
manager: k3s
operation: Update
time: "2022-10-19T12:48:42Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:status:
f:accessModes: {}
f:capacity:
.: {}
f:storage: {}
f:phase: {}
manager: k3s
operation: Update
subresource: status
time: "2022-10-19T12:48:42Z"
name: builds
namespace: gitlab
resourceVersion: "4847301"
uid: 09f9a7b0-558a-4142-b9b3-e1318aff223a
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: longhorn-storage
volumeMode: Filesystem
volumeName: pvc-09f9a7b0-558a-4142-b9b3-e1318aff223a
配置 Gitlab Runner
[[runners]]
name = "Kubernetes executor"
url = "https://gitlab.netkiller.cn/"
id = 3
token = "5J6amB15rYWie_zGscFC"
token_obtained_at = 2022-10-19T07:16:07Z
token_expires_at = 0001-01-01T00:00:00Z
executor = "kubernetes"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.kubernetes]
host = "https://172.18.200.5:6443"
#cert_file = "/etc/ssl/kubernetes/api.crt"
#key_file = "/etc/ssl/kubernetes/api.key"
ca_file = "/etc/ssl/kubernetes/ca.crt"
tls_verify = true
bearer_token = "eyJhbGciOiJSUzI1NiIsImtpZCI6IktCOHRvYlZOLXFPRmEyb1JWdlQxSzBvN0tvZF9HNFBGRnlraDR5UU1jakkifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiLCJrM3MiXSwiZXhwIjoxNjY2MTg1NDc5LCJpYXQiOjE2NjYxODE4NzksImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJnaXRsYWIiLCJzZXJ2aWNlYWNjb3VudCI6eyJuYW1lIjoiZ2l0bGFiLXJ1bm5lciIsInVpZCI6IjdjYjhmNjM3LTc5YjQtNDc5Yi1hZjAzLTRhOGQ2ZDliM2YzOSJ9fSwibmJmIjoxNjY2MTgxODc5LCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6Z2l0bGFiOmdpdGxhYi1ydW5uZXIifQ.ExhowiFt9k-vFwaxohs_KGF_JmzPIMnUyoOzENN9g250taWu3jwtRasixIckYhO5JavpzZRU1XTdhRmbEQ9hehlY5q6kXtGCZ4AImh-q8gj7lglBLdWvZLMcmaqu3o4pIjb7Cw0qP25y-F79IeDtUPZeLU2b_ocTIM5GPd5g6MxilTzw84GJ_hVevWVtdpTrRg48b64S6crAWpkUgMSVwyqwyjsTkGuEcn_V32fzO9ro7EjumuWxVmOp5TA0mOXl4L8W1C6ZYVT_qGKsMJJ_AS3uZkKqcxT6FmSS0Ta4vjgQsVN1l6OjQNmrIWFBjRzNCiXPhlk5JzR0ZID2J20UQQ"
bearer_token_overwrite_allowed = true
privileged = true
image = ""
namespace = "gitlab"
namespace_overwrite_allowed = ""
pod_labels_overwrite_allowed = ""
service_account = "gitlab-runner"
service_account_overwrite_allowed = ""
pod_annotations_overwrite_allowed = ""
[runners.kubernetes.affinity]
[runners.kubernetes.pod_security_context]
[runners.kubernetes.init_permissions_container_security_context]
[runners.kubernetes.init_permissions_container_security_context.capabilities]
[runners.kubernetes.build_container_security_context]
[runners.kubernetes.build_container_security_context.capabilities]
[runners.kubernetes.helper_container_security_context]
[runners.kubernetes.helper_container_security_context.capabilities]
[runners.kubernetes.service_container_security_context]
[runners.kubernetes.service_container_security_context.capabilities]
[runners.kubernetes.volumes]
[[runners.kubernetes.volumes.host_path]]
name = "cache"
mount_path = "/cache"
host_path = "/tmp/cache"
[[runners.kubernetes.volumes.pvc]]
name = "maven"
mount_path = "/root/.m2"
[[runners.kubernetes.volumes.pvc]]
name = "builds"
mount_path = "/builds"
[runners.kubernetes.dns_config]
[runners.kubernetes.container_lifecycle]
/etc/gitlab-runner/config.toml
bearer_token_overwrite_allowed = false 改为 bearer_token_overwrite_allowed = true
.gitlab-ci.yml
variables: KUBERNETES_SERVICE_ACCOUNT: gitlab KUBERNETES_BEARER_TOKEN: eyJhbGciOiJSUzI1NiIsImtpZCI6IktCOHRvYlZOLXFPRmEyb1JWdlQxSzBvN0tvZF9HNFBGRnlraDR5UU1jakkifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiLCJrM3MiXSwiZXhwIjoxNjY2MTg1MTE3LCJpYXQiOjE2NjYxODE1MTcsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJnaXRsYWItcnVubmVyIiwic2VydmljZWFjY291bnQiOnsibmFtZSI6ImdpdGxhYiIsInVpZCI6Ijc3MGE4ODk1LTcwNjAtNGUwYi04MTc0LTVkNmMzNDUzYTQxOSJ9fSwibmJmIjoxNjY2MTgxNTE3LCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6Z2l0bGFiLXJ1bm5lcjpnaXRsYWIifQ.PO96hsJSG7-h2EJhDoUc6xWCPzDp9AwoZ1-CrsM-GUN73ft7ge9prdarq0of-rzgAP-wn-07r6vIxdcT2G0ZHl-GsElyxaAm5Poz1WX2bQLuLvcpoFVQyGFH1Mkkno5MDCfi2CPfNYUuEBl5vQ_jDooE5dUnildLmbEv7ooxLYeWAPnEd5HmOyXKjC2FiBqKQ88oxkDbmq3Hwbxm-XmTma7T3NqXxST_m7qE6tb2n0RsG3o1lJGEDftdf9bF1eEfAykaa077tzHNNZtQvK87LK69XZVUbOVun_G98-rLL7afSridgP6mhia6CPeful7xvedJ8l4g8V-Ku8qixKb5Yg
[root@agent-5 ~]# cat /etc/gitlab-runner/config.toml
concurrent = 5
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "Kubernetes local cluster"
url = "https://gitlab.netkiller.cn/"
id = 41
token = "y1QnvNhSwMYVX2-z3x4E"
token_obtained_at = 2022-10-20T00:19:08Z
token_expires_at = 0001-01-01T00:00:00Z
executor = "kubernetes"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.kubernetes]
host = "https://172.18.200.5:6443"
ca_file = "/etc/ssl/kubernetes/ca.crt"
bearer_token_overwrite_allowed = true
bearer_token = "eyJhbGciOiJSUzI1NiIsImtpZCI6IktCOHRvYlZOLXFPRmEyb1JWdlQxSzBvN0tvZF9HNFBGRnlraDR5UU1jakkifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiLCJrM3MiXSwiZXhwIjoxNjY2MjMwNjc5LCJpYXQiOjE2NjYyMjcwNzksImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJnaXRsYWIiLCJzZXJ2aWNlYWNjb3VudCI6eyJuYW1lIjoiZ2l0bGFiLXJ1bm5lciIsInVpZCI6IjdjYjhmNjM3LTc5YjQtNDc5Yi1hZjAzLTRhOGQ2ZDliM2YzOSJ9fSwibmJmIjoxNjY2MjI3MDc5LCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6Z2l0bGFiOmdpdGxhYi1ydW5uZXIifQ.e6RvBIUCUlZ4ciVKRQ-i2kEBBTXaPq876L-EzA5NGNWxXmfkuyx_IpGrrzmTMPV9prR3XLQfmyT1OoEgkO8riMLtJL1yPZNiaTo0zhAfuWvvXP3gOnJsFRdcx2PsvA1dxuuhtXNp1Y2BAboU8jV10-OK6WU40i3CKb9OTQpN-BRAlwXylQ-55ZIRT7J3ghpCLfM6BplwZOYfCP6XbHJfgAKD2wy5sl8Ni1XIpsLUOp20TlBaG22k2admcWiRavxyjp68EHMwq3izI5_4qU9hFcYBOsUY-PBC27nGw9ICH0sIRuq4Xs0GQ8oXOzS5Dbja5uKKVq6-bWV3onV13AjcxQ"
image = ""
namespace = "gitlab"
namespace_overwrite_allowed = ""
pull_policy = ["if-not-present"]
image_pull_secrets = ["registry"]
pod_labels_overwrite_allowed = ""
service_account = "gitlab-runner"
service_account_overwrite_allowed = ""
pod_annotations_overwrite_allowed = ""
[runners.kubernetes.affinity]
[runners.kubernetes.pod_security_context]
[runners.kubernetes.init_permissions_container_security_context]
[runners.kubernetes.init_permissions_container_security_context.capabilities]
[runners.kubernetes.build_container_security_context]
[runners.kubernetes.build_container_security_context.capabilities]
[runners.kubernetes.helper_container_security_context]
[runners.kubernetes.helper_container_security_context.capabilities]
[runners.kubernetes.service_container_security_context]
[runners.kubernetes.service_container_security_context.capabilities]
[runners.kubernetes.volumes]
[[runners.kubernetes.volumes.host_path]]
name = "docker"
mount_path = "/var/run/docker.sock"
host_path = "/var/run/docker.sock"
[[runners.kubernetes.volumes.pvc]]
name = "maven"
mount_path = "/root/.m2"
[[runners.kubernetes.volumes.pvc]]
name = "builds"
mount_path = "/builds"
[[runners.kubernetes.volumes.pvc]]
name = "cache"
mount_path = "/cache"
[runners.kubernetes.dns_config]
[runners.kubernetes.container_lifecycle]
[[runners]]
name = "Docker"
url = "https://gitlab.netkiller.cn/"
id = 42
token = "Y7Df44aY8YrRwASUXWE5"
token_obtained_at = 2022-10-20T02:40:25Z
token_expires_at = 0001-01-01T00:00:00Z
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "docker:latest"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
[[runners]]
name = "Shell"
url = "https://gitlab.netkiller.cn/"
id = 43
token = "s2tuoKTrj1s1iv_mfYh5"
token_obtained_at = 2022-10-20T03:18:08Z
token_expires_at = 0001-01-01T00:00:00Z
executor = "shell"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
.gitlab-ci.yml
stages:
- build
- docker
- deploy
variables:
DOCKER_REGISTRY: registry.netkiller.cn
IMAGE: $DOCKER_REGISTRY/$CI_COMMIT_BRANCH/$CI_PROJECT_NAME:$CI_COMMIT_SHORT_SHA-$CI_PIPELINE_ID
KUBERNETES_BEARER_TOKEN: eyJhbGciOiJSUzI1NiIsImtpZCI6IktCOHRvYlZOLXFPRmEyb1JWdlQxSzBvN0tvZF9HNFBGRnlraDR5UU1jakkifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJnaXRsYWIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoiZ2l0bGFiLXJ1bm5lci10b2tlbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJnaXRsYWItcnVubmVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiN2NiOGY2MzctNzliNC00NzliLWFmMDMtNGE4ZDZkOWIzZjM5Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmdpdGxhYjpnaXRsYWItcnVubmVyIn0.pU4-8D4szeL8iud1SvesdN7nV7L3GLaNsa2UbsxkGQ4SDGN85zKTXJl6MtqDsuJB9HBUlOTMnyEa0gCbgHOJlR3fd2HcegitrRLeybvUuotniiLpCPO7vAO-oS5Fej7oUFBXqZJYIx-xMbFoyt3rnGs273c_yE8avI8EGdEPNhOWRgF_GZBYstvwiEjO2IUDWbutzCTtGloPvJ5Ur0s7drLJkCQvT2nod5tSSnY5R0lpNyD2FodkFR28KU1EgFoHUnH_ERtUAS5qObIETWSwm5SmCnd2Ogjh70DDxmIHSU-saFU0zSqPpZ1oX9hgO9YMkcJXPHOEnqIVEagZ5CSf2w
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- target/
build-job:
stage: build
image: registry.netkiller.cn/common/maven:latest
script:
- mvn clean package -Dautoconfig.skip=true -Dmaven.test.skip=true -Dmaven.test.failure.ignore=true
after_script:
- md5sum target/*.jar
only:
- dev
- test
tags:
- kubernetes
artifacts:
name: "$CI_PROJECT_NAME"
paths:
- target/*.jar
build-docker:
stage: docker
image: docker:latest
before_script:
- echo "$CI_REGISTRY_PASSWORD" | docker login $DOCKER_REGISTRY --username $CI_REGISTRY_USER --password-stdin
after_script:
- docker images | grep $CI_PROJECT_NAME
script:
- docker build -t $IMAGE -f Dockerfile .
- docker push $IMAGE
only:
- dev
- test
tags:
- kubernetes
deploy-job:
stage: deploy
variables:
GIT_STRATEGY: none
before_script:
- kubectl -n test get pod | grep $CI_PROJECT_NAME
script:
- kubectl set image deployment/${CI_PROJECT_NAME} ${CI_PROJECT_NAME}=${IMAGE} -n ${CI_COMMIT_BRANCH}
after_script:
- kubectl -n test get pod | grep $CI_PROJECT_NAME
only:
- dev
- test
tags:
- shell
environment:
name: $CI_COMMIT_BRANCH
url: $CI_COMMIT_BRANCH.netkiller.cn/api/monitor/health
自建 Maven 仓库，需要配置 settings.xml，这时就需要制作一个 Maven 镜像，同事有 COPY 命令把 settings.xml 文件复制到 /usr/share/maven/conf/settings.xml 目录
[root@netkiller jdk11]# ls
Dockerfile build.sh settings.xml
[root@netkiller jdk11]# cat Dockerfile
FROM maven:3.8.6-openjdk-11
COPY settings.xml /root/.m2/settings.xml
COPY settings.xml /usr/share/maven/conf/settings.xml
[root@netkiller jdk11]# cat build.sh
docker build -t "registry.netkiller.cn/share/maven:3.8.6-openjdk-11" .
docker push registry.netkiller.cn/share/maven:3.8.6-openjdk-11
JaCoCo Java Code Coverage Library https://www.jacoco.org/jacoco/index.html
pom.xml 中必须有单元测试依赖
<dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <scope>test</scope> </dependency>
不能跳过单元测试
<plugin> <artifactId>maven-surefire-plugin</artifactId> <configuration> <skip>false</skip> </configuration> </plugin>
添加 JaCoCo 插件
<plugin> <groupId>org.jacoco</groupId> <artifactId>jacoco-maven-plugin</artifactId> <executions> <execution> <goals> <goal>prepare-agent</goal> </goals> </execution> <execution> <id>report</id> <phase>test</phase> <goals> <goal>report</goal> </goals> </execution> </executions> </plugin>
最后运行 mvn test 调试一下，输入类似下面
[INFO] ------------------------< cn.netkiller:config >------------------------- [INFO] Building config 0.0.1-SNAPSHOT [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- jacoco-maven-plugin:0.8.7:prepare-agent (default) @ config --- [INFO] argLine set to -javaagent:/Users/neo/.m2/repository/org/jacoco/org.jacoco.agent/0.8.7/org.jacoco.agent-0.8.7-runtime.jar=destfile=/Users/neo/workspace/microservice/config/target/jacoco.exec [INFO] [INFO] --- maven-resources-plugin:3.2.0:resources (default-resources) @ config --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Using 'UTF-8' encoding to copy filtered properties files. [INFO] Copying 1 resource [INFO] Copying 6 resources [INFO] [INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ config --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-resources-plugin:3.2.0:testResources (default-testResources) @ config --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Using 'UTF-8' encoding to copy filtered properties files. [INFO] Copying 1 resource [INFO] [INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ config --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ config --- [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] [INFO] Results: [INFO] [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] [INFO] --- jacoco-maven-plugin:0.8.7:report (report) @ config --- [INFO] Loading execution data file /Users/neo/workspace/microservice/config/target/jacoco.exec [INFO] Analyzed bundle 'config' with 1 classes [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 3.335 s [INFO] Finished at: 2021-10-22T15:52:36+08:00 [INFO] ------------------------------------------------------------------------
配置持续集成流水线 .gitlab-ci.yml 文件
cache:
untracked: true
stages:
- build
- test
- deploy
test-job:
stage: test
variables:
GIT_STRATEGY: none
only:
- tags
- development
- testing
script:
- mvn test
after_script:
- lrsync 'zito-admin/target/site/*' www@report.netkiller.cn:/opt/netkiller.cn/report.netkiller.cn
- wechat -t 1 代码覆盖率报告 http://report.netkiller.cn/jacoco/index.html
Maven 项目 parent 文件
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>cn.netkiller</groupId>
<artifactId>test</artifactId>
<version>0.0.1${project.branch}${project.phase}</version>
<packaging>pom</packaging>
<name>test</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>18</maven.compiler.source>
<maven.compiler.target>${maven.compiler.source}</maven.compiler.target>
<selenium.version>4.8.1</selenium.version>
<project.phase>-SNAPSHOT</project.phase>
<project.branch></project.branch>
</properties>
<distributionManagement>
<repository>
<id>repository</id>
<name>Release repository</name>
<url>https://maven.netkiller.cn/repository/repository/</url>
</repository>
<snapshotRepository>
<id>snapshots</id>
<name>Snapshots repository</name>
<url>https://maven.netkiller.cn/repository/snapshots/</url>
</snapshotRepository>
</distributionManagement>
<modules>
<module>common</module>
<module>selenium</module>
</modules>
</project>
模块
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<packaging>jar</packaging>
<parent>
<groupId>cn.netkiller</groupId>
<artifactId>test</artifactId>
<version>0.0.1-SNAPSHOT</version>
</parent>
<artifactId>common</artifactId>
<!-- <version>${project.parent.version}${project.branch}</version>-->
<properties>
<maven.compiler.source>18</maven.compiler.source>
<maven.compiler.target>18</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
</project>
构建快照版本 common-0.0.1-SNAPSHOT.jar
neo@macbook-pro-neo ~/w/test> mvn clean package deploy [INFO] ------------------------------------------------------------------------ [INFO] Reactor Build Order: [INFO] [INFO] test [pom] [INFO] common [jar] [INFO] selenium [jar] [INFO] [INFO] -------------------------< cn.netkiller:test >-------------------------- [INFO] Building test 0.0.1-SNAPSHOT [1/3] [INFO] --------------------------------[ pom ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ test --- [INFO] [INFO] ------------------------< cn.netkiller:common >------------------------- [INFO] Building common 0.0.1-SNAPSHOT [2/3] [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ common --- [INFO] Deleting /Users/neo/workspace/test/common/target [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ common --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 0 resource [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ common --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 1 source file to /Users/neo/workspace/test/common/target/classes [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ common --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /Users/neo/workspace/test/common/src/test/resources [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ common --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ common --- [INFO] No tests to run. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ common --- [INFO] Building jar: /Users/neo/workspace/test/common/target/common-0.0.1-SNAPSHOT.jar [INFO] [INFO] -----------------------< cn.netkiller:selenium >------------------------ [INFO] Building selenium 0.0.1-SNAPSHOT [3/3] [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ selenium --- [INFO] Deleting /Users/neo/workspace/test/selenium/target [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ selenium --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /Users/neo/workspace/test/selenium/src/main/resources [INFO] [INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ selenium --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 8 source files to /Users/neo/workspace/test/selenium/target/classes [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ selenium --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /Users/neo/workspace/test/selenium/src/test/resources [INFO] [INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ selenium --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 2 source files to /Users/neo/workspace/test/selenium/target/test-classes [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ selenium --- [INFO] [INFO] --- maven-jar-plugin:3.1.1:jar (default-jar) @ selenium --- [INFO] Building jar: /Users/neo/workspace/test/selenium/target/selenium-0.0.1-SNAPSHOT.jar [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary for test 0.0.1-SNAPSHOT: [INFO] [INFO] test ............................................... SUCCESS [ 0.125 s] [INFO] common ............................................. SUCCESS [ 1.422 s] [INFO] selenium ........................................... SUCCESS [ 1.457 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 3.158 s [INFO] Finished at: 2023-04-05T18:32:45+08:00 [INFO] ------------------------------------------------------------------------
构建开发环境快照版本 common-0.0.1-dev-SNAPSHOT.jar
neo@macbook-pro-neo ~/w/t/common> mvn clean package -Dproject.branch=-dev [INFO] Scanning for projects... [INFO] [INFO] ------------------------< cn.netkiller:common >------------------------- [INFO] Building common 0.0.1-dev-SNAPSHOT [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ common --- [INFO] Deleting /Users/neo/workspace/test/common/target [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ common --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 0 resource [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ common --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 1 source file to /Users/neo/workspace/test/common/target/classes [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ common --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /Users/neo/workspace/test/common/src/test/resources [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ common --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ common --- [INFO] No tests to run. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ common --- [INFO] Building jar: /Users/neo/workspace/test/common/target/common-0.0.1-dev-SNAPSHOT.jar [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 1.384 s [INFO] Finished at: 2023-04-05T18:34:50+08:00 [INFO] ------------------------------------------------------------------------
构建RELEASE版本 common-0.0.1.jar
neo@macbook-pro-neo ~/w/t/common> mvn clean package -Dproject.phase= [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary for test 0.0.1: [INFO] [INFO] common ............................................. SUCCESS [ 6.828 s] [INFO] selenium ........................................... SUCCESS [ 6.351 s] [INFO] test ............................................... SUCCESS [ 3.274 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 16.631 s [INFO] Finished at: 2023-04-05T16:16:28+08:00 [INFO] ------------------------------------------------------------------------
构建带有 “-RELEASE” 后缀的版本 common-0.0.1-RELEASE.jar
neo@macbook-pro-neo ~/w/t/common> mvn clean package -Dproject.phase=-RELEASE [INFO] Scanning for projects... [INFO] [INFO] ------------------------< cn.netkiller:common >------------------------- [INFO] Building common 0.0.1-RELEASE [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ common --- [INFO] Deleting /Users/neo/workspace/test/common/target [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ common --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 0 resource [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ common --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 1 source file to /Users/neo/workspace/test/common/target/classes [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ common --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /Users/neo/workspace/test/common/src/test/resources [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ common --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ common --- [INFO] No tests to run. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ common --- [INFO] Building jar: /Users/neo/workspace/test/common/target/common-0.0.1-RELEASE.jar [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 1.656 s [INFO] Finished at: 2023-04-05T18:36:30+08:00 [INFO] ------------------------------------------------------------------------
首先说说什么是数据库结构，什么事版本控制。
数据库结构是指数据库表结构，数据库定义语言导出的DDL语句。主要由CREATE TABLE, DROP TABLE等等构成。
再来说说什么事版本控制，如果你从事开发工作应该会很容易理解，版本控制就是记录每一次变化，可以随时查看历史记录，并可回撤到指定版本。
软件开发过程中需要常常对数据库结构作调整，这是无法避免的，甚至很多想起启动后，需求还不明确，开发人员只能按照所理解需求创建表。需求往往会发生变化，一旦变化，代码需要修改，表结构也避免不了。 我们常常刚改好数据库结构，需求部门有发来通知，不用修改了，维持原有设计。甚至是过了几周再次回撤。
所以我们要将数据库结构的变化进行版本控制，通常的做法是DBA人工管理，但我觉完全可以自动化的工作，没有必要浪费人力资源，且自动化不会犯错更稳定，仅仅需要人工定期查看工作状态即可。
首先下载脚本 https://github.com/oscm/shell/blob/master/backup/backup.mysql.struct.sh
wget https://raw.githubusercontent.com/oscm/shell/master/backup/backup.mysql.struct.sh mv backup.mysql.struct.sh /usr/local/bin chmod +x /usr/local/bin/backup.mysql.struct
创建备份用户
CREATE USER 'backup'@'localhost' IDENTIFIED BY 'chen'; GRANT SELECT, LOCK TABLES ON *.* TO 'backup'@'localhost'; FLUSH PRIVILEGES; SHOW GRANTS FOR 'backup'@'localhost';
配置脚本
BACKUP_HOST="localhost" 数据库主机 BACKUP_USER="backup" 备份用户 BACKUP_PASS="chen" 备份密码 BACKUP_DBNAME="neo netkiller" 版本控制那些数据库，多个数据库使用空格分隔 BACKUP_DIR=~/backup 数据库结构放在那里 GIT=git@gitlab.netkiller.cn:netkiller.cn/db.netkiller.cn.git
初始化仓库
# /usr/local/bin/backup.mysql.struct init Initialized empty Git repository in /www/database/struct/.git/
# /usr/local/bin/backup.mysql.struct
Usage: /usr/local/bin/backup.mysql.struct {init|start|stop|status|restart}
开始脚本
# /usr/local/bin/backup.mysql.struct start
查看状态
# /usr/local/bin/backup.mysql.struct status 9644 pts/1 S 0:00 /bin/bash /usr/local/bin/backup.mysql.struct start
停止脚本
# /usr/local/bin/backup.mysql.struct status
通过 git log 命令查看历史版本
# cd /www/database/struct/
# git status
# On branch master
nothing to commit (working directory clean)
# git log
commit d38fc624c21cad0e2f55f0228bff0c1be981827c
Author: root <root@slave.example.com>
Date: Wed Dec 17 12:33:55 2014 +0800
2014-12-17.04:33:55
这里仅仅将数据库结构版本控制，关于版本控制软件更多细节，延伸阅读 《Netkiller Version 手札》
背景，微服务开发中，常常会用到注册中心和配置中心，目前国内比较流行使用 Nacos，Nacos 的配置是保存在数据库中的，不方便维护。代码的变更与配置的版本是没有强关联的，尤其是并行开发中，我们需要多套开发和测试环境时，配置管理的工作会带来挑战，如果配置中心管理不善，就会出现各种问题，例如相互覆盖，版本不一致等等。
虽然 Nacos 也有历史记录，需要为每个人创建一个帐号，才能实现在版本管理中看到谁在什么时间修改了配置。但是即使这样，配置的版本与代码是没有关联的，我们不清楚不同的代码版本需要那些必要的配置项。
我更趋向让配置与代码版本强关联，实现配置的版本与代码的版本一致，让配置管理与持续集成和部署融合，配置变更由持续部署流水线自行完成，而不是让管理员去 Nacos 后台手工处理。于是变产生了下面的工具。
安装 netkiller-devops 工具
# pip 命令安装：
root@netkiller ~# pip install netkiller-devops
# 如果此前已经安装，可以使用下面命令更新：
root@netkiller ~# pip install netkiller-devops --upgrade
# Docker 方式安装：
root@netkiller ~# docker pull netkiller/netkiller-devops:latest
root@netkiller ~# docker run --rm -it --name=netkiller --entrypoint=sh netkiller-devops:latest
/srv # nacos
Usage: nacos [options] message
Options:
-h, --help show this help message and exit
-s localhost:8848, --server-addr=localhost:8848
localhost:8848
-u USERNAME, --username=USERNAME
-p PASSWORD, --password=PASSWORD
-n public, --namespace=public
-d DATAID, --dataId=DATAID
-g DEFAULT_GROUP, --group=DEFAULT_GROUP
Config:
--push
--show
--save
-f FILE, --file=FILE
.yaml file
-t yaml, --type=yaml
yaml|text|json|xml|Properties
--delete
Homepage: https://www.netkiller.cn Author: Neo <netkiller@msn.com>
Help: https://github.com/netkiller/devops/blob/master/doc/
帮助信息
root@netkiller ~# nacos
Usage: nacos [options]
Options:
-h, --help show this help message and exit
-s http://localhost:8848, --server-addr=http://localhost:8848
Nacos 服务器地址
-u USERNAME, --username=USERNAME
用户名
-p PASSWORD, --password=PASSWORD
密码
-n public, --namespace=public
命名空间
-d DATAID, --dataId=DATAID
配置ID
-g DEFAULT_GROUP, --group=DEFAULT_GROUP
分组
配置管理:
--push 发布配置
--show 查看配置
--save 保存配置
-f FILE, --file=FILE
.yaml 文件
-t yaml, --type=yaml
yaml|text|json|xml|Properties
--delete 删除配置
Homepage: https://www.netkiller.cn Author: Neo <netkiller@msn.com>
Help: https://github.com/netkiller/devops/blob/master/doc/
发布 Nacos 配置
准备配置文件
root@netkiller ~# cat test.yaml
server:
servlet:
context-path: /netkiller
spring:
servlet:
multipart:
max-file-size: 10MB
max-request-size: 10MB
发布 test.yaml 配置文件到 Nacos 配置中心
root@netkiller ~# nacos -s http://nacos.netkiller.cn -u nacos -p nacos -n test -d test --push -f test.yaml
查看 Nacos 配置
root@netkiller ~# nacos -s http://nacos.netkiller.cn -u nacos -p nacos -n test -d test --show
server:
servlet:
context-path: /netkiller
spring:
servlet:
multipart:
max-file-size: 10MB
max-request-size: 10MB
保存 Nacos 配置
root@netkiller ~# nacos -s http://nacos.netkiller.cn -u nacos -p nacos -n test -d test --save -f save.yaml
root@netkiller ~# cat save.yaml
server:
servlet:
context-path: /netkiller
spring:
servlet:
multipart:
max-file-size: 10MB
max-request-size: 10MB
删除 Nacos 配置
root@netkiller ~# nacos -s http://nacos.netkiller.cn -u nacos -p nacos -n test -d test —delete 返回 None 表示配置不存在 root@netkiller ~# nacos -s http://nacos.netkiller.cn -u nacos -p nacos -n test -d test --show None
方案一，使用 Shell 执行器，在 gitlab runner 节点上安装 netkiller-devops 包
stages:
- deploy
deploy-job:
stage: deploy
variables:
NACOS: http://nacos.netkiller.cn
before_script:
- nacos -s $NACOS -u nacos -p nacos -n $CI_COMMIT_BRANCH -d $CI_PROJECT_NAME --show
script:
- nacos -s $NACOS -u nacos -p nacos -n $CI_COMMIT_BRANCH -d $CI_PROJECT_NAME --push -f nacos/$CI_COMMIT_BRANCH.yaml
after_script:
- nacos -s $NACOS -u nacos -p nacos -n $CI_COMMIT_BRANCH -d $CI_PROJECT_NAME --show
only:
- dev
- test
- master
tags:
- shell
方案二、使用 Kubernetes 或者 Docker 执行器
deploy-job-kubernetes:
stage: deploy
image: netkiller/netkiller-devops:latest
variables:
NACOS: http://nacos.netkiller.cn:8848
before_script:
- cat nacos/$CI_COMMIT_BRANCH.yaml
script:
- nacos -s $NACOS -u nacos -p nacos -n $CI_COMMIT_BRANCH -d $CI_PROJECT_NAME --push -f nacos/$CI_COMMIT_BRANCH.yaml
after_script:
- nacos -s $NACOS -u nacos -p nacos -n $CI_COMMIT_BRANCH -d $CI_PROJECT_NAME --show
only:
- dev
- test
- master
tags:
- kubernetes