GitLab - A full DevOps tool

How to purge the pipelines history ?

Situation :

During the CI setup, I've made plenty of tests (most of them failed), now resulting in a useless history of pipelines. How can I get rid of them ?

Details :

This was a strongly requested feature that was finally delivered in December 2018 release of GitLab CE (source).

Solution :

Discover the pipelines API :

Query the API from the GitLab server :
myToken='ZsyqJk1Rz6ZQ3EjyyEQL'; serverName='gitlab.myCompany.tld'; curl --header "PRIVATE-TOKEN: $myToken" "https://$serverName/api/v4/projects"
Query the API remotely :
myToken='ZsyqJk1Rz6ZQ3EjyyEQL'; serverName='gitlab.myCompany.tld'; serverIp='10.2.0.97'; curl -k --proxy "http://10.2.0.20:3128" --header "PRIVATE-TOKEN: $myToken" --header "hostname: $serverName" "https://$serverIp/api/v4/projects" | jq
The -k, --proxy "..." and --header "hostname: www.example.com" hacks are due to specific conditions of my local setup. You won't probably need that .

List projects :

myToken='ZsyqJk1Rz6ZQ3EjyyEQL'; serverName='gitlab.myCompany.tld'; serverIp='10.2.0.97'; curl -s -k --proxy "http://10.2.0.20:3128" --header "PRIVATE-TOKEN: $myToken" --header "hostname: $serverName" "https://$serverIp/api/v4/projects" | jq -r '.[] | .id,.name'
...
7
myGitLabProject
...
myToken='ZsyqJk1Rz6ZQ3EjyyEQL'; serverName='gitlab.myCompany.tld'; serverIp='10.2.0.97'; curl -s -k --proxy "http://10.2.0.20:3128" --header "PRIVATE-TOKEN: $myToken" --header "hostname: $serverName" "https://$serverIp/api/v4/projects/7" | jq
(JSON data)

List project pipelines :

myToken='ZsyqJk1Rz6ZQ3EjyyEQL'; serverName='gitlab.myCompany.tld'; serverIp='10.2.0.97'; curl -s -k --proxy "http://10.2.0.20:3128" --header "PRIVATE-TOKEN: $myToken" --header "hostname: $serverName" "https://$serverIp/api/v4/projects/7/pipelines" | jq
(JSON data)
myToken='ZsyqJk1Rz6ZQ3EjyyEQL'; serverName='gitlab.myCompany.tld'; serverIp='10.2.0.97'; curl -s -k --proxy "http://10.2.0.20:3128" --header "PRIVATE-TOKEN: $myToken" --header "hostname: $serverName" "https://$serverIp/api/v4/projects/7/pipelines" | jq -r '.[] | "\(.id) ==> \(.status)"'
25 ==> success
24 ==> failed
23 ==> failed
22 ==> failed
21 ==> failed
20 ==> failed
19 ==> failed
18 ==> canceled
17 ==> canceled
16 ==> canceled
15 ==> failed
14 ==> failed
13 ==> failed
12 ==> failed
11 ==> canceled
10 ==> canceled
9 ==> failed
8 ==> canceled
This is exactly what I can see in the WebUI : CI/CD | Pipelines

Delete a pipeline :

pipelineId='8'; gitlabProjectId='7'; myToken='ZsyqJk1Rz6ZQ3EjyyEQL'; serverName='gitlab.myCompany.tld'; serverIp='10.2.0.97'; curl -s -k --proxy "http://10.2.0.20:3128" --header "PRIVATE-TOKEN: $myToken" --header "hostname: $serverName" --request "DELETE" "https://$serverIp/api/v4/projects/$gitlabProjectId/pipelines/$pipelineId"

Purge pipelines based on their status :

step 1 : list pipelines having the specified status :
gitlabProjectId='7'; pipelineStatus='failed'; myToken='ZsyqJk1Rz6ZQ3EjyyEQL'; serverName='gitlab.myCompany.tld'; serverIp='10.2.0.97'; curl -s -k --proxy "http://10.2.0.20:3128" --header "PRIVATE-TOKEN: $myToken" --header "hostname: $serverName" "https://$serverIp/api/v4/projects/$gitlabProjectId/pipelines" | jq -r '.[] | select(.status=="'$pipelineStatus'") | .id'
step 2 : same in a for loop to prepare the next step :
gitlabProjectId='7'; pipelineStatus='failed'; myToken='ZsyqJk1Rz6ZQ3EjyyEQL'; serverName='gitlab.myCompany.tld'; serverIp='10.2.0.97'; proxy='http://10.2.0.20:3128'; for pipelineId in $(curl -s -k --proxy "$proxy" --header "PRIVATE-TOKEN: $myToken" --header "hostname: $serverName" "https://$serverIp/api/v4/projects/$gitlabProjectId/pipelines" | jq -r '.[] | select(.status=="'$pipelineStatus'") | .id'); do echo "PIPELINE ID ($pipelineStatus) : $pipelineId"; done
step 3 : delete matching pipelines :
gitlabProjectId='7'; pipelineStatus='failed'; myToken='ZsyqJk1Rz6ZQ3EjyyEQL'; serverName='gitlab.myCompany.tld'; serverIp='10.2.0.97'; proxy='http://10.2.0.20:3128'; for pipelineId in $(curl -s -k --proxy "$proxy" --header "PRIVATE-TOKEN: $myToken" --header "hostname: $serverName" "https://$serverIp/api/v4/projects/$gitlabProjectId/pipelines" | jq -r '.[] | select(.status=="'$pipelineStatus'") | .id'); do echo "Deleting PIPELINE ID ($pipelineStatus) : $pipelineId"; curl -s -k --proxy "$proxy" --header "PRIVATE-TOKEN: $myToken" --header "hostname: $serverName" --request "DELETE" "https://$serverIp/api/v4/projects/$gitlabProjectId/pipelines/$pipelineId"; done

GitLab glossary

.gitlab-ci.yml
YAML file holding GitLab CI / CD pipelines configuration
job (1, 2)
define what to run (for example, code compilation or test runs). Jobs are written to .gitlab-ci.yml
personal access token
ID to authenticate with the GitLab API. Get yours.
pipeline
top-level component of CI and CD. Pipelines have jobs and stages.
runner
isolated (virtual) machine/environment(?) that pick up jobs to run run the code defined in .gitlab-ci.yml
stage
Stages define when and how to run. For example, that tests run only after code compilation.

How to setup Continuous Integration (CI) with GitLab ?

https://docs.gitlab.com/ee/ci/
	https://docs.gitlab.com/ee/ci/introduction/index.html
		https://docs.gitlab.com/ee/ci/yaml/README.html

==========================================8<=========================================================
apt upgrade gitlab-ce
...
...
     _______ __  __          __
    / ____(_) /_/ /   ____ _/ /_
   / / __/ / __/ /   / __ `/ __ \
  / /_/ / / /_/ /___/ /_/ / /_/ /
  \____/_/\__/_____/\__,_/_.___/


Upgrade complete! If your GitLab server is misbehaving try running
	sudo gitlab-ctl restart
==========================================8<=========================================================

Create the .gitlab-ci.yml :
simply create it at the root of the repository
add + commit +push it like any other file



Check the .gitlab-ci.yml :
  1. https://10.2.0.97/safe/safe_sec_tac/-/ci/lint
==========================================8<========================================================= https://10.2.0.97/help/ci/quick_start/README install 'gitlab-runner' : https://docs.gitlab.com/runner/install/linux-repository.html Configuring a Runner gitlab-runner register Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/): https://10.2.0.97/ Please enter the gitlab-ci token for this runner: N6sAJckKFr1g6fFPXed3 Please enter the gitlab-ci description for this runner: [gitrns1]: Runner on GitNRS1 Please enter the gitlab-ci tags for this runner (comma separated): local,gitnrs1
ERROR: Registering runner... failed		runner=N6sAJckK status=couldn't execute POST against https://10.2.0.97/api/v4/runners: Post https://10.2.0.97/api/v4/runners: x509: certificate has expired or is not yet valid
PANIC: Failed to register this runner. Perhaps you are having network problems
==========================================8<========================================================= Let's have a look at the certificate :
openssl s_client -connect 10.2.0.97:443 | less
...
Verification error: certificate has expired
...
Verify return code: 10 (certificate has expired)
...
ss -pan | grep 443 tcp LISTEN 0 511 *:443 *:* users:(("nginx",pid=28588,fd=7),("nginx",pid=28587,fd=7)) find / -name '*nginx.conf' /var/opt/gitlab/nginx/conf/nginx.conf /opt/gitlab/embedded/conf/nginx.conf /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/unicorn-5.4.1/examples/nginx.conf /opt/gitlab/embedded/conf/nginx.conf ==> example ? really used ? /var/opt/gitlab/nginx/conf/nginx.conf # This file is managed by gitlab-ctl. Manual changes will be # erased! To change the contents below, edit /etc/gitlab/gitlab.rb # and run `sudo gitlab-ctl reconfigure`. grep include /var/opt/gitlab/nginx/conf/nginx.conf include /opt/gitlab/embedded/conf/mime.types; include /var/opt/gitlab/nginx/conf/gitlab-http.conf; include /var/opt/gitlab/nginx/conf/nginx-status.conf; less /var/opt/gitlab/nginx/conf/gitlab-http.conf
# This file is managed by gitlab-ctl. Manual changes will be
# erased! To change the contents below, edit /etc/gitlab/gitlab.rb
# and run `sudo gitlab-ctl reconfigure`.
...
server {
	listen *:443 ssl http2;

	server_name gitlab.myCompany.tld;
	...
	ssl on;
	ssl_certificate /etc/gitlab/ssl/gitlab.myCompany.tld.crt;
	ssl_certificate_key /etc/gitlab/ssl/gitlab.myCompany.tld.key;
openssl x509 -noout -in /etc/gitlab/ssl/gitlab.myCompany.tld.crt -dates notBefore=Dec 6 15:32:09 2018 GMT notAfter=Jan 5 15:32:09 2019 GMT openssl x509 -noout -in /etc/gitlab/ssl/gitlab.myCompany.tld.crt -text | grep -E '(Issuer|Subject):' Issuer: CN = gitlab.myCompany.tld Subject: CN = gitlab.myCompany.tld cd /run/shm/; export CN='gitlab.myCompany.tld'; openssl genrsa -out "$CN".key 2048; openssl req -new -x509 -days 365 -sha256 -key "$CN".key -out "$CN".crt -subj "/CN=$CN" openssl x509 -noout -in "$CN".crt -text | grep -E '(Issuer|Subject):' Issuer: CN = gitlab.myCompany.tld Subject: CN = gitlab.myCompany.tld export CN='gitlab.myCompany.tld'; cd /etc/gitlab/ssl; for stuff in "$CN.crt" "$CN.key"; do mv "$stuff" "${stuff}_OLD"; mv "/run/shm/$stuff" .; done sudo gitlab-ctl restart ==========================================8<========================================================= Now that the certificate is ok, let's retry to register our runner : gitlab-runner register Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/): https://10.2.0.97/ Please enter the gitlab-ci token for this runner: N6sAJckKFr1g6fFPXed3 Please enter the gitlab-ci description for this runner: [gitrns1]: Runner on GitNRS1 Please enter the gitlab-ci tags for this runner (comma separated): local,gitnrs1
ERROR: Registering runner... failed		runner=N6sAJckK status=couldn't execute POST against https://10.2.0.97/api/v4/runners: Post https://10.2.0.97/api/v4/runners: x509: cannot validate certificate for 10.2.0.97 because it doesn't contain any IP SANs
PANIC: Failed to register this runner. Perhaps you are having network problems
==========================================8<========================================================= https://geekflare.com/san-ssl-certificate/ export CN='gitlab.myCompany.tld'; workDir='/run/shm'; customConfigFile="$workDir/customConfigFile"; cd "$workDir"; cat <<EOF >"$customConfigFile" [ req ] default_bits = 2048 distinguished_name = req_distinguished_name req_extensions = req_ext [ req_distinguished_name ] #countryName = Country Name (2 letter code) countryName=FR #stateOrProvinceName = State or Province Name (full name) #localityName = Locality Name (eg, city) localityName=Nantes #organizationName = Organization Name (eg, company) organizationName=ACME Corp commonName=$CN [ req_ext ] subjectAltName = @alt_names [alt_names] DNS.1 = foo.com DNS.2 = bar.com DNS.3 = baz.com EOF make the CA key : caKey="$workDir/ca.key"; cd "$workDir" && openssl genrsa -des3 -out "$caKey" 4096 plop make the CA cert : caCert="$workDir/ca.crt";cd "$workDir" && openssl req -config "$customConfigFile" -new -x509 -nodes -sha1 -days 1825 -key "$caKey" -out "$caCert" make cert key openssl genrsa -des3 -out "$CN.key" 4096 plop make CSR : openssl req -config "$customConfigFile" -new -key "$CN.key" -out "$CN.csr" check the SAN in CSR : openssl req -noout -text -in "$CN".csr | grep DNS ==> DNS:foo.com, DNS:bar.com, DNS:baz.com \o/ openssl ca -config "$customConfigFile" -policy policy_anything -out "$CN.crt" -infiles "$CN.csr"
Using configuration from /run/shm/customConfigFile
variable lookup failed for ca::default_ca
139719445565504:error:0E06D06C:configuration file routines:NCONF_get_string:no value:../crypto/conf/conf_lib.c:275:group=ca name=default_ca
==> this is because there is no "default_ca" in the config file ==========================================8<========================================================= retrying : export CN='gitlab.myCompany.tld'; workDir='/run/shm'; customConfigFile="$workDir/customConfigFile"; cd "$workDir"; cp '/usr/lib/ssl/openssl.cnf' "$customConfigFile"; cat <<EOF >>"$customConfigFile" [ req ] default_bits = 2048 distinguished_name = req_distinguished_name req_extensions = req_ext [ req_distinguished_name ] countryName=FR localityName=Nantes organizationName=ACME Corp commonName=$CN [ req_ext ] subjectAltName = @alt_names [alt_names] DNS.1 = foo.com DNS.2 = bar.com DNS.3 = baz.com EOF
Using configuration from /run/shm/customConfigFile
Can't open ./demoCA/private/cakey.pem for reading, No such file or directory
139855953907776:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:74:fopen('./demoCA/private/cakey.pem','r')
139855953907776:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:81:
unable to load CA private key
==========================================8<========================================================= retrying : export CN='gitlab.myCompany.tld'; workDir='/run/shm/cert'; customConfigFile="$workDir/customConfigFile"; caKey="$workDir/ca.key"; caCert="$workDir/ca.crt"; mkdir -p "$workDir" && cd "$workDir"; touch index.txt; echo 01 > serial; cp '/usr/lib/ssl/openssl.cnf' "$customConfigFile"; for keyword in default_bits distinguished_name req_extensions countryName localityName organizationName commonName subjectAltName; do sed -ri 's|^'$keyword'|#'$keyword'|g' "$customConfigFile"; done; cat <<EOF >>"$customConfigFile" [ req ] default_bits = 2048 distinguished_name = req_distinguished_name req_extensions = req_ext [ req_distinguished_name ] countryName = FR localityName = Nantes organizationName = ACME Corp commonName = $CN [ req_ext ] subjectAltName = @alt_names [alt_names] DNS.1 = foo.com DNS.2 = bar.com DNS.3 = baz.com IP = $(hostname -i) EOF sed -ri -e 's|\./demoCA|.|' -e 's|\$dir/private/cakey\.pem|'$caKey'|' -e 's|\$dir/cacert\.pem|'$caCert'|' -e 's|\$dir/newcerts|.|' "$customConfigFile" make the CA key : openssl genrsa -des3 -out "$caKey" 4096 plop make the CA cert : openssl req -config "$customConfigFile" -new -x509 -nodes -sha1 -days 1825 -key "$caKey" -out "$caCert" plop make cert key openssl genrsa -des3 -out "$CN.key" 4096 test make CSR : openssl req -config "$customConfigFile" -new -key "$CN.key" -out "$CN.csr" test check the SAN in CSR : openssl req -noout -text -in "$CN".csr | grep -E 'DNS|IP' ==> DNS:foo.com, DNS:bar.com, DNS:baz.com ==> DNS:foo.com, DNS:bar.com, DNS:baz.com, IP Address:10.2.0.97 \o/ sign the certificate : openssl ca -config "$customConfigFile" -policy policy_anything -out "$CN.crt" -infiles "$CN.csr" plop ==> \o/ remove the password from the certificate private key file mv "$CN.key" "$CN.withPassword.key" && openssl rsa -in "$CN.withPassword.key" -out "$CN.key" test cd /etc/gitlab/ssl && for stuff in "$CN.crt" "$CN.key"; do mv "$stuff" "${stuff}_OLD"; cp "$workDir/$stuff" .; done sudo gitlab-ctl restart ==> KO, still the same error about the certificate having no SAN information :-((( ==========================================8<========================================================= with makeCert.sh (inspired by) CN='gitlab.myCompany.tld'; workDir='/run/shm/certificate'; cd /etc/gitlab/ssl && for stuff in "$CN.crt" "$CN.key"; do mv "$stuff" "${stuff}_OLD"; cp "$workDir/$stuff" .; done sudo gitlab-ctl restart ==> the certificate is OK retry registering the runner :
ERROR: Registering runner... failed		runner=N6sAJckK status=couldn't execute POST against https://10.2.0.97/api/v4/runners: Post https://10.2.0.97/api/v4/runners: x509: certificate signed by unknown authority
PANIC: Failed to register this runner. Perhaps you are having network problems
==========================================8<========================================================= https://stackoverflow.com/questions/44458410/gitlab-ci-runner-ignore-self-signed-certificate#answer-48347895 sudo gitlab-runner register --tls-ca-file /my/path/gitlab/gitlab.myserver.com.pem gitlab-runner register --help | grep 'tls' --tls-ca-file value File containing the certificates to verify the peer when using HTTPS [$CI_SERVER_TLS_CA_FILE] --tls-cert-file value File containing certificate for TLS client auth when using HTTPS [$CI_SERVER_TLS_CERT_FILE] --tls-key-file value File containing private key for TLS client auth when using HTTPS [$CI_SERVER_TLS_KEY_FILE] use + deploy the 'makeCert.sh' seen above CN='gitlab.myCompany.tld'; workDir='/run/shm/certificate'; cd "$workDir"; cat "$CN".crt "$CN".key > "$CN".pem sudo gitlab-runner register --tls-ca-file "$workDir/$CN.pem" https://docs.gitlab.com/runner/commands/#interactive-registration gitlab-runner register
Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
https://10.2.0.97/
Please enter the gitlab-ci token for this runner:
N6sAJckKFr1g6fFPXed3
Please enter the gitlab-ci description for this runner:
[gitrns1]: Runner on GitNRS1
Please enter the gitlab-ci tags for this runner (comma separated):
local,gitnrs1
Registering runner... succeeded				runner=N6sAJckK
Please enter the executor: docker, docker-ssh, parallels, docker-ssh+machine, shell, ssh, virtualbox, docker+machine, kubernetes:
shell
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
==========================================8<========================================================= In the WebUI : Settings | CI/CD | Runners [Expand] ==> my shared runner is here \o/ See also : Admin Area (wrench icon) | Overview | Runners enter 'Edit' mode to 'Restrict projects for this Runner' https://docs.gitlab.com/runner/commands/README.html gitlab-runner --help gitlab-runner list Runtime platform arch=amd64 os=linux pid=23440 revision=692ae235 version=11.9.0 Listing configured runners ConfigFile=/etc/gitlab-runner/config.toml Runner on GitNRS1 Executor=shell Token=5Ri6XJzpUXXJ61Ugx2nX URL=https://10.2.0.97/ https://docs.gitlab.com/runner/commands/#gitlab-runner-verify gitlab-runner verify Runtime platform arch=amd64 os=linux pid=24359 revision=692ae235 version=11.9.0 Running in system-mode. Verifying runner... is alive runner=5Ri6XJzp gitlab-runner --debug run https://docs.gitlab.com/runner/commands/#gitlab-runner-start gitlab-runner start Runtime platform arch=amd64 os=linux pid=24891 revision=692ae235 version=11.9.0 gitlab-runner verify Runtime platform arch=amd64 os=linux pid=25070 revision=692ae235 version=11.9.0 Running in system-mode. Verifying runner... is alive runner=5Ri6XJzp https://10.2.0.97/help/ci/runners/README.md#allowing-runners-with-tags-to-pick-jobs-without-tags If pipeline is still "stuck", check the "Run untagged jobs" box in runner configuration ==========================================8<========================================================= In the WebUI : CI/CD | Jobs | click on any "Failed" tag :
Running with gitlab-runner 11.9.0 (692ae235)
	on Runner on GitNRS1 5Ri6XJzp
Using Shell executor...
Running on gitrns1...
Reinitialized existing Git repository in /home/gitlab-runner/builds/5Ri6XJzp/0/safe/safe_sec_tac/.git/
Fetching changes...
fatal: remote origin already exists.
Clean repository
fatal: unable to access 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@gitlab.myCompany.tld/safe/safe_sec_tac.git/': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
ERROR: Job failed: exit status 1
curl -i https://gitlab.myCompany.tld curl: (60) SSL certificate problem: unable to get local issuer certificate ... curl -ki https://gitlab.myCompany.tld HTTP/2 302 server: nginx date: Thu, 11 Apr 2019 10:30:15 GMT content-type: text/html; charset=utf-8 content-length: 107 location: https://gitlab.myCompany.tld/users/sign_in ...

Need to install my "own CA" root certificate (source) :

workDir="$HOME/certificate"; caCert='ca-cert'; mkdir -p /usr/local/share/ca-certificates/extra; cp "$workDir/$caCert.crt" /usr/local/share/ca-certificates/extra/; update-ca-certificates
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
ll /etc/ssl/certs | grep extra
lrwxrwxrwx 1 root root   50 Apr 11 12:41 ca-cert.pem -> /usr/local/share/ca-certificates/extra/ca-cert.crt	mine  !!!
curl -i https://gitlab.myCompany.tld
HTTP/2 302		Just noticed this is serving HTTP 2
server: nginx
date: Fri, 12 Apr 2019 12:51:10 GMT
content-type: text/html; charset=utf-8
content-length: 97
...
This is a valid response from my webserver !!!

So now, let's try again submitting a pipeline :

Which gives :
Running with gitlab-runner 11.9.0 (692ae235)
	on Runner on GitNRS1 5Ri6XJzp
Using Shell executor...
Running on gitrns1...
Reinitialized existing Git repository in /home/gitlab-runner/builds/5Ri6XJzp/0/safe/safe_sec_tac/.git/
Fetching changes...
fatal: remote origin already exists.
Clean repository
From https://gitlab.myCompany.tld/safe/safe_sec_tac
	* [new branch]	master	-> origin/master
Checking out 2aed53bb as master...
Skipping Git submodules setup
$ true
Job succeeded
Now, I _just_ have to make it do valuable work rather than running true