GitLab - A full DevOps tool

mail

Data is missing in the API results

Situation

Objects (projects, whatever) I am sure they DO exist are not listed when querying the API.

Details

This is because the API results are paginated.

Solution

You'll have to specify extra parameters in the query :
myToken='ZsyqJk1Rz6ZQ3EjyyEQL'; serverName='gitlab.myCompany.tld'; serverIp='10.2.0.97'; curl -s -k --proxy "http://10.2.0.20:3128" --header "PRIVATE-TOKEN: $myToken" --header "hostname: $serverName" "https://$serverIp/api/v4/projects?per_page=50&page=1" | jq -r '.[] | .id,.name'
mail

How to purge the pipelines history ?

Situation

During the CI setup, I've made plenty of tests (most of them failed), now resulting in a useless history of pipelines. How can I get rid of them ?

Details

This was a strongly requested feature that was finally delivered in December 2018 release of GitLab CE (source).

Solution

Discover the pipelines API :

Query the API from the GitLab server :
myToken='ZsyqJk1Rz6ZQ3EjyyEQL'; serverName='gitlab.myCompany.tld'; curl --header "PRIVATE-TOKEN: $myToken" "https://$serverName/api/v4/projects"
Query the API remotely :
myToken='ZsyqJk1Rz6ZQ3EjyyEQL'; serverName='gitlab.myCompany.tld'; serverIp='10.2.0.97'; curl -k --proxy "http://10.2.0.20:3128" --header "PRIVATE-TOKEN: $myToken" --header "hostname: $serverName" "https://$serverIp/api/v4/projects" | jq
The -k, --proxy "" and --header "hostname: www.example.com" hacks are due to specific conditions of my local setup. You won't probably need that .

List projects :

myToken='ZsyqJk1Rz6ZQ3EjyyEQL'; serverName='gitlab.myCompany.tld'; serverIp='10.2.0.97'; curl -s -k --proxy "http://10.2.0.20:3128" --header "PRIVATE-TOKEN: $myToken" --header "hostname: $serverName" "https://$serverIp/api/v4/projects" | jq -r '.[] | .id,.name'

7
myGitLabProject
  • You can also get a project ID from the WebUI : open the project main page (where you can see the root of your repository and the contents of the readme.md file). The project ID is displayed on the top of the page, right under the project name.
  • If a project is not listed in the API results, it may be because it appears on another page.
myToken='ZsyqJk1Rz6ZQ3EjyyEQL'; serverName='gitlab.myCompany.tld'; serverIp='10.2.0.97'; curl -s -k --proxy "http://10.2.0.20:3128" --header "PRIVATE-TOKEN: $myToken" --header "hostname: $serverName" "https://$serverIp/api/v4/projects/7" | jq
(JSON data)

List project pipelines :

myToken='ZsyqJk1Rz6ZQ3EjyyEQL'; serverName='gitlab.myCompany.tld'; serverIp='10.2.0.97'; curl -s -k --proxy "http://10.2.0.20:3128" --header "PRIVATE-TOKEN: $myToken" --header "hostname: $serverName" "https://$serverIp/api/v4/projects/7/pipelines" | jq
(JSON data)
myToken='ZsyqJk1Rz6ZQ3EjyyEQL'; serverName='gitlab.myCompany.tld'; serverIp='10.2.0.97'; curl -s -k --proxy "http://10.2.0.20:3128" --header "PRIVATE-TOKEN: $myToken" --header "hostname: $serverName" "https://$serverIp/api/v4/projects/7/pipelines" | jq -r '.[] | "\(.id) ==> \(.status)"'
25 ==> success
24 ==> failed
23 ==> failed
22 ==> failed
21 ==> failed
20 ==> failed
19 ==> failed
18 ==> canceled
17 ==> canceled
16 ==> canceled
15 ==> failed
14 ==> failed
13 ==> failed
12 ==> failed
11 ==> canceled
10 ==> canceled
9 ==> failed
8 ==> canceled
This is exactly what I can see in the WebUI : CI/CD | Pipelines

Delete a pipeline :

pipelineId='8'; gitlabProjectId='7'; myToken='ZsyqJk1Rz6ZQ3EjyyEQL'; serverName='gitlab.myCompany.tld'; serverIp='10.2.0.97'; curl -s -k --proxy "http://10.2.0.20:3128" --header "PRIVATE-TOKEN: $myToken" --header "hostname: $serverName" --request "DELETE" "https://$serverIp/api/v4/projects/$gitlabProjectId/pipelines/$pipelineId"

Purge pipelines based on their status :

step 1 : list pipelines having the specified status :

gitlabProjectId='7'; pipelineStatus='failed'; myToken='ZsyqJk1Rz6ZQ3EjyyEQL'; serverName='gitlab.myCompany.tld'; serverIp='10.2.0.97'; curl -s -k --proxy "http://10.2.0.20:3128" --header "PRIVATE-TOKEN: $myToken" --header "hostname: $serverName" "https://$serverIp/api/v4/projects/$gitlabProjectId/pipelines" | jq -r '.[] | select(.status=="'$pipelineStatus'") | .id'

step 2 : same in a for loop to prepare the next step :

gitlabProjectId='7'; pipelineStatus='failed'; myToken='ZsyqJk1Rz6ZQ3EjyyEQL'; serverName='gitlab.myCompany.tld'; serverIp='10.2.0.97'; proxy='http://10.2.0.20:3128'; for pipelineId in $(curl -s -k --proxy "$proxy" --header "PRIVATE-TOKEN: $myToken" --header "hostname: $serverName" "https://$serverIp/api/v4/projects/$gitlabProjectId/pipelines" | jq -r '.[] | select(.status=="'$pipelineStatus'") | .id'); do echo "PIPELINE ID ($pipelineStatus) : $pipelineId"; done

step 3 : delete matching pipelines :

gitlabProjectId='7'; pipelineStatus='failed'; myToken='ZsyqJk1Rz6ZQ3EjyyEQL'; serverName='gitlab.myCompany.tld'; serverIp='10.2.0.97'; proxy='http://10.2.0.20:3128'; for pipelineId in $(curl -s -k --proxy "$proxy" --header "PRIVATE-TOKEN: $myToken" --header "hostname: $serverName" "https://$serverIp/api/v4/projects/$gitlabProjectId/pipelines" | jq -r '.[] | select(.status=="'$pipelineStatus'") | .id'); do echo "Deleting PIPELINE ID ($pipelineStatus) : $pipelineId"; curl -s -k --proxy "$proxy" --header "PRIVATE-TOKEN: $myToken" --header "hostname: $serverName" --request "DELETE" "https://$serverIp/api/v4/projects/$gitlabProjectId/pipelines/$pipelineId"; done
mail

GitLab glossary

.gitlab-ci.yml
YAML file holding GitLab CI / CD pipelines configuration
jobs
  • define what to do (compile code, run tests, )
  • are made of a list of commands
  • are written in .gitlab-ci.yml
  • are executed by runners
personal access token
ID to authenticate with the GitLab API. Get yours.
pipeline
  • top-level component of CI / CD
  • Pipelines have
    1. jobs
    2. stages
runner
  • isolated + ephemeral VM / cloud instance that executes jobs
  • each runner, as it becomes available, continually sends requests to the GitLab instance, asking to be assigned jobs
stage (vegastack.com, medium.com)
.gitlab-ci.yml has stages and stage keywords
  • define when to run jobs : conditional execution of groups of tasks. For example :
    1. compile code
    2. if ok, run tests
  • stages / jobs :
    • jobs in the same stage run in parallel
    • jobs in the next stage run after the jobs from the previous stage complete successfully
mail

How to setup Continuous Integration (CI) with GitLab ?

https://docs.gitlab.com/ee/ci/
	https://docs.gitlab.com/ee/ci/introduction/index.html
		https://docs.gitlab.com/ee/ci/yaml/README.html

==========================================8<=========================================================
apt upgrade gitlab-ce
...
...
     _______ __  __          __
    / ____(_) /_/ /   ____ _/ /_
   / / __/ / __/ /   / __ `/ __ \
  / /_/ / / /_/ /___/ /_/ / /_/ /
  \____/_/\__/_____/\__,_/_.___/


Upgrade complete! If your GitLab server is misbehaving try running
	sudo gitlab-ctl restart
==========================================8<=========================================================

Create the .gitlab-ci.yml :
simply create it at the root of the repository
add + commit +push it like any other file



Check the .gitlab-ci.yml :
  1. https://10.2.0.97/safe/safe_sec_tac/-/ci/lint
==========================================8<========================================================= https://10.2.0.97/help/ci/quick_start/README install 'gitlab-runner' : https://docs.gitlab.com/runner/install/linux-repository.html Configuring a Runner gitlab-runner register Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/): https://10.2.0.97/ Please enter the gitlab-ci token for this runner: N6sAJckKFr1g6fFPXed3 Please enter the gitlab-ci description for this runner: [gitrns1]: Runner on GitNRS1 Please enter the gitlab-ci tags for this runner (comma separated): local,gitnrs1
ERROR: Registering runner... failed		runner=N6sAJckK status=couldn't execute POST against https://10.2.0.97/api/v4/runners: Post https://10.2.0.97/api/v4/runners: x509: certificate has expired or is not yet valid
PANIC: Failed to register this runner. Perhaps you are having network problems
==========================================8<========================================================= Let's have a look at the certificate :
openssl s_client -connect 10.2.0.97:443 | less

Verification error: certificate has expired

Verify return code: 10 (certificate has expired)
ss -punta | grep 443 tcp LISTEN 0 511 *:443 *:* users:(("nginx",pid=28588,fd=7),("nginx",pid=28587,fd=7)) find / -name '*nginx.conf' /var/opt/gitlab/nginx/conf/nginx.conf /opt/gitlab/embedded/conf/nginx.conf /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/unicorn-5.4.1/examples/nginx.conf /opt/gitlab/embedded/conf/nginx.conf ==> example ? really used ? /var/opt/gitlab/nginx/conf/nginx.conf # This file is managed by gitlab-ctl. Manual changes will be # erased! To change the contents below, edit /etc/gitlab/gitlab.rb # and run `sudo gitlab-ctl reconfigure`. grep include /var/opt/gitlab/nginx/conf/nginx.conf include /opt/gitlab/embedded/conf/mime.types; include /var/opt/gitlab/nginx/conf/gitlab-http.conf; include /var/opt/gitlab/nginx/conf/nginx-status.conf; less /var/opt/gitlab/nginx/conf/gitlab-http.conf
# This file is managed by gitlab-ctl. Manual changes will be
# erased! To change the contents below, edit /etc/gitlab/gitlab.rb
# and run `sudo gitlab-ctl reconfigure`.

server {
	listen *:443 ssl http2;

	server_name gitlab.myCompany.tld;
	
	ssl on;
	ssl_certificate /etc/gitlab/ssl/gitlab.myCompany.tld.crt;
	ssl_certificate_key /etc/gitlab/ssl/gitlab.myCompany.tld.key;
openssl x509 -noout -in /etc/gitlab/ssl/gitlab.myCompany.tld.crt -dates notBefore=Dec 6 15:32:09 2018 GMT notAfter=Jan 5 15:32:09 2019 GMT openssl x509 -noout -in /etc/gitlab/ssl/gitlab.myCompany.tld.crt -text | grep -E '(Issuer|Subject):' Issuer: CN = gitlab.myCompany.tld Subject: CN = gitlab.myCompany.tld cd /run/shm/; export CN='gitlab.myCompany.tld'; openssl genrsa -out "$CN".key 2048; openssl req -new -x509 -days 365 -sha256 -key "$CN".key -out "$CN".crt -subj "/CN=$CN" openssl x509 -noout -in "$CN".crt -text | grep -E '(Issuer|Subject):' Issuer: CN = gitlab.myCompany.tld Subject: CN = gitlab.myCompany.tld export CN='gitlab.myCompany.tld'; cd /etc/gitlab/ssl; for stuff in "$CN.crt" "$CN.key"; do mv "$stuff" "${stuff}_OLD"; mv "/run/shm/$stuff" .; done sudo gitlab-ctl restart ==========================================8<========================================================= Now that the certificate is ok, let's retry to register our runner : gitlab-runner register Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/): https://10.2.0.97/ Please enter the gitlab-ci token for this runner: N6sAJckKFr1g6fFPXed3 Please enter the gitlab-ci description for this runner: [gitrns1]: Runner on GitNRS1 Please enter the gitlab-ci tags for this runner (comma separated): local,gitnrs1
ERROR: Registering runner... failed		runner=N6sAJckK status=couldn't execute POST against https://10.2.0.97/api/v4/runners: Post https://10.2.0.97/api/v4/runners: x509: cannot validate certificate for 10.2.0.97 because it doesn't contain any IP SANs
PANIC: Failed to register this runner. Perhaps you are having network problems
==========================================8<========================================================= https://geekflare.com/san-ssl-certificate/ export CN='gitlab.myCompany.tld'; workDir='/run/shm'; customConfigFile="$workDir/customConfigFile"; cd "$workDir"; cat <<EOF >"$customConfigFile" [ req ] default_bits = 2048 distinguished_name = req_distinguished_name req_extensions = req_ext [ req_distinguished_name ] #countryName = Country Name (2 letter code) countryName=FR #stateOrProvinceName = State or Province Name (full name) #localityName = Locality Name (eg, city) localityName=Nantes #organizationName = Organization Name (eg, company) organizationName=ACME Corp commonName=$CN [ req_ext ] subjectAltName = @alt_names [alt_names] DNS.1 = foo.com DNS.2 = bar.com DNS.3 = baz.com EOF make the CA key : caKey="$workDir/ca.key"; cd "$workDir" && openssl genrsa -des3 -out "$caKey" 4096 plop make the CA cert : caCert="$workDir/ca.crt";cd "$workDir" && openssl req -config "$customConfigFile" -new -x509 -nodes -sha1 -days 1825 -key "$caKey" -out "$caCert" make cert key openssl genrsa -des3 -out "$CN.key" 4096 plop make CSR : openssl req -config "$customConfigFile" -new -key "$CN.key" -out "$CN.csr" check the SAN in CSR : openssl req -noout -text -in "$CN".csr | grep DNS ==> DNS:foo.com, DNS:bar.com, DNS:baz.com \o/ openssl ca -config "$customConfigFile" -policy policy_anything -out "$CN.crt" -infiles "$CN.csr"
Using configuration from /run/shm/customConfigFile
variable lookup failed for ca::default_ca
139719445565504:error:0E06D06C:configuration file routines:NCONF_get_string:no value:../crypto/conf/conf_lib.c:275:group=ca name=default_ca
==> this is because there is no "default_ca" in the config file ==========================================8<========================================================= retrying : export CN='gitlab.myCompany.tld'; workDir='/run/shm'; customConfigFile="$workDir/customConfigFile"; cd "$workDir"; cp '/usr/lib/ssl/openssl.cnf' "$customConfigFile"; cat <<EOF >>"$customConfigFile" [ req ] default_bits = 2048 distinguished_name = req_distinguished_name req_extensions = req_ext [ req_distinguished_name ] countryName=FR localityName=Nantes organizationName=ACME Corp commonName=$CN [ req_ext ] subjectAltName = @alt_names [alt_names] DNS.1 = foo.com DNS.2 = bar.com DNS.3 = baz.com EOF
Using configuration from /run/shm/customConfigFile
Can't open ./demoCA/private/cakey.pem for reading, No such file or directory
139855953907776:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:74:fopen('./demoCA/private/cakey.pem','r')
139855953907776:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:81:
unable to load CA private key
==========================================8<========================================================= retrying : export CN='gitlab.myCompany.tld'; workDir='/run/shm/cert'; customConfigFile="$workDir/customConfigFile"; caKey="$workDir/ca.key"; caCert="$workDir/ca.crt"; mkdir -p "$workDir" && cd "$workDir"; touch index.txt; echo 01 > serial; cp '/usr/lib/ssl/openssl.cnf' "$customConfigFile"; for keyword in default_bits distinguished_name req_extensions countryName localityName organizationName commonName subjectAltName; do sed -ri 's|^'$keyword'|#'$keyword'|g' "$customConfigFile"; done; cat <<EOF >>"$customConfigFile" [ req ] default_bits = 2048 distinguished_name = req_distinguished_name req_extensions = req_ext [ req_distinguished_name ] countryName = FR localityName = Nantes organizationName = ACME Corp commonName = $CN [ req_ext ] subjectAltName = @alt_names [alt_names] DNS.1 = foo.com DNS.2 = bar.com DNS.3 = baz.com IP = $(hostname -i) EOF sed -ri -e 's|\./demoCA|.|' -e 's|\$dir/private/cakey\.pem|'$caKey'|' -e 's|\$dir/cacert\.pem|'$caCert'|' -e 's|\$dir/newcerts|.|' "$customConfigFile" make the CA key : openssl genrsa -des3 -out "$caKey" 4096 plop make the CA cert : openssl req -config "$customConfigFile" -new -x509 -nodes -sha1 -days 1825 -key "$caKey" -out "$caCert" plop make cert key openssl genrsa -des3 -out "$CN.key" 4096 test make CSR : openssl req -config "$customConfigFile" -new -key "$CN.key" -out "$CN.csr" test check the SAN in CSR : openssl req -noout -text -in "$CN".csr | grep -E 'DNS|IP' ==> DNS:foo.com, DNS:bar.com, DNS:baz.com ==> DNS:foo.com, DNS:bar.com, DNS:baz.com, IP Address:10.2.0.97 \o/ sign the certificate : openssl ca -config "$customConfigFile" -policy policy_anything -out "$CN.crt" -infiles "$CN.csr" plop ==> \o/ remove the password from the certificate private key file mv "$CN.key" "$CN.withPassword.key" && openssl rsa -in "$CN.withPassword.key" -out "$CN.key" test cd /etc/gitlab/ssl && for stuff in "$CN.crt" "$CN.key"; do mv "$stuff" "${stuff}_OLD"; cp "$workDir/$stuff" .; done sudo gitlab-ctl restart ==> KO, still the same error about the certificate having no SAN information :-((( ==========================================8<========================================================= with makeCert.sh (inspired by) CN='gitlab.myCompany.tld'; workDir='/run/shm/certificate'; cd /etc/gitlab/ssl && for stuff in "$CN.crt" "$CN.key"; do mv "$stuff" "${stuff}_OLD"; cp "$workDir/$stuff" .; done sudo gitlab-ctl restart ==> the certificate is OK retry registering the runner :
ERROR: Registering runner... failed		runner=N6sAJckK status=couldn't execute POST against https://10.2.0.97/api/v4/runners: Post https://10.2.0.97/api/v4/runners: x509: certificate signed by unknown authority
PANIC: Failed to register this runner. Perhaps you are having network problems
==========================================8<========================================================= https://stackoverflow.com/questions/44458410/gitlab-ci-runner-ignore-self-signed-certificate#answer-48347895 sudo gitlab-runner register --tls-ca-file /my/path/gitlab/gitlab.myserver.com.pem gitlab-runner register --help | grep 'tls' --tls-ca-file value File containing the certificates to verify the peer when using HTTPS [$CI_SERVER_TLS_CA_FILE] --tls-cert-file value File containing certificate for TLS client auth when using HTTPS [$CI_SERVER_TLS_CERT_FILE] --tls-key-file value File containing private key for TLS client auth when using HTTPS [$CI_SERVER_TLS_KEY_FILE] use + deploy the 'makeCert.sh' seen above CN='gitlab.myCompany.tld'; workDir='/run/shm/certificate'; cd "$workDir"; cat "$CN".crt "$CN".key > "$CN".pem sudo gitlab-runner register --tls-ca-file "$workDir/$CN.pem" https://docs.gitlab.com/runner/commands/#interactive-registration gitlab-runner register
Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
https://10.2.0.97/
Please enter the gitlab-ci token for this runner:
N6sAJckKFr1g6fFPXed3
Please enter the gitlab-ci description for this runner:
[gitrns1]: Runner on GitNRS1
Please enter the gitlab-ci tags for this runner (comma separated):
local,gitnrs1
Registering runner... succeeded				runner=N6sAJckK
Please enter the executor: docker, docker-ssh, parallels, docker-ssh+machine, shell, ssh, virtualbox, docker+machine, kubernetes:
shell
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
==========================================8<========================================================= In the WebUI : Settings | CI/CD | Runners [Expand] ==> my shared runner is here \o/ See also : Admin Area (wrench icon) | Overview | Runners enter 'Edit' mode to 'Restrict projects for this Runner' https://docs.gitlab.com/runner/commands/README.html gitlab-runner --help gitlab-runner list Runtime platform arch=amd64 os=linux pid=23440 revision=692ae235 version=11.9.0 Listing configured runners ConfigFile=/etc/gitlab-runner/config.toml Runner on GitNRS1 Executor=shell Token=5Ri6XJzpUXXJ61Ugx2nX URL=https://10.2.0.97/ https://docs.gitlab.com/runner/commands/#gitlab-runner-verify gitlab-runner verify Runtime platform arch=amd64 os=linux pid=24359 revision=692ae235 version=11.9.0 Running in system-mode. Verifying runner... is alive runner=5Ri6XJzp gitlab-runner --debug run https://docs.gitlab.com/runner/commands/#gitlab-runner-start gitlab-runner start Runtime platform arch=amd64 os=linux pid=24891 revision=692ae235 version=11.9.0 gitlab-runner verify Runtime platform arch=amd64 os=linux pid=25070 revision=692ae235 version=11.9.0 Running in system-mode. Verifying runner... is alive runner=5Ri6XJzp https://10.2.0.97/help/ci/runners/README.md#allowing-runners-with-tags-to-pick-jobs-without-tags If pipeline is still "stuck", check the "Run untagged jobs" box in runner configuration ==========================================8<========================================================= In the WebUI : CI/CD | Jobs | click on any "Failed" tag :
Running with gitlab-runner 11.9.0 (692ae235)
	on Runner on GitNRS1 5Ri6XJzp
Using Shell executor...
Running on gitrns1...
Reinitialized existing Git repository in /home/gitlab-runner/builds/5Ri6XJzp/0/safe/safe_sec_tac/.git/
Fetching changes...
fatal: remote origin already exists.
Clean repository
fatal: unable to access 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@gitlab.myCompany.tld/safe/safe_sec_tac.git/': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
ERROR: Job failed: exit status 1
curl -i https://gitlab.myCompany.tld curl: (60) SSL certificate problem: unable to get local issuer certificate ... curl -ki https://gitlab.myCompany.tld HTTP/2 302 server: nginx date: Thu, 11 Apr 2019 10:30:15 GMT content-type: text/html; charset=utf-8 content-length: 107 location: https://gitlab.myCompany.tld/users/sign_in ...

Need to install my "own CA" root certificate (source) :

workDir="$HOME/certificate"; extraDir='/usr/local/share/ca-certificates/extra'; caCert='ca-cert'; mkdir -p "$extraDir"; cp "$workDir/$caCert.crt" "$extraDir"; update-ca-certificates
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
ll /etc/ssl/certs | grep extra
lrwxrwxrwx 1 root root   50 Apr 11 12:41 ca-cert.pem -> /usr/local/share/ca-certificates/extra/ca-cert.crt	mine  !!!
curl -i https://gitlab.myCompany.tld
HTTP/2 302		Just noticed this is serving HTTP 2
server: nginx
date: Fri, 12 Apr 2019 12:51:10 GMT
content-type: text/html; charset=utf-8
content-length: 97
This is a valid response from my webserver !!!

So now, let's try again submitting a pipeline :

Which gives :
Running with gitlab-runner 11.9.0 (692ae235)
	on Runner on GitNRS1 5Ri6XJzp
Using Shell executor...
Running on gitrns1...
Reinitialized existing Git repository in /home/gitlab-runner/builds/5Ri6XJzp/0/safe/safe_sec_tac/.git/
Fetching changes...
fatal: remote origin already exists.
Clean repository
From https://gitlab.myCompany.tld/safe/safe_sec_tac
	* [new branch]	master	-> origin/master
Checking out 2aed53bb as master...
Skipping Git submodules setup
$ true
Job succeeded
Now, I _just_ have to make it do valuable work rather than running true