GitLab - A full DevOps tool

Sample .gitlab-ci.yml (source) :

stages:							all the stages this pipeline has, listed in execution order
  - build
  - test
  - deploy

job1:
  stage: build						which stage this job runs in
  script:
    - echo "This job compiles code."

job2:
  stage: test						which stage this job runs in
  script:
    - echo "This job tests the compiled code. It runs when the build stage completes."

job3:							when a job defines no stage, it defaults to test
  script:
    - echo "This job also runs in the test stage."

job4:
  stage: deploy						which stage this job runs in
  script:
    - echo "This job deploys the code. It runs when the test stage completes."
  environment: production

• per_page
• page

Discover the pipelines API :

List projects and get the project ID :

List project pipelines :

Delete pipelines :

nbPipelinesToKeep=5; projectId=123456; myToken='ZsyqJk1Rz6ZQ3EjyyEQL'; serverName='gitlab.myCompany.tld'
for pipelineId in $(curl -s --header "PRIVATE-TOKEN: $myToken" "https://$serverName/api/v4/projects/$projectId/pipelines?per_page=1000&page=1" | jq -r '.[] | .id' | tail -n +$((nbPipelinesToKeep+1))); do
	echo -n "deleting pipeline #$pipelineId ... "
	curl -s --header "PRIVATE-TOKEN: $myToken" --request "DELETE" "https://$serverName/api/v4/projects/$projectId/pipelines/$pipelineId" && echo '👍' || echo '🚩'
done

.gitlab-ci.yml

• YAML file holding GitLab CI / CD pipelines configuration

jobs

• are made of a list of commands and define what to do (compile code, run tests, )
• are written in .gitlab-ci.yml
• are executed by runners
  • jobs in the same stage can execute in parallel if they run on different runners
  • if you have only one runner, jobs can run in parallel if the runner's concurrent setting is greater than 1
• There are manual jobs : jobs that can only be launched by a user action.

personal access token

ID to authenticate with the GitLab API. Get yours.

pipeline

• top-level component of CI / CD
• Pipelines have
  1. jobs
  2. stages

rules (rules)

• are conditions to include/exclude jobs to/from pipelines (i.e. run those jobs or not)
• are evaluated
  • when the pipeline is created
  • in order
  • when a match is found
    • the corresponding job is included to / excluded from the pipeline
    • no more rules are checked
    if none of the rules matches, the job is not added to the pipeline

runner (https://docs.gitlab.com/ci/runners/, https://docs.gitlab.com/runner/)

• isolated + ephemeral VM / cloud instance that executes jobs
• each runner, as it becomes available, continually sends requests to the GitLab instance, asking to be assigned jobs

stage (vegastack.com, medium.com)

.gitlab-ci.yml has stages and stage keywords (see sample .gitlab-ci.yml)

• define when to run jobs : conditional execution of groups of tasks. For example :
  1. compile code
  2. if ok, run tests
• stages / jobs :
  • jobs in the same stage run in parallel
  • jobs in the next stage run after the jobs from the previous stage complete successfully
• stages defaults to

  .pre
  build
  test
  deploy
  .post

https://docs.gitlab.com/ci/

==========================================8<=========================================================
apt upgrade gitlab-ce
...
...
     _______ __  __          __
    / ____(_) /_/ /   ____ _/ /_
   / / __/ / __/ /   / __ `/ __ \
  / /_/ / / /_/ /___/ /_/ / /_/ /
  \____/_/\__/_____/\__,_/_.___/


Upgrade complete! If your GitLab server is misbehaving try running
	sudo gitlab-ctl restart
==========================================8<=========================================================

Create the .gitlab-ci.yml :
simply create it at the root of the repository
add + commit +push it like any other file



Check the .gitlab-ci.yml :

	
https://10.2.0.97/safe/safe_sec_tac/-/ci/lint
	
Validate


==========================================8<=========================================================

https://10.2.0.97/help/ci/quick_start/README


install 'gitlab-runner' :	https://docs.gitlab.com/runner/install/linux-repository.html


Configuring a Runner


gitlab-runner register
	Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
		https://10.2.0.97/
	Please enter the gitlab-ci token for this runner:
		N6sAJckKFr1g6fFPXed3
	Please enter the gitlab-ci description for this runner:
		[gitrns1]: Runner on GitNRS1
	Please enter the gitlab-ci tags for this runner (comma separated):
		local,gitnrs1


ERROR: Registering runner... failed		runner=N6sAJckK status=couldn't execute POST against https://10.2.0.97/api/v4/runners: Post https://10.2.0.97/api/v4/runners: x509: certificate has expired or is not yet valid
PANIC: Failed to register this runner. Perhaps you are having network problems


==========================================8<=========================================================
Let's have a look at the certificate :

openssl s_client -connect 10.2.0.97:443 | less

Verification error: certificate has expired

Verify return code: 10 (certificate has expired)



ss -punta | grep 443
	tcp	LISTEN	0	511	*:443				*:*				users:(("nginx",pid=28588,fd=7),("nginx",pid=28587,fd=7))


find / -name '*nginx.conf'
	/var/opt/gitlab/nginx/conf/nginx.conf
	/opt/gitlab/embedded/conf/nginx.conf
	/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/unicorn-5.4.1/examples/nginx.conf


/opt/gitlab/embedded/conf/nginx.conf
	==> example ? really used ?
/var/opt/gitlab/nginx/conf/nginx.conf
	# This file is managed by gitlab-ctl. Manual changes will be
	# erased! To change the contents below, edit /etc/gitlab/gitlab.rb
	# and run `sudo gitlab-ctl reconfigure`.

grep include /var/opt/gitlab/nginx/conf/nginx.conf
	include /opt/gitlab/embedded/conf/mime.types;
	include /var/opt/gitlab/nginx/conf/gitlab-http.conf;
	include /var/opt/gitlab/nginx/conf/nginx-status.conf;


less /var/opt/gitlab/nginx/conf/gitlab-http.conf
# This file is managed by gitlab-ctl. Manual changes will be
# erased! To change the contents below, edit /etc/gitlab/gitlab.rb
# and run `sudo gitlab-ctl reconfigure`.

server {
	listen *:443 ssl http2;

	server_name gitlab.myCompany.tld;
	
	ssl on;
	ssl_certificate /etc/gitlab/ssl/gitlab.myCompany.tld.crt;
	ssl_certificate_key /etc/gitlab/ssl/gitlab.myCompany.tld.key;



openssl x509 -noout -in /etc/gitlab/ssl/gitlab.myCompany.tld.crt -dates
	notBefore=Dec  6 15:32:09 2018 GMT
	notAfter=Jan  5 15:32:09 2019 GMT


openssl x509 -noout -in /etc/gitlab/ssl/gitlab.myCompany.tld.crt -text | grep -E '(Issuer|Subject):'
	Issuer: CN = gitlab.myCompany.tld
	Subject: CN = gitlab.myCompany.tld


cd /run/shm/; export CN='gitlab.myCompany.tld'; openssl genrsa -out "$CN".key 2048; openssl req -new -x509 -days 365 -sha256 -key "$CN".key -out "$CN".crt -subj "/CN=$CN"

openssl x509 -noout -in "$CN".crt -text | grep -E '(Issuer|Subject):'
	Issuer: CN = gitlab.myCompany.tld
	Subject: CN = gitlab.myCompany.tld

export CN='gitlab.myCompany.tld'; cd /etc/gitlab/ssl; for stuff in "$CN.crt" "$CN.key"; do mv "$stuff" "${stuff}_OLD"; mv "/run/shm/$stuff" .; done


sudo gitlab-ctl restart
==========================================8<=========================================================
Now that the certificate is ok, let's retry to register our runner :


gitlab-runner register
	Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
		https://10.2.0.97/
	Please enter the gitlab-ci token for this runner:
		N6sAJckKFr1g6fFPXed3
	Please enter the gitlab-ci description for this runner:
		[gitrns1]: Runner on GitNRS1
	Please enter the gitlab-ci tags for this runner (comma separated):
		local,gitnrs1

ERROR: Registering runner... failed		runner=N6sAJckK status=couldn't execute POST against https://10.2.0.97/api/v4/runners: Post https://10.2.0.97/api/v4/runners: x509: cannot validate certificate for 10.2.0.97 because it doesn't contain any IP SANs
PANIC: Failed to register this runner. Perhaps you are having network problems


==========================================8<=========================================================

https://geekflare.com/san-ssl-certificate/

export CN='gitlab.myCompany.tld'; workDir='/run/shm'; customConfigFile="$workDir/customConfigFile"; cd "$workDir"; cat <<EOF >"$customConfigFile"
[ req ]
default_bits       = 2048
distinguished_name = req_distinguished_name
req_extensions     = req_ext
[ req_distinguished_name ]
#countryName                 = Country Name (2 letter code)
countryName=FR
#stateOrProvinceName         = State or Province Name (full name)
#localityName               = Locality Name (eg, city)
localityName=Nantes
#organizationName           = Organization Name (eg, company)
organizationName=ACME Corp
commonName=$CN
[ req_ext ]
subjectAltName = @alt_names
[alt_names]
DNS.1   = foo.com
DNS.2   = bar.com
DNS.3   = baz.com
EOF


make the CA key :
	caKey="$workDir/ca.key"; cd "$workDir" && openssl genrsa -des3 -out "$caKey" 4096
	plop


make the CA cert :
	caCert="$workDir/ca.crt";cd "$workDir" && openssl req -config "$customConfigFile" -new -x509 -nodes -sha1 -days 1825 -key "$caKey" -out "$caCert"


make cert key
	openssl genrsa -des3 -out "$CN.key" 4096
	plop

make CSR :
	openssl req -config "$customConfigFile" -new -key "$CN.key" -out "$CN.csr"

check the SAN in CSR :
	openssl req -noout -text -in "$CN".csr | grep DNS
		==> DNS:foo.com, DNS:bar.com, DNS:baz.com
		\o/


openssl ca -config "$customConfigFile" -policy policy_anything -out "$CN.crt" -infiles "$CN.csr"

Using configuration from /run/shm/customConfigFile
variable lookup failed for ca::default_ca
139719445565504:error:0E06D06C:configuration file routines:NCONF_get_string:no value:../crypto/conf/conf_lib.c:275:group=ca name=default_ca
	==> this is because there is no "default_ca" in the config file

==========================================8<=========================================================

retrying :

export CN='gitlab.myCompany.tld'; workDir='/run/shm'; customConfigFile="$workDir/customConfigFile"; cd "$workDir"; cp '/usr/lib/ssl/openssl.cnf' "$customConfigFile"; cat <<EOF >>"$customConfigFile"
[ req ]
default_bits       = 2048
distinguished_name = req_distinguished_name
req_extensions     = req_ext
[ req_distinguished_name ]
countryName=FR
localityName=Nantes
organizationName=ACME Corp
commonName=$CN
[ req_ext ]
subjectAltName = @alt_names
[alt_names]
DNS.1   = foo.com
DNS.2   = bar.com
DNS.3   = baz.com
EOF

Using configuration from /run/shm/customConfigFile
Can't open ./demoCA/private/cakey.pem for reading, No such file or directory
139855953907776:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:74:fopen('./demoCA/private/cakey.pem','r')
139855953907776:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:81:
unable to load CA private key


==========================================8<=========================================================

retrying :

export CN='gitlab.myCompany.tld'; workDir='/run/shm/cert'; customConfigFile="$workDir/customConfigFile"; caKey="$workDir/ca.key"; caCert="$workDir/ca.crt"; mkdir -p "$workDir" && cd "$workDir"; touch index.txt; echo 01 > serial; cp '/usr/lib/ssl/openssl.cnf' "$customConfigFile"; for keyword in default_bits distinguished_name req_extensions countryName localityName organizationName commonName subjectAltName; do sed -ri 's|^'$keyword'|#'$keyword'|g' "$customConfigFile"; done; cat <<EOF >>"$customConfigFile"
[ req ]
default_bits       = 2048
distinguished_name = req_distinguished_name
req_extensions     = req_ext
[ req_distinguished_name ]
countryName = FR
localityName = Nantes
organizationName = ACME Corp
commonName = $CN
[ req_ext ]
subjectAltName = @alt_names
[alt_names]
DNS.1   = foo.com
DNS.2   = bar.com
DNS.3   = baz.com
IP      = $(hostname -i)
EOF


sed -ri -e 's|\./demoCA|.|' -e 's|\$dir/private/cakey\.pem|'$caKey'|' -e 's|\$dir/cacert\.pem|'$caCert'|' -e 's|\$dir/newcerts|.|' "$customConfigFile"


make the CA key :
	openssl genrsa -des3 -out "$caKey" 4096
	plop


make the CA cert :
	openssl req -config "$customConfigFile" -new -x509 -nodes -sha1 -days 1825 -key "$caKey" -out "$caCert"
	plop

make cert key
	openssl genrsa -des3 -out "$CN.key" 4096
	test

make CSR :
	openssl req -config "$customConfigFile" -new -key "$CN.key" -out "$CN.csr"
	test

check the SAN in CSR :
	openssl req -noout -text -in "$CN".csr | grep -E 'DNS|IP'
		==> DNS:foo.com, DNS:bar.com, DNS:baz.com
		==> DNS:foo.com, DNS:bar.com, DNS:baz.com, IP Address:10.2.0.97
		\o/

sign the certificate :
	openssl ca -config "$customConfigFile" -policy policy_anything -out "$CN.crt" -infiles "$CN.csr"
		plop
		==> \o/

remove the password from the certificate private key file
	mv "$CN.key" "$CN.withPassword.key" && openssl rsa -in "$CN.withPassword.key" -out "$CN.key"
	test

cd /etc/gitlab/ssl && for stuff in "$CN.crt" "$CN.key"; do mv "$stuff" "${stuff}_OLD"; cp "$workDir/$stuff" .; done


sudo gitlab-ctl restart
	==> KO, still the same error about the certificate having no SAN information :-(((

==========================================8<=========================================================

with makeCert.sh (inspired by)

CN='gitlab.myCompany.tld'; workDir='/run/shm/certificate'; cd /etc/gitlab/ssl && for stuff in "$CN.crt" "$CN.key"; do mv "$stuff" "${stuff}_OLD"; cp "$workDir/$stuff" .; done

sudo gitlab-ctl restart
	==> the certificate is OK

retry registering the runner :

ERROR: Registering runner... failed		runner=N6sAJckK status=couldn't execute POST against https://10.2.0.97/api/v4/runners: Post https://10.2.0.97/api/v4/runners: x509: certificate signed by unknown authority
PANIC: Failed to register this runner. Perhaps you are having network problems

==========================================8<=========================================================
https://stackoverflow.com/questions/44458410/gitlab-ci-runner-ignore-self-signed-certificate#answer-48347895

sudo gitlab-runner register --tls-ca-file /my/path/gitlab/gitlab.myserver.com.pem

gitlab-runner register --help | grep 'tls'
	--tls-ca-file value		File containing the certificates to verify the peer when using HTTPS [$CI_SERVER_TLS_CA_FILE]
	--tls-cert-file value	File containing certificate for TLS client auth when using HTTPS [$CI_SERVER_TLS_CERT_FILE]
	--tls-key-file value	File containing private key for TLS client auth when using HTTPS [$CI_SERVER_TLS_KEY_FILE]


use + deploy the 'makeCert.sh' seen above



CN='gitlab.myCompany.tld'; workDir='/run/shm/certificate'; cd "$workDir"; cat "$CN".crt "$CN".key > "$CN".pem
sudo gitlab-runner register --tls-ca-file "$workDir/$CN.pem"


https://docs.gitlab.com/runner/commands/#interactive-registration
gitlab-runner register
Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
https://10.2.0.97/
Please enter the gitlab-ci token for this runner:
N6sAJckKFr1g6fFPXed3
Please enter the gitlab-ci description for this runner:
[gitrns1]: Runner on GitNRS1
Please enter the gitlab-ci tags for this runner (comma separated):
local,gitnrs1
Registering runner... succeeded				runner=N6sAJckK
Please enter the executor: docker, docker-ssh, parallels, docker-ssh+machine, shell, ssh, virtualbox, docker+machine, kubernetes:
shell
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

==========================================8<=========================================================
In the WebUI : Settings | CI/CD | Runners [Expand]

	==> my shared runner is here \o/


See also : Admin Area (wrench icon) | Overview | Runners

enter 'Edit' mode to 'Restrict projects for this Runner'



https://docs.gitlab.com/runner/commands/README.html
gitlab-runner --help
gitlab-runner list
	Runtime platform				arch=amd64 os=linux pid=23440 revision=692ae235 version=11.9.0
	Listing configured runners		ConfigFile=/etc/gitlab-runner/config.toml
	Runner on GitNRS1				Executor=shell Token=5Ri6XJzpUXXJ61Ugx2nX URL=https://10.2.0.97/

https://docs.gitlab.com/runner/commands/#gitlab-runner-verify
gitlab-runner verify
	Runtime platform				arch=amd64 os=linux pid=24359 revision=692ae235 version=11.9.0
	Running in system-mode.
	Verifying runner... is alive	runner=5Ri6XJzp

gitlab-runner --debug run

https://docs.gitlab.com/runner/commands/#gitlab-runner-start
gitlab-runner start
	Runtime platform				arch=amd64 os=linux pid=24891 revision=692ae235 version=11.9.0

gitlab-runner verify
	Runtime platform				arch=amd64 os=linux pid=25070 revision=692ae235 version=11.9.0
	Running in system-mode.

	Verifying runner... is alive	runner=5Ri6XJzp



https://10.2.0.97/help/ci/runners/README.md#allowing-runners-with-tags-to-pick-jobs-without-tags
If pipeline is still "stuck", check the "Run untagged jobs" box in runner configuration

==========================================8<=========================================================
In the WebUI : CI/CD | Jobs | click on any "Failed" tag :
Running with gitlab-runner 11.9.0 (692ae235)
	on Runner on GitNRS1 5Ri6XJzp
Using Shell executor...
Running on gitrns1...
Reinitialized existing Git repository in /home/gitlab-runner/builds/5Ri6XJzp/0/safe/safe_sec_tac/.git/
Fetching changes...
fatal: remote origin already exists.
Clean repository
fatal: unable to access 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@gitlab.myCompany.tld/safe/safe_sec_tac.git/': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
ERROR: Job failed: exit status 1



curl -i https://gitlab.myCompany.tld
	curl: (60) SSL certificate problem: unable to get local issuer certificate
	...

curl -ki https://gitlab.myCompany.tld
	HTTP/2 302
	server: nginx
	date: Thu, 11 Apr 2019 10:30:15 GMT
	content-type: text/html; charset=utf-8
	content-length: 107
	location: https://gitlab.myCompany.tld/users/sign_in
	...

Need to install my "own CA" root certificate (source) :

1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.

lrwxrwxrwx 1 root root   50 Apr 11 12:41 ca-cert.pem -> /usr/local/share/ca-certificates/extra/ca-cert.crt	mine  !!!

HTTP/2 302		Just noticed this is serving HTTP 2
server: nginx
date: Fri, 12 Apr 2019 12:51:10 GMT
content-type: text/html; charset=utf-8
content-length: 97

So now, let's try again submitting a pipeline :

Running with gitlab-runner 11.9.0 (692ae235)
	on Runner on GitNRS1 5Ri6XJzp
Using Shell executor...
Running on gitrns1...
Reinitialized existing Git repository in /home/gitlab-runner/builds/5Ri6XJzp/0/safe/safe_sec_tac/.git/
Fetching changes...
fatal: remote origin already exists.
Clean repository
From https://gitlab.myCompany.tld/safe/safe_sec_tac
	* [new branch]	master	-> origin/master
Checking out 2aed53bb as master...
Skipping Git submodules setup
$ true
Job succeeded