Add a CA certificate to GitLab running in a podman container

Adding a CA certificate to GitLab which is running in a podman container (also works with docker containers, just replace podman with docker):

1. Login to the podman container 
2. Copy/install the CA certificates (in this case Digi-Issuing-CA01-G3.pem & Digi-Root-CA01-G3.pem)
3. Restart the podman container

Example

euprdgitlab655:~ #
euprdgitlab655:~ # podman exec -it gitlab /bin/bash
root@ad24f5df0102:/#
root@ad24f5df0102:/#
root@ad24f5df0102:/# ls /etc/gitlab/
gitlab-secrets.json gitlab.rb ssh_host_ecdsa_key ssh_host_ecdsa_key.pub ssh_host_ed25519_key ssh_host_ed25519_key.pub ssh_host_rsa_key ssh_host_rsa_key.pub ssl trusted-certs
root@ad24f5df0102:/#
root@ad24f5df0102:/# ls /etc/gitlab/trusted-certs/
07ac5923.0 Digi-Issuing-CA01-G2.pem Digi-Root-CA-G2.pem e0c0effb.0
root@ad24f5df0102:/#
root@ad24f5df0102:/# ls -lah /etc/gitlab/trusted-certs/
total 8.0K
drwxr-xr-x 2 root root 101 Mar 27 12:44 .
drwxrwxr-x 4 root root 250 Oct 15 2024 ..
lrwxrwxrwx 1 root root 19 Mar 27 12:44 07ac5923.0 -> Digi-Root-CA-G2.pem
-rw-r--r-- 1 root root 2.6K Sep 14 2021 Digi-Issuing-CA01-G2.pem
-rw-r--r-- 1 root root 2.3K Sep 14 2021 Digi-Root-CA-G2.pem
lrwxrwxrwx 1 root root 24 Mar 27 12:44 e0c0effb.0 -> Digi-Issuing-CA01-G2.pem
root@ad24f5df0102:/#
root@ad24f5df0102:/#
root@ad24f5df0102:/# vi /etc/gitlab/trusted-certs/Digi-Root-CA-G3.pem
root@ad24f5df0102:/# vi /etc/gitlab/trusted-certs/Digi-Issuing-CA-G3.pem
root@ad24f5df0102:/#
root@ad24f5df0102:/# ls -lah /etc/gitlab/trusted-certs/
total 16K
drwxr-xr-x 2 root root 158 Jul 14 10:16 .
drwxrwxr-x 4 root root 250 Oct 15 2024 ..
lrwxrwxrwx 1 root root 19 Mar 27 12:44 07ac5923.0 -> Digi-Root-CA-G2.pem
-rw-r--r-- 1 root root 2.3K Jul 14 10:16 Digi-Issuing-CA-G3.pem
-rw-r--r-- 1 root root 2.6K Sep 14 2021 Digi-Issuing-CA01-G2.pem
-rw-r--r-- 1 root root 2.3K Sep 14 2021 Digi-Root-CA-G2.pem
-rw-r--r-- 1 root root 1.9K Jul 14 10:16 Digi-Root-CA-G3.pem
lrwxrwxrwx 1 root root 24 Mar 27 12:44 e0c0effb.0 -> Digi-Issuing-CA01-G2.pem
root@ad24f5df0102:/#
root@ad24f5df0102:/# exit
exit
euprdgitlab655:~ #
euprdgitlab655:~ #
euprdgitlab655:~ #
euprdgitlab655:~ # podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ad24f5df0102 reg.subdomain.domain.tld/gitlab/gitlab-ee:18.0.3-ee.0 /assets/wrapper 3 months ago Up 3 weeks (healthy) 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:2222->22/tcp gitlab
aa22bdf8c33a docker.io/library/nginx:1.27.5 nginx -g daemon o... 3 months ago Up 3 weeks 0.0.0.0:8443->8443/tcp nginx
euprdgitlab655:~ #
euprdgitlab655:~ # podman stop gitlab
WARN[0010] StopSignal SIGTERM failed to stop container gitlab in 10 seconds, resorting to SIGKILL
gitlab
euprdgitlab655:~ # podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ad24f5df0102 reg.subdomain.domain.tld/gitlab/gitlab-ee:18.0.3-ee.0 /assets/wrapper 3 months ago Up 2 seconds (starting) 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:2222->22/tcp gitlab
aa22bdf8c33a docker.io/library/nginx:1.27.5 nginx -g daemon o... 3 months ago Up 3 weeks 0.0.0.0:8443->8443/tcp nginx
euprdgitlab655:~ #
euprdgitlab655:~ #

[...] *wait* [...]

euprdgitlab655:~ #
euprdgitlab655:~ # podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ad24f5df0102 reg.subdomain.domain.tld/gitlab/gitlab-ee:18.0.3-ee.0 /assets/wrapper 3 months ago Up 7 minutes (healthy) 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:2222->22/tcp gitlab
aa22bdf8c33a docker.io/library/nginx:1.27.5 nginx -g daemon o... 3 months ago Up 3 weeks 0.0.0.0:8443->8443/tcp nginx
euprdgitlab655:~ #

Backup GitLab running in a Container and encrypt the backup

Many Gitlab instances run in a docker or podman container. The following is a bash script which 

1. fetches the gitlab-configs and the gitlab-data from inside of the container, 
2. then creates a key (symmetric), 
3. which is again encrypted with a asymmetric key (certificate)
4. and encrypted the gitlab-config & data with that
5. And it deletes old backup data which is older then 7 days

 

gitlab040:/srv/gitlab # cat gitlab-backup.sh
#!/bin/bash
# remove tmp files 
rm -rf /srv/gitlab/tmp

# create tmp directory
mkdir -p /srv/gitlab/tmp

# generate backups
docker exec -t gitlab /bin/sh -c 'umask 0077; tar cfz /secret/gitlab/backups/$(date +"%Y-%m-%d-%H-%M")_config_gitlab_backup.tgz -C / etc/gitlab'
docker exec -t gitlab gitlab-backup create CRON=1 BACKUP=$(date +"%Y-%m-%d-%H-%M")_data

# locate backup files
BACKUPCONFIG=$(ls -Art /srv/gitlab/backup-config/*config_gitlab_backup.tgz | tail -n 1)
BACKUPDATA=$(ls -Art /srv/gitlab/backup-data/*data_gitlab_backup.tar | tail -n 1)

# generate symmetric key
openssl rand -base64 32 > /srv/gitlab/tmp/symmetric_keyfile.key

# encrypt symmetric key with asm-key
openssl rsautl -encrypt -inkey /srv/gitlab/public_key.pem -pubin -in /srv/gitlab/tmp/symmetric_keyfile.key -out /srv/gitlab/tmp/symmetric_keyfile.enc

# generate hashes
sha1sum $BACKUPCONFIG >> /srv/gitlab/tmp/$(basename $BACKUPCONFIG).sha1sum
sha1sum $BACKUPDATA >> /srv/gitlab/tmp/$(basename $BACKUPDATA).sha1sum

# encrypt backup files
openssl enc -in $BACKUPCONFIG -out /srv/gitlab/tmp/$(basename $BACKUPCONFIG).enc -e -aes256 -kfile /srv/gitlab/tmp/symmetric_keyfile.key
openssl enc -in $BACKUPDATA -out /srv/gitlab/tmp/$(basename $BACKUPDATA).enc -e -aes256 -kfile /srv/gitlab/tmp/symmetric_keyfile.key

# archive backup files
cd /srv/gitlab/tmp
tar -cvzf /backup/gitlab-backup-$(date +"%Y-%m-%d-%H-%M").tgz *.enc *.sha1sum 1>/dev/null

# remove tmp files
rm -rf /srv/gitlab/tmp

# delete old config backups
find /srv/gitlab/backup-config -type f -mtime +7 -name '*config_gitlab_backup.tgz' -delete
find /srv/gitlab/backup-data -type f -mtime +7 -name '*data_gitlab_backup.tar' -delete
find /backup -type f -mtime +30 -name 'gitlab-backup-*.tgz' -delete 

Certbot Cloudflare DNS Challenge error --dns-cloudflare-credentials

When certbot is showing you the error "certbot: error: unrecognized arguments: --dns-cloudflare-credentials" you might find the following helpful:

Problem

root@linlxc033:~#
root@linlxc033:~# certbot certonly --dns-cloudflare --dns-cloudflare-credentials /root/tls/certbot/cloudflare-api.key --preferred-challenges dns-01 -d "somedomain.com" -d "*.somedomain.com"
usage:
  certbot [SUBCOMMAND] [options] [-d DOMAIN] [-d DOMAIN] ...

Certbot can obtain and install HTTPS/TLS/SSL certificates.  By default,
it will attempt to use a webserver both for obtaining and installing the
certificate.
certbot: error: unrecognized arguments: --dns-cloudflare-credentials /root/tls/certbot/cloudflare-api.key
root@linlxc033:~#
root@linlxc033:~#


Solution

Install the required package. This is the solution if you have installed python3-certbot as an apt package:

sudo apt -y install python3-certbot-dns-cloudflare

This is the solution if you have installed certbot as a snap:

sudo snap set certbot trust-plugin-with-root=ok
sudo snap install certbot-dns-cloudflare

Example

root@linlxc033:~#
root@linlxc033:~# sudo apt -y install python3-certbot-dns-cloudflare
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
[...]
root@linlxc033:~#
root@linlxc033:~# sudo snap set certbot trust-plugin-with-root=ok
root@linlxc033:~# sudo snap install certbot-dns-cloudflare
certbot-dns-cloudflare 2.8.0 from Certbot Project (certbot-eff✓) installed
root@linlxc033:~#
root@linlxc033:~#
root@linlxc033:~# certbot certonly --dns-cloudflare --dns-cloudflare-credentials /root/tls/certbot/cloudflare-api.key --preferred-challenges dns-01 -d "somedomain.com" -d "*.somedomain.com"
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for somedomain.com and *.somedomain.com   
Unsafe permissions on credentials configuration file: /root/tls/certbot/cloudflare-api.key
Waiting 10 seconds for DNS changes to propagate

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/somedomain.com/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/somedomain.com/privkey.pem
This certificate expires on 2024-08-04.
These files will be updated when the certificate renews.
Certbot has set up a scheduled task to automatically renew this certificate in the background.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you like Certbot, please consider supporting our work by:
 * Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
 * Donating to EFF:                    https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
root@linlxc033:~#
root@linlxc033:~#


Overview of public interfaces for SOC/IT-Security staff

In case of an IT-security incident, emergency oder if a new critical vulnerability (like log4j in December 2021) arises, it is good to be prepared, so you can quickly answer questions like:

• "Are we affected?"
• "Do we use this technology?"
• "Where do we use this vulnerable protocol?"
  
• "To whom is the attack surface exposed to?"
• "Are there mitigations in place?"
• "Is is exploitable without authentication in our setup?"
• "Which is the best place to place a first mitigation?"
  
• etc..

An overview like the following can and will be helpful for your IT-security staff or your Security Operations Center SOC:

System	Internet Facing	Protocol	Authentication	Security	Used Products/Vendors	Logs send to SIEM	Contact Person	Known Weaknesses
Websites	Yes, exposed to all public-ip-addresses	HTTPS (TCP:443) & HTTP (TCP:80 - HTTP 301 Redirect to HTTPS)	None	Web Application Firewall	F5 BigIP LoadBalancer WAF & Apache Container on OpenShift	Yes	Link to CMDB	Websites may contain 3rd party code, SBOM see CMDB
Managed File Transfer	Yes, but limited to dedicated public ip-addresses of partners	HTTPS (TCP:443)	HTTPS Tokens	Web Application Firewall	F5 BigIP LoadBalancer WAF IPSwitch	Yes	Link to CMDB	Runs on VM as appliance, OS might not be hardend from vendor
Citrix	Yes, exposed to all public-ip-addresses	HTTPS (TCP:443)	MFA	Netscaler WAF	Citrix Systems + Okta MFA	Yes	Link to CMDB	NetScaler WAF Ruleset might be out-of-date
Mailserver	Yes, exposed to all public-ip-addresses	SMTP (TCP:25)	None	AntiSpam Mailgatway & AV-Sandbox	Cisco E-Mail Security	Yes	Link to CMDB	Mailgateways run on Hardware, might not be hardended from vendor
SSLVPN S2E	Yes, exposed to all public-ip-addresses	HTTPS (TCP:443)	Mutual TLS Certbased + MFA	Azure DDoS	FortiGate SSLVPN Azure VM + Okta MFA	Yes	Link to CMDB	Possible FortiGate FortiOS SSLVPN Vulnerabilities
M365 ActiveSync	Yes, exposed to all public-ip-addresses	HTTPS (TCP:443)	Mutual TLS Certbased	Azure DDoS	Microsoft 365 + Intunes	Yes	Link to CMDB	Not part of own Vulnerability-Scanner
VPN S2S	Yes, but limited to dedicated public ip-addresses of partners	IPSec UDP:500 & UDP:4500 & ESP	IPsec IKEv2 Certbased Auth	Azure DDoS	FortiGate SSLVPN Azure VM		Link to CMDB	-
DMARC SaaS	Yes, exposed to all public-ip-addresses	DNS (UDP:53), HTTP (TCP:80), HTTPS (TCP:443), SMTP (TCP:25)	None	-	dmarcadvisor.com SaaS	No	Link to CMDB	Not part of own Vulnerability-Scanner
DNS Server	Yes, but limited to dedicated public ip-addresses of partners	DNS (UDP:53 & TCP:53)	None	Azure Network Security Groups	RHEL Bind	Yes	Link to CMDB	-
ISP Routers	Yes, but limited to dedicated public ip-addresses of ISP routers	BGP (TCP:179), BFD, Ping (ICMP:0/8)	BGP MD5 Auth	-	Extreme Networks XOS	Yes	Link to CMDB
etc..	etc..	etc..	etc..	etc..	etc..	etc..	etc..	etc..

 

Of course you can add many more columns like e.g.:

• "SBOM technologys used" (for example: RHEL, Apache Tomcat, OpenSSL, log4j, puppet, ansible, splunk universal forwarder, appdynamics,..)
• Direct links to your Firewall Management System, WAF or SIEM
• "Is it part of our vulnerability scanner?"
• "Is the vulnerability scanner scanning it authenticated?"
• "Is the system/application hardended?"
• and so on :-)

This list will help in case of an IT-security emergency to sort out the first steps in order to mitigate and fix the issue of the public exposed interfaces (like to the internet or to business partners). However this is only one of many steps necessary - always "asume breach" and make sure an attacker controlling a client or server still is unable to spread (unnoticed) in your companies (cloud) network.

Git for Windows - Git uses certificate from internal PKI

If you initially want to git clone a repository from an internal git system from a windows system and the internal git system uses a certificate from a internal PKI, you'll receive an error: SSL certificate problem: self signed certificate in certificate chain

PS C:\Users\flo\Splunk>
PS C:\Users\flo\Splunk> git clone https://git.domain.tld/ansible-playbooks/carbonblack-threatfeeds.git
Cloning into 'carbonblack-threatfeeds'...
fatal: unable to access 'https://git.domain.tld/ansible-playbooks/carbonblack-threatfeeds.git/': SSL certificate problem: self signed certificate in certificate chain
PS C:\Users\flo\Splunk>
PS C:\Users\flo\Splunk>

Check your windows git settings with git config -l. In the attribute http.sslcainfo is the ca-cert-bundle-file, which is used by git as certification authority certificate store.

PS C:\Users\flo\Splunk>
PS C:\Users\flo\Splunk> git config -l
diff.astextplain.textconv=astextplain
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
http.sslbackend=openssl
http.sslcainfo=C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt
core.autocrlf=true
core.fscache=true
core.symlinks=false
pull.rebase=false
credential.helper=manager-core
credential.https://dev.azure.com.usehttppath=true
init.defaultbranch=master
user.email=flo@some.mail
user.name=flo
core.editor="C:\Users\flo\AppData\Local\Programs\Microsoft VS Code\Code.exe" --wait
PS C:\Users\flo\Splunk>
PS C:\Users\flo\Splunk>

So append your CA certificate in the file "C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt". Then it works:

PS C:\Users\flo\Splunk>
PS C:\Users\flo\Splunk> git clone https://git.domain.tld/ansible-playbooks/carbonblack-threatfeeds.git
Cloning into 'carbonblack-threatfeeds'...
remote: Enumerating objects: 104, done.
remote: Counting objects: 100% (104/104), done.
remote: Compressing objects: 100% (58/58), done.
remote: Total 104 (delta 50), reused 93 (delta 40), pack-reused 0 eceiving objects:  34% (36/104)
Receiving objects: 100% (104/104), 20.03 KiB | 2.86 MiB/s, done.
Resolving deltas: 100% (50/50), done.
PS C:\Users\flo\Splunk>
PS C:\Users\flo\Splunk>


List FortiGate Certificates via CLI - CA certificates and local Certificates

You can either use the GUI of the FortiGate to list all certificates, or use the CLI. Either using the commands:

Using the "get" command

config vdom
edit root   #<--- your management vdom/your vdom of choice
get vpn certificate ca

FGT50E00000000 (root) #
FGT50E00000000 (root) # get vpn certificate ca
== [ Fortinet_Wifi_CA ]
name: Fortinet_Wifi_CA
== [ Fortinet_CA ]
name: Fortinet_CA
== [ ACCVRAIZ1 ]
name: ACCVRAIZ1
== [ AC_RAIZ_FNMT-RCM ]
name: AC_RAIZ_FNMT-RCM
== [ Actalis_Authentication_Root_CA ]
name: Actalis_Authentication_Root_CA
[...]

Using the "show" command

The show command might not be very helpful, because it does not necessarily show all certificates:

FGT50E00000000 (root) #
FGT50E00000000 (root) # show vpn certificate ca
config vpn certificate ca
end
FGT50E00000000 (root) # show full-configuration vpn certificate ca
config vpn certificate caend

FGT50E00000000 (root) # show full-configuration | grep -f 'vpn certificate ca'
config vpn certificate ca <---
end
FGT50E00000000 (root) #

Using the "fnsysctl" command

Using the fnsysctl command might be helpful:

FGT50E00000000  #
FGT50E00000000 # fnsysctl ls -la /etc/cert/local/
drwxr-xr-x    2 0    0   Wed Dec 25 21:43:14 2019        0 .
drwxr-xr-x    6 0    0   Wed Sep 18 20:39:27 2019        0 ..
-rw-------    1 0    0   Wed Sep 18 20:35:46 2019     2250 root_2020jan_sub.domain.tld.cer
-rw-------    1 0    0   Wed Sep 18 20:35:46 2019     1704 KEY-FILE
-rw-------    1 0    0   Wed Sep 18 20:35:46 2019     1407 root_Fortinet_CA_SSL.cer
-rw-------    1 0    0   Wed Sep 18 20:35:47 2019     1704 KEY-FILE
-rw-------    1 0    0   Wed Sep 18 20:35:47 2019     1419 root_Fortinet_CA_Untrusted.cer
-rw-------    1 0    0   Wed Sep 18 20:35:47 2019     1704 KEY-FILE
-rw-------    1 0    0   Wed Sep 18 20:35:47 2019     4285 root_Fortinet_Factory.cer
-rw-------    1 0    0   Wed Sep 18 20:35:47 2019     1679 KEY-FILE[...]
FGT50E00000000  #
FGT50E00000000  # fnsysctl ls -la /etc/cert/ca
drwxr-xr-x    2 0    0   Wed Dec 25 21:41:28 2019        0 .
drwxr-xr-x    6 0    0   Wed Sep 18 20:39:27 2019        0 ..
-rw-------    1 0    0   Wed Sep 18 20:35:55 2019      119 ca_bundle_ver
-rw-------    1 0    0   Tue Jan 14 20:06:15 2020 1972 root_AC_RAIZ_FNMT-RCM.cer
-rw-------    1 0    0   Tue Jan 14 20:06:15 2020 2772 root_ACCVRAIZ1.cer
-rw-------    1 0    0   Wed Sep 18 20:35:55 2019     2041 root_ACEDICOM_Root.cer
-rw-------    1 0    0   Tue Jan 14 20:06:15 2020 2049 root_Actalis_Authentication_Root_CA.cer
-rw-------    1 0    0   Tue Jan 14 20:06:14 2020 1521 root_AddTrust_External_Root.cer[...]