Fix blocked ldap user in GitLab container using GitLabs shell

If you are running GitLab in a docker container and your are using some directory service, for example ActiveDirectory with LDAPS for authentication, you might face the challenge, that when a user is moved in ActiveDirectory to another ad-group or the ad-group which is used as user-filter is deleted, then GitLab marks the user as "blocked".

Unblock the ldap user in GitLab

1. Connect to the docker host server
2. Open a connection to GitLabs Shell using docker exec -it <container-name> gitlab-rails console -e production
3. Find the user in GitLabs Shell using user = User.find_by_email("someone@e-mail")
4. Check the Users state using user.state
5. Unblock the user using user.state = "active"
6. Save using user.save
7. Exit

Example:

prdrhel8180:/ #
prdrhel8180:/ # docker exec -it gitlab gitlab-rails console -e production
--------------------------------------------------------------------------------
Ruby: ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
GitLab: 15.3.1-ee (518311979e3) EE
GitLab Shell: 14.10.0
PostgreSQL: 12.10
------------------------------------------------------------[ booted in 37.73s ]
Loading production environment (Rails 6.1.6.1)
irb(main):001:0> user = User.find_by_email("someone@e-mail")
=> nil
irb(main):002:0> user = User.find_by_email("someone@e-mail.com")
=> #<User id:55 @someone>
irb(main):003:0> user.state
=> "ldap_blocked"
irb(main):004:0> user.state = "active"
=> "active"
irb(main):005:0> user.save
=> true
irb(main):006:0> exit
prdrhel8180:/ #
prdrhel8180:/ #

Fix the LDAP user filter

If the user was blocked due to a deleted AD group, which was used as ldap user filter, then you have to fix the LDAP connect from GitLab to ActiveDirectory. GitLab will log this in /var/log/gitlab/gitlab-rails/application.log as:

2023-02-02T01:30:18.098Z: LDAP account "cn=lastname\, firstname,ou=deleted-users,ou=someou,dc=internal,dc=domain,dc=local" does not exist anymore, blocking GitLab user "Lastname, Firstname" (firstname.lastname@domain.local)

prdrhel8180:/ #
prdrhel8180:/ # docker exec -it gitlab cat /etc/gitlab/gitlab.rbgitlab_rails
gitlab_rails['ldap_servers'] = YAML.load <<- br=""> someldap: #
label: 'LDAP'
host: 'some-ldaps-vip.internal.domain.local'
port: 636
uid: 'sAMAccountName'
[...]
user_filter: '(|(memberOf=CN=SomeGroup,OU=Groups,OU=SomeOU,DC=internal,DC=domain,DC=local)(memberOf=CN=SomeGroup2,OU=Groups2,OU=SomeOU2,DC=internal,DC=domain,DC=local))'
prdrhel8180:/ #
prdrhel8180:/ #

The user_filter has to be adjusted to the new AD group, which includes the blocked user(s).

New LAPS version explained

Microsoft will release a new version of Local Administrator Password Solution (LAPS), which   provides new Azure AD features as well as new Active Directory OnPrem features and some migration features from the old version to the new one.

A video explaining everything in detail can be found here:

This video includes a nice overview showing how LAPS is working internally using CSP (lapscsp.dll), PowerShell (lapspsh.dll) or GPOs and LAPS core logic (laps.dll) which then reads and updates the expiry of accounts as well as updates their password, either in Azure Active Directory or in Windows Server Active Directory on premise:

Source: https://learn.microsoft.com/en-us/windows-server/identity/laps/laps-concepts

LAPS can be used as solution against pass-the-hash (https://attack.mitre.org/techniques/T1550/002) and lateral-traversal attacks (https://attack.mitre.org/tactics/TA0008), as well as for securing user help desk access or recover to devices with a fine-grained security model and for RBAC in Azure AD.

 

Security Testing - Review LAPS

It is a good idea to review your companys Microsoft's Local Administrator Password Solution LAPS installation. Leo Loobeek published a nice powershell-script which helps in finding groups which are specifically delegated by sysadmins and finding users with "All Extended Rights" that can view passwords, and viewing all computers with LAPS enabled.

https://github.com/leoloobeek/LAPSToolkit
https://github.com/leoloobeek/LAPSToolkit/blob/master/LAPSToolkit.ps1

• Get-LAPSComputers
  Displays all computers with LAPS enabled, password expriation, and password if user has access
  
  
• Find-LAPSDelegatedGroups
  Searches through all OUs to see which AD groups can read the ms-Mcs-AdmPwd attribute
  
  
• Find-AdmPwdExtendedRights
  Parses through ExtendedRights for each AD computer with LAPS enabled and looks for which group has read access and if any user has "All Extended Rights". Sysadmins may not be aware the users with All Extended Rights can view passwords and may be less protected than the users in the delegated groups. An example is the user which adds a computer to the domain automatically receives the "All Extended Rights" permission. Since this function will parse ACLs for each AD computer, this can take very long with a larger domain.

I've seen installations, where LAPS was implemented to improve security. However due to wrong configuration way to many people where able to read passwords in Active Directory (Mitre Att&ck Tactic Credential Access)

Quick win securing Azure AD

An easy quick win for securing Azure Active Directory passwords is the feature "Azure AD Password Protection". This helps you in mitre att&cks tactic credential access, for example in the technique brute force and its sub-techniques password guessing, password spraying, credential stuffing, etc.

Users are recommended to avoide simple passwords and instead should use pass-sentences. Password breaches of the recent past reveal that the majority still chose simple passwords. Azure AD Password Protection finds (audit mode) or enforces (enforce mode) stronger passwords for everybody. There is a hidden global banned password list which is applied to every user in the Azure AD tenant. Additionally you can block custom words like your companys name, your companys slogan, the founders or CEOs name, most used childerens names or your country or famous sport team names, which are often used as weak passwords.

Microsoft promises the "password validation algorithm" automatically detects/blocks variants and combinations like "password!1", "!password", "p@ssw0rd" and so on.

This is also available for on-premise Active Directory using an agent:

 

However this needs an additional license, you need to install and agent on your domain controllers, you need to reboot the domain controllers and you need Azure AD. Also you do not see the changing content of Microsofts global password list and there is no enforcement based on Active Directory groups or OUs, so you for example cant just enforce it to priviledged accounts but must enforce it for everybody.

Of course I highly recommend to use Multi Factor Authentication MFA everywhere.

How to increase IT security of a company using quick wins

How to increase my companys IT security? Of course there are many, many, many topics, processes, systems, parameters, awareness and a lot more to implement, adjust, train, improve or get rid of. Following the different available frameworks like MITREs Att&ck, the recommendations from NIST or BSI etc will get you there. However they require a lot of time and some companys want to implement some 'quick wins', before they do they adopt a whole framework (which they should).

So some of those 'quick wins' are:

• Implement Multi-Factor-Auth or 2FA. Trying to balance security with comfort will let you win your users, for example by implementing MFA or 2FA using PushTokens. 
• Raising awareness by regularly sending internal phishing mails will make them learn and understand, not to open or click on everything. 
• Implementing zero trust or microsegmentation will make lateral movement hard. 
• Regularly scanning your whole environment for vulnerabilities and configurations issues with regular patching and improving hardening will dramatically reduce your attack surface.
• Restrict administrative permissions, regularly checking if they are still necessary, implementing JIT and securing your directory services using special jump servers with MFA/2FA will help further. Also don't use Microsofts AD Built-in default groups because very often the have to many unnecessary permissions

Some more are described in an article from Microsoft security blog. I personally don't agree on all of them, but it is a possible approach:

(Picture from Microsoft security blog)

 

 

Keeping in mind the point of view from an attacker might help:

 (Picture from Microsoft security blog)

Account Lockout Policy - A possible threat

Most companys use an account lockout policy for their directory service like Microsoft Active Directory, LDAP-system, eDirectory or their own environment solution. Locking out accounts is helpful when it comes to password guessing attacks like brute force (https://attack.mitre.org/techniques/T1110/) or similar ones.

However: An attacker is able to use lockout policies, too.

The following scenario is not so old, but was already used in the last two years:
1. Attacker gains access to environment
2. Attacker creates its own administrative accounts
3. Attacker brute forces all other administrative accounts
👉This way, the attacker makes sure, that all legitime administrators are locked out.

This is an advanced threat and a not very common attack, yet. The solution for this is to make sure, that not all accounts are affected by the lockout policy. Instead very few dedicated emergancy accounts, which are normally not used and stored in a safe location, should be excluded from it. Those accounts (as all administrative accounts) should be monitored closely in your SIEM or logging systems.