List FortiGate Certificates via CLI - CA certificates and local Certificates

You can either use the GUI of the FortiGate to list all certificates, or use the CLI. Either using the commands:

Using the "get" command

config vdom
edit root   #<--- your management vdom/your vdom of choice
get vpn certificate ca

FGT50E00000000 (root) #
FGT50E00000000 (root) # get vpn certificate ca
== [ Fortinet_Wifi_CA ]
name: Fortinet_Wifi_CA
== [ Fortinet_CA ]
name: Fortinet_CA
== [ ACCVRAIZ1 ]
name: ACCVRAIZ1
== [ AC_RAIZ_FNMT-RCM ]
name: AC_RAIZ_FNMT-RCM
== [ Actalis_Authentication_Root_CA ]
name: Actalis_Authentication_Root_CA
[...]

Using the "show" command

The show command might not be very helpful, because it does not necessarily show all certificates:

FGT50E00000000 (root) #
FGT50E00000000 (root) # show vpn certificate ca
config vpn certificate ca
end
FGT50E00000000 (root) # show full-configuration vpn certificate ca
config vpn certificate caend

FGT50E00000000 (root) # show full-configuration | grep -f 'vpn certificate ca'
config vpn certificate ca <---
end
FGT50E00000000 (root) #

Using the "fnsysctl" command

Using the fnsysctl command might be helpful:

FGT50E00000000  #
FGT50E00000000 # fnsysctl ls -la /etc/cert/local/
drwxr-xr-x    2 0    0   Wed Dec 25 21:43:14 2019        0 .
drwxr-xr-x    6 0    0   Wed Sep 18 20:39:27 2019        0 ..
-rw-------    1 0    0   Wed Sep 18 20:35:46 2019     2250 root_2020jan_sub.domain.tld.cer
-rw-------    1 0    0   Wed Sep 18 20:35:46 2019     1704 KEY-FILE
-rw-------    1 0    0   Wed Sep 18 20:35:46 2019     1407 root_Fortinet_CA_SSL.cer
-rw-------    1 0    0   Wed Sep 18 20:35:47 2019     1704 KEY-FILE
-rw-------    1 0    0   Wed Sep 18 20:35:47 2019     1419 root_Fortinet_CA_Untrusted.cer
-rw-------    1 0    0   Wed Sep 18 20:35:47 2019     1704 KEY-FILE
-rw-------    1 0    0   Wed Sep 18 20:35:47 2019     4285 root_Fortinet_Factory.cer
-rw-------    1 0    0   Wed Sep 18 20:35:47 2019     1679 KEY-FILE[...]
FGT50E00000000  #
FGT50E00000000  # fnsysctl ls -la /etc/cert/ca
drwxr-xr-x    2 0    0   Wed Dec 25 21:41:28 2019        0 .
drwxr-xr-x    6 0    0   Wed Sep 18 20:39:27 2019        0 ..
-rw-------    1 0    0   Wed Sep 18 20:35:55 2019      119 ca_bundle_ver
-rw-------    1 0    0   Tue Jan 14 20:06:15 2020 1972 root_AC_RAIZ_FNMT-RCM.cer
-rw-------    1 0    0   Tue Jan 14 20:06:15 2020 2772 root_ACCVRAIZ1.cer
-rw-------    1 0    0   Wed Sep 18 20:35:55 2019     2041 root_ACEDICOM_Root.cer
-rw-------    1 0    0   Tue Jan 14 20:06:15 2020 2049 root_Actalis_Authentication_Root_CA.cer
-rw-------    1 0    0   Tue Jan 14 20:06:14 2020 1521 root_AddTrust_External_Root.cer[...]

VMWare ESXi VM CPU Performance Over Commitment "CPU Stuck"

Many VMWare ESXi installations make the same mistake: They overcommit vCPUs, don't monitor CPU metrics like %RDY, %CSTP and don't know why their virtual machines are slow or have sometimes performance issues, especially in load situations. Sometimes you can find hints like "kernel BUG: soft lockup - CPU stuck for 22seconds" in your logs, but most aren't aware of anything.

What can be the cause of this issue?

A very good explanation about ESXi CPU Scheduling can be found here: https://www.youtube.com/watch?v=8jeBIvzyB80 

It explains how the hypervisor ESXi schedules the physical CPUs to the virtual vCPUs. And here is the issue. For example:

Picture from YouTube Video "The vSphere CPU Scheduler" of "TrainerTests"

 As shown in the screenshot overcommiting the physical CPU by assigning to many vCPUs to VMs may decrease the VMs performance, because you waste time slots.

This can be monitored by monitoring the following CPU metrics:

Which ESXi metrics should be monitored?

·         %USED tells you how much time did the virtual machine spend executing CPU cycles on the physical CPU.

·         %RDY (should be very low) is a very important performance indicator. Start with this one. This one defines how much time your virtual machine wanted to execute CPU cycles but could not get access to the physical CPU. It tells you how much time did you spend in a “queue”. Expect this value to be better than 5% (this equals 1000ms in the vCenter Performance Graphs read about it here)

·         %CSTP (should be 0.0%) tells you how much time a virtual machine is waiting for a virtual machine with multiple vCPU to catch up. If this number is higher than 1% you should consider lowering the amount of vCPU in your virtual machine.

·         %WAIT It is the percentage of time the virtual machine was waiting for some VMkernel activity to complete (such as I/O) before it can continue.

·         %IDLE (should be high) The percentage of time a world is in idle loop.
From http://www.vstellar.com/2015/10/09/understanding-cpu-over-commitment/

👉 High %RDY indicates that vCPUs are waiting for actual physical CPUs to schedule their threads
👉 High %CSTP indicates that ESXi is asking the vCPUs to wait – A.K.A. co-stopping them for scheduling purposes -> decrease vCPUs of VM

Small/Medium Businesses - New network devices (switches, routers,..) - Minimum ToDo list

Most small/medium businesses don't do much configuration, monitoring, cfg-baselineing or follow best practises with their network devices like switches, routers, wireless controllers, access points etc. Here is a short list of things you should do at minimum:

1. Extend your monitoring of your network devices

Not only ping them, check their uptime by snmp, but:
1.1 Monitor all uplinks (e.g. SNMP bandwidth)
1.2 Monitor all important ports (ports of the servers, firewalls, storage, etc; again e.g. with SNMP bandwidth)
1.3 Monitor device health, fan status, temperature, etc
1.4 Monitor the routing table, especially if you use dynamic routing protocols and/or have many routes
1.5 Monitor utilization of cpu, mem, i/o, etc..
1.6 Monitor everything with secure protocols like SSH, SNMPv3 AuthPriv AES+SHA
1.7 Send SNMP traps from devices to your monitoring system
1.8 Send Syslog from your devices to your monitoring system & logging solution

2. Harden your network

2.1 Disable telnet
2.2 Disable http
2.3 Implement ACLs for allowing access only from dedicated trusted hosts
2.4 Implement ACLs for dynamics routing protocols like BGP, OSPF, etc
2.5 Use LDAPS/Radius authentication for LDAP/AD-authentication for device mgmt
2.6 Send Syslog from your devices to your monitoring system & logging solution
2.7 Disable SNMPv1/v2c
2.8 Use DHCP-Snooping for Rouge DHCP server protection
2.9 Use ARP Spoofing Protection
2.10 Think about disabling link-layer discovery protocols like LLDP, CDP, EDP, etc
2.11 Allow local admin account login only if LDAPS/Radius server is not reachable
2.12 Delete default users, groups and communities

3. Authentication & dynamische vlan assignment

3.1 Use IEEE 802.1x with certificates (at least two AAA Radius Serves (e.g. FreeRadius) with EAP-TLS)
3.2 Use rfc3580 for dynamic vlan assignment
3.3 Think about using either a quarantine fallback vlan for not authenticated clients or a guest vlan with internet access only
3.4 Think of using DHCP Snooping (forwarding) for your devices which does device fingerprinting

4. Documentation

4.1 Create a layer1 and layer2 network plan (e.g. in visio)
4.2 Create a layer2 and layer3 network plan (e.g. in visio)
4.3 Use the l2&l3 plan as background for your monitoring system in a map to have a live-overview

5. Testing

5.1 Test your loop protection (m/r/stp, loop-protect, elrp, + broadcast limit thresholds like max 200 broadcasts per second, etc) in a maintenance window
5.2 Test your "CrossVlan Protection" in a maintenace window. By "CrossVlan" I mean not wanted connections between to vlans, which should be separated (m/r/stp, loop-protect, extra VLAN which is tagged on all ports and sends ELRP or similar loop protection protocols, etc)
5.3 Test your monitoring alerting - is an alert really send when e.g. an important uplink is full or disconnected, if an important lacp lag is down, etc (test using simulation, e.g. via jPerf, Observer etc)
5.4 Check and test if all best practises of the vendor are applied

6. IP-Subnetting

Yes, so many small and medium companys still have a huge flat layer2 network per site :(
6.1 The more subnets, the more a network issue stays only in that tiny subnet
6.2 The smaller the subnet, the less background noise
6.3 Microsegmentation is key! The smaller the subnet and the more it is separated (using private vlans, ACLs, a firewall, filtering device, host firewalls, a microsegmentation solution, NSX-T or something similar), the more it is protected and lateral movement gets harder.

There are many more things, like using LACP instead of static link aggregation groups, using LACP Mode Fast instead of the default slow, using Bidirectional Forwarding Detection "BFD" for everything, using multi-chassis link-aggregation (like MC-LAG, MLAG, etc) instead of Stacking (firmware-updates & reboots mostly cost the whole stack-topology to reboot, which is not the case in MLAG), using Out of Band management, and much more.

The listed items are the things which should be done at minimum.

FortiGate allows Ping from "not trusted hosts" since FortiOS 6.0

Recently I discovered something after updating a FortiGate cluster, which I intensively monitor, not only via working monitoring queries, but also doing some negative monitoring*: Since FortiOS 6.0 the Fortinet FortiGate firewall answers to source addresses via ICMP Type:8 EchoRequest "Ping", which are not included in the trustedhosts (config system admin -> edit admin -> set trusthost1). This is due to a change in FortiOS, which allows ping from more addresses. To limit ping, you have to use LocalIn-Policies (which always should be used, too).

*negative monitoring = I recommend not only to monitor what is allowed (like your SNMPv3 AuthPriv queries, your SSH or HTTPS-Calls), but also to monitor what is forbidden, in order to find some possible bugs or misconfigurations, if suddenly something is answering what really shouldn't. This can be done by trying to use insecure protocols like Telnet, SNMPv1/v2c, HTTP, etc.. or by sending requests from not allowed source ips or by trying to login using deleted/altered accounts or with wrong credentials.


Fortinet documentation:

In versions 5.x and below, trusted hosts configured by an administrator user only allow access from certain IP addresses configured in trusted hosts, to all services configured on the interface, including ping.

From version 6.0 onwards ping service on management interfaces are not included within the scope of trusted hosts. This means that you will be able to ping the interface from an IP that is not included within trusted hosts.

In order to only allow trusted hosts to be able to ping the interface and deny everyone else, you will need to configure a Local In Policy as below.

CLI configuration:

System > Administrators >
config system admin
    edit "admin"
        set trusthost1 172.26.73.48 255.255.255.255
        set accprofile "super_admin"
        set vdom "root"
    next
end

Configuring address and address group as per the trusted hosts:

config firewall address
    edit "trusted-1"
        set type ipmask
        set comment ''
        set visibility enable
        set associated-interface ''
        set color 0
        set allow-routing disable
        set subnet 172.26.73.48 255.255.255.255
    next
end

config firewall addrgrp
    edit "trusted_grp"
        set member "trusted-1"
        set comment ''
        set visibility enable
        set color 0
    next
end

Configuring Firewall local in policies:

config firewall local-in-policy
    edit 2
        set intf "port1"
        set srcaddr "trusted_grp"
        set dstaddr "all"
        set action accept
        set service "PING"
        set schedule "always"
        set status enable
        set comments ''
    next
    edit 1
        set intf "port1"
        set srcaddr "all"
        set dstaddr "all"
        set action deny
        set service "PING"
        set schedule "always"
        set status enable
        set comments ''
    next
end

Before configuring the local in policy:

diagnose sniffer packet any 'host 172.26.73.78 and icmp' 4
interfaces=[any]
filters=[host 172.26.73.48 and icmp]
3.647787 port1 in 172.26.73.78 -> 10.5.22.114: icmp: echo request
3.647850 port1 out 10.5.22.114 -> 172.26.73.78: icmp: echo reply
4.651341 port1 in 172.26.73.78 -> 10.5.22.114: icmp: echo request
4.651383 port1 out 10.5.22.114 -> 172.26.73.78: icmp: echo reply
5.657949 port1 in 172.26.73.78 -> 10.5.22.114: icmp: echo request
5.657992 port1 out 10.5.22.114 -> 172.26.73.78: icmp: echo reply

After configuring the local in policies:

diagnose sniffer packet any 'host 172.26.73.78 and icmp' 4
interfaces=[any]
filters=[host 172.26.73.48 and icmp]
4.264950 port1 in 172.26.73.78 -> 10.5.22.114: icmp: echo request
8.904217 port1 in 172.26.73.78 -> 10.5.22.114: icmp: echo request
13.906576 port1 in 172.26.73.78 -> 10.5.22.114: icmp: echo request
18.893835 port1 in 172.26.73.78 -> 10.5.22.114: icmp: echo request

trace_id=22 func=print_pkt_detail line=5497 msg="vd-root:0 received a packet(proto=1, 172.26.73.78:1->10.5.22.114:2048) from port1. type=8, code=0, id=1, seq=335."
id=20085 trace_id=22 func=init_ip_session_common line=5657 msg="allocate a new session-00874fe6"
id=20085 trace_id=22 func=vf_ip_route_input_common line=2591 msg="find a route: flag=80000000 gw-10.5.22.114 via root"
id=20085 trace_id=22 func=fw_local_in_handler line=409 msg="iprope_in_check() check failed on policy 1, drop

Reply for the trusted host:

diagnose sniffer packet any 'host 172.26.73.48 and icmp' 4
interfaces=[any]
filters=[host 172.26.73.48 and icmp]
7.239647 port1 in 172.26.73.48 -> 10.5.22.114: icmp: echo request
7.239743 port1 out 10.5.22.114 -> 172.26.73.48: icmp: echo reply
8.261081 port1 in 172.26.73.48 -> 10.5.22.114: icmp: echo request
8.261122 port1 out 10.5.22.114 -> 172.26.73.48: icmp: echo reply
9.276261 port1 in 172.26.73.48 -> 10.5.22.114: icmp: echo request
9.276321 port1 out 10.5.22.114 -> 172.26.73.48: icmp: echo reply
10.294536 port1 in 172.26.73.48 -> 10.5.22.114: icmp: echo request
10.294588 port1 out 10.5.22.114 -> 172.26.73.48: icmp: echo reply

Diag debug flow on FortiOS 6.0.9
Source IP of ping is not configured in trustedhosts

2020-02-26 08:38:47 id=20085 trace_id=255 func=init_ip_session_common line=5684 msg="allocate a new session-00018a90"
2020-02-26 08:38:47 id=20085 trace_id=255 func=vf_ip_route_input_common line=2591 msg="find a route: flag=80000000 gw-10.128.36.35 via root"
2020-02-26 08:38:47 id=20085 trace_id=256 func=print_pkt_detail line=5519 msg="vd-root:0 received a packet(proto=1, 10.128.36.35:1->10.240.161.178:0) from local. type=0, code=0, id=1, seq=1156."


Diag debug flow on FortiOS 5.6.12
Source IP of ping is not configured in trustedhosts

2020-02-26 08:46:05 id=20085 trace_id=10 func=print_pkt_detail line=5375 msg="vd-root received a packet(proto=1, 10.240.161.178:1->10.128.36.4:2048) from mgmt1. type=8, code=0, id=1, seq=1166."
2020-02-26 08:46:05 id=20085 trace_id=10 func=init_ip_session_common line=5534 msg="allocate a new session-fd0ad6af"
2020-02-26 08:46:05 id=20085 trace_id=10 func=vf_ip_route_input_common line=2574 msg="find a route: flag=80000000 gw-10.128.36.4 via root"
2020-02-26 08:46:05 id=20085 trace_id=10 func=fw_local_in_handler line=402 msg="iprope_in_check() check failed on policy 0, drop"

Source: https://kb.fortinet.com/kb/documentLink.do?externalID=FD44156

Advanced Endpoint Protection Testing from MITRE using ATT&CK

If you are searching for test results about current Advanced Endpoint Protection/Endpoint Detection and Response tools: MITRE is transparently testing some of them using the great ATT&CK map.

Round 1 Testing

In round 1 the following AEP/EDR products were tested:

• Carbon Black
• Crowdstrike
• Endgame
• GoSecure
• Windows Defender ATP
• RSA NetWitness
• SentinelOne

The following vendors/products will follow:

• Cyberreason
• F-Secure Countercept
• Fireeye Endpoint Security
• McAfee MVision
• Palo Alto WildFire Traps XDR 

Example

As an example you can look at the results of the Microsoft Windows Defender ATP results. You can see all techniques which were tested and how the product worked. You even can see screenshots of it:

Screenshot of Microsoft Defender ATP of MITRE ATT&CK Evaluation Round1 Testing 1.A.1 User Execution T1204:

Summary

This is amazing work done by MITRE! 👍 It provides transparency of the Advanced Endpoint Protection AEP/Endpoint Detection and Response EDR products, lets you compare them, check which techniques they protect against and how they work in terms of forensics.

Round2 is already running. 👌

Solution for Skybox connection issue to Fortinet FortiGate or FortiManager

If you are using the Skybox (https://www.skyboxsecurity.com/) solution for your environment, during the initial setup there might be an issue in the connection from the Skybox to your Fortinet FortiGate firewalls or your or FortiManager firewall management system. Skybox uses HTTPS (XML API with SOAP) and/or SSH to connect to the Fortinet systems.

If your hardening of the Fortinet devices changed the default minimum Diffie-Hellman Exchange-Bits from 2048 to 3072, 4096 or 8192Bits, then your Skybox is not able to connect to them, after Skybox currently does not support more than 2048Bit for DH.

FortiGate Hardening of DH-Bits:

config sys global
 set dh-params 8192
end

See also the Skybox documentation "ReferenceGuide", e.g. here "Reference Guide v10.801" or an overview of the documentation of all last versions.

Malware using PowerShell - PowerShell Logging "Script Block Logging"

More and more so called "fileless malware" uses powershell in order to execute malicious actions. In order to find possible malicious powershell commands or ps-scripts, it is very useful to log them, automatically send them to your SIEM and analyze them. Also if possible, disable Powershell for your users, but this is in the real-world sometimes hard or even not possible.

1. Disable PowerShell for users if possible via GPO
2. Fileless malware using PowerShell - PowerShell Logging using Script Block Logging

https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_logging_windows?view=powershell-6#enabling-script-block-logging

Enabling Script Block Logging

When you enable Script Block Logging, PowerShell records the content of all script blocks that it processes. Once enabled, any new PowerShell session logs this information. It's recommended to enable Protected Event Logging.

Using Group Policy

To enable automatic transcription, enable the Turn on PowerShell Script Block Logging feature in Group Policy through Administrative Templates -> Windows Components -> Windows PowerShell.

Using the Registry

Run the following function:

PowerShell

function Enable-PSScriptBlockLogging
{
    $basePath = 'HKLM:\Software\Policies\Microsoft\Windows' +
      '\PowerShell\ScriptBlockLogging'

    if(-not (Test-Path $basePath))
    {
        $null = New-Item $basePath -Force
    }

    Set-ItemProperty $basePath -Name EnableScriptBlockLogging -Value "1"
}

Powershell-Commands will be logged in Windows Eventlog with Event-ID 4104.

Use OpenSSL to decrypt private key

Use OpenSSL to decrypt private key

openssl rsa -in *encrypted-key-file* -out *decrypted-key-file*

Example:

Encrypted private key file: /etc/ssl/private/sub.domain.tld_201908.enc.key 
New decrypted private key file: /etc/ssl/private/sub.domain.tld_201908.key 

root@eulinxhost119:/etc/ssl/private#
root@eulinxhost119:/etc/ssl/private# openssl rsa -in ./sub.domain.tld_201908.enc.key -out sub.domain.tld_201908.key
Enter pass phrase for ./sub.domain.tld_201908.enc.key:
writing RSA key
root@eulinxhost119:/etc/ssl/private#
root@eulinxhost119:/etc/ssl/private#

Opening the old encrypted private key: File starts with:
-----BEGIN ENCRYPTED PRIVATE KEY----- 

 Opening the new unencrypted private key: File starts with:
-----BEGIN RSA PRIVATE KEY-----

Use OpenSSL to verify the private key

openssl rsa -in certkey.key –check

Example:

root@eulinxhost119:/
root@eulinxhost119:/etc/ssl/private# openssl rsa -in sub.domain.tld_201908.key -check
RSA key ok
writing RSA key
-----BEGIN RSA PRIVATE KEY-----
[...]
-----END RSA PRIVATE KEY-----
root@eulinxhost119:/
root@eulinxhost119:/

OpenSSL documentation and examples

OpenSSL man-page: https://www.openssl.org/docs/manmaster/man1/openssl.html
OpenSSL examples: https://wiki.openssl.org/index.php/Command_Line_Utilities

Notepad++ edit previously opened sessions/files

Sometimes I have so many or too large files opened with Notepad++ (GitHub) --> it stops starting the next time and hangs.

To resolve this, simply either delete or edit the session.xml-file:
C:\Users\\AppData\Roaming\Notepad++\session.xml 
or
%AppData%\Notepad++\session.xml 

Each entry starting with "<File lang.." and ending with "/>" is a previously opened file. Example:
<File lang="Normal Text" mapIsWrap="
8...] 
firstVisibleLine="0"/> 

By deleting entries like this, you throw out that the file is automatically opened the next time when starting Notepad++.
🚩Hint: Delete the last entry, this resolves most times the issue.

SonicWALL SD-NAT (loopback NAT) for internal published services

If you're having a small network with a SonicWALL firewall and you've published a service using destination NAT (DNAT) to the internet (or another interface/networks), you sometimes want to to access the same published service using the same (public) address from the inside.

Example:

When SystemB is in the same subnet or same network zone as the API-Gateway and you want to access the API-Gateway address from SystemB, you can either use SplitDNS (internal DNS resolution resolves the public FQDN to the internal address), or only use public ip-addresses in that network/zone, or provide the API-Gateway multiple virtual addresses, or you use a "lookback NAT" which is a source- and destination-NAT at the same time.

The SonicWALL then changes the destination ip-address of every DMZ packet from the external ip-address of the API-Gateway to the internal ip-address of the API-Gateway and at the same time, changes the source ip-address from SystemB to its own internal interface ip-address. Changing the source-address makes sure that the API-Gateway does not sends the traffic directly back to SystemB, but instead to the SonicWALL, which then changes back the destination-NAT for the answered packets.

In the SonicWALL NAT-Table this setup looks like this:

Of course you have to setup the suiting access-rules in the SonicWALL as well. Keep in mind the SonicWALL internal traffic flow first checks the access rules, then changes the address using the NAT table. So you may have to setup Rules like DMZ>DMZ (or LAN>LAN) but as a destination the WAN-address objects!

I've set this up in many setups for over 10 years with very different SonicWALLs, FortiGates and other Firewalling-devices, and it works well (e.g. tested with SonicWALL on SonicOS3/4/5/6 from TZ-series, NSA-series to SuperMassive-series).
🚩However keep in mind, opening services using NAT might be a possible security-hole (for example VPN might help), publishing services to the internet always should go through an application reverse proxy like a WAF, API-Gateway or something else, the published services should be hardend, closely monitored and be located in a DMZ, have strong encryption and authentication (mutual TLS). Also I personally prefer to use SplitDNS and different virtual addresses in order to keep original ip-addresses (not having to use X-Forwarded-For which might cause lots of logging-volume on your logs/Logging-System/SIEM) etc.

Hardening FortiGate connection to FortiAnalyzer

The FortiGate uses OFTP (Fortinets proprietary Optimized Fabric Transfer Protocol, not Odette File Transfer Protocol) to transfer data between FortiGates and FortiAnalyzer. For encryption SSLv3 or TLS is used. In order to harden the connection, use the following commands:

Cfg for FortiAnalyzer 

 1. Set TLS to minimum version 1.2:


2. Change FAZ certificate to one using the correct FQDN from your/a public CA:



Debugging can be done using the fowlloing commands:

Oftpd daemon test usage:   diagnose test application oftpd <integer>

• 1: show PID
• 2: show statistics and state
• 3: show connected device name and IP
• 4: show detailed session state
• 5: show oftp request statistics
• 6: show cmdb device cache
• 7: show logfwd thread stats
• 8: show tasklist statistics
• 9: show unreg dev cache
• 10: log cluster bridge stats
• 20: show forticlent end-user stats
• 21: rebuild forticlient end-user avatar table
• 90: reload un-reg device tree
• 99: restart daemon

FAZ looks like this:

(global)# get

dh-params           : 8192

enc-algorithm       : high

fgfm-ssl-protocol   : tlsv1.2

oftp-ssl-protocol   : tlsv1.2

ssl-low-encryption  : disable

ssl-protocol        : tlsv1.2

ssl-static-key-ciphers: disable

webservice-proto    : tlsv1.2

Cfg for FortiGate

1. Change TLS ciphers to high:

config log fortianalyzer setting

    set enc-algorithm high

end

2. Change Global TLS-Settings

config system global

    set ssl-static-key-ciphers disable

    set dh-params 8192

    set strong-crypto enable

end

Important Splunk update for timestamp issue starting 01.01.2020

Splunk has identified an issue regarding timestamps, which are intensivly used for data correlation in many Splunkbased logging and SIEM systems. Affected versions are Splunk Enterprise, Splunk Light and Splunk Cloud. This issue has potential significant impact on data ingestion - including causing inaccurate, unsearchable, or prematurely-deleted data - starting January 1, 2020.

Cause of the issue: Timestamps using two-digit years will stop being correctly recognized. Full details around this issue, including workarounds and product fixes, are documented in Release Notes for each Splunk Version: https://docs.splunk.com/Documentation/Splunk/latest/ReleaseNotes/FixDatetimexml2020
 

Possible Solutions:

1. Manual Change of "datetimes.xml"

Change on all Splunk systems (search heads, indexers, heavy forwards etc) the file "datetimes.xml" with the following file: http://download.splunk.com/products/ingest2020/datetime.zip
In order to do that, put the downloaded in $SPLUNK_HOME/etc. (mostly found in /opt/splunk/etc). Then restart the system (in a Indexer Cluster a rolling-restart is possible). Until a splunk patch is available, the warning will be shown, that this file is not part of the splunk manifest. This will be fixed in the future splunk versions.

2. Update to a version with a fix

Splunk will ship minor releases with fixes, soon:

Major Release --- Minor Release with patch
6.6 --- 6.6.12.1 (not yet released)
7.0 --- 7.0.13.1 (not yet released)
7.1 --- 7.1.10 (not yet released)
7.2 --- 7.2.9.1 (not yet released)
7.3 --- 7.3.3 (Installationguide)
8.0 --- 8.0.1 (not yet released)

FortiOS XSS in DHCP-Monitor

Fortinet has released PSIRT FG-IR-19-184 (CVE-2019-6697) about a vulnerability in FortiOS of the FortiGate firewall. A DHCP packet may contain a stored XSS in the hostname parameter field.

Affected Versions and Update

Affected Products are FortiOS 6.2.1 and below & FortiOS 6.0.6 and below.

FortiOS 6.2.2 and FortiOS 6.0.7 fix the vulnerability

PoC/Vulnerability Details from ssd-disclosure.com

Source: https://ssd-disclosure.com/archives/3987/ssd-advisory-fortigate-dhcp-stored-xss

An unauthenticated attacker can trigger a Stored XSS Vulnerability via a malicious DHCP packet in the Fortigate DHCP Monitor.

This can happen if Device Detection is enabled through Network >Interface > Edit Interface > Device Detection

  


 When this option is enabled the attacker may perform the following steps in order to exploit the vulnerability:

1. Install dhtest or any other tool that can send arbitrary DHCP packets.
   (https://sargandh.wordpress.com/2012/02/23/linux-dhcp-client-simulation-tool/)
2. Send a malicious DHCP packet. For example:
   
   #./dhtest-master/dhtest -i eth0 -m 12:34:56:78:90:12 -h "x<svg onload=alert();)>x" [Option] -m : mac address -h : hostname(dhcp option 12). The attacker can inject malicious scripts.

   1 2 3 4

   #./dhtest-master/dhtest -i eth0 -m 12:34:56:78:90:12 -h "x<svg onload=alert();)>x"     [Option]     -m : mac address     -h : hostname(dhcp option 12). The attacker can inject malicious scripts.

Linux show all cronjobs of all users

How to Linux show all cronjobs of all users (not ldap or nis users): Use the following command as root:

for user in $(cut -f1 -d: /etc/passwd); do crontab -u $user -l; done

Example:

linuxhost001:~ # for user in $(cut -f1 -d: /etc/passwd); do crontab -u $user -l; done
no crontab for bin
no crontab for daemon
no crontab for ftp
no crontab for lp
no crontab for mail
no crontab for man
no crontab for messagebus
no crontab for news
no crontab for nobody
no crontab for nscd
no crontab for openslp
no crontab for polkitd
no crontab for postfix
no crontab for root
no crontab for rpc
no crontab for sshd
no crontab for statd
no crontab for systemd-bus-proxy
no crontab for systemd-timesync
no crontab for uucp
no crontab for wwwrun
no crontab for ntp
no crontab for nagios
no crontab for radiusd 

Intel NUC running VMware ESXi 6.5.3

Because there are some people on the internet asking for experience on "is my Intel NUC supporting ESXi version x.y.z?", so I will provide short feedback regarding that: Using an Intel NUC NUC7i7DNHE "Intel NUC 8th Gen Commercial" (https://www.intel.de/content/www/de/de/products/boards-kits/nuc.html) I'm running VMware ESXi 6.5 update 3 without any additional drivers necessary.

VMware updates can be found here: http://www.vmware.com/patchmgr/download.portal

Apple iPhone/iPad iOS IPSec IKEv2 Proposals

When setting up VPN-tunnel from an Apple iPhone or iPad running iOS using IPSec with IKEv2 you need to know, which IPSec proposals the iPhone/iPad/iOS device are supporting/offering:

Offered proposals from iOS

Testing with an iPhone running iOS 12.4.1 and iPad 13.1.2:

• AES256-SHA256-DH14 (2048-bit MODP Group) <------ (✔ okay)
• AES256-SHA256-DH19 (256-bit random ECP group) <------ (✅ recommended)
• AES256-SHA256-DH5 (1536-bit MODP Group) <------ (❌not recommended)
• AES128-SHA1-DH2 (1024-bit MODP Group) <------ (❌not recommended)
• 3DES-SHA1-DH2 (1024-bit MODP Group) <------ (❌not recommended)

Recommendation

Therefore I recommened 🔒✅ to use for your IPSec IKEv2 proposals:

• IKEv2 Phase1: AES-CBC-256 with SHA2-256 and DH-Grp 19 (ECP 256bit)
• IKEv2 Phase2: AES-CBC-256 with SHA2-256 and DH-Grp 19 (ECP 256bit)

DH-Grp 19 ECP 256Bit > DH-Grp 14 RSA 2048Bit
-> For example see BSI recommendation for crypto IPSec page 13 section 3.2.4  or NIST recommendation page 9 line 264

-> Details for ECP (Elliptic Curve from NIST) for IKEv1/v2 see RFC5903 or IANA ipsec registry

Details to reverse engineering

iPhone iOS 12.4.1 IKEv2 RAW output:

2019-10-27 16:25:15.519164 ike 4: incoming proposal:
2019-10-27 16:25:15.519176 ike 4: proposal id = 1:
2019-10-27 16:25:15.519185 ike 4:   protocol = IKEv2:
2019-10-27 16:25:15.519195 ike 4:      encapsulation = IKEv2/none
2019-10-27 16:25:15.519205 ike 4:         type=ENCR, val=AES_CBC (key_len = 256)
2019-10-27 16:25:15.519215 ike 4:         type=INTEGR, val=AUTH_HMAC_SHA2_256_128
2019-10-27 16:25:15.519224 ike 4:         type=PRF, val=PRF_HMAC_SHA2_256
2019-10-27 16:25:15.519234 ike 4:         type=DH_GROUP, val=MODP2048.
2019-10-27 16:25:15.519246 ike 4: proposal id = 2:
2019-10-27 16:25:15.519255 ike 4:   protocol = IKEv2:
2019-10-27 16:25:15.519264 ike 4:      encapsulation = IKEv2/none
2019-10-27 16:25:15.519274 ike 4:         type=ENCR, val=AES_CBC (key_len = 256)
2019-10-27 16:25:15.519283 ike 4:         type=INTEGR, val=AUTH_HMAC_SHA2_256_128
2019-10-27 16:25:15.519293 ike 4:         type=PRF, val=PRF_HMAC_SHA2_256
2019-10-27 16:25:15.519303 ike 4:         type=DH_GROUP, val=ECP256.
2019-10-27 16:25:15.519314 ike 4: proposal id = 3:
2019-10-27 16:25:15.519323 ike 4:   protocol = IKEv2:
2019-10-27 16:25:15.519332 ike 4:      encapsulation = IKEv2/none
2019-10-27 16:25:15.519342 ike 4:         type=ENCR, val=AES_CBC (key_len = 256)
2019-10-27 16:25:15.519353 ike 4:         type=INTEGR, val=AUTH_HMAC_SHA2_256_128
2019-10-27 16:25:15.519365 ike 4:         type=PRF, val=PRF_HMAC_SHA2_256
2019-10-27 16:25:15.519374 ike 4:         type=DH_GROUP, val=MODP1536.
2019-10-27 16:25:15.519384 ike 4: proposal id = 4:
2019-10-27 16:25:15.519392 ike 4:   protocol = IKEv2:
2019-10-27 16:25:15.519400 ike 4:      encapsulation = IKEv2/none
2019-10-27 16:25:15.519408 ike 4:         type=ENCR, val=AES_CBC (key_len = 128)
2019-10-27 16:25:15.519416 ike 4:         type=INTEGR, val=AUTH_HMAC_SHA_96
2019-10-27 16:25:15.519424 ike 4:         type=PRF, val=PRF_HMAC_SHA
2019-10-27 16:25:15.519432 ike 4:         type=DH_GROUP, val=MODP1024.
2019-10-27 16:25:15.519443 ike 4: proposal id = 5:
2019-10-27 16:25:15.519451 ike 4:   protocol = IKEv2:
2019-10-27 16:25:15.519459 ike 4:      encapsulation = IKEv2/none
2019-10-27 16:25:15.519466 ike 4:         type=ENCR, val=3DES_CBC
2019-10-27 16:25:15.519474 ike 4:         type=INTEGR, val=AUTH_HMAC_SHA_96
2019-10-27 16:25:15.519482 ike 4:         type=PRF, val=PRF_HMAC_SHA
2019-10-27 16:25:15.519490 ike 4:         type=DH_GROUP, val=MODP1024.

 

Windows Server 2016 Backup - error with user of network share

If you are running a Windows Server 2016, are using the integrated Windows Server Backup utility and you want to save the backup to a remote destination, which is a CIFS/SMB network share, then during the setup the following error might occur:

"The user name being used for accessing the remote share folder is not recognized by the local computer."

Soltuion

1. Create a local user at your Windows Server, which has the same username and password as the cifs/smb user
2. Add this user to the "Backup Operators" user group:

3. Go back to setup the backup job using the "Previous" button and click next/finish again -> Your scheduled backup job using a cifs/smb network share should now be created successfully.

Outlook connection protocol, auth, encryption & status

A quick way to check or beginn to troubleshoot the used protocol, authentication, encryption and connection status of Microsofts Outlook client as well as to find out which version is running on your Microsoft Exchange server or Microsoft Office 365 server is to use:

1. CTRL + right-click on the outlook icon:
    
2. Select "connection status":
   
3. The following window will appear:
   
   There the server, the proxy-server, the used protocol, the authentication method, the encryption, the status and the exchange-server version can be quickly checked.

More in the Microsoft support article.

Security fixes in PRTG 19.3.51/19.4.52

The current version PRTG 19.3.51/19.4.52 includes some security fixes:

• PRTG Core Server XSS Cross-Site-Scripting
  We fixed potential reflected XSS vulnerabilities with medium severity on the PRTG core server. The potential vulnerabilities affected tag filters, object IDs, and the contact support/feedback page. Please note that the fixed vulnerabilities required a logged in PRTG user account to be exploited.
• Sensors DoS
  We fixed a potential Denial of Service (DoS) vulnerability of the HTTP Full Web Page sensor. Please note that the fixed vulnerability required a logged in PRTG user account with elevated rights to be exploited. (CVE-2019-11074) 

Besides those security fixes some new improvements made it into the release, too. More information can be found in the release notes https://www.de.paessler.com/prtg/history/stable or in the blog of the vendor: https://blog.paessler.com/prtg-release-19.3.51-and-19.4.52-news-roundup. Information regarding previous security issues with proof of concept are listed here: https://sensepost.com/blog/2019/being-stubborn-pays-off-pt.-1-cve-2018-19204/

SonicWALL E-Mail-Security AntiSpam Troubleshooting & Firmware

If you are using SonicWALL E-Mail-Security as your antispam solution there are some useful ways to troubleshoot issues using different logs. They help to understand what happend to a certain mail, why communication to another mailserver is slow/lost and how the SNWL E-Mail-Security or your ES Remote Analyzers work internally.
These logs contain very sensible information in terms of privacy, make sure your data protection & legal processes are fully in place.

1. Which firmware to use?

SonicWALL uses the SNWL E-Mail-Security themselfs. The version of the used SonicWALL E-Mail-Security is written in their SMTP Banner:

1. DNS-Query to find MX-Records:
   > set q=mx
   > sonicwall.com
   Server:  dns9.quad9.net
   Address:  9.9.9.9
   
   Non-authoritative answer:
   sonicwall.com   MX preference = 15, mail exchanger = mail1.sonicwall.com
   sonicwall.com   MX preference = 15, mail exchanger = mail3.sonicwall.com
   sonicwall.com   MX preference = 15, mail exchanger = mail2.sonicwall.com
   
   
2. Connecting to them using a TCP-Session on TCP-Port 25 (e.g. telnet 25 or nc -C 25):
   nc -C mail1.sonicwall.com 25
   220 mail.sonicwall.com ESMTP SonicWall (9.1.2.3763)
   220 mail.sonicwall.com ESMTP SonicWall (9.1.1.3121)
   220 mail.sonicwall.com ESMTP SonicWall (9.1.2.3761)

There you can see which firmware SonicWALL themselfs are using.

2. Audit log

Use the audit log. It helps you most of the time and is very easy to understand.

3. Log level

Set the loglevel to "level 2" or "debug". That gerenates a lot of logs, but is necessary for fully troubleshooting mail-problems, either with other or your own mailservers or with mails "lost" in antispam.

1. Login to the SonicWall and navigate to Manage -> Anti-Spam -> Advanced Settings
   
2. Now select the Log Level 2 and then select the Type of Log file and then click on Download. We have chosen MlfAsgSMTP in the screenshot shown below to download the SMTP Logs, however depending on the issue the desired log files may be selected.
   
3. Save the logs in the desired location.
   Source: https://www.sonicwall.com/support/knowledge-base/how-to-obtain-smtp-logs-from-anti-spam/170503798824694/

You can use the CLI to adjust the maximum log filesize:

Logfile names:
Some log file names, such as those found in the commonlogs directory, contain a two-digit number such as 12.log. The "12" indicates that the log is for the 12th day of the most recent month. Some log file names end with a digit, such as MlfThumbUpdate_2.log. The "2" indicates that this is an older log. The current log is MlfThumbUpdate.log. The next most recent log is MlfThumbUpdate_0.log, followed by MlfThumbUpdate_1.log, and so forth.

The following logs are very useful:

• pmta:logs
• logs:MlfAsgSMTP
• logs:smtp

Use the timestamp, unique-message-id or the destination-address in order to follow a mail. Sometimes the E-Mail-Security sends a mail to localhost on another internal tcp socket. You can see that, because the destination is 127.0.0.1:*highport*. Because sometimes an action is done while the second inspection time, you need to follow that session, which will be documented in the logs, too.

FortiOS update with versions 5.2.14/5.4.12/5.6.10/6.0.5/6.2.1

Fortinet released two "Customer Support Bulletins" (check fortinet support portal) regarding a possible issue with uninterruptible upgrade in FortiGate HA-clusters running version FortiOS 5.2.14 or 5.4.12 or 5.6.10 or 6.0.5 or 6.2.1. Please check the two Customer Support Bulletins before upgrading and plan a possible short downtime for the update.

Securing Layer 2 / Switing Level

Ideas for securing your layer 2 switching infrastructure

No matter if you use Cisco, Extreme Networks, HP or any other vendor, most of them have the same basic security features, which are rarely used, but actually really should be. Here are some ideas, which you should use for ideal protection:

Securing Switch Management Interfaces

• ACLs --> to limit all management interfaces (SSH, SNMP, etc) to very few dedicated ip-addresses
• Use SSH
• Disable telnet
• SNMPv3 AuthPriv AES+SHA
• Disable SNMPv1/v2c
• Disable HTTP/HTTPS
• Delete default users, groups & communities
• Use radius authentication and two local backup-accounts with long passwords (30+ characters), those should be only useable when the radius-servers are not reachable anymore, otherwise they should not work
• Use syslog (reliable & encrypted if possible) --> send these logs to a SIEM and define alerts/SIEM use cases
• Fail2ban or similar login protection
• Use Out-of-band management interfaces
• Think about using serial-to-ip adapters for out-of-band-management, if you have a dedicated seperated out-of-band-mgmt network
• Disable LLDP/CDP/EDP/etc where possible
• Setup IT-Monitoring (PRTG, Icinga2, Nagios,..) and check CPU, RAM, NIC-Utilization & Errors of critial links like LAGs, Uplinks, Mgmt, etc, 802.1x-Auth-Errors-per-Min, Temperature, Fans, Hardware-Status, 
• Setup IT-Monitoring to do negative testing like try to access the switch via Telnet/HTTP/SNMPv1/v2c or SSH/SNMPv3 with default accounts or try to access the switches from a forbidden ip-address (which is not part of the ACL) in order to find possible cfg-errors, holes or cfg mistakes after a switch got replaced, reconfigured or a firmware-update
• Limit Access to VirtualStacking/MultiChassis-Links like MLAG, VSS etc using dedicated ACLs; If possible use internal ip-addresses like 169.254.0.0/16 if you have to use ip-addresses for them

Securing Traffic passing the switches

• Use 802.1x Authentication with certificates from your pki
• Port-Level Traffic Controls and port security mechanisms to provide protection against MAC flooding attacks
• Private VLAN (PVLAN) 
• Access Lists on Switches if necessary, normally done on a firewall
• Spanning Tree Protocol Features 
• Dynamic Host Configuration Protocol (DHCP) Snooping 
• IP Source Guard 
• Dynamic ARP Inspection (DAI) 
• Advanced Integrated Security Features on High-End Catalyst Switches 
• Control Plane Policing (CoPP) Feature 
• CPU Rate Limiters 
• M/BCast Limitations or Warnings 
• Loop Protection as BPDU Guard, Loopguard, ELRP, BCast-Storm-Control, (M/R)STP, Root Guard, etc
• Use VLAN tagging to seperate user-traffic from network traffic like LLDP, (M)STP, OSPF, BGP, BFD, etc. --> Then those protocols are not mixed with user-traffic (example: STP is send untagged while user-traffic is sent tagged)
• Shut down or disable all unused ports on the switch
• Use small subnets, not bigger than /24 for good performance/stability and for small segmentation --> in general: the more segmentation, the better

Not really security related, but I always recommend to use LACP in fast mode for LAGs (let LACPDUs work for you), BFD for faster failover in (dynamic) routing-protocols and protocols like MLAG instead of stacking for better firmware-update-situations (often stacking requires the whole stack-topology to be rebooted). The more you harden your switch infrastructure, the less often they are attackable and the less often you have to update the firmware.

Extend ZFS zpool of Linux Ubuntu NextCloud VM

If you are running a virtual machine with NextCloud v16 on linux ubuntu and you want to extend /mnt/ncdata, then:

1. Make sure your backup is fine, think about creating a vm snapshot
2. Extend second virtual harddrive disk in VMWare virtual machine settings (e.g. from 40Gb to 170Gb)
3. Login using SSH/VMWare-Console, change to root user using sudo su or sudo -i
4. Check the zpool size using zpool list
5. Check the /mnt/ncdata size using df -h
6. Read the new partition size using parted -l with the answer "fix" for the adjustment
7. You can delete the buffer partition 9 using parted /dev/sdb rm 9
8. Extend the first partition using to 100% of the available size parted /dev/sdb resizepart 1 100%
9. Using zpool online -e ncdata /dev/sdb you can adjust the partition to the correct size
10. Check the new zpool size using zpool list
11. Check the new /mnt/ncdata size using df -h

Example with nextcloud 16 on Ubuntu 18.04:

root@nextcloud:/mnt# zpool list
NAME     SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
ncdata  39.8G  15.2M  39.7G         -     0%     0%  1.00x  ONLINE  - 
root@nextcloud:/mnt#
root@nextcloud:/mnt#
root@nextcloud:/mnt# df -Th
Filesystem                     Type      Size  Used Avail Use% Mounted on
[...]
ncdata                         zfs        39G  896K   39G   1% /mnt/ncdata
[...]
root@nextcloud:/mnt#
root@nextcloud:/mnt# parted -l
[...]
Model: VMware Virtual disk (scsi)
Disk /dev/sdb: 183GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name                  Flags
 1      1049kB  42.9GB  42.9GB  zfs          zfs-28cd7
 9      42.9GB  42.9GB  8389kB[...]
root@nextcloud:/mnt#
root@nextcloud:/mnt#
root@nextcloud:/mnt# parted /dev/sdb rm 9
Information: You may need to update /etc/fstab.
root@nextcloud:/mnt#
root@nextcloud:/mnt# parted /dev/sdb resizepart 1 100%
Information: You may need to update /etc/fstab.
root@nextcloud:/mnt#
root@nextcloud:/mnt# zpool online -e ncdata /dev/sdb
root@nextcloud:/mnt#
root@nextcloud:/mnt# zpool list
NAME     SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
ncdata   170G  2.51M   170G         -     0%     0%  1.00x  ONLINE  -
root@nextcloud:/mnt#
root@nextcloud:/mnt# df -h
Filesystem                      Size  Used Avail Use% Mounted on[...]
ncdata                          165G  896K  165G   1% /mnt/ncdata[...]
root@nextcloud:/mnt#



Update 25.12.2020: If you are running Ubuntu 20/Nextcloud VM 20, then follow the instrutions here starting point 7: https://how2itsec.blogspot.com/2020/12/increase-disk-and-zfs-of-nextcloud-vm.html

Hint when VMWare doesnt show a snapshot

If VMWare ESXi/vCenter/your vSphere environment does not show a virtual machine snapshot, however it is there and the virtual machine is using it, here is a little trick which might help you:

1. Make sure you have some free space left in your datastore, in which the vm is stored.
2. Create another virtual machine snapshot 
3. Click on "delete all snapshots"

This can trigger the ESXi to delete all snapshots, including the one which isn't shown. However this does not always work, in which case working with a clone of the VM and vmkfstools can help.