FortiAnalyzer reports - how to group subdomains to only show the root domain

Most Fortinet FortiAnalyzer reports use the full domain including subdomains, e.g. www.google.com or maps.google.com. If you want to group subdomains into the root-domain (e.g. google.com), then you might clone the existing reports & dataset and adjust the new dataset from:

Original Dataset:

select domain, string_agg(distinct catdesc, ', ') as agg_catdesc, sum(requests) as requests from ###(select hostname as domain, catdesc, count(*) as requests from $log where $filter and (eventtype is null or logver>=52) and hostname is not null and catdesc is not null and action!='blocked' group by domain, catdesc /*SkipSTART*/order by requests desc/*SkipEND*/)### t group by domain order by requests desc

Changed Dataset:

select domain, string_agg(distinct catdesc, ', ') as agg_catdesc, sum(requests) as requests from ###( select coalesce (nullifna (root_domain (hostname)), ipstr(dstip)) as domain, catdesc, count(*) as requests from $log where $filter and (eventtype is null or logver>=52) and hostname is not null and catdesc is not null and action!='blocked' group by domain, catdesc /*SkipSTART*/order by requests desc/*SkipEND*/)### t group by domain order by requests desc

Dataset change:

select hostname as domain
->
select coalesce (nullifna (root_domain(hostname)), ipstr(dstip)) as domain

Result screenshots:

Before adjusting the dataset:

After adjusting the dataset:

Source:

https://kb.fortinet.com/kb/microsites/search.do?cmd=displayKC&docType=kc&externalId=fortianalyzer-v560-dataset-referencepdf&sliceId=&docTypeID=DT_PRODUCTDOCUMENTATION_1_1&dialogID=130376293&stateId=1%200%20130378047%27)

Debugging PRTG Enterprise Console Remover

Paessler just released for PRTG a remover for the deprecated Enterprise Console: PRTG Enterprise Console Remover.exe:

You can uninstall standalone installations of the EC via the Windows control panel under Programs and Features.

If the EC was automatically installed as part of a PRTG installation, you can use the PRTG Enterprise Console Removal Tool as of PRTG 20.2.58 to uninstall the EC independently from the PRTG installation.

Click the Windows Start menu and select PRTG Network Monitor, then click Remove PRTG Enterprise Console to remove the EC from your PRTG core server system.
Source: https://kb.paessler.com/en/topic/85851-how-to-uninstall-the-prtg-enterprise-console-from-the-prtg-server

I like to know what programs do => so I traced what actions are done by "PRTG Enterprise Console Remover.exe". This is a screenshot of all the "write" actions with were performed:

The uninstaller checks more paths, registry entries etc, however only those were deleted or modified on my Windows Server 2016 system. I did not have the standalone installation of PRTG Enterprise Console, but the Enterprise Console was automatically installed as part of the PRTG installation.

Restore CHK files

I just had to restore many broken CHK files of a SD card from a hidden FOUND.000 folder. The program "unchk.exe" helped me: http://www.ericphelps.com/uncheck/. As described I had to install the Microsoft Visual Basic 5 Runtime because of an "MSVBM50.dll not found" error, which can be downloaded from Microsoft.

I checked and noticed, that all the big files of the broken SD card were restored by unchk.exe as *.WRI files. However they actually were AVI video files which can be opened with VLC video player, which I didn't know at that time.

When checking unchk.exe with Sysinternals Process Explorer, I've noticed the description says "Identifies and renames CHK files":

So I simply renamed the bigger files from *.CHK to *.AVI using the following powershell-script and it worked fine:

Powershell CHK files to AVI:
Dir *.CHK | rename-item -newname { [io.path]::ChangeExtension($_.name, "AVI") }




Powershell WRI (from unchk.exe) files to AVI:
Dir *.WRI | rename-item -newname { [io.path]::ChangeExtension($_.name, "AVI") }

=> ✅ VLC Video LAN Player was able to open the files

=> ✅ You can directly open the CHK files with VLC Video LAN Player (if they were video files in the first place)

Account Lockout Policy - A possible threat

Most companys use an account lockout policy for their directory service like Microsoft Active Directory, LDAP-system, eDirectory or their own environment solution. Locking out accounts is helpful when it comes to password guessing attacks like brute force (https://attack.mitre.org/techniques/T1110/) or similar ones.

However: An attacker is able to use lockout policies, too.

The following scenario is not so old, but was already used in the last two years:
1. Attacker gains access to environment
2. Attacker creates its own administrative accounts
3. Attacker brute forces all other administrative accounts
👉This way, the attacker makes sure, that all legitime administrators are locked out.

This is an advanced threat and a not very common attack, yet. The solution for this is to make sure, that not all accounts are affected by the lockout policy. Instead very few dedicated emergancy accounts, which are normally not used and stored in a safe location, should be excluded from it. Those accounts (as all administrative accounts) should be monitored closely in your SIEM or logging systems.

Windows 10 Core Isolation deactivation via registry

You can deactivate the Windows 10 Core Isolation function by changing the following Registry Key and therefore do "Tampering with Windows 10 Device Protection Security - Switching off Core Isolation or HypervisorEnforcedCodeIntegrity"

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\DeviceGuard\Scenarios\HypervisorEnforcedCodeIntegrity

Set "Enabled" to

• 0 = deactivated (off)
  
• 1 = activated (on)
  

Microsoft about Windows 10 Core Isolation or HypervisorEnforcedCodeIntegrity: "Core isolation provides added protection against malware and other attacks by isolating computer processes from your operating system and device. Select 'Core isolation details' to enable, disable, and change the settings for core isolation features."

https://support.microsoft.com/en-us/help/4096339/windows-10-device-protection-in-windows-defender-security-center

Mitre ATT&CK tactic: Persistence, Defense Evasion

Mitre ATT&CK sub-technique of T1060 or T1019

Required Permissions: HKEY_LOCAL_MACHINE keys require administrator access to create and modify

OS Credential Dumping - Att&ck T1003

Some places to start to monitor (e.g. for unexpected processes interacting with one of the following), collector forensics and try to protect for OS Credential Dumping Mitre Att&ck T1003 are:

Place ----- ATT&CK Subtechnique ID

LSASS Memory -----T1003.001

Security Account Manager ----- T1003.002

NTDS ----- T1003.003

LSA Secrets ----- T1003.004

Cached Domain Credentials ----- T1003.005

DCSync ----- T1003.006

Proc Filesystem ----- T1003.007

/etc/passwd ----- T1003.008

/etc/shadow ----- T1003.008

Each attack technique is described with examples, as well as possible mitigations and detections.

SIEM IoC regsvr32.exe outbound network connection

An easy to find possible indicator of compromise (IoC) for your SIEM, AEP or EDR could be a outbound network connection from Windows own register server regsvr32.exe (Microsoft Docs or Wiki). Normally the register server never establishes an outbound network connection to the internet. It is a commonly used evasion technique to avoid detection and has its own MITRE Att&ck technique with ID T1117 (or new sub-techniques T1218/010 and can be mapped to the MITRE Att&ck tactics Execution TA0002 and Defense Evasion TA0005.

A starting point can be searching your SIEM logs for network connections from regsvr32.exe to a not RFC1918 private ip address and your IPv6 address space.

Mitigations could be using the Windows firewall to block outbound network connections from regsvr32.exe or as MITRE Att&ck writes:

"Microsoft's Enhanced Mitigation Experience Toolkit (EMET) Attack Surface Reduction (ASR) feature can be used to block regsvr32.exe from being used to bypass whitelisting. Identify and block potentially malicious software executed through regsvr32 functionality by using application whitelisting tools, like Windows Defender Application Control, AppLocker, or Software Restriction Policies where appropriate."

More useful searches for Splunk & Sysmon environments can be found on Github, example: https://github.com/mitre-attack/car/issues/11 and testing if your AEP/EDR/Sysmon or log-collection-tool actually logs regsvr32 events is described here: https://github.com/redcanaryco/atomic-red-team/blob/master/atomics/T1117/T1117.md
 

Sysmon 11 released

Many SIEM installations use sysinternals sysmon as one of many data sources. Mark Russinovich (Microsoft Azure CTO, co-creator of sysinternals) released a video explaining some of the new features of Sysmon 11, which was released on 28th April.

A new useful feature is archiving a file, just before it is deleted. Some attackers delete their tools after gathering information. In order to understand their tools or even search for MD5 or imphashes, the sysmon 11 archiving function can be helpful.


(Source: https://www.youtube.com/watch?v=_MUP4tgdM7s)

Geographically redundant datacenters - their performance issues and designs solutions

Performance problem - Distance and Latency

More and more companys face the challenging requirement to provide available services, which survive regional desasters. In other words: Their datacenters must be located in different geographical regions. High availability solutions, storage systems, systems which do a lot of sequential or write-operations often require very low latency, in order to work stable, especially in peak or high load situations.

I recently saw an application, which had performance issues with some daily jobs, as soon the application server was running in another datacenter, which was connected by dedicated 2x 40Gbit/s lines with about 5 kilometers distance and 0.2ms latency in between the application server and the database servers. As soon as the application server was moved to the same datacenter as the datacase, these jobs only took about 1 hour instead of 8 hours.
👉These 0.2ms latency added up to slow down the application drastically
👉Latency is critical when it comes to performance (see also Bandwidth delay product or Long Fat Pipes LFN in RFC1072)

But: As soon you have the requirement to have datacenters separated hundreds of kilometeres, in order to be geographically redundant, your latency will grow, after you can't change the speed of light.

👉If the datacenters in the example wouldn't be only separated by 5 kilometers, but by 200 kilometers, latency would grow from 0.2ms to 3-5ms (or more), which would be the death of the application, because those jobs would take too long.

That results in: You can't (and shouldn't) stretch applications, high availability clusters, SANs, etc over long pipes, even if they are fat. E.g. even if you have many 100GBit/s directly connected dark fibre connections between your datacenters, the higher the latency, the worse your performance. (Of course bandwidth is important, too, as well as other factors)

How do others do it?

Microsoft released some documents, which talk about possible solutions:
https://docs.microsoft.com/en-us/sharepoint/administration/plan-for-disaster-recovery

• Cold standby. A secondary data center that can provide availability within hours or days.
• Warm standby. A secondary data center that can provide availability within minutes or hours.
• Hot standby. A secondary data center that can provide availability within seconds or minutes.

[...]
Important: Available network bandwidth and latency are major considerations when you are using a failover approach for disaster recovery. We recommend that you consult with your SAN vendor to determine whether you can use SAN replication for SQL databases or another supported mechanism to provide the hot standby level of availability across data centers.

There is also a video of the great Mark Russinovich (Microsoft Azure CTO) https://www.youtube.com/watch?v=X-0V6bYfTpA

(Source: Youtube https://www.youtube.com/watch?v=X-0V6bYfTpA)

In this video Microsoft talks about its Azure datacenter requirements and mentions, that within a region, the network latency perimeter must stay under 2ms. Mark Russinovich also mentions, that therefore Datacenters are within a 100 kilometers range of each other.

There are other useful documents, for example for datacase redundancy, which is for example about business continuity planning, recovery point objective (RPO) and estimated recovery time (ERT), as well as thinking about how to monitoring a failure of a site and how ERT is affected not only by the cluster switch, but also by failure detection time + DNS TTL.
https://docs.microsoft.com/en-us/azure/sql-database/sql-database-designing-cloud-solutions-for-disaster-recovery


There is also a great article from Percona.com-Blog, which talks about this issue: https://www.percona.com/blog/2018/11/15/how-not-to-do-mysql-high-availability-geographic-node-distribution-with-galera-based-replication-misuse/

We had two datacenters.

• The connection between the two was with fiber
• Distance Km ~400, but now we MUST consider the distance to go and come back. This because in case of real communication, we have not only the send, but also the receive packages.
• Theoretical time at light-speed =2.66ms (2 ways)
• Ping = 3.10ms (signal traveling at ~80% of the light speed) as if the signal had traveled ~930Km (full roundtrip 800 Km)
• TCP/IP best at 48K = 4.27ms (~62% light speed) as if the signal had traveled ~1,281km
• TCP/IP best at 512K =37.25ms (~2.6% light speed) as if the signal had traveled ~11,175km

 Given the above, we have from ~20%-~40% to ~97% loss from the theoretical transmission rate. Keep in mind that when moving from a simple signal to a more heavy and concurrent transmission, we also have to deal with the bandwidth limitation. This adds additional cost. All in only 400Km distance.
This is not all. Within the 400km we were also dealing with data congestions, and in some cases the tests failed to provide the level of accuracy we required due to transmission failures and too many packages retry.

[...]
What Is the Right Thing To Do? 

The right solution is easier than the wrong one, and there are already tools in place to make it work efficiently. Say you need to define your HA solution between the East and West Coast, or between Paris and Frankfurt. First of all, identify the real capacity of your network in each DC. Then build a tightly coupled database cluster in location A and another tightly coupled database cluster in the other location B. Then link them using ASYNCHRONOUS replication.

(Source: https://www.percona.com/blog/2018/11/15/how-not-to-do-mysql-high-availability-geographic-node-distribution-with-galera-based-replication-misuse/)

Conclusion

👉 Unfortunately there is no easy solution for this. Each application, system and solution has it's own requirements and possible solutions. But: Don't underestimate the distance between your datacenters and the latency comming with it, even if only a few milliseconds don't sound much.

List FortiGate Certificates via CLI - CA certificates and local Certificates

You can either use the GUI of the FortiGate to list all certificates, or use the CLI. Either using the commands:

Using the "get" command

config vdom
edit root   #<--- your management vdom/your vdom of choice
get vpn certificate ca

FGT50E00000000 (root) #
FGT50E00000000 (root) # get vpn certificate ca
== [ Fortinet_Wifi_CA ]
name: Fortinet_Wifi_CA
== [ Fortinet_CA ]
name: Fortinet_CA
== [ ACCVRAIZ1 ]
name: ACCVRAIZ1
== [ AC_RAIZ_FNMT-RCM ]
name: AC_RAIZ_FNMT-RCM
== [ Actalis_Authentication_Root_CA ]
name: Actalis_Authentication_Root_CA
[...]

Using the "show" command

The show command might not be very helpful, because it does not necessarily show all certificates:

FGT50E00000000 (root) #
FGT50E00000000 (root) # show vpn certificate ca
config vpn certificate ca
end
FGT50E00000000 (root) # show full-configuration vpn certificate ca
config vpn certificate caend

FGT50E00000000 (root) # show full-configuration | grep -f 'vpn certificate ca'
config vpn certificate ca <---
end
FGT50E00000000 (root) #

Using the "fnsysctl" command

Using the fnsysctl command might be helpful:

FGT50E00000000  #
FGT50E00000000 # fnsysctl ls -la /etc/cert/local/
drwxr-xr-x    2 0    0   Wed Dec 25 21:43:14 2019        0 .
drwxr-xr-x    6 0    0   Wed Sep 18 20:39:27 2019        0 ..
-rw-------    1 0    0   Wed Sep 18 20:35:46 2019     2250 root_2020jan_sub.domain.tld.cer
-rw-------    1 0    0   Wed Sep 18 20:35:46 2019     1704 KEY-FILE
-rw-------    1 0    0   Wed Sep 18 20:35:46 2019     1407 root_Fortinet_CA_SSL.cer
-rw-------    1 0    0   Wed Sep 18 20:35:47 2019     1704 KEY-FILE
-rw-------    1 0    0   Wed Sep 18 20:35:47 2019     1419 root_Fortinet_CA_Untrusted.cer
-rw-------    1 0    0   Wed Sep 18 20:35:47 2019     1704 KEY-FILE
-rw-------    1 0    0   Wed Sep 18 20:35:47 2019     4285 root_Fortinet_Factory.cer
-rw-------    1 0    0   Wed Sep 18 20:35:47 2019     1679 KEY-FILE[...]
FGT50E00000000  #
FGT50E00000000  # fnsysctl ls -la /etc/cert/ca
drwxr-xr-x    2 0    0   Wed Dec 25 21:41:28 2019        0 .
drwxr-xr-x    6 0    0   Wed Sep 18 20:39:27 2019        0 ..
-rw-------    1 0    0   Wed Sep 18 20:35:55 2019      119 ca_bundle_ver
-rw-------    1 0    0   Tue Jan 14 20:06:15 2020 1972 root_AC_RAIZ_FNMT-RCM.cer
-rw-------    1 0    0   Tue Jan 14 20:06:15 2020 2772 root_ACCVRAIZ1.cer
-rw-------    1 0    0   Wed Sep 18 20:35:55 2019     2041 root_ACEDICOM_Root.cer
-rw-------    1 0    0   Tue Jan 14 20:06:15 2020 2049 root_Actalis_Authentication_Root_CA.cer
-rw-------    1 0    0   Tue Jan 14 20:06:14 2020 1521 root_AddTrust_External_Root.cer[...]

VMWare ESXi VM CPU Performance Over Commitment "CPU Stuck"

Many VMWare ESXi installations make the same mistake: They overcommit vCPUs, don't monitor CPU metrics like %RDY, %CSTP and don't know why their virtual machines are slow or have sometimes performance issues, especially in load situations. Sometimes you can find hints like "kernel BUG: soft lockup - CPU stuck for 22seconds" in your logs, but most aren't aware of anything.

What can be the cause of this issue?

A very good explanation about ESXi CPU Scheduling can be found here: https://www.youtube.com/watch?v=8jeBIvzyB80 

It explains how the hypervisor ESXi schedules the physical CPUs to the virtual vCPUs. And here is the issue. For example:

Picture from YouTube Video "The vSphere CPU Scheduler" of "TrainerTests"

 As shown in the screenshot overcommiting the physical CPU by assigning to many vCPUs to VMs may decrease the VMs performance, because you waste time slots.

This can be monitored by monitoring the following CPU metrics:

Which ESXi metrics should be monitored?

·         %USED tells you how much time did the virtual machine spend executing CPU cycles on the physical CPU.

·         %RDY (should be very low) is a very important performance indicator. Start with this one. This one defines how much time your virtual machine wanted to execute CPU cycles but could not get access to the physical CPU. It tells you how much time did you spend in a “queue”. Expect this value to be better than 5% (this equals 1000ms in the vCenter Performance Graphs read about it here)

·         %CSTP (should be 0.0%) tells you how much time a virtual machine is waiting for a virtual machine with multiple vCPU to catch up. If this number is higher than 1% you should consider lowering the amount of vCPU in your virtual machine.

·         %WAIT It is the percentage of time the virtual machine was waiting for some VMkernel activity to complete (such as I/O) before it can continue.

·         %IDLE (should be high) The percentage of time a world is in idle loop.
From http://www.vstellar.com/2015/10/09/understanding-cpu-over-commitment/

👉 High %RDY indicates that vCPUs are waiting for actual physical CPUs to schedule their threads
👉 High %CSTP indicates that ESXi is asking the vCPUs to wait – A.K.A. co-stopping them for scheduling purposes -> decrease vCPUs of VM

Small/Medium Businesses - New network devices (switches, routers,..) - Minimum ToDo list

Most small/medium businesses don't do much configuration, monitoring, cfg-baselineing or follow best practises with their network devices like switches, routers, wireless controllers, access points etc. Here is a short list of things you should do at minimum:

1. Extend your monitoring of your network devices

Not only ping them, check their uptime by snmp, but:
1.1 Monitor all uplinks (e.g. SNMP bandwidth)
1.2 Monitor all important ports (ports of the servers, firewalls, storage, etc; again e.g. with SNMP bandwidth)
1.3 Monitor device health, fan status, temperature, etc
1.4 Monitor the routing table, especially if you use dynamic routing protocols and/or have many routes
1.5 Monitor utilization of cpu, mem, i/o, etc..
1.6 Monitor everything with secure protocols like SSH, SNMPv3 AuthPriv AES+SHA
1.7 Send SNMP traps from devices to your monitoring system
1.8 Send Syslog from your devices to your monitoring system & logging solution

2. Harden your network

2.1 Disable telnet
2.2 Disable http
2.3 Implement ACLs for allowing access only from dedicated trusted hosts
2.4 Implement ACLs for dynamics routing protocols like BGP, OSPF, etc
2.5 Use LDAPS/Radius authentication for LDAP/AD-authentication for device mgmt
2.6 Send Syslog from your devices to your monitoring system & logging solution
2.7 Disable SNMPv1/v2c
2.8 Use DHCP-Snooping for Rouge DHCP server protection
2.9 Use ARP Spoofing Protection
2.10 Think about disabling link-layer discovery protocols like LLDP, CDP, EDP, etc
2.11 Allow local admin account login only if LDAPS/Radius server is not reachable
2.12 Delete default users, groups and communities

3. Authentication & dynamische vlan assignment

3.1 Use IEEE 802.1x with certificates (at least two AAA Radius Serves (e.g. FreeRadius) with EAP-TLS)
3.2 Use rfc3580 for dynamic vlan assignment
3.3 Think about using either a quarantine fallback vlan for not authenticated clients or a guest vlan with internet access only
3.4 Think of using DHCP Snooping (forwarding) for your devices which does device fingerprinting

4. Documentation

4.1 Create a layer1 and layer2 network plan (e.g. in visio)
4.2 Create a layer2 and layer3 network plan (e.g. in visio)
4.3 Use the l2&l3 plan as background for your monitoring system in a map to have a live-overview

5. Testing

5.1 Test your loop protection (m/r/stp, loop-protect, elrp, + broadcast limit thresholds like max 200 broadcasts per second, etc) in a maintenance window
5.2 Test your "CrossVlan Protection" in a maintenace window. By "CrossVlan" I mean not wanted connections between to vlans, which should be separated (m/r/stp, loop-protect, extra VLAN which is tagged on all ports and sends ELRP or similar loop protection protocols, etc)
5.3 Test your monitoring alerting - is an alert really send when e.g. an important uplink is full or disconnected, if an important lacp lag is down, etc (test using simulation, e.g. via jPerf, Observer etc)
5.4 Check and test if all best practises of the vendor are applied

6. IP-Subnetting

Yes, so many small and medium companys still have a huge flat layer2 network per site :(
6.1 The more subnets, the more a network issue stays only in that tiny subnet
6.2 The smaller the subnet, the less background noise
6.3 Microsegmentation is key! The smaller the subnet and the more it is separated (using private vlans, ACLs, a firewall, filtering device, host firewalls, a microsegmentation solution, NSX-T or something similar), the more it is protected and lateral movement gets harder.

There are many more things, like using LACP instead of static link aggregation groups, using LACP Mode Fast instead of the default slow, using Bidirectional Forwarding Detection "BFD" for everything, using multi-chassis link-aggregation (like MC-LAG, MLAG, etc) instead of Stacking (firmware-updates & reboots mostly cost the whole stack-topology to reboot, which is not the case in MLAG), using Out of Band management, and much more.

The listed items are the things which should be done at minimum.

FortiGate allows Ping from "not trusted hosts" since FortiOS 6.0

Recently I discovered something after updating a FortiGate cluster, which I intensively monitor, not only via working monitoring queries, but also doing some negative monitoring*: Since FortiOS 6.0 the Fortinet FortiGate firewall answers to source addresses via ICMP Type:8 EchoRequest "Ping", which are not included in the trustedhosts (config system admin -> edit admin -> set trusthost1). This is due to a change in FortiOS, which allows ping from more addresses. To limit ping, you have to use LocalIn-Policies (which always should be used, too).

*negative monitoring = I recommend not only to monitor what is allowed (like your SNMPv3 AuthPriv queries, your SSH or HTTPS-Calls), but also to monitor what is forbidden, in order to find some possible bugs or misconfigurations, if suddenly something is answering what really shouldn't. This can be done by trying to use insecure protocols like Telnet, SNMPv1/v2c, HTTP, etc.. or by sending requests from not allowed source ips or by trying to login using deleted/altered accounts or with wrong credentials.


Fortinet documentation:

In versions 5.x and below, trusted hosts configured by an administrator user only allow access from certain IP addresses configured in trusted hosts, to all services configured on the interface, including ping.

From version 6.0 onwards ping service on management interfaces are not included within the scope of trusted hosts. This means that you will be able to ping the interface from an IP that is not included within trusted hosts.

In order to only allow trusted hosts to be able to ping the interface and deny everyone else, you will need to configure a Local In Policy as below.

CLI configuration:

System > Administrators >
config system admin
    edit "admin"
        set trusthost1 172.26.73.48 255.255.255.255
        set accprofile "super_admin"
        set vdom "root"
    next
end

Configuring address and address group as per the trusted hosts:

config firewall address
    edit "trusted-1"
        set type ipmask
        set comment ''
        set visibility enable
        set associated-interface ''
        set color 0
        set allow-routing disable
        set subnet 172.26.73.48 255.255.255.255
    next
end

config firewall addrgrp
    edit "trusted_grp"
        set member "trusted-1"
        set comment ''
        set visibility enable
        set color 0
    next
end

Configuring Firewall local in policies:

config firewall local-in-policy
    edit 2
        set intf "port1"
        set srcaddr "trusted_grp"
        set dstaddr "all"
        set action accept
        set service "PING"
        set schedule "always"
        set status enable
        set comments ''
    next
    edit 1
        set intf "port1"
        set srcaddr "all"
        set dstaddr "all"
        set action deny
        set service "PING"
        set schedule "always"
        set status enable
        set comments ''
    next
end

Before configuring the local in policy:

diagnose sniffer packet any 'host 172.26.73.78 and icmp' 4
interfaces=[any]
filters=[host 172.26.73.48 and icmp]
3.647787 port1 in 172.26.73.78 -> 10.5.22.114: icmp: echo request
3.647850 port1 out 10.5.22.114 -> 172.26.73.78: icmp: echo reply
4.651341 port1 in 172.26.73.78 -> 10.5.22.114: icmp: echo request
4.651383 port1 out 10.5.22.114 -> 172.26.73.78: icmp: echo reply
5.657949 port1 in 172.26.73.78 -> 10.5.22.114: icmp: echo request
5.657992 port1 out 10.5.22.114 -> 172.26.73.78: icmp: echo reply

After configuring the local in policies:

diagnose sniffer packet any 'host 172.26.73.78 and icmp' 4
interfaces=[any]
filters=[host 172.26.73.48 and icmp]
4.264950 port1 in 172.26.73.78 -> 10.5.22.114: icmp: echo request
8.904217 port1 in 172.26.73.78 -> 10.5.22.114: icmp: echo request
13.906576 port1 in 172.26.73.78 -> 10.5.22.114: icmp: echo request
18.893835 port1 in 172.26.73.78 -> 10.5.22.114: icmp: echo request

trace_id=22 func=print_pkt_detail line=5497 msg="vd-root:0 received a packet(proto=1, 172.26.73.78:1->10.5.22.114:2048) from port1. type=8, code=0, id=1, seq=335."
id=20085 trace_id=22 func=init_ip_session_common line=5657 msg="allocate a new session-00874fe6"
id=20085 trace_id=22 func=vf_ip_route_input_common line=2591 msg="find a route: flag=80000000 gw-10.5.22.114 via root"
id=20085 trace_id=22 func=fw_local_in_handler line=409 msg="iprope_in_check() check failed on policy 1, drop

Reply for the trusted host:

diagnose sniffer packet any 'host 172.26.73.48 and icmp' 4
interfaces=[any]
filters=[host 172.26.73.48 and icmp]
7.239647 port1 in 172.26.73.48 -> 10.5.22.114: icmp: echo request
7.239743 port1 out 10.5.22.114 -> 172.26.73.48: icmp: echo reply
8.261081 port1 in 172.26.73.48 -> 10.5.22.114: icmp: echo request
8.261122 port1 out 10.5.22.114 -> 172.26.73.48: icmp: echo reply
9.276261 port1 in 172.26.73.48 -> 10.5.22.114: icmp: echo request
9.276321 port1 out 10.5.22.114 -> 172.26.73.48: icmp: echo reply
10.294536 port1 in 172.26.73.48 -> 10.5.22.114: icmp: echo request
10.294588 port1 out 10.5.22.114 -> 172.26.73.48: icmp: echo reply

Diag debug flow on FortiOS 6.0.9
Source IP of ping is not configured in trustedhosts

2020-02-26 08:38:47 id=20085 trace_id=255 func=init_ip_session_common line=5684 msg="allocate a new session-00018a90"
2020-02-26 08:38:47 id=20085 trace_id=255 func=vf_ip_route_input_common line=2591 msg="find a route: flag=80000000 gw-10.128.36.35 via root"
2020-02-26 08:38:47 id=20085 trace_id=256 func=print_pkt_detail line=5519 msg="vd-root:0 received a packet(proto=1, 10.128.36.35:1->10.240.161.178:0) from local. type=0, code=0, id=1, seq=1156."


Diag debug flow on FortiOS 5.6.12
Source IP of ping is not configured in trustedhosts

2020-02-26 08:46:05 id=20085 trace_id=10 func=print_pkt_detail line=5375 msg="vd-root received a packet(proto=1, 10.240.161.178:1->10.128.36.4:2048) from mgmt1. type=8, code=0, id=1, seq=1166."
2020-02-26 08:46:05 id=20085 trace_id=10 func=init_ip_session_common line=5534 msg="allocate a new session-fd0ad6af"
2020-02-26 08:46:05 id=20085 trace_id=10 func=vf_ip_route_input_common line=2574 msg="find a route: flag=80000000 gw-10.128.36.4 via root"
2020-02-26 08:46:05 id=20085 trace_id=10 func=fw_local_in_handler line=402 msg="iprope_in_check() check failed on policy 0, drop"

Source: https://kb.fortinet.com/kb/documentLink.do?externalID=FD44156

Advanced Endpoint Protection Testing from MITRE using ATT&CK

If you are searching for test results about current Advanced Endpoint Protection/Endpoint Detection and Response tools: MITRE is transparently testing some of them using the great ATT&CK map.

Round 1 Testing

In round 1 the following AEP/EDR products were tested:

• Carbon Black
• Crowdstrike
• Endgame
• GoSecure
• Windows Defender ATP
• RSA NetWitness
• SentinelOne

The following vendors/products will follow:

• Cyberreason
• F-Secure Countercept
• Fireeye Endpoint Security
• McAfee MVision
• Palo Alto WildFire Traps XDR 

Example

As an example you can look at the results of the Microsoft Windows Defender ATP results. You can see all techniques which were tested and how the product worked. You even can see screenshots of it:

Screenshot of Microsoft Defender ATP of MITRE ATT&CK Evaluation Round1 Testing 1.A.1 User Execution T1204:

Summary

This is amazing work done by MITRE! 👍 It provides transparency of the Advanced Endpoint Protection AEP/Endpoint Detection and Response EDR products, lets you compare them, check which techniques they protect against and how they work in terms of forensics.

Round2 is already running. 👌

Solution for Skybox connection issue to Fortinet FortiGate or FortiManager

If you are using the Skybox (https://www.skyboxsecurity.com/) solution for your environment, during the initial setup there might be an issue in the connection from the Skybox to your Fortinet FortiGate firewalls or your or FortiManager firewall management system. Skybox uses HTTPS (XML API with SOAP) and/or SSH to connect to the Fortinet systems.

If your hardening of the Fortinet devices changed the default minimum Diffie-Hellman Exchange-Bits from 2048 to 3072, 4096 or 8192Bits, then your Skybox is not able to connect to them, after Skybox currently does not support more than 2048Bit for DH.

FortiGate Hardening of DH-Bits:

config sys global
 set dh-params 8192
end

See also the Skybox documentation "ReferenceGuide", e.g. here "Reference Guide v10.801" or an overview of the documentation of all last versions.

Malware using PowerShell - PowerShell Logging "Script Block Logging"

More and more so called "fileless malware" uses powershell in order to execute malicious actions. In order to find possible malicious powershell commands or ps-scripts, it is very useful to log them, automatically send them to your SIEM and analyze them. Also if possible, disable Powershell for your users, but this is in the real-world sometimes hard or even not possible.

1. Disable PowerShell for users if possible via GPO
2. Fileless malware using PowerShell - PowerShell Logging using Script Block Logging

https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_logging_windows?view=powershell-6#enabling-script-block-logging

Enabling Script Block Logging

When you enable Script Block Logging, PowerShell records the content of all script blocks that it processes. Once enabled, any new PowerShell session logs this information. It's recommended to enable Protected Event Logging.

Using Group Policy

To enable automatic transcription, enable the Turn on PowerShell Script Block Logging feature in Group Policy through Administrative Templates -> Windows Components -> Windows PowerShell.

Using the Registry

Run the following function:

PowerShell

function Enable-PSScriptBlockLogging
{
    $basePath = 'HKLM:\Software\Policies\Microsoft\Windows' +
      '\PowerShell\ScriptBlockLogging'

    if(-not (Test-Path $basePath))
    {
        $null = New-Item $basePath -Force
    }

    Set-ItemProperty $basePath -Name EnableScriptBlockLogging -Value "1"
}

Powershell-Commands will be logged in Windows Eventlog with Event-ID 4104.

Use OpenSSL to decrypt private key

Use OpenSSL to decrypt private key

openssl rsa -in *encrypted-key-file* -out *decrypted-key-file*

Example:

Encrypted private key file: /etc/ssl/private/sub.domain.tld_201908.enc.key 
New decrypted private key file: /etc/ssl/private/sub.domain.tld_201908.key 

root@eulinxhost119:/etc/ssl/private#
root@eulinxhost119:/etc/ssl/private# openssl rsa -in ./sub.domain.tld_201908.enc.key -out sub.domain.tld_201908.key
Enter pass phrase for ./sub.domain.tld_201908.enc.key:
writing RSA key
root@eulinxhost119:/etc/ssl/private#
root@eulinxhost119:/etc/ssl/private#

Opening the old encrypted private key: File starts with:
-----BEGIN ENCRYPTED PRIVATE KEY----- 

 Opening the new unencrypted private key: File starts with:
-----BEGIN RSA PRIVATE KEY-----

Use OpenSSL to verify the private key

openssl rsa -in certkey.key –check

Example:

root@eulinxhost119:/
root@eulinxhost119:/etc/ssl/private# openssl rsa -in sub.domain.tld_201908.key -check
RSA key ok
writing RSA key
-----BEGIN RSA PRIVATE KEY-----
[...]
-----END RSA PRIVATE KEY-----
root@eulinxhost119:/
root@eulinxhost119:/

OpenSSL documentation and examples

OpenSSL man-page: https://www.openssl.org/docs/manmaster/man1/openssl.html
OpenSSL examples: https://wiki.openssl.org/index.php/Command_Line_Utilities

Notepad++ edit previously opened sessions/files

Sometimes I have so many or too large files opened with Notepad++ (GitHub) --> it stops starting the next time and hangs.

To resolve this, simply either delete or edit the session.xml-file:
C:\Users\\AppData\Roaming\Notepad++\session.xml 
or
%AppData%\Notepad++\session.xml 

Each entry starting with "<File lang.." and ending with "/>" is a previously opened file. Example:
<File lang="Normal Text" mapIsWrap="
8...] 
firstVisibleLine="0"/> 

By deleting entries like this, you throw out that the file is automatically opened the next time when starting Notepad++.
🚩Hint: Delete the last entry, this resolves most times the issue.

SonicWALL SD-NAT (loopback NAT) for internal published services

If you're having a small network with a SonicWALL firewall and you've published a service using destination NAT (DNAT) to the internet (or another interface/networks), you sometimes want to to access the same published service using the same (public) address from the inside.

Example:

When SystemB is in the same subnet or same network zone as the API-Gateway and you want to access the API-Gateway address from SystemB, you can either use SplitDNS (internal DNS resolution resolves the public FQDN to the internal address), or only use public ip-addresses in that network/zone, or provide the API-Gateway multiple virtual addresses, or you use a "lookback NAT" which is a source- and destination-NAT at the same time.

The SonicWALL then changes the destination ip-address of every DMZ packet from the external ip-address of the API-Gateway to the internal ip-address of the API-Gateway and at the same time, changes the source ip-address from SystemB to its own internal interface ip-address. Changing the source-address makes sure that the API-Gateway does not sends the traffic directly back to SystemB, but instead to the SonicWALL, which then changes back the destination-NAT for the answered packets.

In the SonicWALL NAT-Table this setup looks like this:

Of course you have to setup the suiting access-rules in the SonicWALL as well. Keep in mind the SonicWALL internal traffic flow first checks the access rules, then changes the address using the NAT table. So you may have to setup Rules like DMZ>DMZ (or LAN>LAN) but as a destination the WAN-address objects!

I've set this up in many setups for over 10 years with very different SonicWALLs, FortiGates and other Firewalling-devices, and it works well (e.g. tested with SonicWALL on SonicOS3/4/5/6 from TZ-series, NSA-series to SuperMassive-series).
🚩However keep in mind, opening services using NAT might be a possible security-hole (for example VPN might help), publishing services to the internet always should go through an application reverse proxy like a WAF, API-Gateway or something else, the published services should be hardend, closely monitored and be located in a DMZ, have strong encryption and authentication (mutual TLS). Also I personally prefer to use SplitDNS and different virtual addresses in order to keep original ip-addresses (not having to use X-Forwarded-For which might cause lots of logging-volume on your logs/Logging-System/SIEM) etc.

Hardening FortiGate connection to FortiAnalyzer

The FortiGate uses OFTP (Fortinets proprietary Optimized Fabric Transfer Protocol, not Odette File Transfer Protocol) to transfer data between FortiGates and FortiAnalyzer. For encryption SSLv3 or TLS is used. In order to harden the connection, use the following commands:

Cfg for FortiAnalyzer 

 1. Set TLS to minimum version 1.2:


2. Change FAZ certificate to one using the correct FQDN from your/a public CA:



Debugging can be done using the fowlloing commands:

Oftpd daemon test usage:   diagnose test application oftpd <integer>

• 1: show PID
• 2: show statistics and state
• 3: show connected device name and IP
• 4: show detailed session state
• 5: show oftp request statistics
• 6: show cmdb device cache
• 7: show logfwd thread stats
• 8: show tasklist statistics
• 9: show unreg dev cache
• 10: log cluster bridge stats
• 20: show forticlent end-user stats
• 21: rebuild forticlient end-user avatar table
• 90: reload un-reg device tree
• 99: restart daemon

FAZ looks like this:

(global)# get

dh-params           : 8192

enc-algorithm       : high

fgfm-ssl-protocol   : tlsv1.2

oftp-ssl-protocol   : tlsv1.2

ssl-low-encryption  : disable

ssl-protocol        : tlsv1.2

ssl-static-key-ciphers: disable

webservice-proto    : tlsv1.2

Cfg for FortiGate

1. Change TLS ciphers to high:

config log fortianalyzer setting

    set enc-algorithm high

end

2. Change Global TLS-Settings

config system global

    set ssl-static-key-ciphers disable

    set dh-params 8192

    set strong-crypto enable

end

Important Splunk update for timestamp issue starting 01.01.2020

Splunk has identified an issue regarding timestamps, which are intensivly used for data correlation in many Splunkbased logging and SIEM systems. Affected versions are Splunk Enterprise, Splunk Light and Splunk Cloud. This issue has potential significant impact on data ingestion - including causing inaccurate, unsearchable, or prematurely-deleted data - starting January 1, 2020.

Cause of the issue: Timestamps using two-digit years will stop being correctly recognized. Full details around this issue, including workarounds and product fixes, are documented in Release Notes for each Splunk Version: https://docs.splunk.com/Documentation/Splunk/latest/ReleaseNotes/FixDatetimexml2020
 

Possible Solutions:

1. Manual Change of "datetimes.xml"

Change on all Splunk systems (search heads, indexers, heavy forwards etc) the file "datetimes.xml" with the following file: http://download.splunk.com/products/ingest2020/datetime.zip
In order to do that, put the downloaded in $SPLUNK_HOME/etc. (mostly found in /opt/splunk/etc). Then restart the system (in a Indexer Cluster a rolling-restart is possible). Until a splunk patch is available, the warning will be shown, that this file is not part of the splunk manifest. This will be fixed in the future splunk versions.

2. Update to a version with a fix

Splunk will ship minor releases with fixes, soon:

Major Release --- Minor Release with patch
6.6 --- 6.6.12.1 (not yet released)
7.0 --- 7.0.13.1 (not yet released)
7.1 --- 7.1.10 (not yet released)
7.2 --- 7.2.9.1 (not yet released)
7.3 --- 7.3.3 (Installationguide)
8.0 --- 8.0.1 (not yet released)

FortiOS XSS in DHCP-Monitor

Fortinet has released PSIRT FG-IR-19-184 (CVE-2019-6697) about a vulnerability in FortiOS of the FortiGate firewall. A DHCP packet may contain a stored XSS in the hostname parameter field.

Affected Versions and Update

Affected Products are FortiOS 6.2.1 and below & FortiOS 6.0.6 and below.

FortiOS 6.2.2 and FortiOS 6.0.7 fix the vulnerability

PoC/Vulnerability Details from ssd-disclosure.com

Source: https://ssd-disclosure.com/archives/3987/ssd-advisory-fortigate-dhcp-stored-xss

An unauthenticated attacker can trigger a Stored XSS Vulnerability via a malicious DHCP packet in the Fortigate DHCP Monitor.

This can happen if Device Detection is enabled through Network >Interface > Edit Interface > Device Detection

  


 When this option is enabled the attacker may perform the following steps in order to exploit the vulnerability:

1. Install dhtest or any other tool that can send arbitrary DHCP packets.
   (https://sargandh.wordpress.com/2012/02/23/linux-dhcp-client-simulation-tool/)
2. Send a malicious DHCP packet. For example:
   
   #./dhtest-master/dhtest -i eth0 -m 12:34:56:78:90:12 -h "x<svg onload=alert();)>x" [Option] -m : mac address -h : hostname(dhcp option 12). The attacker can inject malicious scripts.

   1 2 3 4

   #./dhtest-master/dhtest -i eth0 -m 12:34:56:78:90:12 -h "x<svg onload=alert();)>x"     [Option]     -m : mac address     -h : hostname(dhcp option 12). The attacker can inject malicious scripts.