Messages

Has anyone experienced an issue where ICMP is being sent to the loopback address in high volumes?

Here is example logs"

Code Select Expand

<134>1 2024-08-14T11:56:43+00:00 xxxx.acme.tech filterlog 12611 - [meta sequenceId="7416432"] 66,,,1232f88e5fac29a32501e3f051020cac,lo0,match,pass,out,4,0x0,,64,1281,0,none,1,icmp,596,127.0.0.1,127.0.0.1,datalength=576

When looking at the logs there is a rather large volume of traffic to the loopback from the loopback which does not make sense.


Any thoughts or ideas why this would be happening?

Appliance Specs:

Versions:
OPNsense 24.7.1-amd64
FreeBSD 14.1-RELEASE-p3
OpenSSL 3.0.14
CPU:
12th Gen Intel(R) Core(TM) i5-1240P
Memory:
32GB

Installed Modules:
os-acme-client
os-clamav
os-crowdsec
os-dmidecode
os-etpro-telemetry
os-git-backup
os-haproxy    
os-intrusion-detection-content-et-open
os-intrusion-detection-content-et-pro    
os-intrusion-detection-content-pt-open
os-intrusion-detection-content-snort-vrt
os-iperf
os-maltrail
os-sensei
os-sensei-agent
os-sensei-updater
os-sunnyvalley
os-theme-cicada
os-theme-rebellion
os-vnstat

**Question**
Has there been any change in the blacklist updates in the past week?

**Support**
It seems that Maltrail on OPNsense was fine up to 8/7/2024 as far as over all visibility, but after that time it has progressively diminished to the point that there is no hits at all as of today...


8/7/2024
8/12/2024
overall activity by date (maltrail calendar view)

Error Log: https://github.com/stamparm/maltrail/issues/19272

Is there there a way to create custom rules for IDS in 24.7?

I am specifically wanting to create an alert for GRE.

Code Select Expand

# Alert on GRE traffic to a specific IP
alert ip any any -> {IP ADDRESS} any (msg:"GRE traffic to specific IP"; ip_proto:47; sid:1000002; rev:1;)

# Alert on high-volume GRE traffic
alert ip any any -> any any (msg:"High volume GRE traffic"; ip_proto:47; threshold: type both, track by_src, count 1000, seconds 60; sid:1000003; rev:1;)

The current user defined option does not cut it...

[Old and New Logging Formats for FilterLog and Suricata]

All for those that are planning and or dealing with updating to the latest release. There are changes to the log output for Filter and Suricata Logs (syslog output) as compared to the previous versions:

Old and New Logging Formats for FilterLog and Suricata

FilterLog:

Code Select Expand

Old Format: <134>Jul 16 06:36:52 xxxx.acme.tech filterlog[40156]: 85,,,9f96d956119c25145fc2ce221237f3a5,bridge0,match,pass,out,4,0x0,,64,63281,0,DF,17,udp,52,10.13.37.156,8.8.8.8,36444,53,32

New Format:

Code Select Expand

<134>1 2024-08-05T12:05:54+00:00 xxxx.acme.tech filterlog 54802 - [meta sequenceId="2763892"] 85,,,9f96d956119c25145fc2ce221237f3a5,bridge0,match,pass,out,4,0x0,,63,11771,0,DF,6,tcp,60,10.13.37.156,193.0.6.135,43802,43,0,S,1404651214,,64240,,mss;sackOK;TS;nop;wscale

Suricata: Old Format:

Code Select Expand

<173>Jul 26 13:17:32 xxxx.acme.tech suricata[48264]: [1:2017928:4] ET POLICY check.torproject.org IP lookup/Tor Usage check over TLS with SNI [Classification: Device Retrieving External IP Address Detected] [Priority: 2] {TCP} 192.168.200.69:1247 -> 116.202.120.181:443

New Format:

Code Select Expand

<173>1 2024-08-05T23:17:15+00:00 xxxx.acme.tech suricata 23296 - [meta sequenceId="1335381"] [1:2024364:4] ET SCAN Possible Nmap User-Agent Observed [Classification: Web Application Attack] [Priority: 1] {TCP} 192.168.2.141:56840 -> 192.168.88.2:80

Key Changes:

Date Format: The old format used a more condensed representation (e.g., "Jul 26 13:17:32"), while the new format adopts the ISO 8601 standard (e.g., "2024-08-05T23:17:15+00:00"), including timezone information.
Meta Sequence ID: The new format introduces a [meta sequenceId="XXXXXXX"] field, which can be useful for tracking log sequence and detecting missing logs.
Structure: The overall structure of the log entries has been modified to align with more standardized logging practices, potentially improving compatibility with log analysis tools.

While many already be aware, they may have noticed potential impacts to logging and SIEM solutions.  If you use Graylog, I have create a Github Repository that has Grok Patterns to deal with the updated formatting.

If you need a quick work around, here is the link:
https://github.com/secdoc/OPNsense-24.7-Graylog-Grok-Patterns/tree/main

Have you tried New Jersey server? I am getting a lot of drops from the Georgia server.

The Jitter and latency was even worse with New Jersey...

you have tons of underrun as well for the laptop which is an issue. I'd recommend looking at the cables and monitoring the traffic during the zoom call.

The laptop is a 1Gbps interface connected to a multi-gig(10Gbps) interface, so I kinda expect that to a certain perspective., all of the cables are pre made fiber or CAT6e cables.... from a "cable tester" perspective, all of the cables have been vetted...Would the Rx MAC Errors point to anything? They are apparent on the SW/FW ports...

I am also running smokeping to get some statistics and for the most part the info is showing that I am not seeing the amount of packet loss the various zoom/apps note. I have posted samples of Quad9. Areas where there was specific loss is likely associated with Fiber or ISP work.

Are there other thoughts...

Here is the distro SW error. Definitely seeing Rx MAC Errors on the port associated with the FW interface. Also I am not using these as L3 SW, so not sure how to deal with QoS from a L2 perspective...

The QoS is localized to the particular switch/device. So, I'd recommend investigating the traffic on the Mikrotiks.

you stated a 5gbps Internet, but your connection to ISP is 1gbps Cat6e? none the less, the issue may be on the ports on the switch where you can check to see if you see any errors.

The connection (i.e., 1Gbps is dependent on the device...I tried uploading images of all the switches but the forum was saying the images were too large.)...The ATT ONT connects directly to the FW so it does not show in the switches. I have a 10Gbps SFP Fiber connection to the distro SW as a trunk link.

How are you connecting the DEC850 SFP+ to the ISP?

Also, I see your switches are port-channel'd together. How are they connected to the DEC850?

I am connecting via a CAT6e cable. From testing, if I connect directly to the DEC850 and bypass switching, I do not see the issue, SO i am guessing it is a switching or cable issue, but do not find any definitive things to narrow down the issue...IO was hoping, since the DEC850 is the Router/FW/DHCP host for the networks, I would be able to see some type of report or monitor to see what  I may not be able to see in the switches...

The additional screenshots...

Apologies.  Here are the details on the hardware/environment:

FW:

Code Select Expand


OPNsense DEC850
OPNsense 23.10.2-amd64 Business Edition
FreeBSD 13.2-RELEASE-p7
OpenSSL 1.1.1w
Licensed until 2024-12-20


Switches:

Code Select Expand


Mikrotik Switches running SWOS 2.13
1x CRS326-24
2x CRS312-4C-8XG
2x CRS309-1G-8S
2x CRS305-1G-4S


As far as QoS, I have not specified anything explicit. I have ATT Fiber Internet with a 5Gbps synchronous link. This is a business link, so they prefer Data type services.. I have had them come out and they have replaced ONT and some fiber. When I use services like Zoom or other conferencing services, where latency and jitter  can have an impact, I get poor quality and  get  the following, when looking at Statistics in Zoom for example. I am trying to determine where the issue is. If I use https://packetlosstest.com/ I get the following results in the attachment.

As can be seen, the latency and jitter are pretty bad. Just trying to determine best approach to understand what/where to look. I am also including screenshots of the gateway monitor and switch stats/error pages for reference.

Is there a way to monitor  other interfaces/vlans for packet loss? I have been troubleshooting QoS related issues and seeing high packet loss in  video conferencing apps such as Zoom and cannot narrow down where it is at. I have been able to rule out the ISP and the FW itself as dropping, but do not know if it could be in the switching infrastructure or cabling and not seeing anything specific like the health quality monitoring for the WAN gateway...

Any thoughts or ideas would be greatly appreciated...

That is what I figured but but was hoping that the timeline would be slightly sooner, but understand the delayed rollout from a business perspective...

[after]

Never, because that is not how it works.

Take a look at the Announcements section and you will find that the community editions are named YEAR.1.x and YEAR.7.x while the business editions are named YEAR.4.x and YEAR.10.x.

23.10 has just been announced. a few days after 24.1. So, depending on what you aim for, you can either buy that or wait for 23.4.

I guess my question is then when will the BE get the update to the latest version of OpenSSL? Based on what I read or understood the 3.x version of OpenSSL was only be included within the 24.x release..., So was not sure if that is the case with BE or not...

Thanks for the info...