Topics

1

23.1 Legacy Series / [SOLVED] Suricata prevents access to OPN itself

« on: April 02, 2023, 01:11:12 pm »

Since the upgrade to 23.1 I noticed that a few minutes after booting up I lose all connectivity to OPN itself. As in, I can't get in the web GUI or even SSH into it, and existing sessions disconnect with "broken pipe" errors. Traffic still properly passes through OPN though, and oddly enough it does still respond to pings. I finally figured out this was due to Suricata:

• Reboot OPN simply by power cycling
• Continuously try to SSH into OPN and kill Suricata (while true; do ssh root@opn killall -9 suricata; sleep 3; done)
  
• When it succeeds, stop the loop and open a regular SSH session
• This session keeps working indefinitely, unlike before
• Go into OPN web GUI and change Suricata to stop listening on the interface I would access OPN through
• Start Suricata again
• SSH keeps working, but other networks are once again unable to access OPN
• Suricata isn't logging an alert about this, so it's not somehow a rule that's blocking it (I even added a pass rule for my specific IP)
• I also made sure sshlockout wasn't triggered (list is still empty)

I think there may be a problem with Suricata opening the same interface multiple times. I checked the latest.log and these are the only lines that appear right before I lose connectivity:

Code: [Select]

[meta sequenceId="1"] [102930] <Notice> -- opened netmap:igb3/R from igb3: 0x82a915000
[meta sequenceId="2"] [102930] <Notice> -- opened netmap:igb3^ from igb3^: 0x82a915300
[meta sequenceId="3"] [102930] <Notice> -- opened netmap:igb3^ from igb3^: 0x8556f4000
[meta sequenceId="4"] [102930] <Notice> -- opened netmap:igb3/T from igb3: 0x8556f4300

[meta sequenceId="5"] [102930] <Notice> -- opened netmap:igb2/R from igb2: 0x8804f4000
[meta sequenceId="6"] [102930] <Notice> -- opened netmap:igb2^ from igb2^: 0x8804f4300
[meta sequenceId="7"] [102930] <Notice> -- opened netmap:igb2^ from igb2^: 0x8ab2f4000
[meta sequenceId="8"] [102930] <Notice> -- opened netmap:igb2/T from igb2: 0x8ab2f4300

I'm guessing the /R and /T are for receive and transmit, but then what are the 2 lines with ^? I think those being present (either twice, or besides the R/T variants) might cause Suricata to kind of lose track of traffic destined for OPN and it disappears into a black hole.

I'm not entirely sure if this is a problem with Suricata itself, or simply the config OPN generates for it. So I'll try here first. =]

2

22.1 Legacy Series / VLANs stop working after reboot

« on: July 15, 2022, 03:32:39 pm »

Every time I reboot OPN my VLANs stop having any outbound connectivity, i.e. through OPN. To get it working again, all I have to do is go to /ui/interfaces/vlan, change the parent interface to something else, click the Apply button and change it back to the proper interface the exact same way. Alternatively, while a tcpdump on any of the involved interfaces (physical or VLAN) is running it also works, of course due to being in promiscuous mode. That means that letting either OPN or Suricata put the interface in promiscuous mode also works, but that shouldn't even be necessary and might actually cause other problems elsewhere.

The VLANs aren't anything special:

• The parent interface is set to have no IP configuration and doesn't block anything (like bogons). It does overwrite the "global settings" so that VLAN hardware filtering is disabled. This way Suricata doesn't even need to run promiscuously in order to see actual VLAN traffic (it's listening only on the physical interface).
• The VLAN interfaces are of course configured statically.
• The switch that connects to OPN has 2 uplinks: one only with tagged VLANs (blocking untagged) and another for only untagged traffic. That should prevent any network loops or somehow passing untagged traffic to the unconfigured parent for the VLANs.
  

I'm on the latest version of OPN. Any ideas?

3

20.7 Legacy Series / NAT reflection issues with double NAT

« on: September 04, 2020, 06:42:20 pm »

I couldn't find any thread about my specific scenario, hence my creation of a new one.

My OPNsense firewall is behind my ISP's modem router (double NAT) for a couple of reasons. We have one ISP connection shared amongst me and a handful of other tenants here and I don't trust them enough to just put the modem router in bridge mode and put OPN right behind it (since I'd have to physically move it into a public space). Also they have no business directly connecting to my devices over LAN instead of WAN anyways, as I run many devices and having to firewall each of them individually would be too much of a hassle (just for fun: I have 36 static IPs as well as a DHCP scope of 101 addresses). Some don't even have firewalling capabilities without writing something for it yourself.

The double NAT in itself doesn't seem to be a problem though and it's not inherently bad regardless, as long as you understand certain key points. OPN has been assigned as the DMZ of the modem router so I can access any ports I've forwarded to devices in my inner network without issue, and even VPNs work just fine. The problem arises when I access my public IP from within. Since OPN doesn't know about that IP, it sends it to the modem router which in turn doesn't do NAT reflection (I believe). I've been having some issues with packets being dropped upstream over the last few days, and if I access e.g. HTTP on my public IP it sometimes errors out. If it did do NAT reflection then upstream issues shouldn't be a problem.

My goal is to reroute traffic to my public IP back into the port forwards so that when I access e.g. port 80, it ends up at my web server. Ideally the web server also sees the client's LAN IP and not the OPN gateway, but I doubt that's possible because the client expects a reply from the gateway. Setting up split-horizon DNS is not really an option; I have many zones, records and servers/services so it would be a major pain to maintain all that. NAT reflection is really the only way.

I figured it would be as simple as attaching a virtual IP to the external interface and making sure NAT reflection is enabled on the port forward rule as well as Firewall > Settings > Advanced > Reflection for port forwards and Automatic outbound NAT for Reflection. However, when I do that then all traffic from inside to the public IP never gets to its destination. I can see it entering OPNsense through tcpdump (marked as "pass" and not "block"), but when I run tcpdump on the web server I don't see a single request coming in. Once I change the virtual IP to something else (last octet +1) then the exact same tcpdump shows a request from my WAN IP.

I even started messing around with outbound NAT, loopback interfaces, policy-based gateways and static routes but it simply wouldn't work reliably. At one point I was getting requests from OPN's IP on entirely different VLANs. :DD

Any ideas?

4

Web Proxy Filtering and Caching / Missing/outdated root CAs

« on: March 18, 2019, 06:46:03 pm »

The web proxy section seemed most accurate for me to post this in, but I suppose it's actually a "core" issue. It's just most noticeable when using squid.

It seems that the ca-root-nss.crt bundle uses slightly outdated root certs. Trying to navigate to certain pages results in:

Code: [Select]

The following error was encountered while trying to retrieve the URL: https://www.dfrobot.com/*
    Failed to establish a secure connection to 49.51.40.62

The system returned:
    (92) Protocol error (TLS code: X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY)
    SSL Certficate error: certificate issuer (CA) not known: /C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
Now, I know Comodo reissued some of their intermediate certs a couple years ago. They previously used SHA1 but migrated to SHA2 (had to deal with it at work too).

I quickly cooked up a script to download any root CA's cert and put it along with its hashed-name version in /usr/local/openssl/certs. Since squid looks there I can now access mentioned site, but Comodo isn't exactly a lesser known CA so it should probably be fixed at the core. : D
Not sure if you (OPNsense devs) even have any control over the ca-root-nss bundle or if it's purely a BSD thing though.

5

Hardware and Performance / Required disk space for "all" services w/ minimal logging?

« on: February 14, 2019, 08:18:45 pm »

I installed OPNSense on a 16 GB mSATA SSD and mounted a USB pendrive to /var and /tmp to save write cycles, but the latter seems too slow to keep up (especially when using c-icap and clamav) and sometimes even causes processes to go into uninterruptible sleep state (D state in ps output) when they're attempting I/O. Now, the official guide recommends a 120 GB SSD when you want to enable most/all features, but I'm not at all interested in squid access logs and the like.

There's currently 2.3 GB in use on the SSD itself and around 650 MB on the USB. The only log file that gets changed a lot is suricata's due to having multiple servers/ports exposed to the internet, so there are many intrusion attempts. The only services I'm not using are any form of DNS (because I have a dedicated VM for it which hosts about 10 zones) and captive portal (due to this being a non-business setting).

My aim is to remove the USB drive from the equation and keep the SSD from filling more than halfway, but I don't know how log-hungry everything is over a longer period of time. Could I use the disk for a couple of years or would it be wiser to just get a larger one, and if so, what size would be best for this setup?