Messages

[does]

I did try that before changing Suricata/netmap versions, but not afterwards. Surprisingly enough that does work now. :> On a certain OPN version I could run Suricata with just VLAN acceleration disabled though, so is the requirement for disabling all acceleration stuff something from newer OPNs/BSDs, or is it perhaps related to the netmap fixes? Or maybe I was just lucky that it used to work at all?

[prevention]

Little bump after doing a bit more research.

So apparently the igbX^ interfaces are simply there due to IPS mode, to allow traffic to still be seen by OPN and all. Setting Suricata to simple detection mode does resolve the connectivity problem, but then I also don't have any actual prevention. :> I think setting it to IDS only changes one config parameter (possibly one per interface), so I'm pretty sure it's not a config problem and more likely related to Suricata and/or netmap themselves.

I did some other searches around the forum and found a few mentions of the Suricata version I'm on (6.0.9) potentially having problems, so I downgraded it to 6.0.8 as per other people's experiences, but nothing changed for me. I also found some issues related to netmap but not exactly related to mine, e.g. stuff with emulated interfaces/generic mode (which I'm not using). There's a newer version of netmap that's (probably) still in internal testing, but I tried it out anyways since Franco did mention another fix. Unfortunately, again nothing changed. I'm doing full reboots every time just to be sure.

[through]

Since the upgrade to 23.1 I noticed that a few minutes after booting up I lose all connectivity to OPN itself. As in, I can't get in the web GUI or even SSH into it, and existing sessions disconnect with "broken pipe" errors. Traffic still properly passes through OPN though, and oddly enough it does still respond to pings. I finally figured out this was due to Suricata:

• Reboot OPN simply by power cycling
  
• Continuously try to SSH into OPN and kill Suricata (while true; do ssh root@opn killall -9 suricata; sleep 3; done)
  
• When it succeeds, stop the loop and open a regular SSH session
  
• This session keeps working indefinitely, unlike before
  
• Go into OPN web GUI and change Suricata to stop listening on the interface I would access OPN through
  
• Start Suricata again
  
• SSH keeps working, but other networks are once again unable to access OPN
  
• Suricata isn't logging an alert about this, so it's not somehow a rule that's blocking it (I even added a pass rule for my specific IP)
  
• I also made sure sshlockout wasn't triggered (list is still empty)
  

I think there may be a problem with Suricata opening the same interface multiple times. I checked the latest.log and these are the only lines that appear right before I lose connectivity:

Code Select Expand


[meta sequenceId="1"] [102930] <Notice> -- opened netmap:igb3/R from igb3: 0x82a915000
[meta sequenceId="2"] [102930] <Notice> -- opened netmap:igb3^ from igb3^: 0x82a915300
[meta sequenceId="3"] [102930] <Notice> -- opened netmap:igb3^ from igb3^: 0x8556f4000
[meta sequenceId="4"] [102930] <Notice> -- opened netmap:igb3/T from igb3: 0x8556f4300

[meta sequenceId="5"] [102930] <Notice> -- opened netmap:igb2/R from igb2: 0x8804f4000
[meta sequenceId="6"] [102930] <Notice> -- opened netmap:igb2^ from igb2^: 0x8804f4300
[meta sequenceId="7"] [102930] <Notice> -- opened netmap:igb2^ from igb2^: 0x8ab2f4000
[meta sequenceId="8"] [102930] <Notice> -- opened netmap:igb2/T from igb2: 0x8ab2f4300


I'm guessing the /R and /T are for receive and transmit, but then what are the 2 lines with ^? I think those being present (either twice, or besides the R/T variants) might cause Suricata to kind of lose track of traffic destined for OPN and it disappears into a black hole.

I'm not entirely sure if this is a problem with Suricata itself, or simply the config OPN generates for it. So I'll try here first. =]

Unfortunately that also prevents Suricata from inspecting VLAN traffic, so that's not an option for me. :>

[Apply]

Every time I reboot OPN my VLANs stop having any outbound connectivity, i.e. through OPN. To get it working again, all I have to do is go to /ui/interfaces/vlan, change the parent interface to something else, click the Apply button and change it back to the proper interface the exact same way. Alternatively, while a tcpdump on any of the involved interfaces (physical or VLAN) is running it also works, of course due to being in promiscuous mode. That means that letting either OPN or Suricata put the interface in promiscuous mode also works, but that shouldn't even be necessary and might actually cause other problems elsewhere.

The VLANs aren't anything special:

• The parent interface is set to have no IP configuration and doesn't block anything (like bogons). It does overwrite the "global settings" so that VLAN hardware filtering is disabled. This way Suricata doesn't even need to run promiscuously in order to see actual VLAN traffic (it's listening only on the physical interface).
  
• The VLAN interfaces are of course configured statically.
  
• The switch that connects to OPN has 2 uplinks: one only with tagged VLANs (blocking untagged) and another for only untagged traffic. That should prevent any network loops or somehow passing untagged traffic to the unconfigured parent for the VLANs.
  

I'm on the latest version of OPN. Any ideas?

No, reflection in conjunction with double NAT, for which the only solution is NATting LANs (since simply attaching a virtual IP doesn't do the trick). ;]

Yeah I get where you're coming from. If they ultimately decide to drop support for interface groups in these rules then I'll have to live with that, but I find it hard to believe I would be the only one with this exact scenario and resulting problems.

I've created an issue on OPN's GitHub anyways, so we'll see what they think about it. =]

[source]

Yes. this is exactly what I meant when I talked about the confusion when using groups in nat rules. it is not clear how to use the group in translation/target. shortly - no way)

But as I said, just the fact that you can choose those groups means it should be supported in an expected manner. Also the whole reason groups exists is so you don't have to duplicate the same rule, which is exactly the scenario I have here. The groups themselves aren't even the problem, it's that OPN's source IP rotates between the interfaces it has in that group instead of using only the interface actually involved. It doesn't even add VLAN tags if it decides to use one of the VLAN's gateway IPs. This is weird, unexpected and undesired behaviour.

and I do not think that Source Hash will help here. it will assign the translated address to the source, but it is not clear which address it will assign.

Yeah I know, it was just a test to see if I could get it off the round-robin algorithm. But no matter what option I pick, it will insist on using it.

if we consider specifically your ASCII scheme and all servers in the igb1 network, then you can try to change your rule like this:
nat on igb1 inet from (AnyLAN: network) to any -> (igb1: 0) port 1024: 65535

I tried that before though:

That works, but it pretty much defeats the point of an interface group. .... If I decide to add a couple extra VLANs I have to remember to add more NAT rules instead of it being automagically handled by making them part of the proper group (AnyLAN).

or try to abandon nat and try asymmetric routing

I don't think asymmetric routing will work in this case though? There's only 1 path between the client and server.

and one more moment:
in rdr rule
rdr on AnyExternal inet proto tcp from any to <Home_WAN> port = http -> <DBWSRV> port 80 round-robin
i dont see "AnyLAN". why dont you want to  reflect client packet at AnyLAN interfaces?

I'm just mostly testing some stuff, see if it works if I use standard rules for external connections and add NAT reflection/WAN loopback on top of that (without modifying the port forward rules themselves). But even if I change the interface to one of the LANs the same issue with the address rotation remains. I may have to change it to AnyLAN at some point anyways, e.g. if the external is down it may not trigger the rule because it's not passing out of the external interface. But I'll mess with that later.

Just to be clear: the redirection works fine when using only AnyExternal, the only remaining problem is OPN's rotation of source IPs when rewriting requests (which falls under Outbound NAT where I'm using AnyLAN, and not Port Forward). I think I'll bug the developers about that, since there's not a lot you can do about it anyways. But at least you helped me figure out the cause of this (I didn't even know about the pfctl command), so thanks for that. ;]

[round-robin]

ASCII is wonderful  :)

Cool. :>

sorry, one unrelated question: what will vlan do in this configuration?

The VLANs are actually not really relevant, besides the fact they're part of the AnyLAN interface group. For example, I have one VLAN set to bypass a couple filtering options, e.g. the HTTP proxy.

on topic. Am I right? do you have only one port forward rule and it contains only external interfaces ? in this case, try to add intrenal interfaces to this rule. why do we need to drive a packet from the internal network to the external interface and back to the internal ones. let it go from one internal to another

I've added AnyLAN to the port forward rule and changed the outbound rule to work with AnyLAN instead of a specific interface. OPN still rotates between the 3 mentioned interface addresses when rewriting requests.

if this is not enough and nat rule with a interfaces group still does not work, then you will need to look at the actual rules that the opnsense generates for pf. while the GUI does not allow it (I just requested this feature https://github.com/opnsense/core/issues/4331). Please, in the shell, give the command pfctl -snat and share the result. it will be interesting how opnsense registers nat rule with interfaces group

Well, that clearly shows the problem:

Code Select Expand


rdr on AnyExternal inet proto tcp from any to <Home_WAN> port = http -> <DBWSRV> port 80 round-robin
nat on AnyLAN inet from (AnyLAN:network) to (AnyLAN:network) -> (AnyLAN:0) port 1024:65535 round-robin

For some reason it's added with a round-robin flag, but even changing the Pool Options to Source Hash doesn't do the trick. It does have a little note saying "Only Round Robin types work with Host Aliases. Any type can be used with a Subnet.". And sure enough, the pfctl command still says round-robin after changing it for both NAT rules. Which is weird imo, since the source address is still a subnet (AnyLAN:network). I generally find it pretty odd that round-robin is the default, I can't think of a good reason why you would want this behaviour (at least not for the source, you could use it for destination for load-balancing purposes).

[NAT > Port Forward]

can you please descripe your setup in more details?

Let's see if my ASCII art turns out ok:

Code Select Expand


OPN igb0 ------------------------------- ISP modem/router
OPN igb1 ----- switch ----- switch ----- server enp0s31f6
                  |
                  |-------- client
OPN igb2 ------------------------------- server enp6s0


The switch nearest the server is a dumb switch and doesn't support jumbo frames, so VLAN tags would be stripped which is why I have my VLAN trunk on a different interface (connected directly to a dedicated interface bridge on the server). My primary LAN is VLAN-less as I only need access to them from my server anyways.

and now i cant understand how your nat rule relates to forward rule.
what is in your port foward rule?

It's pretty simple:

NAT > Port Forward

• Interface: AnyExternal -- an interface group containing my external interfaces
  
• Protocol: TCP
  
• Source Address: *
  
• Source Ports: *
  
• Destination Address: Home_WAN -- an alias containing the external IP of igb0 (local to the ISP router, for its DMZ setting) as well as my actual external IP, if I just keep it at AnyExternal address (or net) then OPN still sends the request to the ISP router (which is to be expected)
  
• Destination Ports: 80
  
• NAT IP: web server
  
• NAT Ports: 80
  

NAT > Outbound

• Interface: AnyLAN
  
• Source: AnyLAN net
  
• Source Port: *
  
• Destination: AnyLAN net
  
• Destination Port: *
  
• NAT Address: Interface address
  
• NAT Port: *
  
• Static Port: no
  

When doing it like that, my web server receives requests from OPN's interface address on my primary LAN, then VLAN A, followed by VLAN B and then back to primary LAN. If I change the Interface in the Outbound NAT rule to Trusted_LAN instead (which is the name of the actual igb1 interface, not a group), then it only shows requests from OPN's IP on that network.

and most important: i realy think that use of interface groups in nat rule is not a perfect idea. its realy confusing. have you tried to make the NAT rules one by one (one rdr rule for specific client-server traffic -> one related nat rule) and check results?

That works, but it pretty much defeats the point of an interface group. The fact that such groups even appear in the settings of the NAT rules shows that they are/should be supported for that. If I decide to add a couple extra VLANs I have to remember to add more NAT rules instead of it being automagically handled by making them part of the proper group (AnyLAN). I don't really see how using a group would be confusing as long as the name is clear, though.

[AnyLAN]

it works. no messing with virtual IP or VLANs needed.

I'm not messing with VLANs as an attempt to fix the issue in this topic though, I simply need VLANs for my network.

it is not necessary that the ip must be on the firewall for rdr rule (port forward) to work.

It's not, I removed it because it only gave me more issues:

However, when I do that then all traffic from inside to the public IP never gets to its destination. I can see it entering OPNsense through tcpdump (marked as "pass" and not "block"), but when I run tcpdump on the web server I don't see a single request coming in. Once I change the virtual IP to something else (last octet +1) then the exact same tcpdump shows a request from my WAN IP.

The method you described generally works fine, except for this:

getting requests from OPN's IP on entirely different VLANs. :DD

Those VLANs are even assigned to a different interface (igb2) than where my HTTP server is connected to (igb1). Oddly enough only the source IP is changed, it doesn't actually get a VLAN tag. For some reason the outbound NAT rule rotates between OPN's interface addresses.

I figured out the cause of this in the meantime and it's not really a problem with the NAT rule in itself. I have an interface group AnyLAN, which contains my primary LAN interface (igb1) as well as the VLAN trunk interface (igb2). Now if you use that group in the outbound NAT rule for the Interface field and its "net" for Source, OPN seems to regard the IPs of all interfaces in that group as "interface address", instead of only the interface actually participating/receiving the client's packet (primary LAN). This results in OPN simply rotating through those IPs in a linear fashion (first primary LAN, then VLAN A, followed by VLAN B and then back to primary LAN). This seems like a bug to me, could you try reproducing that on your end?

since opnsense knows nothing about real external IP

Well I simply figured that by attaching a virtual IP to the external interface, it would automatically pick up on that address and use it in its auto-reflection rules.

one outbound rule for all traffic from lan to lan:
interface: LAN, source: LAN Net, source port: tcp/*, destination: Lan Net, dest port: tcp/*, NAT address: interface address.

and port forward rules for your services:
eg if want to do this with tcp 80:
Port Forward:source: interface LAN, proto tcp, address LAN net. Destination: *YourRealPublicIP*, ports: HTTP/S, redirect target: *lanaddressofyourhttpserver*, port HTTP/S

Yeah exactly that is one of the earlier things I've tried, but in that case this happens:

getting requests from OPN's IP on entirely different VLANs. :DD

Those VLANs are even assigned to a different interface (igb2) than where my HTTP server is connected to (igb1). Oddly enough only the source IP is changed, it doesn't actually get a VLAN tag. For some reason the outbound NAT rule rotates between OPN's interface addresses.

Pretty much all of them.

[Reflection for port forwards]

I couldn't find any thread about my specific scenario, hence my creation of a new one.

My OPNsense firewall is behind my ISP's modem router (double NAT) for a couple of reasons. We have one ISP connection shared amongst me and a handful of other tenants here and I don't trust them enough to just put the modem router in bridge mode and put OPN right behind it (since I'd have to physically move it into a public space). Also they have no business directly connecting to my devices over LAN instead of WAN anyways, as I run many devices and having to firewall each of them individually would be too much of a hassle (just for fun: I have 36 static IPs as well as a DHCP scope of 101 addresses). Some don't even have firewalling capabilities without writing something for it yourself.

The double NAT in itself doesn't seem to be a problem though and it's not inherently bad regardless, as long as you understand certain key points. OPN has been assigned as the DMZ of the modem router so I can access any ports I've forwarded to devices in my inner network without issue, and even VPNs work just fine. The problem arises when I access my public IP from within. Since OPN doesn't know about that IP, it sends it to the modem router which in turn doesn't do NAT reflection (I believe). I've been having some issues with packets being dropped upstream over the last few days, and if I access e.g. HTTP on my public IP it sometimes errors out. If it did do NAT reflection then upstream issues shouldn't be a problem.

My goal is to reroute traffic to my public IP back into the port forwards so that when I access e.g. port 80, it ends up at my web server. Ideally the web server also sees the client's LAN IP and not the OPN gateway, but I doubt that's possible because the client expects a reply from the gateway. Setting up split-horizon DNS is not really an option; I have many zones, records and servers/services so it would be a major pain to maintain all that. NAT reflection is really the only way.

I figured it would be as simple as attaching a virtual IP to the external interface and making sure NAT reflection is enabled on the port forward rule as well as Firewall > Settings > Advanced > Reflection for port forwards and Automatic outbound NAT for Reflection. However, when I do that then all traffic from inside to the public IP never gets to its destination. I can see it entering OPNsense through tcpdump (marked as "pass" and not "block"), but when I run tcpdump on the web server I don't see a single request coming in. Once I change the virtual IP to something else (last octet +1) then the exact same tcpdump shows a request from my WAN IP.

I even started messing around with outbound NAT, loopback interfaces, policy-based gateways and static routes but it simply wouldn't work reliably. At one point I was getting requests from OPN's IP on entirely different VLANs. :DD

Any ideas?

[SHA-2 Intermediate CAs (under SHA-1 Root)]

Same error for GeoTrust RSA CA 2018

the only way i found to solve the problem was to export the CA from firefox, import it in Trust->Authorities and then restart squid.
it should really interesting to have the script mentioned by Sahbi :)

The script does require a URL to the original cert file though. In the case of GeoTrust RSA CA 2018 you'll find this page via Google, then copy the link from SHA-2 Intermediate CAs (under SHA-1 Root) > second Intermediate CA in the table > Download. This would be: https://www.websecurity.symantec.com/content/dam/websitesecurity/support/digicert/geotrust/ica/GeoTrust_RSA_CA_2018.pem

Anyways, the script in question:

Code Select Expand

#!/bin/sh
certdir=/usr/local/openssl/certs

checkem_ret() {
ret=$?
cmd="$1"
exitval="$2"
if [ -z "$cmd" ]; then cmd='<UNKNOWN>'; fi
if [ -z "$exitval" ]; then exitval=1; fi

if [ $ret -ne 0 ]; then
echo "$cmd returned non-zero exit code ($ret), not proceeding"
exit $exitval
fi
}

if [ -z "$2" ]; then
skripname="$(basename "$0")"
echo "Usage: $skripname <URL to PEM/DER-encoded cert file> <target local file>"
echo "A hardcoded variable \$certdir is prepended to the local file name, this variable is currently set to: $certdir"
echo "Example: $skripname 'https://support.comodo.com/index.php?/Knowledgebase/Article/GetAttachment/970/821027' COMODORSADomainValidationSecureServerCA.crt"
exit 0
fi

certurl="$1"
lfile="$certdir/$2"

if [ ! -d "$certdir" ]; then
mkdir "$certdir"
checkem_ret mkdir 1
fi

if [ -f "$lfile" ]; then
echo "A local cert with the given name already exists, not proceeding: $lfile"
exit 2
fi

# curl's -k flag ignores any certificate errors (useful for downloading self-signed ones like CAcert)
curl -vk -o "$lfile" "$certurl"
checkem_ret cURL 3

if ! grep -q '^-----BEGIN CERTIFICATE-----' "$lfile"; then
echo "Certificate doesn't seem to be PEM/base64 encoded, trying to convert"
bitch64=$(openssl x509 -inform DER -in "$lfile")
checkem_ret openssl 4
echo "Ayy we good y0"
echo "$bitch64" > "$lfile"
fi

certhash=$(openssl x509 -in "$lfile" -hash -noout)
checkem_ret openssl 5

echo "---"
echo "Certificate hash: $certhash"

num=0
while [ -e "$certdir/${certhash}.$num" ]; do
num=$((num + 1))
done
ln -vs "$lfile" "$certdir/${certhash}.$num"

Example usage is given when you run the script without arguments, but for completeness here's how you would/could download the GeoTrust cert:

Code Select Expand

./install_cacert.sh https://www.websecurity.symantec.com/content/dam/websitesecurity/support/digicert/geotrust/ica/GeoTrust_RSA_CA_2018.pem GeoTrustRSACA2018.crt

I prefer to have CA cert names without underscores and with a .crt extension, but you can choose whatever you want really. =] You don't have to restart squid after running this script, by the way.