Messages

121

23.1 Legacy Series / Re: Multi-Wan Setup Failback from Tier 2 to Tier 1 unreliable

« on: August 10, 2023, 09:08:27 pm »

My understanding, for default gateway switching you need:

- Specify Priority, lower numerical value is higher priority
- Tag both as 'Upstream'

"This will select the above gateway as a default gateway candidate."

The 2 fail-over mechanisms are different:

- Firewall rule -> gateway group, uses gateway groups.
- Default gateway switching, the priority/upstream tags in System -> Gateway -> Single

Default gateway switching is going to impact services running on the firewall itself and rules where there is no gateway/gateway group specified.

122

23.1 Legacy Series / Re: Multi-Wan Setup Failback from Tier 2 to Tier 1 unreliable

« on: August 10, 2023, 08:26:41 pm »

If you go to:

Systems -> Gateway -> Single

Mark the Tier 2 as down (Disable) when it's active, apply, I assume it would then fail back to Tier 1?

123

23.1 Legacy Series / Re: Multi-Wan Setup Failback from Tier 2 to Tier 1 unreliable

« on: August 10, 2023, 08:15:43 pm »

Do you have a firewall rule with the specified gateway group setting, i.e to send traffic to the correct gateway group? 

Or are you just relying on the default gateway switching?

EDIT: Oh, missed the below initially... so you do, to the first point
"Relevant Firewall Rules: IPv4 Lan Network Pass rule to Gateway group"

The below would only really have been relevant if you were just relying on gateway switching:

- What is the routing table (netstat -rn) pre/post fail over?
- Systems -> Gateway -> Single, what priority are both gateways set to? Are they both tagged as 'upstream'?

124

23.7 Legacy Series / Re: Firewall rules are ignored.. but a little bit goes pass...

« on: August 10, 2023, 08:07:32 pm »

You keep saying 'random' - but it's really not random at all. 

The firewall only sees part of the connection, so it drops it.   With Keep state enabled, it needs to see all of the connection.

The other potential option, is that you've got packet loss/drops/weirdness somewhere with your Proxmox/USB NIC setup - opnsense is reacting to what it sees, or doesn't see as the case may be.

125

23.7 Legacy Series / Re: Firewall rules are ignored.. but a little bit goes pass...

« on: August 10, 2023, 05:09:21 pm »

Thank you, the hack worked, but how can i resolv a Asymmetric routing?

My OpnSense are running on a Proxmox Server, there has 3x USB 3.0 Network Cards.
The network cards are connected to the Proxmox Server, and created as virtual "network cards" to the virtual machine with VLAN Aware.

Sounds horrendous

I suspect one of the following is happening:

- The previously 'seen' source and destination is not the same both ways, for example:

a.a.a.a -> b.b.b.b but the reply is c.c.c.c -> a.a.a.a, or b.b.b.b -> d.d.d.d

- You have multiple routes/paths to the destination, not all via the firewall, so the devices can bypass the firewall for the initial 'SYN', so when it only sees tcp flags that should occur later in the connection they're dropped as out of state.

By turning off keep state you're telling the firewall to ignore the various sequences that should occur.  Which is generally not a good thing to do, sometimes necessary but can nearly always be avoided.

126

23.7 Legacy Series / Re: Firewall rules are ignored.. but a little bit goes pass...

« on: August 10, 2023, 04:50:50 pm »

Thank you, but why my packets randomly blocked and passed ?

Because they're out of state - A 'SYN' has to be seen before a Reset ACK, see explanation above.  The firewall is not 'seeing' all the packets it should do.

127

23.7 Legacy Series / Re: Firewall rules are ignored.. but a little bit goes pass...

« on: August 10, 2023, 04:45:02 pm »

S = SYN
RA = Reset ACK

The SYN is what is needed to be 'seen' to create the state entry on a stateful firewall, to avoid a state violation it needs to see SYN, SYN/ACK and finally ACK (with the source/destination remaining the same, same interfaces traversed, etc).  All of it.  If it only sees part of the connection, it gets dropped as out of state.

If you only see a Reset ACK via the firewall, no prior SYN SYN/ACK - you'd need to look at the logs - it likely means that the 'SYN' or 'SYN/ACK' is not seen via the firewall - asymmetric routing, possibly.

A hack:

- Edit rule
- Enabled Advanced Features
- Under Keep State, None (possibly Sloppy state might work)
- Save/Apply

... but really you should get to the root cause of the problem, why the devices are not sending the initial SYN, SYN/ACK via the firewall... why is the firewall only seeing the Reset ACK.  Asymmetric routing typically.

128

23.7 Legacy Series / Re: Firewall rules are ignored.. but a little bit goes pass...

« on: August 10, 2023, 04:01:48 pm »

Click the 'i' next to a drop, what does it show?

129

23.7 Legacy Series / Re: Firewall rules are ignored.. but a little bit goes pass...

« on: August 10, 2023, 03:45:54 pm »

Docker?

State violation, quite possibly the Docker container is replying back on a different IP than the 'destination' that is seen by the firewall.  Or, through different interfaces.

The state entry is created on the initial connection into the interface, source IP -> destination IP, for the destination to be able to send traffic back to the source, it has to match the forward path - same source (destination one way) and destination (source one way), same interfaces being traversed, etc.

... otherwise it is dropped as out of state, a state violation.

130

Hardware and Performance / Re: Saturating 1Gbit NIC creates packet loss

« on: August 10, 2023, 03:11:27 pm »

Look into setting up the Shaper, probably FQ (Fair Queue) Codel.

Set an up/down pipe limit of 900Mbps - maximum throughput of gigabit ethernet is 940Mbps, so this gives it some wiggle room.

... and possibly RSS.

How many of your 4x NICs are in use?  Just 2, with a WAN/LAN?

If so, then there is not really much benefit to 2.5Gbps or even 10Gbps interfaces - even on the LAN side. 

If your LAN is just 1 subnet, the only traffic that would probably go to/through the firewall is external/internet traffic - so if you have a 1Gbps WAN connection, the bottleneck is the same. 

Shaping would help to elevate situations where the link is saturated and using a Fair Queuing algo would apply just that - a fair weighted queue, to all traffic, to hopefully prevent loss through saturation.

If, for arguments sake, you have WAN/LAN/GUEST (3 interfaces/zones) then 2.5/10Gbps on the inside would have a benefit - if the 1Gbps WAN is saturated by a LAN/GUEST device, then traffic between LAN <-> GUEST would be unaffected.

Previously when I had a Qotom 4 NIC box, I used to have 1 dedicated for WAN, then a Round Robin LAGG with the other 3x1Gbps NICs with tagged VLANS on (giving the tagged VLANs shared 3Gbps bandwidth).  Used to work ok.

131

Hardware and Performance / Re: Strange issue with net.isr.maxthreads

« on: August 10, 2023, 01:43:04 pm »

Just to add to this, on 23.7 (just upgraded from an older 22.1), I see the same - but only when RSS is enabled:

Code: [Select]

net.isr.maxthreads=-1
net.isr.defaultqlimit=4096
net.isr.bindthreads=1
#net.inet.rss.enabled=1
#net.inet.rss.bits=2
- Could not check for updates
- NTP would fail to sync
- Telnet-ing to an external port would work less than 5% of the time

... when running a telnet test, I could see the SYN/ACK coming back and being retransmitted by the remote end but not 'seen' by the OS (only by the NIC in tcpdump).

I have 2 other boxes slightly different hardware, RSS works without issue.

EDIT: Seems to be some suggestions that RSS (even though it seems to be supported on Intel spec sheet) has some problems on 82574L.

EDIT 2: One of my other boxes also has 82574L and works with RSS just fine.  Weird.

132

23.1 Legacy Series / Re: OpnSense keeps crashing(Version 23.1.11 f1305748e)

« on: July 18, 2023, 02:31:05 pm »

How did you get on?

133

23.1 Legacy Series / Re: OpnSense keeps crashing(Version 23.1.11 f1305748e)

« on: July 09, 2023, 07:36:29 pm »

Just throwing it out there, I've also had issues with recent CPUs that use Intel Speed Shift - seems the N200 also does - seems to be common on FreeBSD 13, there were reports on the FreeBSD lists that it was resolved in FreeBSD 14.

Adding the below to /boot/loader.conf.local and then rebooting

Code: [Select]

hint.hwpstate_intel.0.disabled=1
.. causes it to then use SpeedStep instead, which can then be used with PowerD if you wish.  This fixed my random crashing issues.

134

22.1 Legacy Series / Re: With "Less Secure" depreciated for gmail, how are you getting notifications?

« on: July 21, 2022, 06:02:54 pm »

Yep, just tested - seems to work with app specific password.

I use it with Monit, to forward via email to PushOver for notifications.

135

22.1 Legacy Series / Re: With "Less Secure" depreciated for gmail, how are you getting notifications?

« on: July 21, 2022, 05:57:30 pm »

Can't you just setup an 'app specific password'?

https://support.google.com/mail/answer/185833?hl=en-GB

I assume mine is still working, port 465, SSL enabled, etc, as of 5 days ago: