Topics

1

19.1 Legacy Series / reordering packets under higher traffic

« on: July 05, 2019, 04:55:58 pm »

I'm running OpnSense 19.1 on Xen, connecting a DMZ host to its file server via NFS.
On rare occasions, when a big file is transferred, the nfs connection is broken, and a new tcp connection has to be started.

I've been tcpdumping the traffic in and out of the firewall (TCP segment offloading is disabled on all interfaces to avoid driver trouble), and found the following explanation:
Sometimes, a big PDU sent from the fileserver (split into 364 segments within 9.5ms) isn't forwarded to the destination DMZ host in-order, but instead in the middle of the flow segments are forwarded out-of-order, provoking out-of-order acks and resends, apparently driving the tcp stack mad and ultimately breaking the connection.

The server is a Xeon E5-2620V3, with 4 CPUs assigned to the firewall (low single digit cpu utilization, load rarely reaching 1), and no other machines running on the host. Typical state table size is 450, mbuf usage 800.

While the usage pattern of the system and general load hasn't changed over the last year, the problem started some months ago, which kind of coincides with the upgrade to 19.1 and the hardened kernel.

Why does the firewall start reordering, what can I do to prevent that?

Regards
Andreas

2

19.1 Legacy Series / system crashes

« on: June 21, 2019, 05:33:42 pm »

In the weeks, I observe spontaneous reboots every 3 days or so. This happens on the current master of a carp router pair (slightly different hardware after repair of one router), so this doesn't seem to be hardware related but appears caused by some kernel update in the last months.
The system.log doesn't help, it just logs the fresh boot. External monitoring doesn't show any anormal traffic or load (0.5 load, 2-5% CPU) on the Atom C3758 system with 19.1.9.

Any hints how to narrow this down?

Regards
Andreas

3

19.1 Legacy Series / flowd.log not rotated

« on: May 31, 2019, 10:14:12 am »

I've been running netflow logging a while now flawlessly. Since about two weeks (I guess post-19.1.5), rotation of flowd.log stalls, resulting in a fillled disk and consequently stopping some services for out-of-diskspace reasons. This includes flowd and flowd_aggregate.py. The firewall is monitored, so I could check that flowd_aggregate is killed from out-of-disk, not before.

I upgraded to 19.1.8, and see some strange behaviour of flowd_aggregate:
Usually, flowd.log will reach between 11MB and 13MB before rotating; it takes about 5 minutes for this size. But I also can observe this:

-rw-------  1 root  wheel   3.2M May 31 10:01 flowd.log
-rw-------  1 root  wheel    12M May 31 10:00 flowd.log.000001
-rw-------  1 root  wheel    13M May 31 09:56 flowd.log.000002
-rw-------  1 root  wheel    56M May 31 09:51 flowd.log.000003
-rw-------  1 root  wheel    13M May 31 09:29 flowd.log.000004
-rw-------  1 root  wheel    23M May 31 09:24 flowd.log.000005
-rw-------  1 root  wheel    12M May 31 09:15 flowd.log.000006
-rw-------  1 root  wheel    13M May 31 09:11 flowd.log.000007
-rw-------  1 root  wheel    11M May 31 09:06 flowd.log.000008
-rw-------  1 root  wheel    12M May 31 09:01 flowd.log.000009
-rw-------  1 root  wheel    12M May 31 08:56 flowd.log.000010

So obviously flowd_aggregate stalls sometimes for some minutes, and then continues to work. I can't see any anomalies in cpu load or usage during this period. Nothing in the system log, the last flowd related message is
 May 31 06:47:26 fw05a flowd_aggregate.py: vacuum done

While this is not the total stop of flowd.log rotation I've been suffering from in the last weeks, it still seems suspicious to me. What's going wrong here?

Added: stalling is happening right now, and I can see the flowd_aggregate proces consuming a lot of CPU.

4

19.1 Legacy Series / CARP over LAGG problems

« on: May 03, 2019, 09:25:38 am »

I usually do my opnsense upgrades by first updating the usually-backup machine, disabling carp on the master and updating it as well.
Now when upgrading from 19.1.2 to 19.1.6 (which needs reboot), I found that some VHIDs would go to master and some to backup (net.inet.carp.preempt=0, should be 1 but helpful for debugging here) afterwards. The VHIDs that became master all are on a LAGG interface (directly or VLAN), the others remaining on backup are on physical interfaces. When disabling and enabling carp on the master machine, the situation was resolved. Apparently, the LAGG interface didn't receive carp packets from the master in-time when booting up, so the rebooted machine suspected it needed to become master itself.

After my HA setup was settled and working normally, I started to upgrade the switches one by one. With one switch down, the LAGG interface is still workable, since only one of both physical interfaces looses connection, but CARP seems to increase demotion based on the physical interface, not the resulting LAGG interface. In order to not have CARP failing over unnecessarily (which would affect eg. OpenVPN connections), CARP on the backup needs to be disabled temporarily.

So there seem to be two issues here: CARP expecting traffic before LAGG is ready, and CARP demotion reacting to LAGG slave interfaces instead of the LAGG interface itself.

5

General Discussion / Multicast storm created by firewall

« on: April 09, 2019, 12:08:07 pm »

From time to time, we're suffering from some strange issue:
Triggered by a workstation on LAN1 sending a ws-discovery multicast on port 3702 (or some other service, just as example), some thousand duplicated packets can be seen on LAN2 (with LAN1-address as sender and mcast as destination), with the source MAC address of the backup firewall of a CARP pair.

Or in other words:
The carp backup firewall, which should be listening passively, creates IP Multicast packets with its own LAN2 MAC source address, LAN1 IP source Address of a client, with a rate of about 5000/s and will not stop until the firewall is kicked with pfctl -d;pfctl -e

Hotfix is to drop UDP traffic to specific ports (such as 3702) on the LAN1 network, but a firewall shouldn't create such packets on its own, right? It's 19.1 (had this already with 18.1/18.7), no specific Multicast/IGMP settings or modules.

6

19.1 Legacy Series / 19.1.x ipsec phase2 edit problem with old entries

« on: March 29, 2019, 04:07:15 pm »

I'd like to add a third phase2 entry to an ipsec definition, so I pressed "clone" on an existing phase2 entry. The fields of the page coming up are not prefilled, and if filled and saved the entry won't show up. Looking at a config backup, the entry misses the ikeid, but has a uniqueid instead.

Investigating further, apparently editing is affected as well: parameters shown are not the one reflecting the phase2 entry to edit.
Only adding a fresh entry seems to work.

Version 19.1.2 and 19.1.4 tested, after refreshing with F5.

7

18.7 Legacy Series / unbound domain overrides not evaluated

« on: December 28, 2018, 10:56:19 am »

I've been upgrading from 18.1.x to 18.7.9, and had some trouble with unbound not resolving some domain overrides after that. It would resolve some addresses from cache, and tried some from root servers. I had to edit and save a domain override unmodified to get unbound back to normal work.
This happened on two machines (master and slave).

8

18.7 Legacy Series / Broadcast flood generated by firewall

« on: November 23, 2018, 05:46:24 pm »

There are some smartphones that will connect via wireless to one LAN or another, depending on app needs. Apparently, IOS phones may remember the old IP address, and sending out UDP broadcasts for quite some stuff (SMB, dropbox, spotify) using the old IP address (network A) on a LAN that has another network B.
Even if the iPhone is disconnected, about 4000 packets/s are still broadcasted, originating from the firewall's B network, but broadcasting A-sourced packets.
I have invented block rules
- for specific UDP ports
- for 255.255.255.255 destination
- for any packets that don't originate from that interface's network

Still, these broadcast storms from the firewall persist.
To stop the storm, I need to issue pfctl -d ; pfctl -e

I'm running out of ideas.

card/pfsync pair of opnsense, sometimes the master is the source of the broadcasts, sometimes the backup.

Anybody a clue for me?
Regards
Andreas

9

18.7 Legacy Series / os-bind too minimalistic, corrupting manual config

« on: September 27, 2018, 08:02:14 pm »

For quite a while I had bind9 running on my firewalls (from FreeBsd repo), acting as secondary.
After Upgrade to 18.7, named was gone, and I installed os-bind, not immediately noticing the OpnSense configuration options so I went on configuring as usual.
Unfortunately, my configuration won't survive a reboot, and the config pages for bind are not sufficient to configure.
A free text "custom options" field as in unbound config or "advanced" as in dnsmasq would be very helpful.

10

18.1 Legacy Series / outgoing NAT with interface_address uses carp ip

« on: February 15, 2018, 12:12:10 pm »

After upgrading to 17.x to 18.1.2, the outgoing NAT address translation doesn't work any more as expected.

I have outgoing nat configured to use the interface address on a CARP cluster, which used do use the physical ip address of each machine.
After the upgrade, outgoing traffic uses all VIF ip addresses randomly, making some sites' session handling nonfunctional.

11

German - Deutsch / 17.7.6 CARP-Pakete werden gefiltert

« on: November 01, 2017, 01:33:04 pm »

Nach einem Update eines Firewall-Paars von 17.1.2 über 17.1.11 auf 17.7.6 funktioniert CARP auf dem WAN-Interface nicht mehr, da der Master zwar seine CARP-Announcements wie erwartet erzeugt, diese die Firewall aber nicht verlassen und der Backup daher meint übernehmen zu müssen.
Im Filterlog ist von CARP nichts zu sehen, die Pakete gehen aber sofort raus wenn die Firewall per pfctl -d abgeschaltet ist. ACCEPT-Rules helfen nicht.

Irgendein Hinweis wo das hängenbleiben kann?

12

17.7 Legacy Series / outgoing CARP blocked

« on: October 25, 2017, 10:07:25 am »

I have a pair of opnsense routers with CARP which haven't been updated for quite a time. They were running 17.1.2 fine with 16 VIFs defined on the WAN interface (since it's still impossible to define IP aliases on a VIF), and some more on the LAN and DMZ sides.

After upgrading both machines to 17.7.6 the backup machine does't receive CARP announcements any more on the WAN interface (other interfaces are ok), so it will switch to master (on WAN only), messing up traffic badly.

Checking on the master, I still see CARP announcements generated on the WAN if, but apparently they are not passed out. As soon as I pfctl -d the firewall, I can see CARP arriving at the secondary as well; pfctl -e and announcements are lost again.

I added explicit rules on the WAN interface, allowing CARP from the firewall, and even any traffic from the firewall, no result.

Any hint how to get CARP working again? I'm non-redundant now, giving me a bad feeling after I had a kernel crash on the master lately.

Regards
Andreas

13

17.1 Legacy Series / carp/pfsync but connections interrupted

« on: June 22, 2017, 06:41:04 pm »

As far as I understand carp/pfsync, the backup firewall should take over the running sessions from the master more or less interruption free. On an heavy traffic tcp session, maybe some packets might need retransmission since there's a time gap between pfsyncs, but not more.

Apparently, this assumption is wrong. Even on a virtually idle system, connections are broken. I have a OS 17.1.8 pair (virtualized on Xen 4.4.1), machines are connected via 10GBit. The web server in the DMZ accesses the file storage via NFS4 tcp port 2049. When rebooting the master (for whatever reason, e.g. fw upgrade), the nfs mount might report a stale handle with no chance to recover but reboot the web server VM. I tried to demote the master to slave upfront, with persistence, and promote later back again, but this doesn't seem to help too much.

All other connections might break as well, but the nfs connection is really fatal.

Is there any means to make the carp/pfsync stuff work without breakage? Even a fancy solution improving the nfs breakage only would help. I've been observing this problem since pfSense 1.x/2.x/OS16.7 and it's still annoying.

Regards
Andreas

14

15.7 Legacy Series / Gateway definition quirks

« on: December 22, 2015, 07:40:58 pm »

I encountered this problem with pfsense (they say it's not bug...), and checked with opnsense now getting the same results:

The definition of a gateway implicitely creates a route via that gateway, if a monitoring ip is given. I found this unexpected, and would like to have a checkbox to disable this behaviour (it will create icmp-redirects unless disabled by kernel params).

Anyhow, if this gateway entry is disabled, the routing will still persist, which I'd call a bug. IMHO a disabled entry shouldn't have any side effect.

15

15.7 Legacy Series / Installation problem on Virtualbox with 512MB

« on: December 22, 2015, 07:26:02 pm »

I tried installing on Virtualbox for testing purposes, but the copying would fail at various positions. After trying different NIC and chipset settings, I found that increasing memory from 512MB to 1G did the trick. Apparently, 512MB is not sufficient to install (using 15.7.18-amd64).

Regards,
Andreas