Topics

After upgrading a firewall from 24.1.x to 24.7.7, ftp-proxy doesn't work any more. Analyzing traffic with tcpdump on both LAN and WAN interface:

- in PASV mode, the client sends SYN packets to the port as returned from ftp-proxy, but there's not traffic to the upstream ftp server.
- in active mode, the server sends SYN packets to the port as presented by the PORT command that ftp-proxy issued upstream, but won't forward any traffic to the client.

I have logging enable on both the client-to-ftp-proxy redirect on Port21, as well as client-to-server traffic for the data connection, both log as "pass" when issuing the client data command.

I checked against that FTP server with another client/firewall (different site, a lot simpler firewall setup), also on 24.7.7, which works correctly there.

From the ftp-proxy man pages, I'd expect to see something with

Code Select Expand

pfctl -a ftp-proxy -s rules, but there's nothing while the data connection is stuck.

I'm out of clues now, anybody with an idea?

Regards,
Andreas

After upgrading to 24.1_1, firewall groups cannot be edited any more:
The overview doesn't show the members any more, when editing all fields are empty (no members available in the dropdown list)

When doing regular maintenance on our CARP cluster, I regularly disable CARP on the machine and enter persistent maintenance mode. I'd expect it to never get MASTER until I enable CARP again.

Now, I rebooted the machine (22.1.1), and while it came up I glanced "Timeout on ix2, becoming MASTER" on the console for a second or so until it stepped back to BACKUP.
While I also have layered interfaces (vlan over lagg over 10GBit), this very ix2 interface is just a plain 1GBit onboard Intel NIC, connected to a switch, no VLAN no whistles or bells (upstream internet).

Having double master even for fractions of a second will screw up network traffic more or less badly, so this really isn't good and shouldn't happen, maintenance mode or not.

So how to safely reboot a router without triggering major trouble?

After upgrading from 21.7 to 22.1.1, my firewall shows an error rate of 0.05/s avg on the WAN interface (SuperMicro A2SDI on-board NIC connected to a Juniper switch afaik), which used to be zero before (netstat -i Ierrs). The machine has 4 more connections, all are still at zero errors.

What may be the reason?

We have an opnsense installation (CARP pair), running on 20.1.3 until recently, with 4 ipsec peers (2xIKEv1, 2xIKEv2) and some 20 tunnels defined. This used to run flawlessly, until I upgraded those machines to 20.7.1. Since then, the tunnels will stop working after a while, until a new connect is forced on the tunnel. Strangely, all logging looks normal on both sides of the tunnel, even when the tunnel traffic has stalled (still IKE/ISAKMP traffic, but no more ESP packets)

The situation is a little different between peers, and sometimes there are stable phases for one peer, getting bad again after a while, but none is 100% fine. It will take some seconds to some minutes until the tunnels stall; more traffic seems to speed up the failure.

I reinstalled one firewall with 20.1, and now we have stable performance again. The backup machine is in maintenance mode and still 20.7.1 (with syslog-ng fixed).

When reviewing the updates that happened between 20.1.3 and 20.7.1, strongswan was upgraded from 4.8.2 to 4.8.4 (in April), and the kernel from 11.2 to 12.1. Since IKE/ISAKMP traffic seems normal, I'd suspect some issue in the kernel/pf, but I'm out of clues how to narrow down the reason further.

Any thoughts on this?
Regards,
Andreas

On a pair of 20.1 firewalls coupled with CARP, I have some 27 VIFs defined, total 10 interfaces.
Recently, I added two more VLANs and two interfaces. When enabling the new interfaces on the CARP master (without IP for the start), many VIFs became unresponsive after APPLY. To resolve the issue quickly, I disabled and re-enabled CARP on the master, and operation was re-established.

I'm quite sure that I could add interfaces without any problems "in the old days", but trouble increased over time (hiccup/interrupts for some seconds), now defunct VIFs. This might have several reasons:
- just a matter of many interfaces
- opnsense code to execute the interface operations was changed

Is this a known issue? Something I have to live with?

On a firewall with 20.1.3 (now 20.1.6), I noticed that it's not possible to edit phase2 entries on an existing configuration (unchanged and running for a while, probably older than a year)

When editing, the remote address or network isn't shown, when duplicating a single address phase2 entry with a different target addr, I get "already exists".

Digging further, I find some entries displaying (and probably editing) correctly, others not. A look at config.xml shows the problem: then non-functional entries don't habe a uniqid.

I'm running OpnSense 19.1 on Xen, connecting a DMZ host to its file server via NFS.
On rare occasions, when a big file is transferred, the nfs connection is broken, and a new tcp connection has to be started.

I've been tcpdumping the traffic in and out of the firewall (TCP segment offloading is disabled on all interfaces to avoid driver trouble), and found the following explanation:
Sometimes, a big PDU sent from the fileserver (split into 364 segments within 9.5ms) isn't forwarded to the destination DMZ host in-order, but instead in the middle of the flow segments are forwarded out-of-order, provoking out-of-order acks and resends, apparently driving the tcp stack mad and ultimately breaking the connection.

The server is a Xeon E5-2620V3, with 4 CPUs assigned to the firewall (low single digit cpu utilization, load rarely reaching 1), and no other machines running on the host. Typical state table size is 450, mbuf usage 800.

While the usage pattern of the system and general load hasn't changed over the last year, the problem started some months ago, which kind of coincides with the upgrade to 19.1 and the hardened kernel.

Why does the firewall start reordering, what can I do to prevent that?

Regards
Andreas

In the weeks, I observe spontaneous reboots every 3 days or so. This happens on the current master of a carp router pair (slightly different hardware after repair of one router), so this doesn't seem to be hardware related but appears caused by some kernel update in the last months.
The system.log doesn't help, it just logs the fresh boot. External monitoring doesn't show any anormal traffic or load (0.5 load, 2-5% CPU) on the Atom C3758 system with 19.1.9.

Any hints how to narrow this down?

Regards
Andreas

I've been running netflow logging a while now flawlessly. Since about two weeks (I guess post-19.1.5), rotation of flowd.log stalls, resulting in a fillled disk and consequently stopping some services for out-of-diskspace reasons. This includes flowd and flowd_aggregate.py. The firewall is monitored, so I could check that flowd_aggregate is killed from out-of-disk, not before.

I upgraded to 19.1.8, and see some strange behaviour of flowd_aggregate:
Usually, flowd.log will reach between 11MB and 13MB before rotating; it takes about 5 minutes for this size. But I also can observe this:

-rw-------  1 root  wheel   3.2M May 31 10:01 flowd.log
-rw-------  1 root  wheel    12M May 31 10:00 flowd.log.000001
-rw-------  1 root  wheel    13M May 31 09:56 flowd.log.000002
-rw-------  1 root  wheel    56M May 31 09:51 flowd.log.000003
-rw-------  1 root  wheel    13M May 31 09:29 flowd.log.000004
-rw-------  1 root  wheel    23M May 31 09:24 flowd.log.000005
-rw-------  1 root  wheel    12M May 31 09:15 flowd.log.000006
-rw-------  1 root  wheel    13M May 31 09:11 flowd.log.000007
-rw-------  1 root  wheel    11M May 31 09:06 flowd.log.000008
-rw-------  1 root  wheel    12M May 31 09:01 flowd.log.000009
-rw-------  1 root  wheel    12M May 31 08:56 flowd.log.000010

So obviously flowd_aggregate stalls sometimes for some minutes, and then continues to work. I can't see any anomalies in cpu load or usage during this period. Nothing in the system log, the last flowd related message is
May 31 06:47:26 fw05a flowd_aggregate.py: vacuum done

While this is not the total stop of flowd.log rotation I've been suffering from in the last weeks, it still seems suspicious to me. What's going wrong here?

Added: stalling is happening right now, and I can see the flowd_aggregate proces consuming a lot of CPU.

I usually do my opnsense upgrades by first updating the usually-backup machine, disabling carp on the master and updating it as well.
Now when upgrading from 19.1.2 to 19.1.6 (which needs reboot), I found that some VHIDs would go to master and some to backup (net.inet.carp.preempt=0, should be 1 but helpful for debugging here) afterwards. The VHIDs that became master all are on a LAGG interface (directly or VLAN), the others remaining on backup are on physical interfaces. When disabling and enabling carp on the master machine, the situation was resolved. Apparently, the LAGG interface didn't receive carp packets from the master in-time when booting up, so the rebooted machine suspected it needed to become master itself.

After my HA setup was settled and working normally, I started to upgrade the switches one by one. With one switch down, the LAGG interface is still workable, since only one of both physical interfaces looses connection, but CARP seems to increase demotion based on the physical interface, not the resulting LAGG interface. In order to not have CARP failing over unnecessarily (which would affect eg. OpenVPN connections), CARP on the backup needs to be disabled temporarily.

So there seem to be two issues here: CARP expecting traffic before LAGG is ready, and CARP demotion reacting to LAGG slave interfaces instead of the LAGG interface itself.

From time to time, we're suffering from some strange issue:
Triggered by a workstation on LAN1 sending a ws-discovery multicast on port 3702 (or some other service, just as example), some thousand duplicated packets can be seen on LAN2 (with LAN1-address as sender and mcast as destination), with the source MAC address of the backup firewall of a CARP pair.

Or in other words:
The carp backup firewall, which should be listening passively, creates IP Multicast packets with its own LAN2 MAC source address, LAN1 IP source Address of a client, with a rate of about 5000/s and will not stop until the firewall is kicked with pfctl -d;pfctl -e

Hotfix is to drop UDP traffic to specific ports (such as 3702) on the LAN1 network, but a firewall shouldn't create such packets on its own, right? It's 19.1 (had this already with 18.1/18.7), no specific Multicast/IGMP settings or modules.

I'd like to add a third phase2 entry to an ipsec definition, so I pressed "clone" on an existing phase2 entry. The fields of the page coming up are not prefilled, and if filled and saved the entry won't show up. Looking at a config backup, the entry misses the ikeid, but has a uniqueid instead.

Investigating further, apparently editing is affected as well: parameters shown are not the one reflecting the phase2 entry to edit.
Only adding a fresh entry seems to work.

Version 19.1.2 and 19.1.4 tested, after refreshing with F5.

I've been upgrading from 18.1.x to 18.7.9, and had some trouble with unbound not resolving some domain overrides after that. It would resolve some addresses from cache, and tried some from root servers. I had to edit and save a domain override unmodified to get unbound back to normal work.
This happened on two machines (master and slave).

There are some smartphones that will connect via wireless to one LAN or another, depending on app needs. Apparently, IOS phones may remember the old IP address, and sending out UDP broadcasts for quite some stuff (SMB, dropbox, spotify) using the old IP address (network A) on a LAN that has another network B.
Even if the iPhone is disconnected, about 4000 packets/s are still broadcasted, originating from the firewall's B network, but broadcasting A-sourced packets.
I have invented block rules
- for specific UDP ports
- for 255.255.255.255 destination
- for any packets that don't originate from that interface's network

Still, these broadcast storms from the firewall persist.
To stop the storm, I need to issue pfctl -d ; pfctl -e

I'm running out of ideas.

card/pfsync pair of opnsense, sometimes the master is the source of the broadcasts, sometimes the backup.

Anybody a clue for me?
Regards
Andreas

For quite a while I had bind9 running on my firewalls (from FreeBsd repo), acting as secondary.
After Upgrade to 18.7, named was gone, and I installed os-bind, not immediately noticing the OpnSense configuration options so I went on configuring as usual.
Unfortunately, my configuration won't survive a reboot, and the config pages for bind are not sufficient to configure.
A free text "custom options" field as in unbound config or "advanced" as in dnsmasq would be very helpful.

After upgrading to 17.x to 18.1.2, the outgoing NAT address translation doesn't work any more as expected.

I have outgoing nat configured to use the interface address on a CARP cluster, which used do use the physical ip address of each machine.
After the upgrade, outgoing traffic uses all VIF ip addresses randomly, making some sites' session handling nonfunctional.

Nach einem Update eines Firewall-Paars von 17.1.2 über 17.1.11 auf 17.7.6 funktioniert CARP auf dem WAN-Interface nicht mehr, da der Master zwar seine CARP-Announcements wie erwartet erzeugt, diese die Firewall aber nicht verlassen und der Backup daher meint übernehmen zu müssen.
Im Filterlog ist von CARP nichts zu sehen, die Pakete gehen aber sofort raus wenn die Firewall per pfctl -d abgeschaltet ist. ACCEPT-Rules helfen nicht.

Irgendein Hinweis wo das hängenbleiben kann?

I have a pair of opnsense routers with CARP which haven't been updated for quite a time. They were running 17.1.2 fine with 16 VIFs defined on the WAN interface (since it's still impossible to define IP aliases on a VIF), and some more on the LAN and DMZ sides.

After upgrading both machines to 17.7.6 the backup machine does't receive CARP announcements any more on the WAN interface (other interfaces are ok), so it will switch to master (on WAN only), messing up traffic badly.

Checking on the master, I still see CARP announcements generated on the WAN if, but apparently they are not passed out. As soon as I pfctl -d the firewall, I can see CARP arriving at the secondary as well; pfctl -e and announcements are lost again.

I added explicit rules on the WAN interface, allowing CARP from the firewall, and even any traffic from the firewall, no result.

Any hint how to get CARP working again? I'm non-redundant now, giving me a bad feeling after I had a kernel crash on the master lately.

Regards
Andreas

As far as I understand carp/pfsync, the backup firewall should take over the running sessions from the master more or less interruption free. On an heavy traffic tcp session, maybe some packets might need retransmission since there's a time gap between pfsyncs, but not more.

Apparently, this assumption is wrong. Even on a virtually idle system, connections are broken. I have a OS 17.1.8 pair (virtualized on Xen 4.4.1), machines are connected via 10GBit. The web server in the DMZ accesses the file storage via NFS4 tcp port 2049. When rebooting the master (for whatever reason, e.g. fw upgrade), the nfs mount might report a stale handle with no chance to recover but reboot the web server VM. I tried to demote the master to slave upfront, with persistence, and promote later back again, but this doesn't seem to help too much.

All other connections might break as well, but the nfs connection is really fatal.

Is there any means to make the carp/pfsync stuff work without breakage? Even a fancy solution improving the nfs breakage only would help. I've been observing this problem since pfSense 1.x/2.x/OS16.7 and it's still annoying.

Regards
Andreas