Messages

1

19.1 Legacy Series / Re: reordering packets under higher traffic

« on: July 17, 2019, 03:50:10 pm »

Shared forwarding under Firewall/Settings/Advanced isn't enabled.
No traffic shaping, routing groups, advanced rules or other fancy stuff configured.

2

19.1 Legacy Series / reordering packets under higher traffic

« on: July 05, 2019, 04:55:58 pm »

I'm running OpnSense 19.1 on Xen, connecting a DMZ host to its file server via NFS.
On rare occasions, when a big file is transferred, the nfs connection is broken, and a new tcp connection has to be started.

I've been tcpdumping the traffic in and out of the firewall (TCP segment offloading is disabled on all interfaces to avoid driver trouble), and found the following explanation:
Sometimes, a big PDU sent from the fileserver (split into 364 segments within 9.5ms) isn't forwarded to the destination DMZ host in-order, but instead in the middle of the flow segments are forwarded out-of-order, provoking out-of-order acks and resends, apparently driving the tcp stack mad and ultimately breaking the connection.

The server is a Xeon E5-2620V3, with 4 CPUs assigned to the firewall (low single digit cpu utilization, load rarely reaching 1), and no other machines running on the host. Typical state table size is 450, mbuf usage 800.

While the usage pattern of the system and general load hasn't changed over the last year, the problem started some months ago, which kind of coincides with the upgrade to 19.1 and the hardened kernel.

Why does the firewall start reordering, what can I do to prevent that?

Regards
Andreas

3

General Discussion / Re: Virtual to physical

« on: July 05, 2019, 04:14:01 pm »

The interfaces will probably named differently. The renaming can also occur when moving from one hardware to another.

You can modify the interface names in the config.xml before restoring it on the target, or tweak them afterwards, paying attention to the correct order of interfaces (opt1...optN)

4

19.1 Legacy Series / Re: CARP over LAGG problems

« on: July 05, 2019, 04:03:23 pm »

current settings are:

net.inet.carp.senderr_demotion_factor =0
net.inet.carp.ifdown_demotion_factor = 120
net.pfsync.carp_demotion_factor = 0

5

19.1 Legacy Series / system crashes

« on: June 21, 2019, 05:33:42 pm »

In the weeks, I observe spontaneous reboots every 3 days or so. This happens on the current master of a carp router pair (slightly different hardware after repair of one router), so this doesn't seem to be hardware related but appears caused by some kernel update in the last months.
The system.log doesn't help, it just logs the fresh boot. External monitoring doesn't show any anormal traffic or load (0.5 load, 2-5% CPU) on the Atom C3758 system with 19.1.9.

Any hints how to narrow this down?

Regards
Andreas

6

19.1 Legacy Series / Re: flowd.log not rotated: aggregation too slow

« on: May 31, 2019, 01:43:13 pm »

I started with fresh logging, i.e. deleted /var/log/flowd* and /var/netflow/*, restarted flowd and running  flowd_aggregate.py with some timing added in the console.

I can see that a typical aggregation run will take 70-80 seconds, then rotation is checked, then 15 seconds nap.
This means, that currently the flowd_aggregate process is at 85% of its capacity; there are certainly phases where firewall traffic is heavier than currently.

flowd.log is filled with a rate of about 2.5-5MB/min, /var/netflow/* has about 5MB. Doesn't seem to heavy to me.
Machine as a 8-core C3758CPU, Load avg about 1.3, with CPU at some % (flowd_aggregate will add 12%), SATA-DOM SSD.

7

19.1 Legacy Series / flowd.log not rotated

« on: May 31, 2019, 10:14:12 am »

I've been running netflow logging a while now flawlessly. Since about two weeks (I guess post-19.1.5), rotation of flowd.log stalls, resulting in a fillled disk and consequently stopping some services for out-of-diskspace reasons. This includes flowd and flowd_aggregate.py. The firewall is monitored, so I could check that flowd_aggregate is killed from out-of-disk, not before.

I upgraded to 19.1.8, and see some strange behaviour of flowd_aggregate:
Usually, flowd.log will reach between 11MB and 13MB before rotating; it takes about 5 minutes for this size. But I also can observe this:

-rw-------  1 root  wheel   3.2M May 31 10:01 flowd.log
-rw-------  1 root  wheel    12M May 31 10:00 flowd.log.000001
-rw-------  1 root  wheel    13M May 31 09:56 flowd.log.000002
-rw-------  1 root  wheel    56M May 31 09:51 flowd.log.000003
-rw-------  1 root  wheel    13M May 31 09:29 flowd.log.000004
-rw-------  1 root  wheel    23M May 31 09:24 flowd.log.000005
-rw-------  1 root  wheel    12M May 31 09:15 flowd.log.000006
-rw-------  1 root  wheel    13M May 31 09:11 flowd.log.000007
-rw-------  1 root  wheel    11M May 31 09:06 flowd.log.000008
-rw-------  1 root  wheel    12M May 31 09:01 flowd.log.000009
-rw-------  1 root  wheel    12M May 31 08:56 flowd.log.000010

So obviously flowd_aggregate stalls sometimes for some minutes, and then continues to work. I can't see any anomalies in cpu load or usage during this period. Nothing in the system log, the last flowd related message is
 May 31 06:47:26 fw05a flowd_aggregate.py: vacuum done

While this is not the total stop of flowd.log rotation I've been suffering from in the last weeks, it still seems suspicious to me. What's going wrong here?

Added: stalling is happening right now, and I can see the flowd_aggregate proces consuming a lot of CPU.

8

German - Deutsch / Re: 17.7.6 CARP-Pakete werden gefiltert

« on: May 06, 2019, 10:58:14 am »

Ich denke ich weiß jetzt woran es liegt, siehe https://github.com/opnsense/core/issues/3468

9

19.1 Legacy Series / Re: CARP over LAGG problems

« on: May 03, 2019, 05:15:52 pm »

Isn't the backup unit also on the same switch? Then it should not fail over ...

That's exactly the problem. Via the second switch both FWs have degraded but fully functional LAGG connectivity. CARP shouldn't react to LAGG degradation, but it does.

10

19.1 Legacy Series / CARP over LAGG problems

« on: May 03, 2019, 09:25:38 am »

I usually do my opnsense upgrades by first updating the usually-backup machine, disabling carp on the master and updating it as well.
Now when upgrading from 19.1.2 to 19.1.6 (which needs reboot), I found that some VHIDs would go to master and some to backup (net.inet.carp.preempt=0, should be 1 but helpful for debugging here) afterwards. The VHIDs that became master all are on a LAGG interface (directly or VLAN), the others remaining on backup are on physical interfaces. When disabling and enabling carp on the master machine, the situation was resolved. Apparently, the LAGG interface didn't receive carp packets from the master in-time when booting up, so the rebooted machine suspected it needed to become master itself.

After my HA setup was settled and working normally, I started to upgrade the switches one by one. With one switch down, the LAGG interface is still workable, since only one of both physical interfaces looses connection, but CARP seems to increase demotion based on the physical interface, not the resulting LAGG interface. In order to not have CARP failing over unnecessarily (which would affect eg. OpenVPN connections), CARP on the backup needs to be disabled temporarily.

So there seem to be two issues here: CARP expecting traffic before LAGG is ready, and CARP demotion reacting to LAGG slave interfaces instead of the LAGG interface itself.

11

19.1 Legacy Series / Re: OPNsense 19.1.7 release thread

« on: May 03, 2019, 08:51:50 am »

removed bind912 for me without installing bind914, posted a github issue.

12

General Discussion / Re: Firewall rule for CARP on ipv6?

« on: April 21, 2019, 09:30:27 am »

I had the problem a long time. It turned out to be a problem of the switches to which WAN was connected; IGMP snooping had to be disabled.

13

German - Deutsch / Re: 17.7.6 CARP-Pakete werden gefiltert

« on: April 21, 2019, 09:22:40 am »

Vor kurzem hatte ich einen neuen Anlauf genommen das Problem zu diagnostizieren, war aber noch nicht dazu gekommen diesen Thread zu ergänzen.
Da ich nicht im Produktivsystem testen konnte, hab ich schlußendlich eine weitere identische Firewall gebaut und die Originalkonfiguration eingespielt. Sofort konnte ich die abgehenden CARP-Pakete sehen. 
Daraufhin habe ich im Produktivsystem einen zusätzlichen Switch zwischen Firewall und Upstream (Provider Switch) geschaltet, und konnte auch dort meine Pakete sehen.
Eine längere Konsultation mit dem RZ-Personal führte dann dazu daß dort IGMP Snooping im Coreswitch für uns disabled wurde, und siehe da schon gingen auch die CARP Multicastpakete wieder durch. Ich vermute, daß seinerzeit ein SW-Update im Coreswitch eingespielt wurde was IGMP Snooping falsch umsetzt, nämlich auch ALL-NODES Multicast behandelt.

Warum pfctl -d funktionierte hab ich dann nicht mehr ausprobiert, ich war froh endlich wieder funktionstüchtige Standbyredundanz zu haben.

14

General Discussion / Multicast storm created by firewall

« on: April 09, 2019, 12:08:07 pm »

From time to time, we're suffering from some strange issue:
Triggered by a workstation on LAN1 sending a ws-discovery multicast on port 3702 (or some other service, just as example), some thousand duplicated packets can be seen on LAN2 (with LAN1-address as sender and mcast as destination), with the source MAC address of the backup firewall of a CARP pair.

Or in other words:
The carp backup firewall, which should be listening passively, creates IP Multicast packets with its own LAN2 MAC source address, LAN1 IP source Address of a client, with a rate of about 5000/s and will not stop until the firewall is kicked with pfctl -d;pfctl -e

Hotfix is to drop UDP traffic to specific ports (such as 3702) on the LAN1 network, but a firewall shouldn't create such packets on its own, right? It's 19.1 (had this already with 18.1/18.7), no specific Multicast/IGMP settings or modules.

15

19.1 Legacy Series / Re: 19.1.x ipsec phase2 edit regression from 18.7

« on: March 29, 2019, 05:05:38 pm »

I now have 3 phase2 entries: 2 old and one new.

When clicking on 'clone' on an old entry, I get
<fw-addr>/vpn_ipsec_phase2.php?dup=
the new entry has
<fwaddr>/vpn_ipsec_phase2.php?dup=5c9e3fd2320ae

Checking further:
It seems that p2 entries I entered last year are ok, while the problem only exists with entries that are some years old.
Looking at a config backup, it seems that older entries miss the uniqid property. Seems I have to add the uniqid manually in a config and restore from there?