Messages

1

20.7 Legacy Series / Re: IPSEC traffic stalling after 20.7.1 upgrade

« on: September 04, 2020, 07:10:44 pm »

I did an after-hour test, after upgrading the fw to 20.7.2.

The tunnel traffic still stalls after a while (it did so after about 100MB inbound traffic).

When pinging a remote host, I see ICMP on enc0 entering the tunnel, a corresponding outgoing ESP packet on wan, but no returning packet; there's still communication on port 500, with no anomalies (afaics) in the log.

Switching back to the downgraded fw, same config (synced from the 20.7 machine) works flawlessly.

2

20.7 Legacy Series / Re: IPSEC traffic stalling after 20.7.1 upgrade

« on: September 02, 2020, 11:47:12 am »

I did a packet capture on WAN, no more ESP packets visible.
Didn't capture on other interfaces.

3

20.7 Legacy Series / Re: IPSEC traffic stalling after 20.7.1 upgrade

« on: September 02, 2020, 09:45:05 am »

No hint in the logs on either side of the tunnel (already elevated some log levels)

4

20.7 Legacy Series / Re: IPSEC traffic stalling after 20.7.1 upgrade

« on: September 01, 2020, 04:47:03 pm »

All Tunnel IPV4 policy-based.

5

20.7 Legacy Series / IPSEC traffic stalling after 20.7.1 upgrade

« on: September 01, 2020, 03:52:20 pm »

We have an opnsense installation (CARP pair), running on 20.1.3 until recently, with 4 ipsec peers (2xIKEv1, 2xIKEv2) and some 20 tunnels defined. This used to run flawlessly, until I upgraded those machines to 20.7.1. Since then, the tunnels will stop working after a while, until a new connect is forced on the tunnel. Strangely, all logging looks normal on both sides of the tunnel, even when the tunnel traffic has stalled (still IKE/ISAKMP traffic, but no more ESP packets)

The situation is a little different between peers, and sometimes there are stable phases for one peer, getting bad again after a while, but none is 100% fine. It will take some seconds to some minutes until the tunnels stall; more traffic seems to speed up the failure.

I reinstalled one firewall with 20.1, and now we have stable performance again. The backup machine is in maintenance mode and still 20.7.1 (with syslog-ng fixed).

When reviewing the updates that happened between 20.1.3 and 20.7.1, strongswan was upgraded from 4.8.2 to 4.8.4 (in April), and the kernel from 11.2 to 12.1. Since IKE/ISAKMP traffic seems normal, I'd suspect some issue in the kernel/pf, but I'm out of clues how to narrow down the reason further.

Any thoughts on this?
Regards,
Andreas

6

20.1 Legacy Series / APPLY added interface shakes up firewall

« on: June 05, 2020, 01:38:28 pm »

On a pair of 20.1 firewalls coupled with CARP, I have some 27 VIFs defined, total 10 interfaces.
Recently, I added two more VLANs and two interfaces. When enabling the new interfaces on the CARP master (without IP for the start), many VIFs became unresponsive after APPLY. To resolve the issue quickly, I disabled and re-enabled CARP on the master, and operation was re-established.

I'm quite sure that I could add interfaces without any problems "in the old days", but trouble increased over time (hiccup/interrupts for some seconds), now defunct VIFs. This might have several reasons:
- just a matter of many interfaces
- opnsense code to execute the interface operations was changed

Is this a known issue? Something I have to live with?

7

20.1 Legacy Series / IPSEC phase2 edit not working

« on: May 08, 2020, 03:16:14 pm »

On a firewall with 20.1.3 (now 20.1.6), I noticed that it's not possible to edit phase2 entries on an existing configuration (unchanged and running for a while, probably older than a year)

When editing, the remote address or network isn't shown, when duplicating a single address phase2 entry with a different target addr, I get "already exists".

Digging further, I find some entries displaying (and probably editing) correctly, others not. A look at config.xml shows the problem: then non-functional entries don't habe a uniqid.

8

19.1 Legacy Series / Re: reordering packets under higher traffic

« on: July 17, 2019, 03:50:10 pm »

Shared forwarding under Firewall/Settings/Advanced isn't enabled.
No traffic shaping, routing groups, advanced rules or other fancy stuff configured.

9

19.1 Legacy Series / reordering packets under higher traffic

« on: July 05, 2019, 04:55:58 pm »

I'm running OpnSense 19.1 on Xen, connecting a DMZ host to its file server via NFS.
On rare occasions, when a big file is transferred, the nfs connection is broken, and a new tcp connection has to be started.

I've been tcpdumping the traffic in and out of the firewall (TCP segment offloading is disabled on all interfaces to avoid driver trouble), and found the following explanation:
Sometimes, a big PDU sent from the fileserver (split into 364 segments within 9.5ms) isn't forwarded to the destination DMZ host in-order, but instead in the middle of the flow segments are forwarded out-of-order, provoking out-of-order acks and resends, apparently driving the tcp stack mad and ultimately breaking the connection.

The server is a Xeon E5-2620V3, with 4 CPUs assigned to the firewall (low single digit cpu utilization, load rarely reaching 1), and no other machines running on the host. Typical state table size is 450, mbuf usage 800.

While the usage pattern of the system and general load hasn't changed over the last year, the problem started some months ago, which kind of coincides with the upgrade to 19.1 and the hardened kernel.

Why does the firewall start reordering, what can I do to prevent that?

Regards
Andreas

10

General Discussion / Re: Virtual to physical

« on: July 05, 2019, 04:14:01 pm »

The interfaces will probably named differently. The renaming can also occur when moving from one hardware to another.

You can modify the interface names in the config.xml before restoring it on the target, or tweak them afterwards, paying attention to the correct order of interfaces (opt1...optN)

11

19.1 Legacy Series / Re: CARP over LAGG problems

« on: July 05, 2019, 04:03:23 pm »

current settings are:

net.inet.carp.senderr_demotion_factor =0
net.inet.carp.ifdown_demotion_factor = 120
net.pfsync.carp_demotion_factor = 0

12

19.1 Legacy Series / system crashes

« on: June 21, 2019, 05:33:42 pm »

In the weeks, I observe spontaneous reboots every 3 days or so. This happens on the current master of a carp router pair (slightly different hardware after repair of one router), so this doesn't seem to be hardware related but appears caused by some kernel update in the last months.
The system.log doesn't help, it just logs the fresh boot. External monitoring doesn't show any anormal traffic or load (0.5 load, 2-5% CPU) on the Atom C3758 system with 19.1.9.

Any hints how to narrow this down?

Regards
Andreas

13

19.1 Legacy Series / Re: flowd.log not rotated: aggregation too slow

« on: May 31, 2019, 01:43:13 pm »

I started with fresh logging, i.e. deleted /var/log/flowd* and /var/netflow/*, restarted flowd and running  flowd_aggregate.py with some timing added in the console.

I can see that a typical aggregation run will take 70-80 seconds, then rotation is checked, then 15 seconds nap.
This means, that currently the flowd_aggregate process is at 85% of its capacity; there are certainly phases where firewall traffic is heavier than currently.

flowd.log is filled with a rate of about 2.5-5MB/min, /var/netflow/* has about 5MB. Doesn't seem to heavy to me.
Machine as a 8-core C3758CPU, Load avg about 1.3, with CPU at some % (flowd_aggregate will add 12%), SATA-DOM SSD.

14

19.1 Legacy Series / flowd.log not rotated

« on: May 31, 2019, 10:14:12 am »

I've been running netflow logging a while now flawlessly. Since about two weeks (I guess post-19.1.5), rotation of flowd.log stalls, resulting in a fillled disk and consequently stopping some services for out-of-diskspace reasons. This includes flowd and flowd_aggregate.py. The firewall is monitored, so I could check that flowd_aggregate is killed from out-of-disk, not before.

I upgraded to 19.1.8, and see some strange behaviour of flowd_aggregate:
Usually, flowd.log will reach between 11MB and 13MB before rotating; it takes about 5 minutes for this size. But I also can observe this:

-rw-------  1 root  wheel   3.2M May 31 10:01 flowd.log
-rw-------  1 root  wheel    12M May 31 10:00 flowd.log.000001
-rw-------  1 root  wheel    13M May 31 09:56 flowd.log.000002
-rw-------  1 root  wheel    56M May 31 09:51 flowd.log.000003
-rw-------  1 root  wheel    13M May 31 09:29 flowd.log.000004
-rw-------  1 root  wheel    23M May 31 09:24 flowd.log.000005
-rw-------  1 root  wheel    12M May 31 09:15 flowd.log.000006
-rw-------  1 root  wheel    13M May 31 09:11 flowd.log.000007
-rw-------  1 root  wheel    11M May 31 09:06 flowd.log.000008
-rw-------  1 root  wheel    12M May 31 09:01 flowd.log.000009
-rw-------  1 root  wheel    12M May 31 08:56 flowd.log.000010

So obviously flowd_aggregate stalls sometimes for some minutes, and then continues to work. I can't see any anomalies in cpu load or usage during this period. Nothing in the system log, the last flowd related message is
 May 31 06:47:26 fw05a flowd_aggregate.py: vacuum done

While this is not the total stop of flowd.log rotation I've been suffering from in the last weeks, it still seems suspicious to me. What's going wrong here?

Added: stalling is happening right now, and I can see the flowd_aggregate proces consuming a lot of CPU.

15

German - Deutsch / Re: 17.7.6 CARP-Pakete werden gefiltert

« on: May 06, 2019, 10:58:14 am »

Ich denke ich weiß jetzt woran es liegt, siehe https://github.com/opnsense/core/issues/3468