Messages

I just saw that there actually IS traffic to the upstream ftp server in PASV mode, and the setup works again after I corrected ftp-proxy source address to use the outbound NAT interface address as well.

Case closed.

After upgrading a firewall from 24.1.x to 24.7.7, ftp-proxy doesn't work any more. Analyzing traffic with tcpdump on both LAN and WAN interface:

- in PASV mode, the client sends SYN packets to the port as returned from ftp-proxy, but there's not traffic to the upstream ftp server.
- in active mode, the server sends SYN packets to the port as presented by the PORT command that ftp-proxy issued upstream, but won't forward any traffic to the client.

I have logging enable on both the client-to-ftp-proxy redirect on Port21, as well as client-to-server traffic for the data connection, both log as "pass" when issuing the client data command.

I checked against that FTP server with another client/firewall (different site, a lot simpler firewall setup), also on 24.7.7, which works correctly there.

From the ftp-proxy man pages, I'd expect to see something with

Code Select Expand

pfctl -a ftp-proxy -s rules, but there's nothing while the data connection is stuck.

I'm out of clues now, anybody with an idea?

Regards,
Andreas

Same thing with me. Not only the gateway assignment in the WAN interface was removed, the gateway definition itself was gone.
All other gateways were still present.

After upgrading to 24.1_1, firewall groups cannot be edited any more:
The overview doesn't show the members any more, when editing all fields are empty (no members available in the dropdown list)

Well these are the very two buttons I always check.
Temporarily disable CARP won't survive the reboot (which is just what I need), and persistent CARP maintenance state doesn't prevent CARP becoming MASTER unconditionally either, which is the very reason for this post.

Just checked on a box that never had manual fixes, but that py37-markupsafe leftover nevertheless.
pkg remove py37-markupsafe is your friend ;-)

in this case you might try to disable carp completely.

This is what I'd expect from "maintenance mode". I'm not aware of any other means in opnsense to disable carp. Digging FreeBSD docs, I found sysctl net.inet.carp.allow. So am I supposed to use that tuning or am I missing something? (some "enable/disable CARP" opnSense setting)?

Ok did a while to understand, because I was looking for the problem in the router...

To summarize, if STP is configured on the switch, it will not forward traffic for a while directly after the port is physically up; in consequence the freshly rebooted firewall won't receive CARP packets from the master and assume it's dead.

However, this doesn't explain the initial question: why does the firewall do carp at all? I want CARP disabled in the first place to prevent such very glitches.

Some days before the errors started, the switch was replaced (cabling and switch are in the provider's realm), but the switch was still silent afterwards. Errors began right after upgrade, so I wonder if the driver now can detect some error situation that it couldn't before.

Actually, I have TWO firewalls in CARP configuration. The regular error rate is on the carp master, but there are also some error on the backup.

I think I have seen STP packets, I think I can have them disabled.
But what has spanning tree to do with carp? I'd expect only proto-112 packets to have any impact (and least of all in maintenance mode).

I played with this, and apparently (as Franco stated) they're 'somehow' applied. But this means, if you change settings (tso or so), the lagg members will take the settings only on reboot from global settings, no possibility to tweak the hardware online or individually.

When doing regular maintenance on our CARP cluster, I regularly disable CARP on the machine and enter persistent maintenance mode. I'd expect it to never get MASTER until I enable CARP again.

Now, I rebooted the machine (22.1.1), and while it came up I glanced "Timeout on ix2, becoming MASTER" on the console for a second or so until it stepped back to BACKUP.
While I also have layered interfaces (vlan over lagg over 10GBit), this very ix2 interface is just a plain 1GBit onboard Intel NIC, connected to a switch, no VLAN no whistles or bells (upstream internet).

Having double master even for fractions of a second will screw up network traffic more or less badly, so this really isn't good and shouldn't happen, maintenance mode or not.

So how to safely reboot a router without triggering major trouble?

The internet provider guy says there's no errors on the switch side.

sysctl -A shows
  dev.ix.2.mac_stats.checksum_errs: 15381
  dev.ix.2.mac_stats.rx_errs: 15381
which correspond with 15381 IErrs from netstat.
Since there seems a constant error rate, I captured some 5 minute and filtered with wireshark for errors:

eth.fcs.status=="Bad" || ip.checksum.status=="Bad" || tcp.checksum.status=="Bad" || udp.checksum.status=="Bad"

But the result was zero.

netstat -i output, redacted after 42h uptime:

Name    Mtu Network   Address  Ipkts        Ierrs  Idrop  Opkts        Oerrs  Coll
ix2    1500 <Link#5>  xx:xx:xx 40644829 6335  0       59986652  0        0

The firewall is monitored by checkmk, which calculates the error rate. Attached the current graph (was flat zero for the last 400 days until recently)

I agree this isn't a horribly high error rate, but the firewall was quiet for years until the update. So something must have changed. Maybe some driver issue?

After upgrading from 21.7 to 22.1.1, my firewall shows an error rate of 0.05/s avg on the WAN interface (SuperMicro A2SDI on-board NIC connected to a Juniper switch afaik), which used to be zero before (netstat -i Ierrs). The machine has 4 more connections, all are still at zero errors.

What may be the reason?