Hey, I've hit a brick wall and need some places to look for cracks.
Problem: For testing I've put in pass-all rules on all interfaces, but firewall does not match these rules and so default-denies.
Situation (for completeness sake):
Topsense N100-based device with 4 2.5GBe, on which I've added an m.2 WiFi card for an access point, and USB-ethernet and USB wifi dongles.
The USB devices are just for maintaining GUI access to proxmox and Opnsense as I experiment with networking, and the WiFi card has successfully been passed through to an OpenWRT VM.
I don't expect any of this will be pertinent to the problem.
I've got a fresh OPNSense VM running with 3 interfaces bridged from ProxMox:
1. igc3WAN - home router to 4th port on router to proxmox bridge vmbr3, OPNsense sees it as vtnet0, DHCP client on OPNSense, gets DHCP lease successfully from my home router at 192.168.50.55, can ping gateway and google DNS when firewall brought down with console "pfctl -d"
2. LANvmbr0 - bridge on proxmox, USB-ethernet device, OPNsense sees it as vtnet2, my home router issues DHCP lease just fine at 192.168.50.39, can ping gateway and google DNS when firewall brought down with console "pfctl -d".
3. VMNetOpenWRT - bridge on proxmox, no physical device, OPNsense sees it as vtnet10, DNSMasq on OPNsense successfully issued leases to an OpenWRT and two Linux Mint VMs for testing, all can ping each other.
Below is the result from using the web interface on LANvmbr0 (192.168.50.39) and the (temporary) ruleset in place that I think should have prevented it from being denied. At the same time, pinging my home router from console times out.
If I then use pfctl -d from console I can both ping and get to the web gui on this interface, so its clearly the firewall.
(https://i.ibb.co/r9h9cd0/Log-screen.png) (https://ibb.co/NRqRmTC)
(https://i.ibb.co/XfkXcnPf/interface-rules.png) (https://ibb.co/Pszxq2ys)
(https://i.ibb.co/b5GZbQsT/floating-rules.png) (https://ibb.co/9HdLZcb6)
I have:
1. Rebooted the router after changes
2. Rebooted the proxmox host
3. Dumped the state table
4. Restarted all services from console
What would make this allow-all rule set work so I can complete and test the network and then slowly add firewall rules?
Ok, weird. I decided to wait for replies on this thread, and in the meantime disable the packet filter with pfctl -d and work on other bits of the config. I went to gateways and found the IP for the active gateway was no longer showing (this was previously automatically added in and I'm certain it was showing as 192.168.50.1). Added that IP in, rebooted and the pass-all seems to work.
I...still don't know why this would present in logs as it did.
Maybe a little diagram would help to understand your network better. If you want your rules to show up in the logs, you need to activate logging for them. The IPv6 pass-all rule is probably not doing what you expect. The direction of a rule is important. Most of the time the direction should be "in".
My best guess is that you may not be looking at the correct interface. That's why turning the packet filter off helps. If I had to debug this situation, I'd add one pass-all floating rule with debugging enabled. This way, one can see in the logs which interface the traffic is coming in.
It's not obvious what your IP subnet is, but assuming a /24, the traffic in question should not even hit the FW...
IIRC, you were also playing with bridges on PVE/OPN.
If that traffic is exercising the bridge members, it looks like another case of filtering at the wrong level (members vs bridge).
Quote from: mooh on May 30, 2025, 12:39:06 PMMaybe a little diagram would help to understand your network better. If you want your rules to show up in the logs, you need to activate logging for them. The IPv6 pass-all rule is probably not doing what you expect. The direction of a rule is important. Most of the time the direction should be "in".
My best guess is that you may not be looking at the correct interface. That's why turning the packet filter off helps. If I had to debug this situation, I'd add one pass-all floating rule with debugging enabled. This way, one can see in the logs which interface the traffic is coming in.
Thanks, I'll keep that in mind when debugging future issues (and have done, with an port forward issue I'm having right now that I hope I'll be able to figure out today rather than posting again).
Quote from: EricPerl on May 30, 2025, 07:45:25 PMIt's not obvious what your IP subnet is, but assuming a /24, the traffic in question should not even hit the FW...
IIRC, you were also playing with bridges on PVE/OPN.
If that traffic is exercising the bridge members, it looks like another case of filtering at the wrong level (members vs bridge).
Thanks - I had the pass-all rules on both the member and bridge and was using /24.
Again, such traffic should go between peers directly.
One possibility is that one of them is misconfigured (e.g. /32 netmask) so that it sends everything to the gateway.
Another possibility is that the 2 peers are connected to different members of the bridge.
A properly configured bridge should not perform any filtering at the member level for this reason.
Connections between peers is regular switching and typically not subject to filtering (which is expected at the interface level).
That's what the tunables in the bridging guide are all about.
Quote from: EricPerl on June 02, 2025, 10:29:32 PMAgain, such traffic should go between peers directly.
One possibility is that one of them is misconfigured (e.g. /32 netmask) so that it sends everything to the gateway.
Another possibility is that the 2 peers are connected to different members of the bridge.
A properly configured bridge should not perform any filtering at the member level for this reason.
Connections between peers is regular switching and typically not subject to filtering (which is expected at the interface level).
That's what the tunables in the bridging guide are all about.
Ah - the 192.168.50.x network is on my home router, and the 192.168.50.10 address is my laptop (which ifconfig shows to be using a /24 netmask). So, my laptop .10 is on the same home router network as OPNsense at .39 which sees it as an upstream WAN. That seems to me as analogous to it being connected to my ISP's ONT at the WAN port and my laptop is like some other client on the internet trying to talk to it, so I'd expect that to be firewalled. I'm not surprised that without that default gateway being set that it didn't work, but am surprised that it presented as a default block by firewall and not just a routing error.
I can't believe how badly I misread that original post.
Both the "WAN" and the "LAN" side of OPN get IP via DHCP from your home router???
That's an invalid setup. You can't have overlapping subnets across interfaces.
OPN's LAN should have its distinct range, statically assigned.
I won't comment on the OpenWRT VM for now.
Quote from: EricPerl on June 03, 2025, 07:28:08 AMI can't believe how badly I misread that original post.
Both the "WAN" and the "LAN" side of OPN get IP via DHCP from your home router???
That's an invalid setup. You can't have overlapping subnets across interfaces.
OPN's LAN should have its distinct range, statically assigned.
I won't comment on the OpenWRT VM for now.
My fault entirely, I think I need to start over. I couldn't get port forwarding working so I took a proxmox snapshot and then "factory reset" the OPNSense instance.
Background: My ultimate goal is to have all home devices (including WiFi clients, IP Cameras, a general purpose linux server (self hosted apps behind a reverse proxy), a NAS and a zigbee gateway) on separate VLANs via a managed switch behind this OPNSense appliance plugged directly into my fibre ONT.
So far, separately, I've successfully experimented with Outbound NAT, VLANs shared by proxmox VMs and physical devices on a managed switch, and DHCP. When I understand each bit I'll pull everything down and build it out properly.
The parts I have not yet been able to get working are port forwards and PPPoE via the ONT to my ISP (not yet attempted).
After factory reset on the OPNSense instance I brought up only the following:
1. WAN Interface via proxmox bridge to NIC to home router
2. One VLAN on a separate proxmox bridge attached to a separate NIC, dnsmasq serving DHCP.
3. Test VM on the same proxmox bridge with same VLAN tag, successful DHCP lease from OPNSense.
(https://i.ibb.co/fzTWn9y7/Experimental-network.png)
My intention is to be able to simulate SSH'ing from anywhere to the WAN public IP and have it port forward to another machine. I can currently do this on my home router quite easily. However today I got perpetually stuck with the firewall logs showing redirection, but connection timeout from the ssh client.
That's on the backburner for the moment because the behaviour described in the original post in this thread is back on the "fresh" instance. I have the same pass-all rules in place, but without pfctl -d it blocks all traffic on the WAN interface including the gui from my laptop, and this time the gateway address wasn't missing.
(https://i.ibb.co/Z6DwBSS4/gateways.png)
I've once again gone over the steps including deleting and reinstating interfaces and rules, clearing the state table and rebooting all levels including the home router, but can't seem to get the OPNSense instance to once again pass all traffic. Is there anything that comes to mind to check?
Ok, spun up another OPNSense VM to do a cold install, disabled the first one.
After install:
1. Set WAN to DHCP -> OK, new IP 192.168.50.67
2. Added vlan50 -> OK
3. Enabled vlan50 interface with static ip 10.2.1.1/24 -> GUI connection drops.
4. Console pfctl -d . Check firewall logs - shows default deny on WAN IP, port 443 from laptop IP.
5. Delete vlan50 interface via GUI. pfctl -e. No access to GUI.
6. Console pfctl -d. Delete vlan50 device. pfctl -e. No access to GUI.
Might this be a bug? I should be able to reverse changes and have results reverse too, unless creating VLAN devices and assignments makes changes not undone in deletion?
Edit: Tried same process with proxmox bridge with no physical member. Other VM members on this bridge can still talk to each other just fine with static IPs so everything seems working as it should with the bridge itself. Still knocks out the firewall rules.
The diagram is reasonable. You don't need to assign a member NIC to the "LAN" bridge if you only plan to connect VMs.
meyergru has a Proxmox + OPN post in the tutorials. The 2nd post is about a data center use case (1 NIC).
Feel free to continue to use the USB NIC but you don't have to. You can also adapt as appropriate.
Make that Proxmox bridge VLAN aware. All you have to do for the Linux Mint VM is to specify a VLAN tag = 50 in the network device passed to the VM.
Wrt to your issue:
By default, WAN does not allow access to the GUI (no FW rules).
But if it's the only interface, anti-lockout applies to it.
When you're adding another interface, anti-lockout "moves" to that...
Add a FW rule to WAN to allow GUI access.
Quote from: EricPerl on June 03, 2025, 07:17:17 PMAdd a FW rule to WAN to allow GUI access.
And disable reply-to of the management PC is connected to the same network as WAN and the default gateway.
Quote from: EricPerl on June 03, 2025, 07:17:17 PMThe diagram is reasonable. You don't need to assign a member NIC to the "LAN" bridge if you only plan to connect VMs.
meyergru has a Proxmox + OPN post in the tutorials. The 2nd post is about a data center use case (1 NIC).
Feel free to continue to use the USB NIC but you don't have to. You can also adapt as appropriate.
Thanks, it took a while to wrap my head around that from tutorials but its been consistently working :-)
Quote from: EricPerl on June 03, 2025, 07:17:17 PMMake that Proxmox bridge VLAN aware. All you have to do for the Linux Mint VM is to specify a VLAN tag = 50 in the network device passed to the VM.
They are, and VLANs are working fine otherwise - on the last VM I also successfully brought up an outbound NAT for internet access, and the Mint VM with a VLAN tag could talk easily with a debian SBC I had plugged in to a managed switch on a tagged port.
Interestingly enough an issue I had two days ago was that I accidentally enabled VLAN awareness in proxmox on the WAN bridge and that took down all OPNSense networking. I reversed that change but nothing I did could get it back. I ended up restoring from snapshot which to my mind shouldn't have worked but it did. I don't have enough records on what I did to troubleshoot that again though...
Quote from: EricPerl on June 03, 2025, 07:17:17 PMWrt to your issue:
By default, WAN does not allow access to the GUI (no FW rules).
But if it's the only interface, anti-lockout applies to it.
When you're adding another interface, anti-lockout "moves" to that...
Add a FW rule to WAN to allow GUI access.
That's the part I'm having difficulty understanding - the pass-all should cover that (and it did on the last VM) but it doesn't fire, it still reaches default-deny. The rule moving to the new interface (and not moving back on delete) explains why the anti-lockout doesn't do its thing anymore, but I can't figure out why the pass-all doesn't trigger.
The new auto rules are showing here, so I tried to specifically replicate the last 3 back on the WAN:
(https://i.ibb.co/7T361Wv/new-interface-auto-rules.png) (https://ibb.co/XdnvCsX)
I've substituted (self) for "This Firewall" but can't find a way to replicate "(vtnet0)", the port assigned as [WAN].
No change in behaviour.
Quote from: Patrick M. Hausen on June 03, 2025, 08:17:34 PMQuote from: EricPerl on June 03, 2025, 07:17:17 PMAdd a FW rule to WAN to allow GUI access.
And disable reply-to of the management PC is connected to the same network as WAN and the default gateway.
Thanks - tried that, unfortunately no change.
Ok, what the actual hell.
I deleted the VLAN and brought up a bridge to a physical port for testing - no change in behaviour.
Then I added the VM back in, reasoning that I could just run the management interface through a VM in the interim.
Didn't need pfctl -d. Yes, for some reason it now works as soon as the next interface was added. I have no explanation for this. pfctl -e tells me the firewall is definitely already back up.
I'll need to understand eventually but for now I'm going to work on that port forward test.
Good call from Patrick. I was time pressed and forgot to mention that one.
It's always sane to do this on an internal OPN. Without it, traffic is bouncing off of the edge router LAN GW...
FWIW, the anti-lockout rule is primarily a port forward rule (with associated FW rule).
What you see on that screen is the associated rule.
On WAN, all you need is one rule to allow HTTPS (you don't really the HTTP one is you disable the HTTP->HTTPS redirect or type the full https:// when you access the GUI.
Note that the rule won't fire if the anti-lockout rule are currently on that interface because PF rules take precedence over FW rules.
How about showing the WAN rule you added?
Going back to the vmbr2 bridge (LAN side of OPN).
A bridge is a switch.
If the member is a physical NIC, it's like a physical port on the switch.
When you assign the bridge to a VM, you get a virtual port on the switch.
The traffic on the bridge is going to be dependent on the interface(s) you assign on the OPN side:
* Assign an interface to vtnet5 directly and traffic is not tagged
* Assign an interface to a vlan device parented to vtnet5 and traffic is tagged
When you have vmbr2 passed to the VM with a VLAN tag, it will only be exposed to traffic from the VLAN (BUT the traffic received by the VM is untagged, as with a regular access port on a switch). You don't have to do any VLAN config within the VM.
If you connected a machine to NIC3:
When the OPN LAN interface is assigned to a vlan device, traffic is tagged (and ignore by that machine unless its NIC is configured appropriately).
When the OPN LAN interface is assigned to vtnet5 directly (no VLAN), traffic is not tagged and should flow freely.
Quote from: EricPerl on June 03, 2025, 11:01:43 PMGood call from Patrick. I was time pressed and forgot to mention that one.
It's always sane to do this on an internal OPN. Without it, traffic is bouncing off of the edge router LAN GW...
FWIW, the anti-lockout rule is primarily a port forward rule (with associated FW rule).
What you see on that screen is the associated rule.
On WAN, all you need is one rule to allow HTTPS (you don't really the HTTP one is you disable the HTTP->HTTPS redirect or type the full https:// when you access the GUI.
Note that the rule won't fire if the anti-lockout rule are currently on that interface because PF rules take precedence over FW rules.
How about showing the WAN rule you added?
WRT the anti-lockout - even if the port forward takes precedence over the firewall, one of these should still have triggered a pass and prevented a default-deny, right?
WAN Rules (though its working now after adding in another interface):
(https://i.ibb.co/r2fGYYBt/WAN-Rules.png) (https://ibb.co/DfgDhhBb)
Quote from: EricPerl on June 03, 2025, 11:01:43 PMGoing back to the vmbr2 bridge (LAN side of OPN).
A bridge is a switch.
If the member is a physical NIC, it's like a physical port on the switch.
When you assign the bridge to a VM, you get a virtual port on the switch.
The traffic on the bridge is going to be dependent on the interface(s) you assign on the OPN side:
* Assign an interface to vtnet5 directly and traffic is not tagged
* Assign an interface to a vlan device parented to vtnet5 and traffic is tagged
When you have vmbr2 passed to the VM with a VLAN tag, it will only be exposed to traffic from the VLAN (BUT the traffic received by the VM is untagged, as with a regular access port on a switch). You don't have to do any VLAN config within the VM.
If you connected a machine to NIC3:
When the OPN LAN interface is assigned to a vlan device, traffic is tagged (and ignore by that machine unless its NIC is configured appropriately).
When the OPN LAN interface is assigned to vtnet5 directly (no VLAN), traffic is not tagged and should flow freely.
Thanks for that - I have the VLAN working, I think - I have the VM tag the traffic and the bridge VLAN-aware but untagged as I intend for the interface to handle more than one VLAN.
That said now that the firewall is working I still have no luck getting a port forward to work. I'm trying to forward TCP port 21 (now 192.168.50.67 with the new OPNSense VM) on the WAN IP to port 22 on the Linux VM (still 10.2.1.54 and can ping 10.2.1.54 so its live).
(https://i.ibb.co/9kzGw9HR/Port-forward-attempt.png) (https://ibb.co/W4qsDWvS)
I can see where the incoming connection from my laptop (now 192.168.50.212) is being passed to port 21, but no corresponding rdr or any other traffic showing port 22. Edited: My error, I mistyped the ssh connection command of all things. I can now see when the inbound connection from laptop hits the firewall port 23, when it redirects to 10.2.1.54:22, and outbound from 10:2.1.54:22 back to originating IP as below. Unfortunately, still have an ssh timeout.Where might I look to see where the roadblock is?
(https://i.ibb.co/JRgcqHXb/Live-log-updated.png) (https://ibb.co/jvs8zD7x)
Quote from: devl_ish on June 03, 2025, 09:39:01 PMOk, what the actual hell.
I deleted the VLAN and brought up a bridge to a physical port for testing - no change in behaviour.
Then I added the VM VLAN back in, reasoning that I could just run the management interface through a VM in the interim.
Didn't need pfctl -d. Yes, for some reason it now works as soon as the next interface was added. I have no explanation for this. pfctl -e tells me the firewall is definitely already back up.
I'll need to understand eventually but for now I'm going to work on that port forward test.
Slight correction to prevent confusion.
Delving further into things, I've taken a look at the routes status table and have a question about it - what does "link#n" mean? I've had a google around and can't find an explanation.
(https://i.ibb.co/JRmCn5PK/Routes-Status.png) (https://ibb.co/Fbmn6gtK)
To my newbie mind communication would go like this:
Laptop ssh client -> home router gateway (192.168.50.1) -> Opnsense WAN (now 192.168.50.67) port 23 -(NAT)-> interface 10.2.1.1 -> destination sshd (10.2.1.54 port 22)
...and then the response travels that path in reverse. In that case where along that line might it be misconfigured?
Edit: found https://unix.stackexchange.com/questions/110642/what-does-link1-mean which says under FreeBSD this is the interface in the sequence shown by ifconfig. The routes shown seem to be directing the right subnets to the right interfaces so as far as I can tell that's not causing lost communication.
Does log line:
ServersandNAS 2025-06-04T15:52:55 192.168.50.10:44122 10.2.1.54:22 tcp
...imply that communication from the laptop is being directed to the right place IP and port, but there is no route configured for it to use to get there?
Please attach screenshots using preview or reply (vs. quick reply).
As of now, your 3rd party links are hit and miss...
The 2nd "manual anti-lockout" should be sufficient. As is, you won't see stuff in the logs because logging is not enabled on it.
Nothing should end with a default deny with these rules. More don't have logging and some of them don't have description (makes troubleshooting harder).
In the PF rule, I'd use "WAN Address" and add a description.
Given the live view logs, traffic passed through the FW (in on WAN, out on ServerAndNAS).
The next step is to do a packet capture to see if anything is coming from the ssh server.
It's in interface diagnostics. Capture the WAN and LAN side for port 22 (it would be easier if the WAN port was also the standard 22).
More tomorrow. It's late...
Quote from: EricPerl on June 04, 2025, 08:26:48 AMPlease attach screenshots using preview or reply (vs. quick reply).
As of now, your 3rd party links are hit and miss...
Apologies, they were showing up well in Preview - swapping image host.
Port forward currently:
(https://i.postimg.cc/mgNNnSJf/WAN-Rules-050625-9am.png) (https://postimg.cc/18XqnDyJ)
Route Status:
(https://i.postimg.cc/Wbrp7qZk/Routes-Status.png) (https://postimg.cc/Yhrwr9br)
Log:
(https://i.postimg.cc/CxYbbq1v/live-view-050625-9am.png) (https://postimg.cc/RNs3Bqwf)
Quote from: EricPerl on June 04, 2025, 08:26:48 AMThe 2nd "manual anti-lockout" should be sufficient. As is, you won't see stuff in the logs because logging is not enabled on it.
Nothing should end with a default deny with these rules. More don't have logging and some of them don't have description (makes troubleshooting harder).
I agree - and am still at a loss to explain how the denies happened. Nevertheless its working now after I added in the VLAN, even if I don't understand why that is. I'm concerned that the behaviour will return on the final build or when I add in other VLANs/interfaces.
QuoteIn the PF rule, I'd use "WAN Address" and add a description.
Thanks - done, screenshot above. No change.
QuoteGiven the live view logs, traffic passed through the FW (in on WAN, out on ServerAndNAS).
The next step is to do a packet capture to see if anything is coming from the ssh server.
It's in interface diagnostics. Capture the WAN and LAN side for port 22 (it would be easier if the WAN port was also the standard 22).
I started by changing the port forward to "WAN Address" and changing the port forward to 22 -> 22 and confirming that firewalls are down on both proxmox and the linux VM.
Packet capture analysis with Wireshark (hey, new skill...):
Laptop 192.168.50.1, destination port 22, WAN interface
(https://i.postimg.cc/DwcdNZGC/from-laptop.png) (https://postimg.cc/nXCDjn07)
Laptop 192.168.50.1, destination port 22, VLAN interface
(https://i.postimg.cc/0jqhfjD8/Laptop-IP-vlan.png) (https://postimg.cc/TLCk2RtS)
I reinstated the "This Firewall" destination port forward and opened an SSH connection from another VM on 10.2.1.0/24 to 10.2.1.1 to see what the traffic is supposed to look like:
10.2.1.54, port 22, VLAN interface:
(https://i.postimg.cc/brjZ80c1/target-capture.png) (https://postimg.cc/7GmHn2LL)
As far as I can understand the firewall isn't blocking anything, but any requests from 192.168.50.10 are being stopped somewhere or don't have a route.
QuoteMore tomorrow. It's late...
I really appreciate your efforts on this, I can't imagine you have a lot of time on your hands and I'm grateful for your patience on the bits I've badly explained or missed out. Thank you!
FYI, 'WAN Address' is the IP assigned to the WAN interface likely via DHCP.
'This Firewall' is the collection of IPs assigned to all interfaces.
Using 'This Firewall' makes sense for FW rules where an IP associated with the FW might do.
It makes less sense on a port forward on a single interface where only one of the IPs is relevant.
It looks to me like no traffic is coming back from the Linux VM.
Have you confirmed outbound network connectivity from that VM (IP, ping, DNS, light browsing) as a way to establish some baseline on network connectivity?
Have you confirmed that you can connect to it from its LAN? Or even just from OPN?
Is sshd even installed and running?
systemctl status sshd
Edit:
That last screenshot is ssh into OPN? It looks rather healthy.
Quote from: EricPerl on Today at 01:05:11 AMFYI, 'WAN Address' is the IP assigned to the WAN interface likely via DHCP.
'This Firewall' is the collection of IPs assigned to all interfaces.
Using 'This Firewall' makes sense for FW rules where an IP associated with the FW might do.
It makes less sense on a port forward on a single interface where only one of the IPs is relevant.
Thanks for that, I had assumed that's how it worked so was using it interchangeably for testing, but you're right, I should have used "WAN Address" in the first instant and carried it through.
Quote from: EricPerl on Today at 01:05:11 AMIt looks to me like no traffic is coming back from the Linux VM.
Have you confirmed outbound network connectivity from that VM (IP, ping, DNS, light browsing) as a way to establish some baseline on network connectivity?
Have you confirmed that you can connect to it from its LAN? Or even just from OPN?
Is sshd even installed and running?
systemctl status sshd
Edit:
That last screenshot is ssh into OPN? It looks rather healthy.
Yup, the last screenshot is ssh from another Linux VM on the same VLAN 60 (10.2.1.48 from memory, I've taken it down now) which I used to successfully test whether a "This Firewall" port forward would allow it to ssh to 10.2.1.1 and have its traffic forwarded to 10.2.1.54 (the test VM with sshd running). Worked fine.
10.2.1.54 is able to:ping 10.2.1.1, and ping 8.8.8.8 when I have an Outbound NAT rule enabled, and ping the laptop (192.168.50.10) on the VLAN50 interface ssh to another offsite server when I have an Outbound NAT rule enabled
ufw confirmed down on this machine as well, and the offsite server has its ssh on a non-standard port....and just in the middle of typing that I found the problem.
The linux VM was also bridged to vmbr0, a bridge on proxmox, and had an IP of 192.168.50.30 on that interface. It was not a fresh VM and I had that up to enable updating and installing openssh-server from memory.
When I SSH'd from the laptop to the firewall (192.168.50.67) it correctly NAT'd to 10.2.1.54, but when 10.2.1.54 replied it logically routed back through the other interface (or didn't reply at all, who knows).
I deactivated the interface to vmbr0 and it worked :-)
Many thanks for your help! I'm still keen to know why the rules in the original post didn't fire, but as I can no longer replicate that behaviour I'll have to make that a problem for later.
As the original question remains, do I still (and how) mark this thread as solved?