Port closed from firewall itself on ISP WAN

[matlink]

Hello,
I am trying to replace my SFR box (neufbox, france) by an OPNsense. This OPNsense is currently virtualized on a Proxmox.
However, everything goes well until I try to telnet on some ports (i.e. 53 and 80) from the firewall itself to external IPs (i.e. 9.9.9.9, 8.8.8.8, 1.1.1.1 ) using SSH.

Here is what I found:

• ping works whatever the destination
• if firewall is enabled, telnet to anything external to WAN is port blocked (response is immediate, permission denied)
• if firewall is disabled, telnet is possible only for IPs having a dedicated route in the route table. If there is no route or the target is a network (not /32), then it is port blocked. I see that such routes have the flag 'H' for host, while others miss that flag. The default route has just that difference compared to working routes
• if I use the OPNsense as gateway for another machine, everything works as expected (unless blocked from firewall rules)

What make me think the issue comes from the OPNsense firewall itself is when I disable the firewall and add route to specific IPs (/32), telnet works.

Here is the complete setup:

• Proxmox 7.3-3 on a computer (dell optiplex 3032)
• OPNsense 22.7.10-amd64 in a virtual machine
• OPNsense has 2 network interfaces, one for the LAN, one for the WAN
• packets are tagged on VLAN 2 for the WAN interface, but on the Proxmox itself (OPNsense has no knowledge of that VLAN)
• the switch is a D-link DGS-1210-10P where the proxmox is plugged on a trunked port, PVID 1 and tagged 2
• the fiber ONT is on a port in VLAN 2
• OPNsense is using DHCP on WAN, with spoofed parameters from that guide https://github.com/opnsense/docs/blob/master/source/manual/how-tos/sfr_red_fr_ftth.rst (I do get an IP, ping works, etc.)
• Everything is using IPv4 for now
• CGNAT is enabled, but I doubt it has an impact

What I've tried:

• disable force gateway is disabled, changing this had sometimes made some changes but it is still unclear to me
• network analysis using live view or wireshark/sshdump shows that packets are leaving the WAN interface but never returns. Something looks like a response but on another port
• network analysis on the Proxmox interfaces show that packets never arrives on them.
• adding pass rules for anything to anything on WAN interface in both directions (in/out)
• adding pass rules for antything to anything on floatings rules interface in both directions (in/out)

I'm quite out of ideas for solving this. I don't believe that it comes from ISP nor ONT/DHCP configuration. I've checked a bunch of topics over there, found some related issues but none that solved my issue. This one looks very similar: https://forum.opnsense.org/index.php?topic=12665.0.

Thanks in advance for anyone who would try to help me.
Mat.

[matlink]

After some more investigation here is what I found.
I simulate a DHCP server with another machine, connected to the WAN of the OPNsense. When the server returns a lease in a /31 network, telnet does not work from OPNsense, but I give a /24 network lease, it does work. Here are the 2 differents routes that are created in OPNsense routing table :

when /31:


when /24:


With /24, telnet does work for every destination, not only those having a dedicated route in OPNsense.

Howeever, when manually adding the route when connected to ISP, the /31 route seems to have a higher priority...

[cookiemonster]

Hi.
The firewall rules straight out of installation work with all traffic out of WAN is allowed and anything coming in needs to match an outgoing request, as per normal stateful firewall behaviour. The state violation block suggests the connection coming in is not matching the one that left the firewall, but it also suggests there something outside OPN causing it. Whether is the tagging, proxmox firewalling/ragging or something else is hard to know.

[matlink]

Yes I realized that this default denied packets comes from another equipment and is not really related to the given problem. Mostly the final issue is due to a routing misconfiguration, but I still can't see where.

[matlink]

Interesting thing: If I edit the WAN_DHCP gateway (automatically created by the DHCP configuration) by putting any public IP in "IP address" and mark it as "Far Gateway", everything works as expected.