RESOLVED: client on LAN can't access OPNsense, and OPNsense can't access LAN

[imneedham]

I'm starting with a very simple case so that I can get it working before expanding. I"m also converting a previously working setup, so I know it should work if configure the client and OPNsense correctly. All hosts are hosted in VMware. I've probably overlooked something obvious, but I've looked at a lot of seemly-similar forum posts without any luck. For the record, there is no DNS or DHCP involved (yet). There is VPN involved, but I am NOT using the OPNsense VPN.

My previously working set up was a dual-homed Ubuntu 20 forwarder (10.100.100.1 and 10.10.10.1) with iptables rules blocking all traffic from WAN to LAN while allowing traffic through to the internet and to some specific hosts on other segmented networks (natted) on the WAN side. Example: LAN client 10.10.10.10 (also Ubuntu 20) uses 10.10.10.1 as is' gateway and can access 8.8.8.8, 1.1.1.1, or other internet hosts, and is allowed to ready 10.30.30.10 for DNS and 10.40.40.10 for NTP, but not anything else at 10.x.x.x. This works fine.

I've installed OPNsense 24.2.1 on a VM with IP addresses 10.100.100.2 and 10.10.10.2. (I've added a firewall rule allowing me to access the web interface and ssh from my VPN management pool on the WAN, and this works). I enabled ssh into OPNsense. Before adding a rule to allow ssh access from the WAN, the logs showed it being blocked, and after the rule it shows it being passed. On the LAN client I've changed the gateway from 10.10.10.1 to 10.10.10.2 and reboot. It can now ping the old forwarder at 10.10.10.1 as well as OPNsense at 10.10.10.2. But when I try to ssh to 10.10.10.2 I get "connection refused" with no logs (ssh listens on "all interfaces").

Also from OPNsense I can ping anything on the WAN, and it gets logged (all rules log). But I can't ping (or ssh) anything on the LAN and it is NOT logged. I've also used "pfctl -d" to disable the filtering, but that doesn't help.

So I suspect that the problem is not a firewall rule, but something else I don't have configured correctly in OPNsense. Any suggestions with the info I've provided so far? What additional information would you like to see?

Thanks.

[Patrick M. Hausen]

Are you sure you connected the interfaces in the correct order? In OPNsense the first interface is LAN and the second is WAN. pfSense is the other way round.

[imneedham]

Pretty sure.

Code: [Select]

Interface      Device    VLAN   Link Type      IPv4                 IPv6   Gateway        Routes
LAN (lan)      vmx0             static         10.10.10.2/24                              10.10.10.0/24
WAN (wan)      vmx1             static         10.100.100.2/32             10.100.100.1   default 10.0.60.1


[imneedham]

While it does appear that I have the LAN/WAN interface order correct, I may have set it up in the opposite order at the beginning and then swapped them in a later wizard run. If that's the case, could there still be something mismatched that I need to fix?

[Patrick M. Hausen]

Sorry, no idea in a virtualised environment. In a dedicated hardware based installation people frequently mix up WAN and LAN because OPNsense and pfSense do it "the other way round", respectively. And some manufacturers put labels like "WAN" and "LAN" on the ports ...

I'd watch with tcpdump on OPNsense first, if e.g. a ping to a LAN device is going out the correct interface from the guests point of view. Then double check the assignment of virtual interfaces to port groups in ESXi. Are there VLANs involved? In the end the ESXi host is connected to a switch, probably? Can you do a packet trace on the switch looking for these outbound pings and if they go to the right VLAN?

Something like that - the idea is to trace and watch the packets one step at a time. I don't know any other method.

HTH,
Patrick

[imneedham]

This is resolved. TCPdump on OPNsense showed only outgoing ping/ssh attempts to the LAN client, and showed absolutely nothing when the LAN client attempted ping/ssh/http to OPNsense. I had reboot things several times, but the solution was to shutdown the old LAN forwarding host and then reboot the LAN client host. 

I don't understand what the problem was, but now all attempts from either direction are successful and tcpdump captures it as expected.

Thanks for your help.