OpnSense traffic is being blocked by my router/modem

[hazymat]

I recently migrated from Untangle (gateway / firewall) to OpnSense. I am using OpnSense in its normal "router" mode, as opposed to transparent bridging. As such, the Virgin Media router is in "dumb modem" mode, i.e. routing is turned off.

- First up, when I was running Untangle, I could reboot Untangle multiple times within an hour, or renew its WAN DHCP lease multiple times quickly, and it would always continue to route traffic, i.e. internet connection worked fine. I didn't need to touch the Virgin Media modem.

- When I migrated to OpnSense, it simply didn't work until I turned the Virgin Media modem off for a while (I didn't actually time it at first), then turned it back on. NOTE: a quick reboot did not resolve this. It would need to be off for something like 30 mins. Sounding like a DHCP reservation issue so far, right? Nope - read on. In this situation (i.e. not routing traffic) the modem would hand out an IP address to OpnSense WAN interface absolutely fine, but OpnSense itself (or anything on the LAN) would have no internet connectivity. For example I couldn't ping public IP addresses from OpnSense. After turning off the modem for quite a few minutes, and turning it back on, multiple times, it eventually worked, meaning all devices on the LAN (and OpnSense itself) could ping public IPs and internet connectivity was fine.

- So I thought - great, maybe it just didn't like the change in MAC address at first. But now it has the new MAC address registered in the modem, and the internet connection is finally working with OpnSense, that's it - everything will be fine from now on. But that wasn't the case.

- If I subsequently change some setting that causes OpnSense to renew its DHCP lease from the router, it does indeed get a renewed lease but then the modem no longer continues to allow traffic to be sent through it. In this state, I can ping the modem from OpnSense. This doesn't appear to resolve until I then reboot the modem/router again, ensuring I wait for a certain length of time. (I don't know if that's the time since last DHCP lease handed out, or the modem-off time!)

- However guess what? Every time I bring up Untangle, it works just fine. Even though it's a new MAC address again, and a new DHCP lease. I renew the lease within Untangle... works fine.

- Then I take Untangle down, bring up OpnSense, and it works until I renew its WAN lease then stops. Immediately I shut OpnSense down and bring up Untangle, and it magically starts working again.

- So at this point, it's not the modem thinking "I don't like your gateway's MAC address changes". What else could it be?

- I have ruled out this being caused by a NIC hardware issue, i.e. NIC driver on OpnSense causing problems where driver or hardware on Untangle Untangle being ok: I'm now actually running both Untangle AND OpnSense as VMs within Proxmox, and they are using the same physical NICs. (Of course, I'm making sure not to bring both nodes up at the same time...)

- So I went back to thinking it might be that the Virgin Media modem is a bit sensitive about MAC addresses, so I set the Proxmox MAC address on both Untangle and OpnSense to be exactly the same. That way, I can shut one down and start the other up, and the modem will think it's talking to the same physical device. (Again - being careful to wait until one has completely shut down before bringing the other up)

- I swapped the nodes around multiple times, and sure enough OpnSense is doing something the Virgin Media modem doesn't want to work with, whereas Untangle isn't.

There are no additional settings in Untangle that I have configured, such as MTU. In fact I reinstalled Untangle from scratch to be sure that I was fully in control of all the settings, and did the same testing, and the same happened.

My next step would be to snoop packets between the OpnSense WAN interface and the modem whilst doing a DHCP renewal and then pinging from OpnSense, and comparing that with a packet capture from Untangle doing the same thing.

This should surely at least tell me the difference between the information the Virgin modem sees coming from Untangle vs. OpnSense, so I can get more insight.

I have played with MTU settings in OpnSense, first by introducing an MTU of 1400 then 1000. Neither worked.

To be honest it's a bit difficult to set up packet capture between these two devices, how can I do that?

Ideally someone else here has seen a similar issue and might be able to point me in the right direction? Is there anything I haven't done / thought of? I just cannot work out any other ways this Virgin Media modem might be choosing to be picky!

But as you can see, it's something about OpnSense which I am sure can be resolved with packet configuration or similar.

[hazymat]

Ok so to update this after further testing:

- When I "hot-swap" gateways between Untangle and OpnSense, it *always* works when I go back to Untangle.

- Then when it swap to OpnSense, it always works (like Untangle has prepared the way) but only for a short while. Eventually it stops working (as per observations in previous post).

- I have determined that it stops working after a certain amount of time (as opposed to my triggering a DHCP renewal). I don't know what that time is.

- Strangely OpnSense is getting a different IP address despite having the same MAC. So I'm concluding so far that this fact - and possible the problem - is to do with client identifier.

Note I have set a client identifier in OpnSense.