Hi,
I am trying to setup a firewall behind a mobile network router/gateway. I think the problem is in a specific mobile router, it has taken a long time to come to this conclusion. But I need help to figure out exactly what OPNsense is depending on before I can ask the manufacturer of the mobile device to fix it's problem.
I include two attachments, results for two different mobile routers. For this test I moved the SIM-card from one router to the other to rule out any problem with the subscription.
It is very repeatable, it always stop at the same place and then no indication of any problem at all. It just never completes. I always get "Fetching packagesite.pkg: .........", i.e. the last . and done on the row is missing.
Question: are the . showing the download process, i.e. do I get 90% of the download and then it stops for some reason?
If I do: wget https://opnsense.c0urier.net/FreeBSD:13:amd64/23.1/latest/packagesite.pkg on a linux machine behind the mobile router where OPNsense has a problem, it works without any problem on the linux machine.
So any ideas what I could check for and what to ask the manufacturer of the mobile device?
Thanks
It look like a pure IPv4 connection, so i would rule out the usual IPv6 connectivity issue. It might be an MTU issue, meta.conf is around 163 bytes and packagesite.pkg around 240kb.
I would try another mirror or try pulling the file from console via wget or curl and see whats happening.
https://opnsense.c0urier.net/FreeBSD%3A13%3Aamd64/23.1/latest/
wget seems to not exist in the opnsense installation, but curl worked fine. I got the file quickly without any issues.
Then trying the connectivity audit again from the GUI.
Same result as before, it does not get any further than to "Fetching packagesite.pkg: ........."
Changed to Mirror (default) and rebooted the machine.
Checking System->Firmware->Status show nothing, i.e. looks like it is waiting for something.
Changed to Mirror dns-root.de. This filled in the Status-tab, issued a new connectivity audit.
This time Fetching packagesite.pkg worked fine from the GUI, but no the rest.
Did a new reboot and the ouptut from connectivity audit looks like this:
***GOT REQUEST TO AUDIT CONNECTIVITY***
Currently running OPNsense 23.1.11 at Fri Jul 7 14:21:09 UTC 2023
Checking connectivity for host: mirror.dns-root.de -> 104.21.22.179
PING 104.21.22.179 (104.21.22.179): 1500 data bytes
1508 bytes from 104.21.22.179: icmp_seq=0 ttl=54 time=147.281 ms
1508 bytes from 104.21.22.179: icmp_seq=1 ttl=54 time=110.397 ms
1508 bytes from 104.21.22.179: icmp_seq=2 ttl=54 time=42.243 ms
1508 bytes from 104.21.22.179: icmp_seq=3 ttl=54 time=53.380 ms
--- 104.21.22.179 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 42.243/88.325/147.281/42.743 ms
Checking connectivity for repository (IPv4): https://mirror.dns-root.de/opnsense/FreeBSD:13:amd64/23.1
Updating OPNsense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: .......... done
Processing entries: .......... done
OPNsense repository update completed. 835 packages processed.
All repositories are up to date.
Checking connectivity for host: mirror.dns-root.de -> 2606:4700:3034::6815:16b3
ping: UDP connect: No route to host
Checking connectivity for repository (IPv6): https://mirror.dns-root.de/opnsense/FreeBSD:13:amd64/23.1
Updating OPNsense repository catalogue...
pkg: https://mirror.dns-root.de/opnsense/FreeBSD:13:amd64/23.1/latest/meta.txz: Non-recoverable resolver failure
repository OPNsense has no meta file, using default settings
pkg: https://mirror.dns-root.de/opnsense/FreeBSD:13:amd64/23.1/latest/packagesite.pkg: Non-recoverable resolver failure
pkg: https://mirror.dns-root.de/opnsense/FreeBSD:13:amd64/23.1/latest/packagesite.txz: Non-recoverable resolver failure
Unable to update repository OPNsense
Error updating repositories!
***DONE***
System->Settings->General->Prefer IPv4 over IPv6 is checked (but I assume from the output above that it doesn't help).
After making the test with mirror.dns-root.de and changing back to c0urier.net again, the Firmware->Status-tab is not working at all. This remains also after a reboot. To be able to see the Firmware->Status, I need to move the router first to another path to internet. I.e. not through this mobile router/gateway that opnsense seem to have an issue with.
Very annoying and strange.
Hi,
I have the exact same problem - is your mobile router a Teltonika router by any chance?
I've come the conclusion it's a weird combination of OPNsense, the router end perhaps Proxmox that is the cause of the issue. I have not found a solution.
Please update here, if you ever get to the bottom of this...
-Kent
Quote from: kbhsn4 on July 09, 2023, 08:37:16 PM
I've come the conclusion it's a weird combination of OPNsense, the router end perhaps Proxmox that is the cause of the issue. I have not found a solution.
I did a little more testing - and Proxmox doesn't seem to have any effect. It's simply that traffic originating from OPNsense itself doesn't work, or is extremely slow.
I had a thought that IPv6 might have something to do with it (since my Cable modem is IPv4 only, and the mobile router is IPv4/IPv6 dual-stack capable), but disabling IPv6 anywhere in OPNsense (and selecting 'prefer IPv4 over IPv6' doesn't help either.
I've attached my little test-chart - both my upstream routers were supplying IPs to the clients/firewalls with DHCP. So I merely moved the clients/firewalls around without changing any configs.
Your guess was correct, it is a TRB500.
Update works if I use bridge-mode instead of NAT in this device.
It also works with NAT-mode in TRB500 if I have another router in between TRB500 and the router I try to update OPNsense on.
I have a TRB500 as well.
Have you managed to get Teltonika to acknowledge the problem?
So far I've worked around the problem by putting the modem in bridge mode, and spinning up a virtual OPNsense to do the NAT part (I have a HA setup behind the virtual OPNsense, that needs the NAT to share the public IP) - But that introduces a single point of failure with the virtualized firewall; I can live with that for the time beeing, since the TRB500 is only my backup connection. But it would be preferrable if Teltonika could fix their NAT mode.
It's hard to say if they are working on it or not, the first response I got with the initial questions seems to have come from an employee. Other suggestions may have been from other helpful users in that forum. So far it's been 26 posts on the issue.
However since I have figured out I have another issue that is related to OPNsense I may end up in not using OPNsense at all which means I do not have a need for them to push for fixing the FreeBSD update / TRB in nat-mode combination issue.
The deal-breaker for me with OPNsense is that I can not get full speed with Ookla Speedtest when I route that traffic through OPNsense and TRB500. Speedtest set up 4 TCP-streams to different servers in parallell and it is limited to about 200 Mbps in download speed.
I suspect this is intentional throttle-back of the speed from my ISP. But I get close to full Gbps if I replace OPNsense with anything else. I.e. also Debian in the same hardware as OPNsense (I just switch between two SSD's). To get full speed in the Speedtest I need to use nat-mode in TRB500 and also in next router.
My idea was to get around the problem with FreeBSD update through TRB500 by routing this out to internet through a VPN-connection. I have not tried if this works or not, but it was my next step to test. That was until I realized that I can not get much more than 200 Mbps through OPNsense.