Messages

as an aside, i found out that if the USB dongle is removed and refitted it will show "Misconfigured Gateway". The remedy that works for me is to go into
Interfaces > [WAN_Failover] > change the IPv4 gateway rules > save > apply, then change it back to its original setting > save > apply.

it doesnt seem to matter what you change then change back, just as long as you change *something*

Thanks for your info on this! I am trying to achieve the same, but somehow I seem to miss something.

Do I understand correctly, that you put in different IP addresses in the interface static IP address and the gateway address? In your description it seems, you take the IP address of the dongle itself as interface ip, but the DMZ address as gateway? This always shows me "Misconfigured Gateway IP" under the gateway configuration.

Thanks for your help - much appreciated.

hi Alpha, sorry for the super late reply, it would seem i have overlooked the notification or missed it all together... i assume you have sorted this by now but here's my experience.

"Misconfigured Gateway" is usually when the "CIDR" dropdown is not set correctly for the interface

Interfaces > [WAN_Failover] > Static IPv4  configuration > address field, change the 32 to 24

see how you get on.

Issue solved... I can now at least ping IPs and DNS names, testing is still ongoing, if I run into any issues I shall return. Device shows itself as a USB Ethernet device (hence the `UE0` designation), rather than a Cellular Modem.

• Plug in dongle to any desktop and access WebUI. Details of how to do this should be with the dongle. Windows is super easy as it has `autorun` when dongle is inserted to USB. Linux is a case of decompressing a couple of archives, running an `install.sh` and changing your network connection in the taskbar.
  
• Once WebUI is accessed head to the 'Settings' tab on the top navbar. Here you can define 2G/3G/4G connection preferences if you so desire, but head to 'Device Settings' > 'DMZ' on the left hand navbar. 'Enable' DMZ and set desired IP (i used the default 192.168.6.169) hit Apply
• Navigate to top navbar 'Information' and Left hand navbar 'Device Information' note down the 'IP Address' field. Then plug it into a USB2 slot on your OPNsense instance (max speed is 150mbit, so USB2 is fine)
• Now Access OPNsense WebUI, go to  'Interfaces' > 'Assignments'. In the Assign new interface wait for `UE0` to appear. set a description such as 'WAN_Failover'... hit Add & Save
• Head to the [WAN-Failover] Interface now listed and set 'Enable' interface, 'Lock' to prevent removal. Set 'Static IPv4' and 'None' for IPv6. Fill IPv4 Configuration with the DMZ Address you set earlier (in step 2), set the 'Gateway Rules' field to your WAN_Failover Profile
• Go to 'System' > 'Gateways' > 'Configuration' Add a Gateway and fill in the details as you so wish. IP address needs to be set to the IP Address you found from 'Device Information' (in step 3)... deselect all tick boxes apart from 'Failback States', set a monitor IP (advise to go for a "quad" address such as 8.8.8.8 but note cant be the same one used for your main WAN)... finally set Priority. advise a larger number than your Main WAN. Hit Save & Apply
• At this point, for me, OPN auto-created an 'Interface WAN_FAILOVER Gateway' which isnt delete-able. Cool, whatever, doesnt seem to mess anything up. this entity will stay '(active)', which also seems normal regardless if failover is being used or not
• Edit your WAN GW, Enable 'Failover states' (This along with the 'Failback states' set earlier allows the connections to switch over cleanly)
• After some time you should now have populated RTT/RTTd/Loss fields and a green Status Icon. if so, its connected and the 4G Failover is ready for use theres just two more things to do
• Navigate to 'System' > 'Gateways' > 'Group' Create a neew Entry and set Tier 1 as your WAN and Tier 2 as your WAN Failover. Set 'Trigger Level' to 'Member Down' Leave Pool Options and set Your Description
• Finally, set FW rule(s) for 'Allow All' from LAN(s)

now if you unplug/disable WAN connection it should automagically failover to your 4G Connection. Once WAN is re-instated, it will prefer that connection over the 4G due to the Tiers and Priority settings defined earlier.

Picked this up again.

Wondering if anyone can shed light on it for me. What I have learned is that Vodafone, at least in the UK, seem to lock down their kit pretty well (would need firmware flash to even attempt to fix in the way I would like) I have found that the ZTE K5161z seems to do a bit of NAT at a firmware level. so I cant turn it off and I cant have a "per interface" NAT On/Off in OPN. im pretty sure this will lead to double-NAT situation.

ill give a little bit of insight as to where I am

• I can get OPNsense to recognise and add the dongle on interface `UE0`
• I have a `DMZ` set within the dongle itself, setting can be changed via WebUI on desktop machines.
• ` traceroute` doesnt show a GW hop for traffic pushed through `UE0` unlike my `pppoe` WAN interface which does. however I do not know if opnsense can be selective like that. the pings seem to be too low for a 4G/LTE network. which makes believe although I am requesting opn to use a certain gateway, its actually appears to be defaulting to WAN (which is my default)
• if disconnecting WAN then there is no take over from `UE0` no dns resolution, no pings etc etc. I set opn the way that is given in docs (https://docs.opnsense.org/manual/how-tos/multiwan.html)

from opn shell,

Code Select Expand

[color=#000000]root@OPNsense:~ # traceroute 1.1.1.1[/color]
[color=#000000]traceroute to 1.1.1.1 (1.1.1.1), 64 hops max, 40 byte packets[/color]
[color=#000000] 1  * * *[/color]
[color=#000000] 2  63.130.172.41 (63.130.172.41)  7.082 ms  13.036 ms  7.184 ms[/color]
[color=#000000] 3  90.255.251.33 (90.255.251.33)  16.145 ms  7.314 ms  7.403 ms[/color]
[color=#000000] 4  141.101.71.2 (141.101.71.2)  7.549 ms  7.392 ms[/color]
[color=#000000]    141.101.71.133 (141.101.71.133)  7.648 ms[/color]
[color=#000000] 5  one.one.one.one (1.1.1.1)  7.276 ms  6.925 ms  7.103 ms[/color]
if attempting to define UE0

Code Select Expand

[color=#000000]root@OPNsense:~ # traceroute -s 192.168.6.169 1.1.1.1[/color]
[color=#000000]traceroute to 1.1.1.1 (1.1.1.1) from 192.168.6.169, 64 hops max, 40 byte packets[/color]
[color=#000000] 1  * * *[/color]
[color=#000000] 2  63.130.172.41 (63.130.172.41)  6.997 ms  7.188 ms  7.216 ms[/color]
[color=#000000] 3  90.255.251.33 (90.255.251.33)  7.590 ms  8.553 ms  15.649 ms[/color]
[color=#000000] 4  141.101.71.135 (141.101.71.135)  8.091 ms[/color]
[color=#000000]    141.101.71.2 (141.101.71.2)  11.137 ms[/color]
[color=#000000]    141.101.71.107 (141.101.71.107)  7.648 ms[/color]
[color=#000000] 5  one.one.one.one (1.1.1.1)  6.947 ms  7.404 ms  7.158 ms[/color]

As you can see... Very similar pings
This is the info reported by F41 workstation
https://nc.d3dl3g.uk/s/ZSf76SzkjB94d8d

this is how the `DMZ`is set
https://nc.d3dl3g.uk/s/AANXkwckJai6fc4

This is how the "Device Info" looks
https://nc.d3dl3g.uk/s/TMeWdqrryrqpFqY

my OPN interfaces/GW is set as follows
https://nc.d3dl3g.uk/s/DJAi6oNmebmfJmP
https://nc.d3dl3g.uk/s/8c4znGk2oaj2M3x

GW Seems to be UP and has latency inline with what id expect from a cellular device
https://nc.d3dl3g.uk/s/g8gokR4f2mYH8Jd

Failover group is set
https://nc.d3dl3g.uk/s/LmiC5LZkM39oirE

and FW rule inplace
https://nc.d3dl3g.uk/s/atQWDKa9D6EwXRL

I have noticed are GW doesnt appear in GW list on dashboard, which I find a little strange.
https://nc.d3dl3g.uk/s/cyAfiZT7kENa8nZ

Am I missing something really simple?


As usual, any help gratefully received.

After the 57th (/s) time of turning it off turning it back on again... it works!!!???

no clue as to what solved it unless the ISP needs time to figure out the link has gone down and "resets" something.

Had been disconnected for a little over 30 mins.

Id started to play with openwrt as a replacement/test, maybe it was that that, could have put a digital rocket up its backside...

Now have +900 down/+900 up

VirtIO network interfaces in the OPNsense VM?

Try to set this tunable and reboot: hw.vtnet.csum_disable=1

Onboard = vmbr0 (vtnet1) "WAN"
PCIe card= vmbr1 (vtnet0) "LAN"
Both accessible via OPN, and set "correctly" in Interfaces > Assignments, yes the swapping of 0 and 1 is correct, it was an oversight on my part when setting up Prox and OPN. one day ill change it so they match. i need the mrs and kids out of the house to do that though

Tunable applied
   
Download speed    894.31 Mbps
Upload speed    13.03 Mbps

"updated" to most recent 6.5.13-5-pve

907mbit Down... 14mbit Up
:(

Code Select Expand

apt-get update
apt install -y pve-kernel-6.5
proxmox-boot-tool kernel pin 6.5.
reboot now


apologies... i am on Proxmox 8.1.3, so not the latest...
current kernel =

Code Select Expand

root@pve:~# uname -r
6.5.11-7-pve


Beware of the shotgun: http://catb.org/jargon/html/S/shotgun-debugging.html

Reduce your config down to the absolute minimum - bare metal, NAT only, with default rules. Then add features one by one until you see a drop in speed.

Bart...

not wanting to shotgun, hence my reason for being here ;)
as stated i reduced CTs and VMs, i may very well spool up bare metal, just to test your suggestion. dont really wanna do it on my live build.

Since this is not a bare metal configuration: Did you verify that Proxmox is not the culprit? There have been reported problems with some newer 6.8.4 kernels introduced by PVE 8.2.2 which have lead to the update to 6.8.8-1, but many support forum posts suggest installing 6.5 and pinning it....

do you have link for further reading? worth exploring if its right for my setup due to other pve services before copy pasting your given code :)

Thought I would add my experience seeing as I have a PPPoE connection (1000Mbit down/500Mbit up), also use OPNsense on Proxmox, kernel is v6.8.8-1 although I have an upgrade to 6.8.8-2 waiting on a reboot, plus my WAN NIC is passed as a device, whereas the LAN is a Proxmox bridge device. My speeds always vary but they're pretty much what I would expect, and did not change much, actually improved when I went from bare-metal to VM...

all my NICs (Onboard or PCIe) are passed via bridge. in my particular case WAN is plugged into Onboard, LAN is plugged into PCIe. i must admit all of my testing has been PCIe side. and not directly through Onboard NIC. *however* i migrated to my 2nd pve and i still see the slow upload.  which lends itself to "not a hardware fault"

im struggling to understand what "changed" at that particular time. it seems too far away from a pve update to make sense (to me) my pve updates at 04.00. so 9+ hours before a speed drop after an update "feels" wrong, id have expected it to show almost instantly.
i do agree that if it is kernel or update driven then it would apply to both my PVEs

[Timestamp DlSpeed UlSpeed Latency]

Good evening everyone.

Im a little bit stumped by this issue.
Since Monday 17th at 13.00 (GMT) i have had degraded upload speeds. (see below for rudimentary table, taken from log export in speedtest widget, timestamps are in "epoch" format https://www.epochconverter.com/ )

I have a 900mbit FTTP service, and use mimugmail's speed test widget (https://github.com/mimugmail/opn-repo) to monitor speeds. every hour, on the hour, from within OPN. this is not an asyncronous connection its "900 down/900 up" (FTR i can regularly speed test at >900 down/ >800 up before this issue arose)

My network HW  is ONT > HP Procurve switch > Dell Power Connect switch > Proxmox Server > OPNsense VM

Proxmox and OPNsense fully up-to-date with latest "stable" releases. neither are of the "enterprise" variety.
The issue has spanned OPNsense 24.1.8 into OPNsense 24.1.9 (release date for .9, i believe, was 18/06)

I have confirmed the following... Shut down all other VMs/CTs Within Proxmox, leaving ONLY OPN VM running. this resulted in no change to the poor upload speeds...  iperf3 at any step in above chain results in maximum 1g throughput. when running iperf from ONT side to Proxmox Server i saw a max of 1300 retries but the throughput was there.
i further tested speeds with my ISP router at every stage that there was an ethernet connection... this resulted in >900down, >800up
Given the combination of these 2 results i think i can safely say my line and HW are good.
i talked with ISP, they informed me that they do not apply throttling to the line, regardless of usage or attached router. i have no reason to not believe them at this stage as the testing appears to confirm that the line speed is obtainable, at least with their kit.

My suspicion is now firmly with OPN. No settings were changed in relation to WAN, i did set a port in NAT Port Forward towards my reverse proxy, but thats it, since testing i have disabled this PF and confirmed the attached floating rule has also been disabled.

im not really sure of what to check/adjust. any help greatly appreciated and warmly recieved.

Timestamp   DlSpeed   UlSpeed   Latency
1718600429   908.81   406   12.66
1718604028   843.66   361.76   15.37
1718607627   842.27   255.55   12.28
1718611225   899.97   266.33   9.04
1718614824   878.21   298.25   10.83
1718618436   737.92   277.4   12.54
1718622031   843.74   356.48   11.87
1718625635   837.8   404.11   14.78
1718629224   825   140.19   13.62
1718762425   909.74   22.65   12.45
1718766017   873.95   14.52   9.37
1718769621   841.62   15.98   10.1
1718773228   851.58   15.68   10.72
1718776830   904.69   25.16   14.81
1718780419   877.29   15.86   9.58
1718784027   798.76   7.31   16.35
1718787630   790.9   25.49   13.92
1718791227   487.07   21.18   12.8
1718794827   890.61   20.5   9.54

Speed test widget reports
3341 probes (and counting)
Avg Down:- 809.15 Mbps (min: 14.78 Mbps, max: 939.75 Mbps)
Avg Up:-  308.96 Mbps (min: 0.53 Mbps, max: 827.16 Mbps)
* "Avg Up" is skewed slightly due to multiple <30mbit results over the past couple of days, but only by around 50Mbps


################
Fixed just as quickly as it arose with "no intervention"
################

After the 57th (/s) time of turning it off turning it back on again... it works!!!???

no clue as to what solved it unless the ISP needs time to figure out the link has gone down and "resets" something.

Had been disconnected for a little over 30 mins.

Id started to play with openwrt as a replacement/test, maybe it was that that, could have put a digital rocket up its backside...

Now have +900 down/+900 up

Code Select Expand

ue0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: OPT2 (opt2)
        options=80000<LINKSTATE>
        ether 34:4b:50:00:00:00
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

is this pertinent?

USB Passthrough from Proxmox results in a ue0 interface (Passing Through the actual device rather than the USB Port, as then it wont matter if i switch USB ports in the future it *shouldnt* break)

from OPN CLI i can see usb device is recognised properly but unable to "cu" with it as per provided link

as for PtP stuff, i assume its PPP i need to setup, but i dont get the option to pick a device.

apologies if this all really n00b stuff that i am asking/providing, but for me it is something completely new.

further searching on my end resulted in finding this https://github.com/opnsense/umb-freebsd addon, not sure if it may be relevant or not.

i think i tried to do the cellular set up you linked to but because im in a vm and sharing the USB Dongle via a "network bridge" in proxmox, rather than a straight USB Port forward, i think i have numerous levels of issues going on.

i will first adjust Proxmox pass throughs, then work on OPN. sometime you cant see the forest for the trees, thanks for the 2nd set of eyes

First of all, USB modem support by BSD is not the best, maybe you should search for specific device to see if it will work under BSD.
Then I guess you will do good not to block private networks, as cellular mostly is CGNAT.
I never used USB LTE modems, but i guess you need to set up PtP connection in your sense (providing APN, etc.). Have you done this and missed to provide screenshots?

I have not checked BSD Compatability... I will do that in the first instance

I will unblock Private Networks regardless if device is compatible or not.

I have not set up PtP or anything more than i have provided screenshots for. I shall look into this.