Messages

Thanks for this, I was wondering why I couldn't set my carp ip as source address, also had to create another one in the fe80:: and use it.
I don't see why using a carp VIP in the 2001:... shouldn't be possible but it isn't.

Hello,

I would very much like to move from my current VPN client [legacy] to a VPN instance of type client to benefit of the "Depend on CARP" for my failover routers to actually automagicaly start/stop the openVPN connection whenever they become master/backup

But currently I use the following advanced options for which I can't seem to find a replacement :

Code Select Expand


pull-filter ignore "ifconfig-ipv6 "
pull-filter ignore "route-ipv6 "
route 10.0.0.0 255.0.0.0 net_gateway
route 172.16.0.0 255.240.0.0 net_gateway
route 192.168.0.0 255.255.0.0 net_gateway


My limited understanding is that this VPN is setup on server side to push me a config where everything is routed through this VPN which messes things up on my router.
So my first 2 lines are there to not get ipv6 at all from this VPN because it really messes my ipv6 where for some unknown reason my ipv6 traffic from the FW itself selects the openVPN ipv6 as source ip but goes out through my WAN interface instead of the openVPN one. I would have prefered the FW to select my LAN ipv6 and send it out the WAN interface as I setup my ISP router to delegate me the /64 I have setup on my LAN...

Maybe I could replace those with some other routes I can't figure out in order to keep ipv6 connectivity (which I don't really need) on this vpn and still ensure the firewall uses my WAN to pass all ipv6 traffic except the one from the /48 subnet that is given to me by my VPN provider.

The other one I really believe the VPN provider is pushing something I don't know about that messes all routing and is needed to allow me to have internet over ipv4 using my ISP WAN also.

I have tried the options to "route-no-pull" and "route-no-exec" but I'm really lost...
Is there a way to use a "VPN instance" of type client and continue passing these options (even manually in a config file to verify it works first), and if not are you able to help me figure out other settings that could help me ?

To be honest, I was not able to find a difference between netstat -rn with my VPN up or down even when I left out the pull-filter aboves, so I feel openvpn is messing with routing in some way and I'm too dumb to see it and therefore fix it... What is the actual way to look at all routing tables in opnsense including whatever openVPN has added/modified ?

If anyone can be of assistance that would be great ! thanks in advance

Yeah, I thought I had read something like this... Any chance you have some infos on where exactly this bug is being tracked and if I can follow it's eventual resolution hopefully ?
Still in the proces of setting up my test setup to play around with offloading anf confirm I'm stuck at 2GB/s as my max throughput with opnSense, takes more than anticipates, need a third interface to administer opnsense with the gui, and a fourth to give WAN access else I can't apt isntall iperf3... Will post my findings soon but I expect I'll hit this bug yeah...

For now the HW only has 2.5G NIC but I'll upgrade to 10G when I get the money, might be a while seeing I 'll need managed or at least vlan capable switches and I refuse using power hungry old enterprise HW going forward 😉

So anything above 10G is not needed nor my target but still quite a bit of room between the current 2G and the target 10G.

Faster than 10G can still be useful within this machine as it my NAS with several SSDs and some VMs are using it over this virtual network. Faster "remote" storage is always a plus I figured 😉
Will make another attempt at passing individual vlans to opnsense and trying to play around with my post up rules find out if one setup allows me to enable CRC offloading in opnsense seeing it seemed to make a world of difference at least within the same vlan between an LXC on the Linux bridge and opnsense on the same bridge

Thanks for the reply @Patrick
I tried that too, adding a virtio nic for some of my vmbr0.2 and vmbr0.3 and reassigned those in opnsense. Same behavior though, CRC offloaded means no inter-vlan communication and CRC not offloaded meaning it works but 900MB/s only.

Yesterday night I tried applying the settings found here https://binaryimpulse.com/2022/11/opnsense-performance-tuning-for-multi-gigabit-internet/
It got me up to 2GB/s inter network but still quite far from the 12GB/s I get in the same Linux bridge...

My CPU is a AMD Ryzen 7 5700U, maybe I'm dreaming and routing from 1 Linux bridge vmbr0.2 to the next vmbr0.3 can't happen any faster than 2GB/s on my machine? I already had to give it all CPU cores available and quite a bit of ram...

Best I think would be to find a way to leave CRC on and still make the routing work across VLANS... But I don't know enough to understand which device is it actually offloaded to, seems a Linux bridge has nothing visible from ethtool in terms of offloading features so maybe my above -K on vmbr0 or any of it's VLan is doing nothing anyway...

Hello,


I have a virtualized opnsense router and can't seem to manage to get decent performance while routing packets between vlans.


On PvE I defined vmbr0

Code Select Expand

auto vmbr0

iface vmbr0 inet manual

        bridge-ports bond0

        bridge-stp on

        bridge-fd 0

        bridge-vlan-aware yes

        bridge-vids 1-4094

        pre-up ethtool -G bond0 rx 1024 tx 1024

        pre-up ethtool -K bond0 tx off gso off

        post-up ethtool -K vmbr0 tx off gso off

#Bridge All VLANs to SWITCH


Now I pass vmbr0 to my opnsenseVM as virtio, it extracts vtnet0_vlan2 and vtnet0_vlan3 properly, serves DHCP properly, and routes traffic between the vlans according to fw rules.


For testing I use an LXC attached to vmbr0 using vlan tag 3, and the PvE host itself attached to vmbr2 as follows

Code Select Expand

auto vmbr2

iface vmbr2 inet static

        address 10.2.2.2/24

        gateway 10.2.2.1

        bridge-ports vmbr0.2

        bridge-stp on

        bridge-fd 0

        post-up   ip rule add from 10.2.2.0/24 table 2Vlan prio 1

        post-up   ip route add default via 10.2.2.1 dev vmbr2 table 2Vlan

        post-up   ip route add 10.2.2.0/24 dev vmbr2 table 2Vlan

        pre-up ethtool -G vmbr0.2 rx 1024 tx 1024

        pre-up ethtool -K vmbr0.2 tx off gso off

        post-up ethtool -K vmbr2 tx off gso off

#VMs bridge


I have in opnsense the settings to disable everything: CRC offloading, TSO, LRO and VLAN offloading as well.


All CPU monitoring I can do show that during an iperf3 across vlans there is ample idle time on all CPU (80%) on all 3 nodes involved (it's a homelab nothing else is stressing anything here)


And yet I get 800-900MB/s when crossing vlans...

On the same vlan I get 18-19GB/s

I also managed to get 12GB/s from one VLan to the router but that was only by enabling the CRC offloading in the opnsense virtual router... But enabling CRC offloads breaks inter-vlan communication, the same opnsense VM, no rules changes, CRC offloaded = 12GB/s in one VLan but no Vlan 2 to 3 communication possible, or CRC not offloaded and only 850MB/s...


I'm getting stuck...

The HW NIC behind bonds is an Intel I225V-rev04, it's alone in the bond, later it will be bonded with a gigabit real Tek in case I plug the cable in the wrong NIC


If you have any ideas as to how I should set it up to achieve>10GB/s between VMs and LXCs regardless of the VLAN I put them on, anything would be helpful here I think.


Thanks for the reading and thanks in advance for any idea!

Nevermind, there is a dedicated UI page for query forwarding that works like a charm for my use...

Hello,
I have y opnsense with unbound serving dhcp leases for the "lan." Zone so when I plug my desktop it's then accessible at "desktop.lan"
Great, no I added a remote location (parents house) with it's dedicated unbound and dhcp that is fully working for "parents.lan" , so when I'm there I can easily dig and find the IP of dad.parents.lan and mom.parents.lan
I have setup IPSEC between the 3 and we are on separate subnets so now we can use each other IPs and we get connectivity transparently though the IPsec VPN.

Now when I'm home on my own opnsense  I can't resolve dad.parents.lan (it's empty) as it's a dhcp lease only on the unbound of my parents not mine.
I tried this in /sur/local/etc/unbound.opnsense.d/parents.conf

Code Select Expand


server:
forward-zone:
  name: "parents.lan"
  forward-addr: 192.168.1.90
I also tried with .parents.lan. parents.lan. .parents.lan and even lan. with no luck (starting with a dot makes unbound refuse to start...)
192.168.1.90 being the IP of the parents opnsense of course.

So I can't find a way for my unbound to forward queries to the parents unbound when the query is about *.parents.lan

Any help will be appreciated.

I do believe that the other way around will be impossible though, make the parents DNS try to answer with local data first then query my unbound for *.lan but that I can live with 😉
But having mine ask the parents DNS for *.parents.lan should work I just don't find how...

Thanks in advance

I know y issue is probably on my proxmox networking so maybe stuff more for linux fans than opnSense, but the network experts I know of live around ehre so I try here ;)

I'm facing some strange random networking issues with LXCs on my PVE cluster not able to communicate.


For instance, sometimes, 10.0.10.51 which is a LXC will not be able to communicate with 10.0.1.23 which is one of my switches.

When this occurs, I see no trafic at all coming in on the gateway (using opnSense, made a packet capture, nothing there), meaning the trafic is not leaving the LXC or not leaving the network bridge. I think I did try a packet capture on the pve host of the lxc and did not see any trafic on the vmbr10 either...

I see that often thanks to my uptime-kuma instance runing on this LXC, and can't really understand why, there is a timeout (60 secs) during which uptime isn't able to either ping or curl http the switch, and doing nothing it starts working again a few minutes later...


The LXC in question is a ubuntu jammy attached with a static ip to vmbr10, the pve host is running v8.1.3 on kernel 6.5.11-7-pve.


While this is occurring, I can reproduce using ssh onside the LXC and communication is indeed down, and during this time I was able to ssh onto my opnsense gateway and confirm it is indeed able to ping or clurl the switch no problem, so were my opnSense to recieve the packets from the LXC it would pass them along correctly...


Uptime is running inside docker inside the LXC and I do believe I have similar issues within docker networking itself (some containers timeout between my traefik instance and the gitea container itselft for example...) but that seems unrelated as within docker itself...


The host is a 8365U so powerfull enough, it's sitting arount 30%CPU usage, no swapping with the 32GB of RAM I added, it is quite busy running around 100 containers total, some in LXCs, some in VMs, but overall no slowness or anything besides these random network dropouts*


I recently tries to increase ulimit -n 99999 (it was 1024 everywhere) but it doesn't seem to do any better...


Any idea ?


Here is my /etc/network/interfaces :

Code Select Expand

auto lo

iface lo inet loopback


auto enp1s0

iface enp1s0 inet manual

        mtu 9000

#eth0


auto enp2s0

iface enp2s0 inet manual

        mtu 9000

#eth1


auto enp3s0

iface enp3s0 inet manual

        mtu 9000

#eth2


auto enp4s0

iface enp4s0 inet manual

        mtu 9000

#eth3


auto enp5s0

iface enp5s0 inet manual

        mtu 9000

#eth4


auto enp6s0

iface enp6s0 inet manual

        mtu 9000

#eth5


iface enx00e04c534458 inet manual


auto bond1

iface bond1 inet manual

        bond-slaves enp5s0 enp6s0

        bond-miimon 100

        bond-mode balance-xor

        bond-xmit-hash-policy layer3+4

        mtu 9000

#LAGG_WAN


auto bond0

iface bond0 inet manual

        bond-slaves enp1s0 enp2s0 enp3s0 enp4s0

        bond-miimon 100

        bond-mode balance-xor

        bond-xmit-hash-policy layer3+4

        mtu 9000

#LAGG_Switch



auto vmbr1000

iface vmbr1000 inet manual

        bridge-ports bond0

        bridge-stp on

        bridge-fd 0

        bridge-vlan-aware yes

        bridge-vids 1-4094

        mtu 9000

#Bridge All VLANs to SWITCH


auto vmbr2000

iface vmbr2000 inet manual

        bridge-ports bond1

        bridge-stp on

        bridge-fd 0

        bridge-vlan-aware yes

        bridge-vids 1-4094

        mtu 9000

#Bidge WAN


auto vmbr1000.10

iface vmbr1000.10 inet manual

        mtu 9000

#VMs


auto vmbr1000.99

iface vmbr1000.99 inet manual

        mtu 9000

#VMs


auto vmbr10

iface vmbr10 inet static

        address 10.0.10.9/24

        gateway 10.0.10.1

        bridge-ports vmbr1000.10

        bridge-stp off

        bridge-fd 0

        post-up   ip rule add from 10.0.10.0/24 table 10Server prio 1

        post-up   ip route add default via 10.0.10.1 dev vmbr10 table 10Server

        post-up   ip route add 10.0.10.0/24 dev vmbr10 table 10Server

        mtu 9000


auto vmbr99

iface vmbr99 inet static

        address 10.0.99.9/24

        gateway 10.0.99.1

        bridge-ports vmbr1000.99

        bridge-stp off

        bridge-fd 0

        post-up   ip rule add from 10.0.99.0/24 table 99Test prio 1

        post-up   ip route add default via 10.0.99.1 dev vmbr99 table 99Test

        post-up   ip route add 10.0.99.0/24 dev vmbr99 table 99Test

        mtu 9000




I do have the proper tables created I believe :

Code (bash) Select Expand

root@pve:~ # cat /etc/iproute2/rt_tables.d/200_10Server.conf

200 10Server

root@pve:~ # cat /etc/iproute2/rt_tables.d/204_99Test.conf

204 99Test

root@pve:~ #

Thanks in advance for any help or ideas on how to fix it ;)

[Solved]

Solved : turns out it was a bug in crowdSec v1.5.1 and someone kindly built me a 1.5.2 for BSD that solves it.

I'm stuck, I can manage to get crowdsec working with my private CA emitting certificates on a docker setup, but putting it inside the opnSense plugin fails.

Essentially, I am sure whatever context crowdsec runs it is not trusting my CA on opnsense, it says so in the logs :

Code Select Expand

time="05-07-2023 02:23:38" level=error msg="error while performing request: tls: failed to verify certificate: x509: certificate signed by unknown authority; 4 retries left"

But in fact, I have signed certs with this ca for other machines on the network, and when I use curl from the very same opnSense machine to use https on a server that has a cert signed by my internal CA, it does work properly and recognize the CA... I did have to import my CA in Trust->Authorities for that, sure but now this at least works.

But somehow, crowdsec seems not to use it..

Any idea how to add a CA cert to be trusted by crowdsec ? even looking at the plugin code I can't find what's missing on my machine to have crowdsec trust my CA...

Hello.
Seems I have a similar issue as in this thread :https://forum.opnsense.org/index.php?topic=22942.0

I'm running the latest opnsense all updates done, inside a VM running on latest proxmox.
Trying to start suricata (Intrusion Detection) I get this in the logs :

Code Select Expand

opening devname netmap:vtnet0/R failed: Invalid argument
And the service stops.

I am using a "virtio (paravirtualized)" as the model of networking card for this VM.
I do not wish to pass the real underlying physical NIC (an intel E1000) as I configured it nicely in proxmox (was painful enough once) and I expect to upgrade the machine and the NIC in the future...

Do you believe it's a NIC issue with this paravirtualized model ?
I could choose a "VMWare vmxnet3" model easily in proxmox, but doing this the name of the interfaces in opnsense will change and I'm having issues to re-map them, especially since I have many VLANs and such.

Maybe someone can give me a trick to take a backup, manually rename vtnet0 and vtnet1 to whatever it's new name will be when it's a vmxnet3, that I could do, but without networking, how to pass the file to the opnsense VM, and how to reload the backup without networking...

Thanks in advance for any kind of help.

Thanks everyone for the help !
I'm kinda tempted with .arpa but a bit lazy to change things up right now, especially since the trick by meyergru really did it for me ! Thanks bro, I'm using firefox anyway, so that works wonders for me !

Also I'm not quite sure how using "subdomains" like host.alwaysrepeaded.tld instead of host.tld would help, it's longer to type for sure. I already own a "real" domain but never found a nice way to have the DNS present different answers based on where the query comes from. Also it would be difficult to enable DNSsec I suppose with this as I'd actually be spoofing DNS when on local network... Maybe I could make my own DNS real owner of my own domain instead of the NS of my registrar but their registrat enables me to use letsencrypt... I'm still using unbound and for the firewall itself it can't even make it serve the proper answer, it's serving IPs for it's own name that aren't reachable over most networks... I highly doubt I'd me able to get a DNS challenge working for letsencrypt if I run my own nameserver, haven't found any "opnsense" or similar API in any known ACME client...

So I think I got my answer for now with the browser trick, and if anyone has a nice DNS in mind to run on my opnsense boxes to do views easily and somehow integrate with the dhcp of opnsense, that would be great ;)

Thx for the info !
Will stay away from .local then, though that seemed promising... any advice then what to use ?

Hello,

I first installed my opnSense a few years ago and I chose to have my LAN on a domain called ".lan", but now I hate myself as most of the time browsers don't know this tld and direct me to google or my default search engine when I type router.lan or server.lan in the address bar... unless I explicitely tell them https:// or http:// in front...

It's a pity as .lan is much faster to type as .local, but hey, no I've seen that most browsers know and deal properly with .local

Do you know of an easy way for me to switch to .local ? I'd really like something to keep resolving .lan by simply trating anything.lan as a CNAME of anything.local so my existing setups continue to work the time for me to update all my configs, like my /etc/fstab, my reverse proxies... if it's not CNAME and still myserver.lan gets resolved the same way as myserver.local I'd be happy ;)

Also to note : I have 2 opnSense doing CARP failover and syncing their conf...

if you know a better alternative to .local that would keep working with devices that are trying to use DNSsec or google's DNS like my android phones, feel free to share as well, I'd still like to keep it contained in my opnSense boxes.

Thanks in advance for any input !

Hello,

a long time ago I got states to be synchronized between my 2 firewalls both running latest opnsense and similar hardware.

They are both running on proxmox with a virtual NIC that is a linux bond on the host, WAN has no VLAN but I have several VLANs for LAN, the bond on proxmox trunks them all and I devined all the vlans on the virtual interface in opnsense.

All seems to work well, again I had it working a while ago and can't find what I did to break it...

I have CARP failover that works (althought all sessions gets killed since no states are synced)
I have even recently tried to add the pfsync0 to the carp group (no idea what it does...)
:

Code Select Expand

main$ifconfig pfsync0
pfsync0: flags=41<UP,RUNNING> metric 0 mtu 9000
        pfsync: syncdev: vtnet1_vlan9 syncpeer: 10.0.9.3 maxupd: 128 defer: off
        syncok: 1
        groups: pfsync carp


Code Select Expand

backup$ifconfig pfsync0
pfsync0: flags=41<UP,RUNNING> metric 0 mtu 9000
        pfsync: syncdev: vtnet1_vlan9 syncpeer: 10.0.9.2 maxupd: 128 defer: off
        syncok: 1
        groups: pfsync carp


I have all interfaces in the same order, the same underlying name...

on both I setup the Synchronize Peer IP properly and they both use the same syncdev as you can see.
But when I do a tcpdump on this interface, vtnet1_vlan9 interface, I only see CARP traffix (heartbeats or what they are called in CARP world...)
I see nothing on UDP or PFSYNC protocol that would share states.

In the GUI, I see something strange in interface overview, see the attachment below. I get the same on both frewalls, only errors (and different number of errors but hey...)


If someone has any clue as to a way to debug pfsync0, see what are these erros, maybe get some logs...

I wanted to crast a "fake" pfsync packet and send it out trough vtnet1_vlan9 just to make sure it's not being blocked by some unknown rule that isn't logging, but I don't see many packets being blocked, and I do have a rule on the proper interface to allow any to any on protocol PFSYNC...

Any help in investigating is realy welcome !