Messages

I have an OPNSense 25.1.10 BGP router. The network is something like this:

BGP1: 1.2.3.4/29
BGP2: 2.3.4.5/29
IPBlock: 4.3.2.1/24
LAN: 192.168.10.1/24

The problem I am having is when the router goes to initiate an outbound connection, it chooses an IP from the BGP side, and not the 4.3.2.1 IP from the routed block. This is causing problems with an IPSec VPN and outbound NAT.

How can I have the router use 4.3.2.1 to initiate outbound connections for both itself and to NAT the internal LAN?

All routers involved except the upstream ISP side are OPNSense boxes running 25.1.10. I believe this forum is the correct community to be asking these questions.

Here's how the network is laid out:
BGP1 = 1.2.3.4/28
BGP2 = 2.3.4.5/28
IPBlock = 4.3.2.1/24

The issue I am having is when charon/strongswan goes to send packets to the remote OPNSense router it's picking the BGP Peer IPs instead of the routed IP block to initiate the connection. I can initiate the VPN from the remote side and it will establish and stay up for an hour or so. Eventually OPNSense goes to re-key (or whatever it does) and tries to use the BGP IPs to initiate the traffic instead of my IP Block.

Some other things of note is that 4.3.2.1 is a CARP IP with the static IP being 4.3.2.2.

I think what I need help with is making traffic initiated by OPNSense itself originate from 4.3.2.1 and not 1.2.3.4 or 2.3.4.5.

So I've narrowed down the problem.

On Site A with the BGP router, it's originating the IPSec from the BGP peer side instead of the routed IP block. I assume this is a misconfiguration somewhere on my side. Is there a way to get the router to originate the IPSec connection from the routed IP block instead of the BGP Peer IPs?

I've got a site to site IPSec VPN setup between two routers with static IPs. Site A is a BGP router with a single static IP. Site B has two cable modems with static IPs. The VPN is setup to use any of the static IPs on Site B to connect with. I have MOBIKE enabled which from what I can tell should help in multi-homed situations. I can manually connect the sites and things works great, but at some point the link will go down and not re-establish.

Time wise the link might stay up for 3 hours or it might stay up for 20 hours. I've yet to see it stay up for a day or longer. I am doing some hefty file transfers over the link but I wouldn't expect this to be a problem since the CPU load on both routers is just fine.

Any guidance on where I should look to try and see why the connection keeps breaking? Or is there a way I can tell it to keep trying to re-establish the link if it goes down?

I am looking at having an OPNSense box setup as a sort of SBC using OpenSIPS. I don't want to run a VM as SIP's audio component is realtime and needs 20ms to always be 20ms in order to operate at scale. This also seems like a decent fit since OpenSIPS is a proxy and would blend in with the other proxy components already built into OPNSense. OpenSIPS is a fairly fat application that would need a database service like MariaDB/MySQL on the backend. I'd probably also try to include an RTP proxy like RTPEngine for anchoring and NAT traversal.

So this leads to my first problem, getting MariaDB or MySQL installed for the backend and CDRs. I know the database client libs are installable but the server side is not. Is there a method to get MySQL/MariaDB installed on OPNSense in a way that future updates won't nuke it or disable the service? I looked and there's no plugins for either database in the baked in repos which I kind of expected.

The second issue would be getting opensips compiled and installed. It looks like it doesn't exist in the FreeBSD ports but the OpenSIPS project has a pkg for it at https://github.com/OpenSIPS/opensips/tree/master/packaging/freebsd. I come from the land of Linux and have very little clue what I need to do to get this installable for FreeBSD.

After that, it should be a 'simple' matter of making the plugin </s>. But before I can bother with that I need the underlying software in place first.

Any feedback or pointing me in the right direction would be much appreciated. Thank you!

Interesting approach, thanks for the detailed description. They only give you a /64 PD? That's nasty.

Ohhh, it's even stupider than that. Spectrum's router is already doing an IPv6 PD request but spectrum is only giving it a single /64 even though it's their own router on a business account. So that's why the router is unable to hand off any prefixes behind it. If they had just handed all their OWN routers a /60 they could delegate IPv6 all day long without a problem. Like I said, maybe I just haven't found the right support guy, but I've given up trying. This is hacky but works well enough for me.

What I'm wondering though: What prevents you from getting rid of the SGR? What wizardry does it perform that OPNsense can't emulate? Do they use some kind of VPN for the static IPv4 and don't give you the credentials? Or what's going on there?

Well, the short version is that the Spectrum uses RIP/RIPNG internally to manage routes between all the modems and routers on the network. This is secured by a key which I don't have. Once you have a static IP their router is required because that's how they route the IPs to you. Their router requests a dynamic IP through the modem and then announces the static IP on their network. The only way they will let me use just the modem is if I give up my static IP and use the base 500mb plan. Soon as I want 1GB or a static IP they tell me I have to use their router behind the modem. I'm guessing this router also phones home which might be part of why the edge keeps disconnecting me.

You can optionally set the DNS Server to be the link-local address of the LAN interface if you want to use the local Unbound DNS service. By default the DNS servers obtained from Spectrum when getting the IPv6 prefix will be used.

By default, the LAN interface's GUA will be advertised as a DNS server. The DNS servers obtained from the ISP will only be advertised if Unbound is disabled.

That wasn't my experience. If both DNS fields were left blank the IPv6 DNS servers for Spectrum is what my LAN clients got. Perhaps I have something set improperly elsewhere causing that.

I live in central Florida and have Spectrum aka Charter as the local cable provider. For all business accounts they mandate a 2-box solution here, one modem and one gloriously garbage router. We also have a static IPv4 which requires their router to provide the static IP.

The problem is that the Spectrum router does not provide an IPv6 prefix delegation to anything downstream like OPNSense. No amount of bridge mode or anything else seems to make a difference. Whether this is a hardware or back-end config issue I don't know. Finding anyone at Spectrum support who even knows what IPv6 is let alone a prefix delegation is near impossible. So, I got creative and a little petty.

What I found is that if you have an intermediary switch or VLAN that goes between the Spectrum Modem and the Router that you can pull an IPv6 prefix delegation directly from Spectrum's modem without affecting the static IP. The problem is that Spectrum will kill routing to this prefix after a while. That's where a somewhat unexplained cron command to periodically kick the interface will keep the routing alive. It's a little hacky but works surprisingly well. This guide will assume you have some basic understanding of OPNSense and how you manage the interfaces, setup VLANS or switches, and an understanding of basic networking. Chances are if you care enough about IPv6 that you want it to work then you likely already have the skillset.

Here's how the Spectrum two-box network topology is re-connected:
Modem <---> Intermediary VLAN/Switch <---> Spectrum Garbage Router <---> WAN VLAN/Switch/OPNSense

It's important that the intermediary switch/VLAN be isolated since we need DHCP to work. While you can probably plug all this into one switch and have it work it would be less then ideal. IPv6 tends to be too smart for it's own good and IPv4 tends to be too dumb. The simplest way to do this would be to have three 1gig or faster NIC ports in your OPNSense box and a small cheap 5-8 port 1gig or faster dumb switch. Connect the Spectrum modem and router's WAN port to the dumb switch. Now connect one port from the OPNSense machine to the switch (SpectrumDHCP/opt1), connect a second port to the router's LAN port (SpectrumStatic/WAN), and connect a third port to your LAN Switch. No VLANs needed at all.

For the WAN interface I named it "SpectrumStatic" and configured the static IPv4 on it. This interface will connect to the Spectrum Router directly or the VLAN/Switch you setup. If you have a static IPv6 you can configure it here otherwise set IPv6 to 'None'.

On the OPT1 interface I named it "SpectrumDHCP" and set IPv4 to 'None' and IPv6 to 'DHCPv6. This interface should connect to the Intermediary Switch or VLAN. Towards the bottom of the configuration screen set the Prefix Delegation size to 64, Check "Send IPv6 Prefix Hint", then save. If you have a static IPv6 you can enable "Request only an IPv6 Prefix" to keep things simpler for OPNSense. What we are really after here is that prefix delegation.

On the LAN interface set IPv6 to 'Track Interface'. Towards the bottom of the configuration screen set the IPv6 interface to 'SpectrumDHCP', IPv6 Prefix ID to 0, and check Allow Manual adjustments of Router Advertisements, then save.

Lastly, go to Service --> Router Advertisements, and click on the name of your LAN interface. Set Router Advertisements to "Unmanaged" and check Advertise Default Gateway. You can optionally set the DNS Server to be the link-local address of the LAN interface if you want to use the local Unbound DNS service. By default the DNS servers obtained from Spectrum when getting the IPv6 prefix will be used.

You might need to reboot the router for everything to straighten itself out. I've noticed that OPNSense/FreeBSD gets a little finicky when you turn IPv6/IPv4 on and off or start moving interfaces around.

You can verify everything is working by going to Interface --> Overview, and looking at the different IP's assigned to each interface. Under SpectrumDHCP you should see "IPv6 Prefix" listed. If you do, you have your prefix delegation and all your LAN clients should have working IPv6.

Now for the catch. For some reason every few hours Spectrum or OPNSense or some combination thereof will decide to just break. Things will still look normal but the LAN can no longer reach the internet on IPv6. What I found is if I go into the Interfaces Overview, expand the SpectrumDHCP interface, and click 'reload', things will just start working. So it'd work for a while but at some point, usually in the next 24 hours, it will break. It has never worked for more then a day on its own.

The workaround I came up with is to used the "Periodic Interface Reset" command from the OPNSense cron jobs. You can set this up by going to System --> Settings --> Cron. Click on the "+" button to add a cron job to OPNSense. Change the Hours field to "*/4", the command to "Periodic interface reset", the parameters to 'opt1' since that's where our SpectrumDHCP interface is at, and give it a description to help you remember why it's there like "Reset Spectrum for IPv6 weirdness", then save. This will kick the SpectrumDHCP interface every 4 hours which keeps things chugging along just fine for me. If you find you can go longer or shoter, just change the 4 in "*/4" to whatever hourly interval you want. If you want it to run every hour, just change it to "*". Every two hours would be "*/2". Every 12 hours "*/12", etc.

So that's how you get a Static IPv4 with DHCP IPv6 and prefix delegation working on a Spectrum two-box solution with OPNSense. Fun fun. Now for some random notes and things:

Both WAN ports on the modem and Spectrum router are 2.5G capable. So while using a 1-gig switch is technically a bottleneck between the router and modem it won't matter unless you have more then 1-gig of internet. The good news is inexpensive 2.5G 5-port switches exist so it's not much of an issue to remedy. There's also no need to over-think dumb switches. They're dumb and all use the same 2 or 3 chipsets. Just don't get the most expensive or cheapest one. Anything middle of the road will do the job just fine.

Smart or Layer 3 managed switches with VLANs are nice but they're pricey and really not needed for the average small-office home-office setup.

When it comes to things like gateway groups and rules you will need to manage both the SpectrumStatic and SpectrumDHCP zones in the firewall. The one downside to this solution is potentially twice the management. I try to use floating rules when at all possible to simplify this.

My final note on all of this is to avoid the temptation to pull more then one IPv4/IPv6 from Spectrum. Multiple IP's from a single ISP's modem to a single OPNSense router do nothing for resiliency or redundancy. All it really does is over-complicate router management with no benefit. It's a "just because you can doesn't mean you should" sort of thing. My opinion is that it's really only worth managing multiple WAN connections when you have two different discrete ISP connections coming into the building. For example having Spectrum cable internet and FiOS fiber internet would be where I want multiple WANs in OPNSense. When everything is just going through a single modem only one IPv4 and one IPv6 is all you need for most setups. However, you can still pull another DHCP IPv4 from the spectrum modem and also get a /64 of IPv6 without prefix delegation from the Spectrum router. Enough to do some isolated homelab stuff or whatever floats your boat.

I've got IPv6 setup on 23.1.9 and the WAN interface is not responding to IPv6 neighbor solicitations for traffic it's routing from the LAN. Below is the output of tcpdump on the WAN interface where a host from the LAN is trying to ping google at 2607:f8b0:4008:801::200e:.

Code Select Expand


01:54:05.130607 IP6 fe80::225:90ff:fe47:7a27 > fe80::2a9e:fcff:fe60:61a: ICMP6, echo request, seq 188, length 8
01:54:05.131275 IP6 fe80::2a9e:fcff:fe60:61a > fe80::225:90ff:fe47:7a27: ICMP6, echo reply, seq 188, length 8
01:54:05.295478 IP6 WWWW:XXXX:YYYY:ZZZZ:3eec:efff:fe12:4b7 > 2607:f8b0:4008:801::200e: ICMP6, echo request, seq 2972, length 64
01:54:05.312970 IP6 fe80::2a9e:fcff:fe60:61a > ff02::1:ff12:4b7: ICMP6, neighbor solicitation, who has WWWW:XXXX:YYYY:ZZZZ:3eec:efff:fe12:4b7, length 32
01:54:06.139438 IP6 fe80::225:90ff:fe47:7a27 > fe80::2a9e:fcff:fe60:61a: ICMP6, echo request, seq 189, length 8
01:54:06.140158 IP6 fe80::2a9e:fcff:fe60:61a > fe80::225:90ff:fe47:7a27: ICMP6, echo reply, seq 189, length 8


You can also see that the link-local from opnsense can communicate just fine with the link-local default gateway.

Any ideas what I am doing wrong or where to look? I looked at some of the other IPv6 threads and tried disable Block Bogons but it didn't make a difference. I wasn't seeing that firewall rule fire anyways but desperation is running rampant.

Here's my IPv6 configuration.
WAN:
- DHCPv6
- Prefix Size 64 (My ISP will only give me a single /64)
- Send Prefix Hint
- Request Only a Prefix didn't make a difference, and OPNsense still ended up with an internet IP even after a reboot?

LAN:
- Track Interface
- Set to WAN
- Prefix ID 0
- Allow adjustment of router advertisements

Router Advertisement:
- Set to Unmanaged
- advertise default gateway
- mostly every else is default

For some reason I'm onling getting a link-local IPv6 default gateway:

::/0                           fe80::225:90ff:fe47:7a27   UGDAe 1024 13


I would have expected there to be a real IPv6 IP with the UG flags there too.

So I moved the LAN interface off of the ix0 NIC and onto an unused em0 NIC. Once I did this and rebooted the LAN nic (now em0) is getting a link-local IP and it's different then the link-local IP of the WAN.

So my network now looks like this:
Spectrum Modem <---> Spectrum Router Box <---> OpnSense ix0_vlan14 WAN <---> OPNsense em0 LAN

So on my box (23.1.9), if the LAN and WAN were on the same interface with the same MAC even though they were in different VLANs the link-local would not get created for the LAN.

Now everything is able to pull an IPv6 from the delegation, but I'm still not able to ping out for some reason. Hmmm.

I don't have any bridge's on this box.

My network is essentially a flat small office network like this:

Spectrum Modem <---> Spectrum Router Box <---> OpnSense ix0_vlan14 WAN <---> OPNsense ix0 LAN

I have a single 10-gig connection to my switch and I'm shipping 3 VLANs over it. The only two VLANs using IPv6 is the WAN and LAN. The third VLAN is an IPv4 private admin network with no routing going through it.

I did notice that if I changed the LAN IPv6 to 'SLACC' for a second, apply, and then change it back to Track Interface that I will get a link-local address on the LAN interface. The link-local on the LAN and WAN interface are the same. Not sure if that's normal or not.

So I figured out the Prefix Delegation issue. On Spectrum where you have a separate modem and router from them you have to log into their router and enable dhcp-pd support. It's not enabled by default cause reason?! Anyways, once you log into Spectrum's router, you have to click on 'Advanced', then go to 'Router Settings', and lastly click on the 'DHCPv6' tab. Once you are there you'll see all the IPv6 settings and can enable DHCP-PD.

Now when I look under 'Interfaces --> Overview --> WAN' I can see that there's an IPv6 delegated prefix listed below the IPv6 address and it matches the /64 that the router allows for delegation.

Now the problem I have is there's no link-local address listed under the LAN when I go to the Interfaces Overview. I have the IPv6 configuration type set to Track Interface and the WAN interface selected. I would expect there to be a link-local listed.

So I have Spectrum Business cable internet and their setup consists of two pieces, one modem and then their own separate router/gateway. This is apparently 'required' since it's a business account and we have a static IPv4. All lies but hey, gotta run what ya brung.

OPNSense successfully pulls an IPv6 through DHCP and can ping the internet just fine. The issue is when it goes to get a prefix delegation (like a /60 or /56) the response is that no prefixes are available. I'm now stuck using the /64 that was assigned to OPNSense for everything on the LAN now. The problem is I can't seem to figure out how to get that to work.

I have no other VLANs I want to give IPv6 to, just the LAN. My first idea was to just manually configure the DHCPv6 daemon on LAN but that would mean whenever OPNSense gets a different IPv6 assignment the DHCPv6 daemon would need to be updated.

Is there a better way to implement this or am I just stuck hoping that the IPv6 assignment doesn't change much?

I spent a few days hating life trying to figure out why my defined BGP Peer with an MD5 password would not connect but a peer without a password would connect just fine. The short version is even though you can type in the MD5 password into the Neighbor definition it does not actually setup the TCP-MD5 connection within the kernel. This is crucial because the TCP-MD5 connection happens at layer 3 which is below where BGP operates at. The MD5 Password field combined with the Local Initiator IP in the neighbor definition just tells the FRR back-end what kind of interface/MD5 pair to hook to. You'll get log errors about zebra_client_message() failing and VNC encountered an error and exited. Basically nothing that says "Hey, I can't find this interface/MD5 pair". Joy...

The solution is to manually load the MD5 key into the kernel and have a script that runs on reboot to reload the keys.

BTW, getting to this point involved a lot of frustration and swearing. This information REALLY should be noted in the OPNsense documentation somewhere for the FRR/BGP neighbors using a MD5 password. I'm posting it here for the next guy who doesn't feel like digging through a dozen or so bug reports going back to 2017 while pouring over Wireshark/PCAPs and using a Cisco sandbox router to get relevant error logs to figure it out. For comparison Cisco says "MD5_DIGEST_MISSING:Dropping packets" when this happens. Useful logging is useful. However, lets get on with the fix.

First lets define the basic network between OPNSense/FRR and a password-protected BGP neighbor:

• BGP1: 172.16.30.1/28
• BGP1 Password: password54321
• OPNSense IP: 172.16.30.2/28

When defining the BGP Neighbor with an MD5 password in the Web GUI here's what you would type in:

• Create or Edit your BGP Neighbor and fill out all the other stuff you need
• Enable 'Advanced Mode' in the upper left corner
• Type the password into the BGP MD5 Password field, I.E. 'password54321'
• Enter the IP address facing the neighbor under the Local Initiator IP field, I.E. '172.16.30.2'
• Save your changes and apply settings/reload config

I assume if you're reading this that most of you have gotten to this point. You'll notice the neighbor status is stuck at "Active" and you'll get log errors about zebra_client_message() failed along with other weird errors. Nothing very descriptive of what's going on at all.

Now what you need to do is load the keys into the kernel itself and setup the TCP-MD5 connection. Everything below here requires SSH/shell access to the OPNSense box and basic shell knowledge. You can look up how to enable SSH and get to a shell through the normal OPNSense docs website. The first thing we're going to do is create a /root/setkey-bgpmd5.conf config file for 'setkey' to load:

Code Select Expand

#!/sbin/setkey -f
# Flush out old keys to be clean if you want, might break IPSEC things
#flush;
#
### BGP1 Neighbor
# Setup TCP connection with MD5 digest key
add 172.16.30.1 172.16.30.2 tcp 0x1000 -A tcp-md5 "password54321" ;
add 172.16.30.2 172.16.30.1 tcp 0x1000 -A tcp-md5 "password54321" ;


Once you have created the above file you can run it manually by typing 'setkey -f /root/setkey-bgpmd5.conf'. You'll now be able to connect to the BGP neighbor with an MD5 password no problem. It's beautiful.

The last thing to do is make sure that the keys are reloaded upon reboot. To do that I utilized the @reboot functionality of cron. I added the following to the end of the /etc/crontab file:

Code Select Expand

#
# Load the TCP-MD5 keys needed for BGP peers on reboot
@reboot root (/sbin/setkey -f /root/setkey-bgpmd5.conf) > /dev/null


You can now reboot and verify that you are still able to reconnect to your BGP neighbor. Hopefully this post helps the next guy save a few days of their time. If nothing else, it's something for me to find next time I do this and forget how. You will also need to check that the above crontab entry still exists after updates. Sometimes the updates will clear out the old /etc/crontab.

Let me know if there's a different/better approach to this. I'm all ears.