Messages

Actually, it appears a reboot was required to clear out the installed default routes.  Its now working as desired/expected.

So we finally got the opportunity to implement this.  We've deleted all gateways except our primary upstream (BGP peers) and marked both of those as default.  Yet, the kernel default route to one of the gateways remains, and is the selected route (ignoring the BGP-provided default route).

How do we fix this going forward?

Hi all:

I have a failover pair of OPNsense routers running in my small ISP.  I use CARP+pfsync for all my user-facing networks.  My backup node is a bit weaker than my primary (working on hardware upgrades), and my demand has grown quickly (a good problem to have!) such that my backup node can't quite keep up.  So if my primary is functional, I do want it handling all the traffic.

Recently, my backup node has on its own changed its CARP priority to -240, which of course makes it master.  The primary router was at a priority of 0 at all times.

entering carp maintenance mode, then exiting on the secondary node did cause it to change to a priority of 0, which caused it to become the backup as expected.

What causes a CARP node to change its priority to -240 without human input?  How can I get logs/troubleshooting details on that?

Thanks!
--Jim

Hello:

I run a small ISP, and I'm using OPNsense in a failover pair as my core ISP router.  I have some public IP address ranges assigned, and I am announcing them via BGP to my upstreams.  I also receive via BGP default (and some additional) routes.

Currently, in my route table, the kernel route is inserted by the gateways.  This overrides the default route from BGP.  Unfortunately, this means if BGP session drops to a peer, the default route remains in place..but I do NOT want that.  I want my default route managed by BGP.

I tried deleting the gateways, but "bad things" happened.  Is there some way to disable the gateway system, or at least suppress insertion of the default route into the kernel routing table?

Hello all:

I'm relatively new to OPNsense (used pfSense years ago), and I own a small ISP.  We just switched our core routing function to an OPNsense pair (our Mikrotik failed, and I wasn't partuarly happy with it anyway).

One big question I still have is on kea dhcp6 HA support.  What I learned the hard way that most docs don't mention is in order for PD (prefix delegation) to work, a static route for the delegated prefix must be added to the router's routing table and pointing to the IPv6 WAN IP of the client who just received the PD prefix.  I assume OPNsense does this.

Where most implementations appear to fail is in an HA cluster, the HA backup is not able to take over for the primary unless it too has all these prefixes in its routing table (or there is automated provisions for adding them upon becoming active).  Does OPNsense w/ Kea DHCP6 support this?

The second issue is related: if I had to reboot the primary and I manually failed over to the backup (pressed the "CARP Persistent maintenance mode" button), and the backup took over for a bit.  I did my work on the primary which included a couple reboots.  Now I want to switch back.  Would the backup have a full copy of the routes too?  Is there some mechanism to rebuild the route table either from the peer or from the lease history?

I have learned that a dhcp6 backup without a current installed PD routes table is not very useful, and HA isn't really HA in that case...This was the case with pfSense (at that time, ISC DHCP6 was the only option).

Thanks!

Thank you for your replies.

First question, ix0: it is unused.  All interfaces are vlan-tagged on ix0, so I have ix0_vlan140, ix0_vlan52, etc.  ix0 (no vlan) has no traffic or direct use.

All traffic will arrive to the same firewall (FW), so yes, in this case that is fine. 

Early this AM, paid support for OPNsense pointed out the synproxy option was set, and clearing that fixed the dropped packets issue....

Thank you for your replies!

So, with BGP routing, the return path of the packet is always up to the network.  As the ISP, I have no control over which route the packet will come back, and in 50-60% of the cases, it will come back through a different provider than it was sent on.  This is normal, and every other ISP I've spoken to sees the same thing.  This is how BGP works...You announce yourself.  You choose which interface to send a packet out.  Each hop along the way chooses which of its routes to use to send it toward its destination.  The destination generates a return packet.  Each hop along the way chooses which route to use to deliver it to you.  This path is rarely the same it took to the destination.

I will likely have 3 paths eventually, and possibly an IX thrown in there for good measure.

However, for some of this testing, I did disable one of my providers so all traffic is going out a single WAN and coming back through the same (the internet only sees me connected through the one provider now).  Still, I see the same disconnect issue.

I also disabled the tag-which-path-to-return-on option in opnsense....forget where that is.  I believe I also set the state table (it actually defaulted, if I recall correctly) to not care about the interface for state matching; the IP/Port is good enough.

As to "default interface" -- if you're referring to the "LAN" interface as defined by opnsense, yes I am using that -- as my office / mgnt network.  All customer facing networks are on opt networks.  Also, one of my BGP peers / feeds is connected on WAN (the one that I am currently using for testing), my second interface is on opt1.  OPNsense doesn't really provide a way for not using the default LAN/WAN...

Years ago, I ran pfsense (I think I set that up before OPNsense was out or at least, before it was considered stable).  I did run with WAN and LAN, and things worked well, but I only had one WAN at the time (I did BGP-announce it).


Oh, I also realize I forgot to mention this is an HA Pair with CARP, pfSync, and xmlsync running.  Switching thus far has worked very well.

So, with the firm requirement that Internet traffic may return on a different interface than it left, how do I configure OPNsense correctly?

So, we found a recommendation to add  floating allow-all rule, and did that, but its only somewhat helping.

I also disabled one of our upstream ISPs so we're not multi-wan anymore, but that too did not make a difference.

We aren't seeing much of a pattern to go on...Just some sites don't work and others do.  Even though we've committed the cardinal sin and just "allowed all from everywhere"...I'm still very, very confused as to why we can't get to wordpress.com and other such sites.

Hello all:

I'm having a heck of a time getting my OPNsense install working correctly.  I am a small ISP with two BGP feeds, one on WAN and the other Opt1.  I have a couple other OPT networks for customer facing IP addresses (I have my own ARIN-assigned IP blocks).  I only have one network that is NATted (LAN -- my internal network); all others have public IPs routed via BGP. 

Right now, to try and troubleshoot this issue, I have Allow IPv4+IPv6 all rules on all my interfaces.  Yet, I still have significant chunks of the internet that don't work (currently facebook.com and wordpress.com are easy examples).  When I look at my log live view, I see the default deny (automatic) rule is catching and blocking them.  I  have been trying to figure out why for quite some time.

I'm not really sure for a clean way to post my config here, so here are some screenshots.

I (and my customers) would greatly appreciate some help figuring this out.

Thanks!