Messages

Hello all:

I'm relatively new to OPNsense (used pfSense years ago), and I own a small ISP.  We just switched our core routing function to an OPNsense pair (our Mikrotik failed, and I wasn't partuarly happy with it anyway).

One big question I still have is on kea dhcp6 HA support.  What I learned the hard way that most docs don't mention is in order for PD (prefix delegation) to work, a static route for the delegated prefix must be added to the router's routing table and pointing to the IPv6 WAN IP of the client who just received the PD prefix.  I assume OPNsense does this.

Where most implementations appear to fail is in an HA cluster, the HA backup is not able to take over for the primary unless it too has all these prefixes in its routing table (or there is automated provisions for adding them upon becoming active).  Does OPNsense w/ Kea DHCP6 support this?

The second issue is related: if I had to reboot the primary and I manually failed over to the backup (pressed the "CARP Persistent maintenance mode" button), and the backup took over for a bit.  I did my work on the primary which included a couple reboots.  Now I want to switch back.  Would the backup have a full copy of the routes too?  Is there some mechanism to rebuild the route table either from the peer or from the lease history?

I have learned that a dhcp6 backup without a current installed PD routes table is not very useful, and HA isn't really HA in that case...This was the case with pfSense (at that time, ISC DHCP6 was the only option).

Thanks!

Thank you for your replies.

First question, ix0: it is unused.  All interfaces are vlan-tagged on ix0, so I have ix0_vlan140, ix0_vlan52, etc.  ix0 (no vlan) has no traffic or direct use.

All traffic will arrive to the same firewall (FW), so yes, in this case that is fine. 

Early this AM, paid support for OPNsense pointed out the synproxy option was set, and clearing that fixed the dropped packets issue....

Thank you for your replies!

So, with BGP routing, the return path of the packet is always up to the network.  As the ISP, I have no control over which route the packet will come back, and in 50-60% of the cases, it will come back through a different provider than it was sent on.  This is normal, and every other ISP I've spoken to sees the same thing.  This is how BGP works...You announce yourself.  You choose which interface to send a packet out.  Each hop along the way chooses which of its routes to use to send it toward its destination.  The destination generates a return packet.  Each hop along the way chooses which route to use to deliver it to you.  This path is rarely the same it took to the destination.

I will likely have 3 paths eventually, and possibly an IX thrown in there for good measure.

However, for some of this testing, I did disable one of my providers so all traffic is going out a single WAN and coming back through the same (the internet only sees me connected through the one provider now).  Still, I see the same disconnect issue.

I also disabled the tag-which-path-to-return-on option in opnsense....forget where that is.  I believe I also set the state table (it actually defaulted, if I recall correctly) to not care about the interface for state matching; the IP/Port is good enough.

As to "default interface" -- if you're referring to the "LAN" interface as defined by opnsense, yes I am using that -- as my office / mgnt network.  All customer facing networks are on opt networks.  Also, one of my BGP peers / feeds is connected on WAN (the one that I am currently using for testing), my second interface is on opt1.  OPNsense doesn't really provide a way for not using the default LAN/WAN...

Years ago, I ran pfsense (I think I set that up before OPNsense was out or at least, before it was considered stable).  I did run with WAN and LAN, and things worked well, but I only had one WAN at the time (I did BGP-announce it).


Oh, I also realize I forgot to mention this is an HA Pair with CARP, pfSync, and xmlsync running.  Switching thus far has worked very well.

So, with the firm requirement that Internet traffic may return on a different interface than it left, how do I configure OPNsense correctly?

So, we found a recommendation to add  floating allow-all rule, and did that, but its only somewhat helping.

I also disabled one of our upstream ISPs so we're not multi-wan anymore, but that too did not make a difference.

We aren't seeing much of a pattern to go on...Just some sites don't work and others do.  Even though we've committed the cardinal sin and just "allowed all from everywhere"...I'm still very, very confused as to why we can't get to wordpress.com and other such sites.

Hello all:

I'm having a heck of a time getting my OPNsense install working correctly.  I am a small ISP with two BGP feeds, one on WAN and the other Opt1.  I have a couple other OPT networks for customer facing IP addresses (I have my own ARIN-assigned IP blocks).  I only have one network that is NATted (LAN -- my internal network); all others have public IPs routed via BGP. 

Right now, to try and troubleshoot this issue, I have Allow IPv4+IPv6 all rules on all my interfaces.  Yet, I still have significant chunks of the internet that don't work (currently facebook.com and wordpress.com are easy examples).  When I look at my log live view, I see the default deny (automatic) rule is catching and blocking them.  I  have been trying to figure out why for quite some time.

I'm not really sure for a clean way to post my config here, so here are some screenshots.

I (and my customers) would greatly appreciate some help figuring this out.

Thanks!