Messages

I am unable to set the correct DNS server in the subnet configuration. I input the correct IP address for my DNS server, which is not on OPNsense, save & apply, but when I go back into the subnet configuration screen, the IP address has been overwritten with that of the OPNsense LAN interface. I have not tested any further to see what address would actually be provided to the client, though. Bug?

It's probably not a very common use case, but I need the ability to make policy routing decisions to destination networks owned by specific organizations. On pfS I was using pfBNG to resolve BGP Autonomous System numbers to network blocks, as this isn't a core feature for pFs either.

There are some old threads here and here discussing it, and the base code contributed by NilsS that I built off of can be found there.

What I've done since then is made it work with Python 3, but also made it a bit easier to use and added a method to call it via http. This means you can self-host the script, and feed/update OPNsense aliases automatically.

Personally, I run it as a Docker container, but there's no requirement to use Docker. All you need is python with Flask ("pip install Flask", if you don't have it already). There are some public services that can do this as well, but I prefer to run my own just in case I accidentally run into API limits or the service unexpectedly disappears.

I readily admit that I can barely code my way out of a paper bag. I feel like what I did with Flask is clunky as hell, but it works. PRs welcome.

https://github.com/ddimick/asn-to-ip
https://hub.docker.com/r/ddimick/asn-to-ip

[would]

As noted, the agent is not needed to cleanly shut down a FreeBSD/OPN guest. And although having the IPs in the PVE console would be nice to have, it's doesn't really add much value.

But what would be a compelling use case is the ability to quiesce a FreeBSD/OPN guest for clean snapshots.

While there is a partially functional port, fsfreeze doesn't yet work. So I guess I'm bumping this just to add visibility that there is work in progress on an agent.

That sounds like it could be an issue with XMLRPC. Are you using the CARP VIP IP address for anything in the System/High Availability setup?

I use a floating rule to allow DNS queries on multiple inside interfaces. I pick the interfaces I want, set direction to in, and then just the the built-in "This Firewall" alias as the destination.

http://prntscr.com/j2yjoa

I don't think the guide mentions how to get ZeroTier to assign a default gateway to clients via DHCP.

You need to add a managed route to 0.0.0.0/0 pointed at your OPNsense interface's ZeroTier IP address (see http://prntscr.com/j1fee3 for an example).

DNS was also a bit tricky to control. I wound up adding a port forward NAT rule on the OPNsense ZeroTier interface to capture all traffic destined to port 53 and redirect it to Unbound.

Edit: Also of course you need firewall rules on your OPNsense ZeroTier interface to permit whatever traffic you're trying to pass. And you need to ensure an outbound NAT rule exists for your ZeroTier network if you want Internet access through it.

I'm successfully using *.domain.com as my CN (along with DNS-01 validation).

I've narrowed this down to having Netflow enabled and capturing local on both master/backup. As soon as you enable it on the backup, the HAproxy health checks on the master immediately fail. Disable Netflow on the backup, and they immediately start working again.

Steps to reproduce:

1. Run two instances of OPNsense.
2. Configure high availability (I'm using CARP and XMLRPC sync, but I am not using states sync).
3. Configure HAproxy on the primary. Observe that health checks show your backends/frontends as UP.
4. Enable Netflow on master.
5. Observe that HAproxy health checks still work.
6. Enable Netflow on backup.
7. Observe that HAproxy health checks now show your backends/frontends as DOWN.

I'm issuing wildcard certs with plugin v1.13 without any problems.

Are you running OPNsense under VMware? If so, you may find this post helpful.

I still have no solution for this. I've rebuilt my backup opnsense from scratch and it still behaves exactly the same.

Is anyone running both carp and haproxy successfully?

I'm running two OPNsense 18.1.3 systems with LAN/WAN/DMZ interfaces and CARP VIP.

LAN CARP VIP 192.168.1.1/17
LAN OPNsense-1 192.168.1.2/17
LAN OPNsense-2 192.168.1.3/17

DMZ CARP VIP 192.168.254.1/24
DMZ OPNsense-1 192.168.254.2/24
DMZ OPNsense-2 192.168.254.3/24

I'm running haproxy only on my master OPNsense-1 system. When OPNsense-1 is running by itself, everything works great.

When I boot OPNsense-2, everything still works great except all my haproxy HTTP health checks running on OPNsense-1 fail.

If I shut down OPNsense-2, the haproxy HTTP health checks on OPNsense-1 immediately start working again.

All traffic through either OPNsense system works fine in both scenarios. The only thing that stops working are the haproxy health checks.

What could be causing this behavior?

I assume that you've configured both OPNsense instances on ESXi? I'm thinking of trying this myself, apart from your current (fixed) problem is there any gotchas with this config?

Yes, both instances are running on the same ESXi 6.5.0U1 host. I have virtual redundancy, but not physical. I give each OPNsense instace 1 vCPU and 1GB RAM, which is more than enough to push 100mb over OpenVPN. If you want to use Intrusion Detection, add more RAM and maybe another vCPU.

The CARP oddity also caused some behavioral issues with DHCP failover, but fixing one also fixed the other.

There is a cosmetic issue where ESXi will complain that there is a mismatch between the guest OS (Other/FreeBSD 64-bit) configured on the VM, and the actual installed OS (FreeBSD 11.1-RELEASE-p6), but that's the case for any FreeBSD 11 guest system running open-vm-tools and, as far as I can tell, doesn't actually cause any problems.

I haven't seen any other issues yet, but I'm relatively new to OPNsense.

I have fixed this issue. I set Net.ReversePathFwdCheckPromisc to 1 in ESXi and the issue is now resolved. I don't understand why I didn't see this problem with pfSense running on the same ESXi host, vSwitch, and port group, but hopefully this will help others in the future.

I'm replacing a working pfSense HA setup, so I'm reasonably sure the network is set up properly. I have two OPNsense 18.1.1 VMs running on ESXi 6.5. ESXi is configured to permit promiscuous mode, MAC address changes, and forged transmits. There are no other systems running CARP on my network.

Every three seconds the system log reports:

Code Select Expand

Feb 7 12:08:04 opnsense: /usr/local/etc/rc.carpbackup: Carp cluster member "192.168.0.5 - (1@em1)" has resumed the state "BACKUP" for vhid 1
Feb 7 12:08:04 configd.py: [8b269f1d-df1c-4809-871a-4f3ee75db2ba] Carp backup event
Feb 7 12:08:04 opnsense: /usr/local/etc/rc.carpmaster: Carp cluster member "192.168.0.5 - (1@em1)" has resumed the state "MASTER" for vhid 1
Feb 7 12:08:04 configd.py: [30b8e115-bc25-4339-9f7f-92176a1a471f] Carp master event
Feb 7 12:08:04 kernel: ifa_maintain_loopback_route: deletion failed for interface em1: 3
Feb 7 12:08:04 kernel: carp: 1@em1: MASTER -> BACKUP (more frequent advertisement received)
Feb 7 12:08:04 kernel: carp: 1@em1: BACKUP -> MASTER (master timed out)

Code Select Expand

em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:0c:29:30:b0:1d
        hwaddr 00:0c:29:30:b0:1d
        inet 192.168.0.2 netmask 0xffff8000 broadcast 192.168.127.255
        inet 192.168.0.5 netmask 0xffff8000 broadcast 192.168.127.255 vhid 1
        inet6 fe80::20c:29ff:fe30:b01d%em1 prefixlen 64 scopeid 0x2
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        carp: BACKUP vhid 1 advbase 1 advskew 0
        groups: Inside

What am I missing?

Edit: There is no carp0 interface listed when I look at ifconfig output.