Topics

Hi all,

We maintain a number of multi-site locations where we've used IPSec and fully-meshed the networks (ie: every location can talk to every other location). The obvious issue with this is the upkeep. Doing the math, if I have 11 locations, that means that every firewall has 10 IPSec tunnels. (Locations * (Locations -1)). So, in the instance of 11 locations, I'm maintaining 110 individual tunnels.

And this is where cloud-meshed VPN solutions (SD-WAN) enters the show. Instead of having to maintain 110 tunnels, I could maintain 11, and the cloud-meshed VPN solution would handle the routing. Or, worst case, I can manage the routes at the centralized console. Point is, the routing becomes easier.

For some of my locations, this is a non-starter. Those locations deal with protected information and an SD-WAN solution leaves too high a possible risk for someone to silently add their own node onto the network and then have time and access to all of the information they can vacuum up without our notice. Yes, I know that these solutions often offer multi-factor authentication at the cloud level. But even multi-factor is being hacked these days.

And this brought us around to ZeroTier. One of my engineers was complaining about what a pain in the rear maintaining all the tunnels was, and how SD-WAN would be so much better a solution. So I decided to give it a try with ZeroTier.

I picked a location that is more or less a "test site" - no issues if that site gets hacked, there's nothing there for them to find. And for the other site (and eventually, sites), I picked a few of our staff's home networks that are in a similar situation - if they get hacked... well, that sucks, but they have firewalls and backups.

I followed this guide and tried to keep the configuration as simple as possible. Get it working reliably first, and then start building rules and filters as necessary. But I didn't get that far.

My experience was that after

• updating OPNsense to the latest version
• getting the zerotier plugin installed
• connecting it to my zerotier portal through VPNs >> ZeroTier and using the API I was given from the ZeroTier portal
• adding the site and approving it in the portal
• creating the network interface and assigning it the appropriate IP address from ZeroTier (note: /24, not /32!)

I would see an uptime of about 45 seconds to 2 minutes, after which my network monitoring software would go from "all green" to "all red" pinging the remote hosts.

I tried adding managed route(s) (example: network 192.168.92.0/24 routes through 10.147.17.197).
I tried rebooting firewalls.
My experience did not change.

I read through the ZeroTier manual. No luck.

I not only disabled the IPSec tunnels that were in place before, I deleted them out of a few firewalls and rebooted. No change.

So, I'm now curious if I did something very obviously wrong, or if this experience is relatable to others? I'm open to the idea of an SD-WAN solution if it's stable, but I'm not going to sacrifice stability for convenience.

Opinions/Comments welcome!

Hi all,

I was referencing this documentation the other day to get up country-based IP filtering.

The documentation states "In OPNsense, goto Firewall:Aliases and select the GeoIP settings tab. Enter the URL you have created into the URL box and click Apply, and that's it."

When I go to Firewall:Aliases on an OpnSense v20.1 server, I see this tab. However, on a v19.7.6 firewall, there is no GeoIP tab. On this one, when you create an alias you can choose GeoIP as an alias type, and then from there select the countries you want to block. Then, in theory, you make a block rule in Firewall:Rules:Floating, selecting all of your outside interfaces, and then block anything with a source of the GeoIP Alias.

I see that in v20 I can still do this with the aliases, as type GeoIP.

So, I'm just looking for clarification. Is the Alias method OK to use? Is the GeoIP method the one we're supposed to use? I don't want to assume that the alias method is working to block inbound traffic from undesirable countries and then find out that it doesn't actually work without MaxMind and the GeoIP tab.

Thanks, in advance!

Hi all,

In the latest release (20.1), there's a hard-coded style element on line 1132 of the output:

Code Select Expand


1129: <style>
1130:  tr.phase1_tr > td {
1131:      font-weight: bolder;
1132:      background-color: #FBFBFB;
1133:  }


Because this is defined after all the css files are rendered, this is breaking themes. For instance, on dark themes (rebellion, tucan) the background is near white and so is the font.

Can this be moved into main.scss or another css file so it can be changed by theme designers?

Thanks!
Anomaly0617

Hi there,

In the past, I have done this with pfS and the Squid Reverse Proxy tool. But it's been a few years.

I've got OPNsense 19.7-amd64 up and running at a location. I'd like to set up a number of web servers on the LAN side and have NGINX reverse proxy the traffic in to them based on the headers.

I'm not looking to load balance at this point, just setting it so that:

http://sitea.com proxies in to http://192.168.xxx.yyy:80
http://siteb.com:8091 proxies in to http://192.168.xxx.zzz:8091
http://sitec.com proxies in to http://192.168.xxx.aaa:80
https://sited.com proxies in to https://192.168.xxx.bbb:443
https://sitee.com proxies in to https://192.168.xxx.ccc:443

So, I'm starting with one server, siteb.com above. I followed this tutorial, but I doubt I did it correctly. So here's what I've got....

Services: NGINX: Configuration: General
-- Enable nginx: Checked

Services: NGINX: Configuration: Upstream Server (1 Entry)
-- Description: Server1
-- Server: 192.168.xxx.zzz
-- Port: 8091
-- Server Priority: 1

Services: NGINX: Configuration: Upstream (1 Entry)
-- Description: com_siteb
-- Server Entries: Server1
-- Load Balancing Algo: Weighted Round Robin
-- Enable TLS: Checked
-- TLS: Servername override: [Blank]
-- TLS: Supported Versions: [Nothing Selected]
-- TLS: Session Reuse: [Not Checked]
-- TLS: Trusted Certificate: [Nothing Selected]

Services: NGINX: Configuration: HTTP(S): Location (1 Entry)
--Description: com_siteb
--URL Pattern: ^http://siteb.com
--Match Type: Case Insensitive Match ("~*")
--URL Rewriting: [Nothing Selected]
--Enable Security Rules: [Not Checked]
--Learning Mode: [Checked]
--Block XSS Score: [Blank]
--Block SQL Injection Score: [Blank]
--Custom Security Policy: [Nothing Selected]
--Upstream Servers: com_siteb
--Path Prefix: [Blank]
--Cache: Directory: [none]
--File System Room: [Blank]
--Index File: [Blank]
--Automatic Index: [Not Checked]
--Basic Authentication: [Blank]
--Basic Credentials List: [None]
--Enable Advanced ACLs: [Unchecked]
--IP ACL: [None]
--Force HTTPS: [Unchecked]
--Enable HTTP/2 Preloading: [Unchecked]
--Pass Request To Local PHP Interpreter / Threat Upstream As FastCGI: [Unchecked]
--(PHP) Router Script: [Blank]

Services: NGINX: Configuration: HTTP(S): URL Rewriting (1 Entry)
--Short description (to display): root
--Original URL Pattern (Regex): /
--New URL Pattern: $1
--Flag: Redirect

Services: NGINX: Logs: HTTP Access Logs: com_siteb
--Time: 22/Dec/2019:18:32:25 -0500
--IP: [Redacted, but it's mine]
--Username: -
--Status: 404
--Size: 5431
--Referer: http://siteb.com/
--User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36
--Forwarded For: -
--Requested Line: GET /favicon.ico HTTP/1.1

Services: NGINX: Logs: HTTP Error Logs: com_siteb
--Date: 2019/12/22
--Time: 18:32:24
--Severity: error
--Number: 35259#100189
--Message: *1 "/usr/local/etc/nginx/html/index.html" is not found (2: No such file or directory), client: [Redacted], server: com_siteb, request: "GET / HTTP/1.1", host: "siteb.com"

[System Information]

Hi all,

I've got a weird one, but I've now seen it at different locations and it's concerning.

Please note that this one is very similar to this post, also by me, from a while back.

First, some specs because well, everyone loves specs:

System Information

Code Select Expand


Name frasvrfw.{redacted}
Versions OPNsense 19.7.6-amd64
FreeBSD 11.2-RELEASE-p14-HBSD
OpenSSL 1.0.2t 10 Sep 2019
Updates Click to check for updates.
CPU Type Intel(R) Xeon(R) CPU X3450 @ 2.67GHz (8 cores)
CPU usage
Load average 0.53, 0.38, 0.28
Uptime 1 days 03:06:07
Current date/time Tue Dec 17 10:33:25 EST 2019
Last config change Mon Dec 16 3:31:41 EST 2019
State table size 1 % ( 8526/814000 )
MBUF Usage 1 % ( 7100/506546 )
Memory usage 13 % ( 1099/8144 MB )
SWAP usage 0 % ( 0/8192 MB )
Disk usage 2% / [ufs] (1.4G/101G)

Interfaces

Code Select Expand


   GUEST 1000baseT <full-duplex> 192.168.25.254
   LAN 1000baseT <full-duplex> 192.168.3.1
   PLC 1000baseT <full-duplex> 192.168.20.1
   Phones 1000baseT <full-duplex> 192.168.9.1
   Printers 1000baseT <full-duplex> 192.168.6.1
   SAN 1000baseT <full-duplex> 192.168.10.1
   SANBACKUPS 1000baseT <full-duplex> 192.168.16.254
   SECUREWIFI 1000baseT <full-duplex> 192.168.5.1
   SECURITY 1000baseT <full-duplex> 192.168.50.1
   WAN 1000baseT <full-duplex> {redacted}

Ok, with that out of the way, I've got a Dell PowerEdge R210 running as an OpnSense firewall/gateway between multiple networks (separated into VLANs and their own subnets) with the OpnSense firewall at the center.

We've now had this happen twice -- 16-Dec-2019 between 3:20 AM and 7:20 AM, and about a month ago where there was a 35 minute power outage.... so I'm no longer thinking this is a "fluke."

The trigger:
The WAN connection drops at the provider end (in other words, between the internet provider's fiber router and their closest node)

The symptom:
OpnSense stops routing all traffic -- even the internal traffic between internal subnets such as from the LAN to the PRINTER networks. This continues to occur even after the WAN connection is restored and the internet is available again.

The workaround resolution:
Reboot the firewall and all the problems go away... until the next time the internet connection drops.

In both instances, the power in the area has gone out, and our on-site battery backups and generator have kept the building up and running throughout. So my OpnSense logs do not show that the firewall has rebooted. But if I go to Reporting -> Health and look at any metric, ie:

Packets -> LAN
Packets -> WAN
Packets -> IPSec
Quality -> Gateway

They are all flat-lined between those times.

So the question is, what would cause OpnSense to stop routing traffic between internal networks when the WAN connection drops, and is there a way to fix it, short of setting up a check_internet script that reboots the firewall if it can't get to something really common like google?

Thanks in advance, all!

Hi there,

Just wondering if anyone is actively working on generating a pkg for zabbix4.4 -- I accidentally updated my Zabbix server to 4.4.1 (meant to just go to 4.2.x) and I'm seeing errors in the zabbix server log like:

11068:20191117:195807.381 cannot process proxy "[SiteID]": protocol version 4.0 is not supported anymore

So, totally my fault for upgrading too far, but since I did and it made some database changes, I'm now hesitant to go backwards to 4.2.

If someone can walk me through compiling from source on FreeBSD, I'm generally fluent in POSIX environments. I'd be happy to submit whatever I compile.

Thanks!
-Paul

Hi all,

Those of you that have been around the *Sense world for awhile probably know me. I've been a fan for a number of years, and have a lot of posts on the pf as well as the opnSense forums. Heck, I may even have a few on m0n0wall, since I go back that far on distros of this firewall.

If you're just now reading this, this is an awesome firewall solution and I've found nothing that compares to it in the commercial world. There's a balance of functionality and UI that I've never seen at this level in a commercial product. So what's below is more for the folks that have been running some version of *Sense for a little while and is not representative of the product of a whole.

Ok, disclaimer over. Now to the real topic:

I have somewhere in the neighborhood of 30-50 *Sense installations in production. These span everything from a terrible DSL connection in the middle of nowhere's-ville to a 200 x 200 Mbps redundant fiber connection in a controlled data center environment. I'm seeing this problem across the board, on all installations. That means everything pf 2.3.2 and newer, and I'm guessing everything opnSense 18.1.x and newer. In other words, I'm thinking this is a FreeBSD problem and not a *Sense problem.

Symptom:

The firewall is set up to reboot on a schedule. After the scheduled reboot, the firewall does not respond to anything remotely. ICMP traffic does not respond, packets are not forwarded, rules are not processed. It's as if the firewall is stuck in the default state of "deny all" for all packets.

Resolution A (a bad one):

Go to the console and log in if necessary. Reboot the firewall using the appropriate menu option. Let the firewall reboot. Check again and see if you can access the outside world, and the outside world can access you.

Resolution B (possibly better?)

Run a script that checks to see if the firewall can reach the gateway device (DSL modem, DOCSIS router, fiber router, etc.) and/or a common website (Google DNS, for instance). If it cannot, issue a reboot command. Possible side-effect: This could reboot the firewall in the middle of the day with no warning to your end users.

Anyone else run into this problem? The most common hardware we use are Dell PowerEdge R2{x}0's as firewalls, so its possible it is specific to those, but I believe I've also seen this behavior on the micro-firewalls we have running made by a reputable Amazon reseller ending in "LI". ;-)

DISCLAIMER: If you can't tell, this email has been formatted to avoid getting a nasty letter from that one company that bought one of the *Sense distributions and has been known for sending nasty letters to people trying to help.

Hi all,

I haven't seen this here yet, but perhaps I missed it. If so, let me know and I'll see myself out. ;-)

I've recently been exploring the idea that access log data from squid could be piped into MySQL (see here). My thought is NOT that the OpnSense server would host this MySQL data, but rather that I could push that MySQL data to a MySQL server internally for further analysis. All of this comes from the desire of management-types that do not want to read through data, but rather would like to see the data from a 1000 meter view and then tunnel down into the data they want.

For instance, if last week squid allowed 98.5% of traffic and blocked 1.5% of traffic for employees, that's one "1000 meter" view. Then if management wants to drill down into the 1.5% of traffic to see what is being blocked and who is attempting to access that information, they can.

Ultimately, this comes around to being able to ask squid on opnsense to send the log file to MySQL. And the link above seems to indicate that squid is possibly capable of doing so. The question is, has anyone done it, is it possible with OpnSense on the BSD platform (as the link above is for a debian platform), and if the answer to that question is yes, what would it take to get the functionality incorporated into Squid for OpnSense?

An ancillary question that I could see coming about would be a way to point multiple OpnSense servers to a single MySQL database (again, internally, over the VPN tunnel for instance) and the ability to see the multi-site view. Why is the internet in [location] always so slow? Lets see what their browsing patterns are like....

So, how far-fetched is this idea? In the short term I'm considering deploying proxy servers out to each location,  but in the longer term, I'm looking for a way to manage the data in a way that isn't cumbersome.

Hi all,

I've been around the community for a few years now, but I'm a pfSense convert like most of us here. I've used pf/OpnSense for going on 10 years (?) now. So, not exactly a newb, but I generally stay pretty quiet.

I have a small municipality who is running pfSense. I'm in the process of converting all the firewalls over to OpnSense. The local Sheriff's Office IT only uses Cisco, and nothing else. Since we have to interface with them for various agency records, this means they have Cisco appliances in key buildings sitting right next to the pf/OpenSense firewalls, plugged into the same ISP router, and with their own external and internal IP address. This creates two points of entry into the networks instead of one, which makes it doubly difficult for me to take responsibility for keeping the network safe. So, on this last expansion run (for ancillary stations) I suggested that we just set up an IPSec VPN tunnel between the county and the existing pfSense/OpnSense firewalls that are on-site at each location.

You'd think I dropped a bomb on them.

The present (valid) argument for why this is not feasible is that the OpnSense firewall platform is not FIPS 140-2 certified. Looking purely at the technical requirements, I think it'd pass with no problem, but the question is, what does it cost to make a firewall FIPS compliant? Is this something the OpnSense community should consider pursuing? Is HardenedBSD going to make this easier for us, assuming the development of OpnSense eventually gets there?

I can foresee running into this problem with other industries that have a standards-based auditing system of firewalls, examples being PCI, SOX, HIPAA, etc. so it'd be nice to hammer this one out before those come up.

Any/all responses are appreciated.

Hi there,

I'm still struggling to implement BiNAT over various IPSec Phase 2 tunnels. Here's how it's handled in pfSense:

Mode: Local Network
Type: LAN Subnet (Mine is 192.168.121.0/24)
Address: [Blank]
NAT/BiNAT Translation Type: Network
NAT/BiNAT Network: 172.16.254.0/24
Remote Network Type: Network
Remote Network Address: 172.16.246.0/24

So, whenever traffic goes out to the 246 network, it should appear to come from 172.16.254.[ip]
Whenever traffic comes in from the 246 network, it should appear to come from 172.16.246.[ip], even though on their end it's likely something like 192.168.1.[ip], and we have BiNAT set up there too.

Lastly (and most importantly) Whenever traffic comes goes out to the 10.0.143.0/24 network, it should appear to come from 192.168.121.0/24 because that is a branch office and it has no BiNAT defined in the Phase 2. There's no chance of a conflict and therefore no need to BiNAT.

If I try the same thing in OPNSense, it looks like this:

Mode: Tunnel IPv4
Description: Customer Name
Local Network Type: Network
Local Network Address: 172.16.246.0/24
Remote Network Type: Network
Remote Network Address: 172.16.254.0/24

Then I create a rule in Firewall >> Nat >> One to One
Interface: IPSec
External IP: 172.16.254.0/24
Internal IP: 192.168.121.0/24
Destination IP: * (Any)

... but this takes over all IPSec traffic going out and makes it appear to come from 172.16.254.0/24 in the firewall logs.

Is there a way to just set BiNAT settings in the Phase 2 settings and be done with it?

[This works, but it negates the documentation I see here]

Hi all!,

Long time monowall/pf/OPNSense user here. I'm a network engineer for a managed service provider in Ohio.

I'm converting firewalls at customers from using pfSense to OPNSense as upgrades are required. I've discovered something through trial and error, but need to know if it's the proper way to be doing things...

For customers, we use BiNAT VPN tunnels extensively. This is because it's incredibly common to run into customers with 192.168.1.0/24 networks or 192.168.0.0/24 networks, and we need to be able to monitor their stuff over an encrypted tunnel from our office. We utilize rules on our side so they can only see the network monitoring server and everything else is blocked. On our side, however, I can see their whole subnet. So it's common for me to have a setup that looks like this:

Customer Side: 192.168.1.0/24 binat to 172.16.212.0/24
Our Side: 192.168.254.0/24 binat to 172.16.254.0/24

So the tunnel on their end is looking for a remote subnet of 172.16.254.0/24, and maintains a local subnet of 192.168.1.0/24 with BiNAT to 172.16.212.0/24.

The tunnel on our end is looking for a remote subnet of 172.16.212.0/24, and maintains a local subnet of 192.168.254.0/24.

On the customer's Firewall >> Rules >> IPSec it looks like IPv4 * * * * * (Allow IPSec Traffic)

On our end in Firewall >> Rules >> IPSec it looks quite different, only allowing customer VPNs to get to one IP address.  :)

So the question became, how do I make this occur in OPNSense? The Phase 1 always establishes with no issue, it's always the Phase 2 that is broken. So, here's what I've tried so far on my Phase 2 Tunnel configuration:

• I tried LocalNet as 192.168.1.0/24, RemoteNet as 172.16.254.0, and Manual SPD as 172.16.212.0/24. No joy.
• I tried LocalNet as 172.16.212.0/24, RemoteNet as 172.16.254.0, and Manual SPD as 192.168.1.0/24. No joy.
• I tried LocalNet as 192.168.1.0/24, RemoteNet as 172.16.254.0, and Manual SPD I left blank. I then tried going to Firewall >> Nat >> One-to-One >> Created a BiNAT that looks like IPSec, External is 172.16.212.0/24, Internal is 192.168.1.0/24, Dest. is Any. This works, but it negates the documentation I see here:
  https://forum.opnsense.org/index.php?topic=989.0
  https://github.com/opnsense/core/issues/369

So, is the issue just that we need an updated tutorial or documentation?

Thanks in advance!