Messages

Create an alias:

Code Select Expand

Local subnets
192.168.0.0/16

Substitute for your subnets.

Create one allow rule on each (vlan) interface:

Code Select Expand


protocol IPv4
source blah blah
destination (important!) tick the box (so destination = NOT)
select the Local subnets alias you created above


Create one allow rule for TCP/UDP 53 (DNS) with destination the interface IP (just select it from the dropdown), on each interface

Create one allow rule for UDP 123 (NTP) with destination the interface IP, on each interface.

Solved, You are welcome  ;D

Advanced version: Create an alias with the used ports (if you want to limit outgoing traffic based on specific ports). When creating the allow rule (the local subnets one) on the interfaces, select the port alias as well.

You are double welcome  ;D

Most SIP run on UDP, so the firewall has no way of knowing where to forward replies to your packets. There are (extremely ugly) hacks to work around this, proxying the SIP packets, waiting for the stars to align etc. etc., but this are the methods that will work no matter what:

1) Forward the UDP port in use (usually 5060, since you have captures, you know the port already) from the office to the client (important, don't leave it wide open) to the client (softphone in your case) from the WAN. Not usually recommended, depends on your threat assessment (ie if the phone is on a separate VLAN, not talking to anything else, there isn't too much harm in going this route). Here be dragons.
2) Get a PBX system and use *that* to connect to the remote office then your softphone to that. Usually PBX systems are (supposedly) kept up to date. Can get pricey, depending on what you are doing and you still have to figure out how to actually get it connected through another option.
3) Use a VPN to connect to the remote office, initiated on your opnsense. Easy, set up and forget type of thing.
4) Use a VPN on the softphone (I'm assuming softphone = SIP client on PC, otherwise on the phone) to get it directly connected to the office. Easy, set up and forget type of thing.
5) Variation of 4: Get connected through an encryption layer, they use TCP to do their thing (=stateful, hence OPNsense can deal with it).
6) Use straight up TCP.

Since running authentication and calls over the public internet unencrypted isn't considered best practice these days*, I would go with either a VPN or encrypted connection to the remote office. The percentage of the remote office not knowing how to set that up are pretty high though (>99%).


* actually scratch that, running everything wide open is considered best practice judging by the clusterf***s that I see in production.

Is this a trick question? There is a domain option right there in DHCP.

I don't think you quite understand how let's encrypt works. You aren't verifying the TXT records, they verify them through their server *then* sign the certificate. Your client only sees a "OK" or "not OK" response that is coming from them (and the certificate of course).

Otherwise you would generate a *.paypal.com certificate that correctly validates on your end user's browser (rendering the entire PKI useless).

A TXT record override could be useful, but for any other reason other than let's encrypt.

With regards to ZFS: I've just done 2 conversions from vanilla "FreeBSD on ZFS" to OPNSense (opnsense-bootstrap) and they both work as expected, ran the script, rebooted, was in webgui.

Can you write short guide/tutorial how to use ZFS in OPNsense? Did you install FreeBSD first and then all this OPNsense stuff or did you have a script to convert OPNsense to ZFS?

FreeBSD was already installed on ZFS, then it was a simple copy/paste from https://github.com/opnsense/update#opnsense-bootstrap, so the quick guide is:
1) Get FreeBSD installed on ZFS
2) Run the script  ;D

[isn't]

Active FTP: Client connects to the server, decides on a port to use for the data channel (your log is showing the command-side channel) and connects.
Passive FTP: Client connects to the server, server tells it "my public IP is XYZ, use port ABC", your client connects to this for its data channel. This is where everything falls apart in your setup, see below.

That being said: The only way to "easily" punch through a NAT is setting up the FTP server for explicit FTP over TLS  (there isn't any other way to use FTP, I don't care what the RFCs say) with a limited port range for the data channel (EDIT:) in passive mode. Your server will need to answer with its "true" public IP, and a limited port range that is port forwarded to it. Accessing it from the internal network is a matter of correctly doing the translation (handled by OPNSense,shouldn't be an issue, at least wasn't in my tests).

Rule on LAN1: Source LAN1 subnet
Destination: tick (=not) + the LAN2 subnet

That means ICMP packets destined for anything other than LAN2 get accepted. Adjust to the protocols/ports you want to filter.

With regards to ZFS: I've just done 2 conversions from vanilla "FreeBSD on ZFS" to OPNSense (opnsense-bootstrap) and they both work as expected, ran the script, rebooted, was in webgui.

What's your WAN connection type? If opnsense isn't "aware" (ie directly responsible) for your WAN as in the case of PPPoE for example, and you are running another router in front of it (ie your ISP's modem), opnsense sees a packet for the external IP, figures it's not mine, and pushes it "upstream" (to your ISP's modem) that is too dumb to do the actual reflection back (sends the packet to your ISP, your ISP can't figure out how to do routing from the source to the destination etc.etc.).

https://linux.die.net/man/8/mtr

[SERVER: 127.0.0.53]

; <<>> DiG 9.11.3-1ubuntu1.5-Ubuntu <<>> www.google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 5346
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;www.google.com.                        IN      A

;; Query time: 100 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Thu Mar 28 09:11:13 CDT 2019
;; MSG SIZE  rcvd: 43

What is that server? I talked about it a couple of replies back.

Feels that there are other things fighting the configuration. Assuming the gateway (the PIA one) is correctly set up, I don't see why it wouldn't work. Also assuming a default configuration + customizations that I mentioned, the gateway is there, the routes are there, the rules are there, dunno why it's not cooperating.

Other than mtr'ing the entire network to see why and where packets gets stuck, I don't think there is any other solution.

thanks for reply. Upstream switch is managed cisco and "Disable preempt" is NOT ticked on master, but IS ticked on backup node. I did that because according to help line underneath - when firewall is booted, it will switch to master and I didn't wanted this feature on backup node. Or I did I get it wrong? Should I have it unticked on both nodes?

Ah, not even a "dumb" switch, just plain ol' dumb (without the quotes)  ;D

"well, there's your problem"  ;)

unticked preempt as mentioned by mimugmail on both

There is also worth mention that there is one cable missing on backup node - waiting for our networkers to configure it on their switch. So maybe the situation will change if all interfaces are up.

Get that cable plugged in otherwise it will trigger a failover.

So if I understand correctly - in this case of one cable missing on  backup node, if I unplug cable from master, the master node will not fail over all interfaces, but only the one that failed. Otherwise, with all interfaces up on both nodes, if I unplug cable on master, it will failover all interfaces. Is that correct?

no, see above, everything fails over at the same time. In your particular case the switch see the IP>MAC association on both ports, and figures that the "first one must be right, so I'll continue using it".

And what part of firewall ensures that all interfaces will failover, is it kernel? AFAIK the carp is running independent on interfaces, so one carp per interface. And on the switch side, both interfaces should see each other, ie. same vlan with no restrictions.

Does it matter what ensures that all will failover? they will, trust me, I've run this configuration for years. Yes, both members should see each other with no "filtering" (unicast,anycast,anything) between the two on the switch side.

That's not how it's supposed to work. There is an issue with your switch upstream of opnsense. In case of a cable being unplugged everything "fails over": the backup (correctly) assumes the master role, notifies your upstream switch (correctly) that your "WAN" IP is now handled by the slave, clients behind your cluster (correctly) see that their gateway's IP (LAN) switched and start (correctly) using that. Return packets arriving at your upstream switch (on opnsense's wan side) go through your switch's routing table and are (wrongly) spit out at the old interface associated with the master's MAC address.

Cisco also has (many) factory backdoors, but we don't want that around here  ;).

Is the upstream switch a "dumb" switch or a managed switch?

You missed pinging the VPN gateway's private IP (after the tunnel is set up) from opnsense + LAN.

[public]

Slowly getting somewhere... (there is an issue with your routes)

Which of the following is true?
1) You want everything to go over the tunnel, except "local" destinations (ie your cable modem)
2) You want only specific traffic from specific interfaces to go over the tunnel (ie everything from LAN to the internet goes over the VPN)

Skimming the thread, I'm under the impression that it's 1 from above, so to save some time I'll use that:
1) add a route for your VPN provider's public network:
Network address: 10.0.0.0/24 (example, adjust for your provider's public IP range, ie the IP you connect to before the tunnel is up)
gateway: WAN_DHCP
Description: Finding the VPN provider (to make it easier on you in the future)
2) system > gateways >  OPENVPNCLIENT_VPN4 > make it the default
3) Edit the LAN rule (any to any) and change gateway to default.
4) Make sure NAT is auto
4) Repeat all the pinging, DNS resolving posting the results.