Messages

1

24.1 Legacy Series / Re: Missing PPPoE manual disconnect/connect

« on: July 08, 2024, 11:31:38 am »

Thanks for the instructions. I did some testing in the past with these settings and sadly i never found it working reliably. It almost never worked. Maybe this has to do with the PPPoE connection needing a VLAN tag.

I will change to letting the modems dial in and use a switch, CARP and 1:1 NAT to expose the Connection to the firewalls CARP IP.
But I need a lot of testing for that because double NAT (VOIP Server -> FW -> Modem) does not fare well with our VOIP Setup somehow. So it will take some time to do this.

2

24.1 Legacy Series / Re: [SOLVED]Weird ARP entry originating from opnsense

« on: July 08, 2024, 10:50:10 am »

I found the cause of the issue.
There seems to be an issue with unreliable paket routing somewhere in the network stack of the virtualization Layer or somethin like that.
It seems like, sometimes VLAN Tags are not respected somewhere along the path between the client and the DHCP Server.
I fixed it by changing the way the DHCP Server is connected to the different VLANs. Now it has an Interface on every VLAN and we don't rely on relaying anymore. Works like a charm at the cost of some IP Addresses and configuring a bunch of interfaces.

3

24.1 Legacy Series / Re: Missing PPPoE manual disconnect/connect

« on: July 08, 2024, 10:44:21 am »

I would like to get back to this problem. I had a feeling this would bite me and I was right.

We use a HA setup. When we switch to the secondary node, we need to disable the PPPoE connection on the primary, in ordner for the secondary node to dial in. (I know this is not ideal and we are working on a solution for this)
In the past I could use the connect/disconnect buttons in the overview to do this. This had the effect, that the interface stayed up but was disconnected.
Now I need to use "disable" in the interface settings. This causes the Interface to be down. In this state the Interface cannot be used to configure e.g. an OpenVPN Server for it.

So if I am running on the secondary node and need to make a change to the config, I do this on the primary and then sync. This does not work anymore because now when I open e.g. OpenVPN configs, the binding Interface for the Service automatically jumps to the first available Interface because the PPPoE Interface isn't available anymore.

This really screws with the configuration of Services. I bet there might be some other things going wrong too.
Please consider reimplementation of "connect/disconnect" functionality for dialup Connections like PPPoE.

4

24.1 Legacy Series / Re: Weird ARP entry originating from opnsense

« on: July 03, 2024, 03:10:04 pm »

I checked if the weird MAC address switches to the other FW by doing a manual Failover and yes, now the other FW answers with this MAC address.

Maybe CARP uses this range of MAC addresses for decoupling the CARP address from the physical interface to enable instant migration to another host?

If so, the question remains, why sometime (a lot of times) DHCPOFFER packets don't arrive at the client, when the DHCP Server clearly sends them out.
There are no blocked UDP packets on UDP Port 67/68 on the interfaces.
Additionally: Why can't I see that MAC address on any interface?

5

24.1 Legacy Series / [SOLVED]Weird ARP entry originating from opnsense

« on: July 03, 2024, 11:27:24 am »

So, this seems really weird and i hope it's just some misunderstanding on my side but here is the story:

For some time we had weird troubles with DHCP and DHCP relaying not working really well. Sometimes the replys didn't reach the requesting device. Very frustrating.
I started a little digging and found, that the dhcp replys were not directed at the relay server on the opnsense, but at an unknown MAC Address, 00:00:5E:00:01:0B
Digging further i found that MAC Address on the switchport that is connected to the opnsense FW, BUT none of the interfaces on the FW own that MAC Address.
When i list all MAC addresses on the FW, there is no match, see attachment 1
So I tested if i could "reach" that MAC address in the hopes that it was just some stuck address on the switches ARP Table. To my surprise something really weird came up. I used arping to send ARP requests for an IP address on the FW, on a Net that my PC is directly connected to. The firewall responded. The TCP Header in the response shows SRC MAC address is the correct interface on the FW, but the ARP reply part of that packet lists the wrong, offending MAC address as destination for that IP. See attachment 2

I have absolutely NO IDEA WTF is happening here. Rebooting the FW didn't help. Next i will try HA failover to the other FW, to see if that phenomenon persists. If not, it might be a faulty Adapter. If it persists i am truly lost.

For some more understanding of the architecture of our Network:
The FW is a virtualised system on Proxmox. The LAN facing interface is a Mellanox 10G Adapter, directly connected to the VM via PCI passthrough.
On this interface, every LAN is a VLAN interface as a child of that Mellanox Card.
We use a HA setup. Both FW consist of the same Hardware.
Every IP address on the FW is either a CARP address or a n IP Alias (if we need more than one adress on that interface) except for the IP by which we reach the Web GUI, which is a fixed address, one for master, one for slave.

Please feel free to ask any questions if something is unclear.

6

24.1 Legacy Series / Re: Missing PPPoE manual disconnect/connect

« on: June 14, 2024, 08:47:22 am »

Thanks for clarification, I thought I was missing something.
Will it come back?
It was a very nice function to easily stop and start connections when troubleshooting. Enabling and disabling the interface works, but it's less convenient.

7

24.1 Legacy Series / Missing PPPoE manual disconnect/connect

« on: June 12, 2024, 05:32:35 pm »

Hi

We recently upgraded from a 23 version of opnSense to the latest 24.4_5 business version. Before the update, there was an Option under "Interfaces - Overview" right of the <PPPoE IF> entry, called disconnect. Since the overview was heavily overhauled, this option seems to be gone.
Has it moved somewhere else, or do i really need to disable the pppoe interface in the interface settings?

Sometimes i just need a quick disconnect and reconnect the interface, so this is somewhat annoying.
Any tips are appreciated.
thanks!

8

Virtual private networks / Single IPSec Tunnel shows as 2 in Overview and Dashboard

« on: September 14, 2023, 10:03:45 am »

Hi all

I recently set up a single IPSec Tunnel, but when i look at the Status Overview Page, I see two Tunnels, of which one is offline. The blanked out addresses are the same on both connections. Screenshots attached.

I don't understand why that is.
Can anyone help me out?

Thanks!

9

High availability / Re: HA failover user connection interrupted

« on: September 12, 2023, 11:36:04 am »

I have successfully reconfigured all the interfaces via the config file.
It looks like, now the states are synced properly.

Thanks!

10

High availability / Re: HA failover user connection interrupted

« on: September 07, 2023, 01:31:10 pm »

I think I found the issue.

When we first set up the HA pair, opnsense was on version 21.x something, or even version 20.
We updated, as versions rolled along and at some point I fiddled too much with the secondary device. I had to set it up entirely new.
Somewhere between major updates, the naming scheme for devices seems to have changed. So our primary device has interfaces with the old naming scheme, while the secondary device has these network devices set up with the new names.
pfsync is clearly unable to assign the synced connections to the appropriate interfaces.

So, how can i change the names of the interfaces? Do I have to delete every interface, create a new one and reassign it? Will I be able to keep all the firewall rules?

Thats really nasty  :'(

11

High availability / Re: Question: WAN failover losing SIP trunk registration

« on: September 07, 2023, 10:07:30 am »

True, but only if the dialup connection is handled by another router instead of the opnsense firewall.
If opnsense does the dialup, the provider will see the MAC of that interface on their endpoint.

12

High availability / Re: HA failover user connection interrupted

« on: September 07, 2023, 09:56:26 am »

Yep, opnsense runs in a VM on that machine. Proxmox is the host OS. This makes it a heck of a lot easier to backup, restore, update and handle it.

It's the only VM on that host, because this machine is dedicated for firewall for obvious reasons.

Sadly, i can't pass-through this interface directly into the VM, because it is one port of two onboard network ports. I would loose access to the Host, if i did this.
The Onboard Controller is a Broadcom NetXtreme BCM5720 2-port Gigabit Ethernet

The other network ports for WAN an VLAN are dedicated hardware and passed through.
For completeness: There are no firewall rules applied to the VM on the host system (service deactivated. Proxmox supports firewalling)

Is it likely that the visualization layer is the culprit though?

I'm currently thinking about how i can test this... live... without making a mess during office hours
LOL i could put a USB network adapter and passthrough that and see what happens. 

13

High availability / Re: HA failover user connection interrupted

« on: September 07, 2023, 08:46:43 am »

Soooooo, thanks very much for the suggestions. Yesterday i got sidetracked @work, but now I found something that might be suspicious.

Config summary:

• We use a hardware interface called vtnet0 for pfsync. On this interface, there is a "allow everything everywhere" rule.
• vtnet0 is connected via direct cable connection to the equivalent interface on the secondary device, no switch involved.
• IP on that interface is 10.0.0.1 and on secondary it is 10.0.0.2.
• On the primary, synchronize peer is 10.0.0.2 and on secondary, sync peer is 10.0.0.1

I did a packet capture on pfsync0 via

Code: [Select]

tcpdump -i pfsync0I immediately noticed, that there are a whole lot more packets captured on the primary device, than there are on the secondary.
This doesn't make sense, because packets outgoing on primary should also be captured incoming on secondary and vice versa. Packet count should therefore be equal on both devices, right?

So i did a capture to pcap file to analyse it better, but WTH?
"The file "pfsync.pcap" contains record data that wireshark doesn't support. (pcap: network type 246 unknown or unsupported)
So I'm a bit stuck on detailed analysis.

14

High availability / Re: HA failover user connection interrupted

« on: September 04, 2023, 06:01:13 pm »

Thanks!
pfsync runs on a dedicated interface, which has an "allow everything" rule.

I will check the 2 other points tomorrow.

15

High availability / <Solved>HA failover user connection interrupted

« on: September 04, 2023, 01:28:43 pm »

Hi all

We use 2 pfsense firewalls in HA setup, with CARP, state table sync and config sync (manual).
Opnsense Version 23.4.2

When I want to update, I activate "Persistent CARP Maintenance Mode" to switch to the secondary device. This works quite flawlessly except for two things:

1. User connections between devices seem to get interrupted
We use devices for recording working time. They are connected to a server via a TCP/IP connection. This connection is actively monitored by the server, to prevent manipulation or something. Every slight disruption causes the server to regard that device as offline.
When I switch to the secondary firewall, ALL of these devices still can be pinged and stay connected to the network, but they lose their server connection. So i guess this might be the secondary firewall not knowing the state of these connections.

How can i test and analyse this?

2. How to: Hot failover of WAN connection
How can i implement an automatic handover of pppoe connections to the secondary firewall? Assigning a CARP IP to these connections does not seem to work. I could not get it to work and frankly i found the information about WAN failover a little bit confusing and unclear. Maybe someone can help me out?