Messages

Got it solved.
I had a virtual IP enabled for the interfaces.  Looks like FRR uses this information for route table population as well.

Thanks again for the input!

Thanks for the replies.  After some digging I found the root of the problem but I don't know why it's happening or how to fix it.  Suggestions would be appreciated.

The problem is related to FRR not removing the original IP address information for the interface and installing two connected routes for the interface in it's RIB (the original subnet and the new one).  Below is the specific example I can replicate:

Current State:
Interface vtnet2(opt1) = 10.1.1.1/24
FRR RIB: Contains one entry for vtnet2
     Connected route 10.1.1.0/24 via "Directly Attached" interface "vtnet2"

Future state:
I change the IP address and/or mask of vtnet2 to something different.
Interface vtnet2(opt1) = 10.10.10.1/29
FRR RIB: Contains two entries for vtnet2
     Connected route 10.1.1.0/24 via "Directly Attached" interface "vtnet2"
     Connected route 10.10.10.0/29 via "Directly Attached" interface "vtnet2"

FRR is not dropping the old IP address information and essentially blackholes traffic destined for 10.1.1.0/24.

I've tried rebooting, deleting vtnet2 and re-adding it (with FRR stopped or running), deleting vtnet2 then rebooting then re-adding, etc, etc.  Only way FRR removes the subnet from the RIB is if I shutdown the interface.  Even removing the IP address info (setting IPv4 Configuration Type to none) doesn't remove the route.

Looking for some guidance on an issue I've experienced in both the pfsense and opnsense worlds.  I was a long time user of pfsense and switched ~2 years ago when the feature parity I cared about became available and love it.  However, I just experienced an issue that I've experienced in pfsense multiple times over the years which makes me think its FreeBSD related (this is the first time I've tried making this type of change in opnsense).

I seem to always run into an issues where traffic stops flowing through an interface when making changes to that interface configuration.  Two specific examples with diagrams attached:

1.
As-is state: DMZ interface is configured for 192.168.1.1/24.  Layer 2 adjacent client is 192.168.1.12/24.
Future: I change the mask to /29 on the firewall and change the mask and IP client to be within the /29.  Post change the client can only ping the DMZ firewall interface, nothing behind it.  Firewall can ping the client from the DMZ interface AND from any other interface on the firewall (Client can NOT ping any other interface on the firewall or any device behind those interfaces).  Similar to the firewal  I even created a floating/quick permit any any on all interfaces to make sure I'm not crazy.

2.
As-is state: One of the physical firewall interfaces has two sub-interfaces created (the physical "untagged" interface is not being used, just sub-interfaces 10 and 11). 
Future: Per the diagram, I try adding a new sub-interface and all three basically stop working (similar behavior to #1).  Nothing is changed on the switch or to the trunk.  I've also tried removing one sub-interface and creating a new sub-interface with a different vlan and the behavior is basically the same.  This is what I experienced in pfsense as well all the time.  Only way I've solved it in the past is to rebuild the box which is why I feel it might be a FreeBSD related issue.

Anyone experience anything similar with fixes?
Can anyone tell me where in FreeBSD and/or opnsense config file is with this info so I can see what the GUI is applying to the config?  Don't mind updating via CLI in the event the GUI is jumbling the data up.

Thanks Fright, clearing the logs solved the problem.

Had nothing to do with sorting the date as I could see the entire log file but I completely missed the giant "clear log" button at the bottom of the screen (thus why I tried deleting the radius.log file via CLI  :) )

Hello,
Hoping someone can help.  I have FreeRADIUS running on my box and have run into an issue with the logging in the GUI.

In short, the GUI is no longer displaying updated log info. (see attached)

I have logging enabled for all authentications but when I go to the logging tab I don't see any recent authentications that were made.  Latest info was from yesterday (Dec 29) for a single client.

When I go into CLI and look at /var/logs/radius.log for that same client, I can see multiple authentications since yesterday.  It almost seems like the GUI is reading a different file for the log information and it's not getting updated.

The behavior started when I changed the setting "Log to file or syslog" to syslog for testing, and then changed it back to file.

I tried uninstalling/reinstalling the FreeRADIUS plugin, deleting the radius.log file to let it build a new one, and toggling logging settings on/off with service restarts in-between.  Nothing has worked.

Appreciate any help anyone can give.

Glad you got it working.  Much better than the backup workarounds you were looking to do.  :)

hi dkanzlemar
When you say a clone of each other does that mean it's virtual and you literally cloned it?  If so, that's why your interfaces are all messed up.  I would suggest spinning up a clean opnsense box so the interfaces match and you can restore the config directly.

You do have a couple alternatives though.
1. Assuming this is a virtual setup, use something like Veeam to backup the whole VM.  Restoring from Veeam is a lot easier IMO.

2. Run backups from a remote computer using a shell script and an SSH key instead of the stuff built into opnsense.  For example:

opnsense ip address: 10.1.1.1
network share ip address: 10.2.2.1
file with user ssh key: ssh_private_key.ppk
(Make sure to paste your ssh key into the "authorized keys" field of the user inside opnsense as well)

Shell script contents:
ssh -o StrictHostKeyChecking=no -i ssh_private_key.ppk USERNAME@10.1.1.1 cat /conf/config.xml > //10.2.2.1/MYSHARE/BACKUPFILENAME.xml

Run the script manually or set it as a cron job/scheduled task.

3. Get HA working.  Not clear why your APs would have issues unless its actually CARP having issues.  CARP can be challenging in virtual setups and you could use routing as an alternative.

Hi orso,
Its difficult to picture what your problem is, sounds like traffic is flowing the way it should.  If you can provide screenshots would be helpful.

I'm envisioning a setup similar to the attached image on the left.  If you are just trying to have your LAN devices access the VPS on TCP/25 (or whatever) all you need is a firewall rule on the LAN and you are done so long as the VPS has opnsense setup as its default gateway.

If you are trying to allow internet access to VPS on TCP/25 its a little more complex but easy enough.  Create a NAT/port forward rule for any traffic hitting your WAN interface on TCP/25 and redirect it to 10.2.2.2. (Attached on right.)  You also need to create a WAN firewall rule allowing Any to 10.2.2.2 on TCP/25. 

If you are just trying to get VPS internet access (or general network connectivity) you need to make sure the VPS has its default gateway set to the opnsense IP for your dmz zone, proper rules are created on the dmz zone, and your outbound NAT statement includes the VPS subnet in the source (attached on bottom right)

Hi Baliste,
You need to configure some routing between your boxes.  You can do this easily enough with static routes.

Look at the routing table in opnsense1.  Willing to bet it doesn't have 10.0.200.0/24 in it.
Opnsense2 only needs a default route/gateway pointing to the LAN ip address of opnsense1 (I assume 192.168.64.1).

Let's say opnsense2's WAN address is 192.168.64.100.  In opnsense1 create a static route for 10.0.200.0/24 pointing to 192.168.64.100 and you should be good.

Make 192.168.64.100 a gateway
System --> Gateways --> Single

Create a route for 10.0.200.0/24 pointing to opnsense2
System --> Routes --> Configuration --> add new
Network address: 10.0.200.0/24
Gateway: 192.168.64.100

Only other thing to call out, make sure NAT is disabled on opnsense2.

May the schwartz be with you.

Hi Spiky,

You are really broaching two topics on your question.  One is VLANs one is subnetting.

If I'm reading the behavior you are seeing correctly it all sounds correct. (see left side of attachment)

In other words, the VLAN 4 ip addres on the switch will only know about/listen for traffic coming from devices in 10.4.4.1-254.  Anything connected to VLAN 4 that does NOT have an IP address in the same subnet will not respond.  This is how subnetting works, period.

On the VLAN side, they are just tags to assign different subnets to.  The layer 3 switch will allow the VLANs to communicate with one another unless ACLs are applied.

To further explain the example (see right side of attachment) the two PCs can communicate but the switch won't be able to talk to them.  And in case you are wondering, the answer is NO, I did not fat-finger PC2's IP address.  Since VLAN 1 knows nothing about these two PCs (because they are connected to VLAN 4 and VLAN 4 is in a different subnet) it doesn't matter what address they have.

I would suggest you youtube a couple subnetting videos.

I had the same issue on one of two boxes running in HA except I'm only ipv4.  I ended up reverting back to 22 to resolve the issue.

Two pieces of info to hopefully help the devs:
1. I run BGP between my opnsense boxes and the LAN instead of using CARP.  The box that would fail passing traffic to the WAN continued to advertise the default route via BGP and still had 0.0.0.0/0 as a zebra (K) route installed in the RIB.  Like teapot, pinging an internet address from the opnsense box results in "no route to host" even though the default is in the RIB.  Rebooting fixed the issue for a short period of time...24 hours sounds close enough but I didn't track it that close.

2.  I upgraded the day 23.1 came out and I applied the additional updates immediately when they were available.  I plan on waiting a couple weeks before upgrading again wondering if there was an issue with the original 23.1.0 upgrade path.

You only have 1x internet circuit and 1x modem, right?  Both of those are single points of failure, so I assume you want to use CARP simply to have redundancy below the modem.  If that's the case I would suggest taking the ATT modem out of bridge mode and just doing a static NAT for all the traffic to opnsense.  (see attached for reference).

Assume your modem WAN IP is 2.2.2.2.  Configure the modem so that all traffic destined for 2.2.2.2 is NATd (port-forwarded) to the 10.1.1.2 CARP VIP of your opnsense boxes.  Make sure to disable any firewall that may exist on the modem and verify you can't access the modem's web interface over the internet so it doesn't get hacked.

Not sure if this satisfies what exactly you are trying to accomplish but it would work.

Your description is difficult to visualize.  You make it sounds like the admin PC is accessing opnsense across the WAN but it has a 10.x.x.x RFC1918 address.  Can you provide a diagram?

TheGreatBellend,

Do you need the modem?  Sounds like its normally a router that NATs and you are just bypassing its normal functionality by putting it into bridge mode.  Can you connect the ATT circuit directly to your own gear?  Or is the modem doing some conversion like coax to ethernet?

If you can remove the modem, and the circuit is larger than a /30 (255.255.255.252), you can just hook the circuit up to a switch (or even a vSwitch on the R740XD if both opnsense boxes are running inside it) and place the WAN interface of each opnsense box in the same vlan...and you are done.

If you can't, the only other thing I can think of (and I've never seen this functionality before...) is if the modem can assume two public IP addresses and bridge each IP address to a different opnsense MAC address.

Hi Eloïse,

I think you are really asking two questions here.
1. How do you make the LTE circuit a failover circuit.
2. How can you apply this failover logic to a set of opnsense boxes running in HA.

For question 1, I think what you are looking for is a gateway group.  You essentially create a group that contains your different gateways (ONT and LTE) and give each gateway a priority so one is preferred over another until it fails or has performance issues like packet loss and latency.
https://docs.opnsense.org/manual/multiwan.html

For question 2, in order for multiple boxes to share a WAN connection the circuits must be larger than a /30 (255.255.255.252) since you need to use 3x IP address (1x for service provider, 1x for opnsense1, 1x for opnsense2).  Assuming this is not a problem, you would simply create the same gateway group on each box where ONT is preferred until it fails which activates LTE.  Would look something like the attached.