Messages

I'm missing something, all the postings I've found didn't wor (but I may have missed some).

I have two opnsense systems that are local and trying to set up as a test bed for a future WAN deployment.

Right now I have them connected via an ethernet interface (VLAN 23) and am running ospf on each to exchange routes. That works fine for the subnets attached to each (that are selected).

But I want one of them to advertise it's default gateway to the other so that this other (downstream) opnsense sees this VLAN 23 connection as a default gateway.

More specifically:
   OPNsense A (upstream, internet connected) :
    WAN connection (external internet) is up, marked as upstream and a gateway.  It's the only (ipv4) gateway
    OSPF - set to advertise default gateway (with a metric of 33 so it shows oddly)
    Various settings of Route Redistribution tried, in particular Connected and Statically
    Area 0.0.0.166, stub, no-summary (can't selected anything but stub or worse)
    Networks: VLAN for interconnect's subnet, plus LAN subnet
    Interfaces: VLAN for interconnect and LAN
   Opnsense B:
    WAN connection currently disabled
    Interconnect VLAN does NOT inherit a default gateway from DHCP (I have it explicitly off in DHCP option 3 for A)
    No default gateay under system, gateway
    OSPF no redistribution set (though i've tried several)
    Area: 0.0.0.166, stub, no-summary
    Networks: subnet of LAN, subnet of interconnected VLAN (i.e. same as set in A)
    Interfaces: VLAN for interconnect, LAN

I end up with all the routes exactly as expected to the various VLAN's.

Under diagnostics, OSPF, I see a "N IA" entry for 0.0.0.0/0 cost 2 for the address on VLAN 23 for A.  Metric is not the 33(+/-) I expected.

On B a shell netstat -r shows no default gateway.

So it's seeing a default gateway (though inter-area not redistributed), but not adding it to the route table.

Nothing interesting in the frr/ospf logs (loading done)

Listing 0.0.0.0/0 on A as a subnet seems to have no impact.

I can distribute the actual WAN's subnet and that works fine, but it doesn't yield a default gateway, only an external subnet's visibility.

Am I missing something?  Do I have to create a static route to the default gateway (right now it's a WAN DHCP from ISP, but it's present and marked upstream)?

PS. Least someone ask this is preparation for a solar powered remote device which will have a cellular modem and also a RF connection.  That RF connection will normally provide the default gateway when it's up, but when down the cellular should, so I want to inject the default via OSPF otherwise it gets "stuck" if it comes in via DHCP as due to the hardware involved the ethernet interface can still be up if the RF connection is down, but OSPF will go away, which will take its routes away, and I want 0.0.0.0 included.

I probably should just step away as I'm not sure I am being helpful but a few comments.

First, I still do not see an actual problem you want solved, other than saying Home Assistant stops responding. I don't know kuma, but I assume it's doing some kind of web oriented monitoring (as opposed to pings?), but my first suggestion is get down lower and figure out what exactly is not responding -- does it ping steadily for example, and just the web is not responding?  If you have it functionally doing something, let's say you set up a script to turn a light on and off every 3 seconds, does that keep working?  Is the issue that HA is not keeping up, and is hanging -- or just that its user interface is hanging.

And I still don't see why any of this is related to a router if you have only a WAN and LAN.  I presume you have NOTHING on the WAN side other than the ISP gear, so with EVERYTHING else on the LAN side the firewall should not be involved.

If you have a LAN and a WAN and that's it, all that other stuff (IGMP proxy, mDNS repeater, UPnP (a security issue)... all these sound like nonsense, as if you have just one subnet (a LAN) and not  multiple VLAN's (other subnets) there should be no need for any of this stuff. UPnP is when you want to allow someone or some thing OUTSIDE your lan to reach in through your firewall and touch something inside.  Is that where some of your problem lies?

The SSDP packets... not sure, that's a public address, so why it would be appearing on a LAN interface is puzzling, assuming that is even what that means.

Maybe someone else can make sense of this.  I am sorry I have not been able to help more, but my advice is get someone to review the whole configuration, as the bits and pieces that are coming through make no sense.

I had a USB to ethernet adapter sitting around and plugged it in, I got a working interface I could assign to LAN and then use the web.  Not all USB to ethernet adapters will work, but if you have some sitting around it may be worth a try.

Is there any way to put the old router back and remove opnsense?

If it still fails the same way you learn a lot.  If it doesn't fail, the packet trace idea is perhaps the best place to go.

You mention running IGMP proxy WAN and LAn, you also talk about a reverse proxy.  Is your access to Home Assistant somehow from the internet and not from your local LAN? What happens if the internet is down (say back with the old router) -- did anything break?

I thought the issue was you can't connect to MQTT -- did that start working?

My suggestion is not at odds with Patrick's but is a different dimension -- pick ONE thing that is a failure you think relates to opnsense, one single thing, and describe it fully, and see if you can get more details including packet traces.   The errors shown look like zigbee related (when I search for bellows for example I get a lot of hits about zigbee).  Are all your poblems actually originating with zigbee?   (The UI -- you mean the HA UI not the Zigbee2MQTT UI?)

Finally, a lot of what you are talking about looks like WAN related stuff.  HA is mostly local (with some cloud integrations of course), and everything you've mentioned should be local.  It would be very helpful if in picking something to concentrate on, pick something unrelated to the internet.  Don't access HA from the internet, access it locally (you are, right?  Not with something thru nabu casa or some proxy?

But... pick one thing that fails and figure out what you can.  MQTT (the service, not zigbee2mqtt) is pretty straightforward -- if HA can't talk to HA, do as I suggested and see if you can, see what happens.   But pick one thing.  And for us to help, try to stick with one specific failure, not jump around.  It will help a lot.

I can't think of any reason you can't map any number of SSID's to the same VLAN.  People do it all the time for different authentication for example.

I apologize for being dense but do you have an example of the actual problem occurring?

For example, if Home Assistant is supposed to connect to the MQTT server, do you have a log of that failing that shows HOW it connects?    Like from the integration page, showing it uses IP (vs name)?

If MQTT connection is the problem AND you are using static IP, we can stop talking about DNS.

Alternatively, run MQTT Explorer and enter the IP and credentials, e.g. as below, and use explicit IP addresses from a PC on that same network, and see if it can connect.  If it can't -- easy to debug.  If it can, see what's different about HA.

I really suggest using explicit IP addresses and not names if this is all internal and on the same network, as that takes mDNS and DNS out of the picture.

Any suggestions on how I could troubleshoot the issue using the diagnostic tools in opnsense?

At first glance, data between HA, NR and Mosquito would appear to be all local, on-net (i.e. same subnet, same VLAN) and so does not even pass through OPNsense.

@cookiemonster's question is good, but unless I have just missed a clue, everything you are saying implies OPNsense is not involved.  Can you go through your setup and think about it and what you changed and share any theory of even how OPNsense plays any role in the setup described?

For example, you have IP addresses -- is ANYTHING using DNS names, and maybe OPNsense is now the DNS server and different?

Is there any chance around the same time that something changed on the host -- is this HAOS?  Or if it's housed in your own linux box, did something like apparmour change?

Please don't take this wrongly, but so far it's kind of like saying "I turned on the back yard light and my toilet overflowed, what's wrong with my light".  :)

You have to find the logical connection between the two, then I think people can help debug what's wrong with that aspect.

If they are properly built.

I was looking at some GPS ntp units on amzon, they are built into small metal enclosure with long wired antenna, $300(us)+

No idea about those.  That's 5-10 times what some people spend doing it on the cheap, so definitely has more potential.

"You don't get what you don't pay for"   :)

Most people with HA get an MQTT server by installing the Mosquito add-on.  Be sure it's started, check it's visible, make sure you can ping from the HA instance to the Mosquito IP (should be the same, but check it pings) and see if the port is up, and as mentioned if it's connected via static IP make sure it didn't change.

MQTT Explorer can be helpful as you can connect independently to the MQTT server to make sure it's up.

GPS (receiver) is accurate to approx 100ns. Is the DCF77 broadcast any better?

If they are properly built.  The reason I add that is a lot of people are getting USB sticks and putting them in something like an rPi and expecting it to rival real GPS receiver accuracy.  Often they don't even have PPS connected, but even if they do (I built one and did) you still have the erratic and slow nature of a cheap-as-dirt USB system and rPi hardware.  YES, they may be better than the internet.  But don't if you expect a GPS source to have that sort of accuracy you need serious hardware with it.

Which is really moot as all the PC's and systems most people are going to hook up don't benefit from that kind of accuracy anyway.

But it's nice to have a source that stays up if the internet is down, and may be more accurate.  I built one, not knocking them at all, just have the right expectations.

Testing new firewall hardware with 10g ports, and a bit limited on 10g devices where I can test performance.

I installed the iperf plugin on 25.7.5, plugin version shows 1.0_2.

This is going to sound like a dumb question, but if I create a server (e.g. from the GUI), and then in the shell I run a separate client, what does this do:

Code Select Expand

# LAN interface is ixl0, 192.168.130.1
# VLAN 136 interface is ixl0_vlan136, address 192.168.136.1  (server created from gui at port 56650)
#
iperf3 -c 192.168.136.1 -p 56550 -t 10 -bind-dev ixl0

I was hoping this would pass traffic through the firewall logic to see if I could load it down, without worrying about NIC speed (I lack enough 10g devices to do a proper test at the moment).

What it does is report traffic passing at 48Gbps but the firewall logs show nothing, so I assume this is taking some shortcut.

Is there a way to force traffic to flow through the firewall portions of opnsense to see if I can load up the processor?

Or do I need actual, physical devices on the two interfaces (and limited to the interface speed)?

Or... put another way, is there a good way to put some stress on a new configuration artificially to make sure things are working well?

I should note that I've got hardware to test at 2.5g and that comes out just shy of 2.5gbps as I would expect.  I don't have anything handy to run linux on both ends at 10g.

Just to put this to bed in case anyone shows up here....

While turning off offloading and using the plugin seemed to work, this all seemed like a bad idea for the device I want to be very stable so...

The GMKTek box is on its way back, and I bought a Minis Forum MS-01, with real honest Intel adapters.

OPNsense comes up with no hacks, I can use the SFP ports and a DAC cable for a better connection.

I'm not a huge fan in Minis Forum due to their support reputation, but this actual piece of hardware looks like it may be a good choice.

And avoiding Realtek and going with Intel I think is a very good choice!

The Beelink I had running fine at 1g was going to be a backup, but I think I will relegate it to "waiting for a purpose" and use my ancient Z170 system with its X550-T2 new Intel adapter as my backup firewall, again getting interfaces that just work without patching.

So... if you show up here with Realtek stuff, just say "no" and move on.  They can work, but one's time and blood pressure are worth a change!

Does the EQi12 use realtek nic?

Yes.  per Beelink support some GTi's have intel:

To clarify regarding our GTi series:

GTi12 / GTi13 / GTi14 – Equipped with Intel I226-V 2.5G LAN

GTi15 – Equipped with Intel E610 10G LAN

I reran the install of the plugin, and I see there is a line in the UI between that message and the output from the plugin, so please ignore that aspect of the above.  In looking with fresh eyes, I think the main thing that may have helped me was sort of a disclaimer to clearly state it may not be about OPNsense.

I'm a bit surprised your Skylake is so high. Of course, my measurements and/or memory could be off.

I'm using three different UPS' for the measurements (due to where they PC's were), so it's also possible their readouts are not the best.