Firewall randomly going down.

[dcol]

This appears in the log when the firewall goes down. This happens 4-8 times a day.
Funny thing is. this system does not use IPv6 at all, so where could this come from? What does this mean?
Firewall uses a static IP on the WAN. No DHCP

2023-08-15T10:34:33-04:00   Error   opnsense   /usr/local/etc/rc.linkup: The command '/bin/kill -'TERM' '77804''(pid:/var/dhcpd/var/run/dhcpdv6.pid) returned exit code '1', the output was 'kill: 77804: No such process'

Any help is greatly appreciated.

[franco]

Disable WAN IPv6 and LAN tracking?

I doubt these are related. It's the same benign message as in your other thread.


Cheers,
Franco

[dcol]

I was looking for reasons the firewall was disconnecting from the internet randomly throughout the day and saw this in the logs which corresponds with the outage times. If these messages are benign then I will look for other causes. The internet closes down for 2-3 minutes every time it happens. I was looking to see what was causing this.
IPv6 is not used in this firewall

[CJ]

What type of internet connection?  Do you have gateway monitoring enabled?

[dcol]

No gateway monitoring. This is a very basic firewall setup with only one static IPv4. Goes down 4-8 times a day for 2-4 minutes at a time. The ISP said it isn't them.

[jclendineng]

If you happen to have breezeline there are multiple people in Columbus Ohio area with ISP related issues and of course they will always tell you there isn’t an issue. Just FYI your ISP is ALWAYS lying to you, always assume ISP issue. I’ll say this looks a lot like just that, I’d always suggest disabling all v6 in WAN interface.

[dcol]

This user has Comcast. IPv6 is disabled.

[franco]

Why would it try starting DHCPv6 then? It sounds like IPv6 globally disabled but not in WAN and LAN.


Cheers,
Franco

[CJ]

No gateway monitoring. This is a very basic firewall setup with only one static IPv4. Goes down 4-8 times a day for 2-4 minutes at a time. The ISP said it isn't them.

I meant do you have cable, fiber, etc.  Line noise on something shared like a cable connection can cause symptoms like you're describing but are too transient for an ISP to notice.  That's why I suggested setting up gateway monitoring.  This will also help you determine if the connection is completely down or just a lot of packet loss.

[dcol]

Went down twice last night between Midnight and 3AM. For about one minute each time.
I turned on GW monitoring. Shows RTT-2.2ms RTTd-.3ms and Loss at 0%
This is Comcast cable. We are trying to get them to put in fiber since they have it everywhere around them.
My 4 other locations do not have this issue and Comcast says it is not them, of course.

I put in a new firewall about 3 months ago because of this issue, so that pretty much eliminates the firewall except I am using the same config downloaded from the old one. I do not see anything suspicious in that file.

The firewall log only shows the same thing over and over again and happens during the outages. I want to get rid of this error. so I know if it is the cause of the outages.
2023-08-16T12:12:06-04:00   Error   opnsense   /usr/local/etc/rc.linkup: The command '/bin/kill -'TERM' '77804''(pid:/var/dhcpd/var/run/dhcpdv6.pid) returned exit code '1', the output was 'kill: 77804: No such process'

This is my only clue. It shows the pid coming from dhcpdv6 and happens every time during these outages. Franco says this message is benign, but it happens at the same time as the outages. I am not sure if this error is the cause or result of the outages
Any ideas

[franco]

First thing I would do is look for dmesg clues. If the interface is detached for example it's already an outside factor. Otherwise a full log during such a cycle would shed more light on it. Individual errors are likely not going to reveal the underlying issue. It could also be short lease times, defunct IPv6 rollout on the ISP side (I mentioned making sure to disable IPv6 modes in WAN and LAN for example), etc.

[dcol]

The error I mentioned shows up whenever the interface goes down.
I looked at dmesg from the console and it has a ton of these
igc0: link state changed to UP
igc0: link state changed to DOWN
igc0: link state changed to UP
igc0: link state changed to DOWN
igc0: link state changed to UP
igc0: link state changed to DOWN
igc0: link state changed to UP
igc0: link state changed to DOWN
igc0: link state changed to UP
igc0: link state changed to DOWN
igc0: link state changed to UP
igc0: link state changed to DOWN
igc0: link state changed to UP
igc0: link state changed to DOWN
igc0: link state changed to UP
igc0: link state changed to DOWN
igc0: link state changed to UP
igc0: link state changed to DOWN
igc0: link state changed to UP
igc0: link state changed to DOWN
igc0: link state changed to UP
There is no timestamp on anything here so I have no idea when these happened
igc0 is the WAN interface

[franco]

That's the equivalent of someone unplugging the WAN cable and plugging it back in so we are getting closer to observing what you describe.

What's plugged into the WAN port? A modem or a router (bridge mode or routed).

Is this the full snippet from dmesg or filtered for these events? Just in case Suricata/Zenarmor is involved via Netmap.


Cheers,
Franco

[dcol]

A Comcast modem is plugged into the WAN port. This is filtered from dmesg. The display was too large to upload so I posted the most recent dsmeg info. See attached
The 4 NIC ports are built-in to the mini-pc
Thank you Franco, I really appreciate your assistance!

[franco]

Ok so Zenarmor or Suricata IPS running here. Could you try without and see? igc(4) is pretty new and we've had a number of reports that it can have issues with Netmap. There is also the emulated mode that could be tried...

If it's not Netmap the link flaps because of the modem. The quickest way to verify is to add a switch between the modem and the WAN port so that the WAN port will always be physically up.

If the outages still happen then it's the ISP or the modem.


Cheers,
Franco