Messages

Hi there,

Since the update of Zenarmor to 1.12 I've some strange issues after rebooting my OPNsense appliance:

- Wireguard tunnels don't come up
My WG tunnels are not comming up. When I do a packet capture, I see incomming and outgoing packets on the WG tunnels. But the outgoing packets have all 0.0.0.0 as source IP.

- Name queries for Zenarmors rDNS are routed through the wrong interface
Zenarmors PTR request (for client names in the report), which should be send to my internal DNS resolver 10.0.1.6, are routed through the WAN interface with destination 10.0.1.6. Therefore reverese lookup is not working anymore. But only for Zenarmor. For all other clients and servers in my network DNS is running fine.

I have to disable the "Start on boot" switch for packet inspection to solve both issues. When I start packet inspection manual after reboot, everything works fine.

Any ideas?
Thanks.

Jas Man

Hi,

The reason for the loss of connectivity is that when Zenarmor packet engine opens the interface in the netmap mode, netmap re-initializes that interface, causing a DOWN/UP link event.
Seeing an interface DOWN/UP event, OPNsense fires IPv4/IPv6 address re-configuration. For IPv4, this takes milliseconds, bur for IPv6, due to auto-configuration, WAN tracking etc, this process might take about 15-60 seconds during which time you might lose WAN connectivity.
After this, everything should be back to normal.
We're aware of this issue, however the solution involves working with 3rd parties (netmap team, OPNsense team etc).
For the final solution, several options are on the table and we're working on them.
I hope this helps clarify the situation

So happy to read this explanation.  :)
I thought that I'd a missconfiguration in my OPNsense due to the long "outages" during IFs down/up.

Seems that this was caused by a faulty patch cable. The WAN IF went up and down, and because of the frequency the lighttpd run into a failure.
I've replaced the cable and since one week no issues so far.

Hey,
Since I've configured IPv6 on my OPNsense 22.7.4, the web GUI is sometimes not available.
According to the logs this happens after the rc.newwanipv6 has run due to a connection change, and the lighthttpd is restarted (can't bind to socket: 127.0.0.1:443: Address already in use).

System.log

Code Select Expand

<13>1 2022-09-17T09:05:34+02:00 FIREWALLNAME php 68729 - [meta sequenceId="45"] plugins_configure vpn (,wan)
<13>1 2022-09-17T09:05:34+02:00 FIREWALLNAME php 68729 - [meta sequenceId="46"] plugins_configure vpn (execute task : ipsec_configure_do(,wan))
<13>1 2022-09-17T09:05:34+02:00 FIREWALLNAME php 68729 - [meta sequenceId="47"] plugins_configure vpn (execute task : openvpn_configure_do(,wan))
<11>1 2022-09-17T09:05:34+02:00 FIREWALLNAME php 68729 - [meta sequenceId="48"] /usr/local/etc/rc.newwanipv6: Resyncing OpenVPN instances for interface WAN.
<13>1 2022-09-17T09:05:34+02:00 FIREWALLNAME php 68729 - [meta sequenceId="49"] plugins_configure newwanip (,wan)
<13>1 2022-09-17T09:05:34+02:00 FIREWALLNAME php 68729 - [meta sequenceId="50"] plugins_configure newwanip (execute task : dnsmasq_configure_do())
<13>1 2022-09-17T09:05:34+02:00 FIREWALLNAME php 68729 - [meta sequenceId="51"] plugins_configure newwanip (execute task : ntpd_configure_do())
<13>1 2022-09-17T09:05:34+02:00 FIREWALLNAME php 35924 - [meta sequenceId="52"] plugins_configure newwanip (execute task : opendns_configure_do())
<13>1 2022-09-17T09:05:34+02:00 FIREWALLNAME php 35924 - [meta sequenceId="53"] plugins_configure newwanip (execute task : openssh_configure_do(,wan))
<11>1 2022-09-17T09:05:34+02:00 FIREWALLNAME php 68729 - [meta sequenceId="54"] /usr/local/etc/rc.newwanipv6: The command '/usr/local/sbin/ntpd -g -c '/var/etc/ntpd.conf'' returned exit code '1', the output was ''
<13>1 2022-09-17T09:05:34+02:00 FIREWALLNAME php 68729 - [meta sequenceId="55"] plugins_configure newwanip (execute task : opendns_configure_do())
<13>1 2022-09-17T09:05:34+02:00 FIREWALLNAME php 68729 - [meta sequenceId="56"] plugins_configure newwanip (execute task : openssh_configure_do(,wan))
<13>1 2022-09-17T09:05:34+02:00 FIREWALLNAME php 68729 - [meta sequenceId="57"] plugins_configure newwanip (execute task : unbound_configure_do(,wan))
<13>1 2022-09-17T09:05:34+02:00 FIREWALLNAME php 35924 - [meta sequenceId="58"] plugins_configure newwanip (execute task : unbound_configure_do(,wan))
<13>1 2022-09-17T09:05:35+02:00 FIREWALLNAME php 68729 - [meta sequenceId="59"] plugins_configure newwanip (execute task : vxlan_configure_do())
<13>1 2022-09-17T09:05:35+02:00 FIREWALLNAME php 35924 - [meta sequenceId="60"] plugins_configure newwanip (execute task : vxlan_configure_do())
<13>1 2022-09-17T09:05:35+02:00 FIREWALLNAME php 68729 - [meta sequenceId="61"] plugins_configure newwanip (execute task : webgui_configure_do(,wan))
<13>1 2022-09-17T09:05:35+02:00 FIREWALLNAME php 35924 - [meta sequenceId="62"] plugins_configure newwanip (execute task : webgui_configure_do(,wan))
<11>1 2022-09-17T09:05:36+02:00 FIREWALLNAME php 35924 - [meta sequenceId="63"] /usr/local/etc/rc.newwanipv6: The command '/usr/local/sbin/lighttpd -f /var/etc/lighty-webConfigurator.conf' returned exit code '255', the output was '2022-09-17 09:05:36: (network.c.540) can't bind to socket: 127.0.0.1:443: Address already in use'

WebGUI log

Code Select Expand

<27>1 2022-09-17T09:05:35+02:00 FIREWALLNAME lighttpd 67897 - [meta sequenceId="1"] (server.c.2097) server stopped by UID = 0 PID = 14049
<27>1 2022-09-17T09:05:36+02:00 FIREWALLNAME lighttpd 16538 - [meta sequenceId="2"] (server.c.1588) server started (lighttpd/1.4.66)
<27>1 2022-09-17T09:05:36+02:00 FIREWALLNAME lighttpd 16538 - [meta sequenceId="3"] (gw_backend.c.378) child signalled: 1
<27>1 2022-09-17T09:05:36+02:00 FIREWALLNAME lighttpd 16538 - [meta sequenceId="4"] (gw_backend.c.378) child signalled: 1
<27>1 2022-09-17T10:52:08+02:00 FIREWALLNAME lighttpd 16538 - [meta sequenceId="1"] (gw_backend.c.283) establishing connection failed: socket: unix:/tmp/php-fastcgi.socket-1: Connection refused
<27>1 2022-09-17T10:52:08+02:00 FIREWALLNAME lighttpd 16538 - [meta sequenceId="2"] (gw_backend.c.283) establishing connection failed: socket: unix:/tmp/php-fastcgi.socket-0: Connection refused
<27>1 2022-09-17T10:52:08+02:00 FIREWALLNAME lighttpd 16538 - [meta sequenceId="3"] (gw_backend.c.993) all handlers for /index.php? on .php are down.

A restart of the web GUI (/usr/local/etc/rc.restart_webgui) fails too. I've to kill and start the lighthttpd process to restore it.
The web GUI is configured to listen on two interfaces only. Any thoughts what could be the issue here?

Jas

Thank you @franco.
I will close my feature request #5915 with your answer.

Hi again,

The reset of the counters happens every 15 minutes. It's caused by the scheduled filter reload script, which is executed every 15 minutes when a schedule rule is active. I created a schedule rule some weeks ago, and I didn't saw the connection between this two things until today.

I guess it is not a bug, more an expected behaviour. Not really nice, but OK when you know it.

Make it sense to open a feature request for this?

Hey,

I'm often looking onto the "Inspect" page of my firewall rules to check if a rule has been hit.
Now I noticed that the statistics get reset after a short time (1 or 2 hours). As far as I can remember the counters have been never reseted in the past, only if I reloaded the rules.

Is this a new behaviour in 22.1?

Jas Man

It seems that I've latency issues with some CDNs at site A.
Site B is using a different ISP, and they don't have this latency issues.

I was not able to find the cause so far.
And the only difference between the both location is, that the last three hops to the CDN servers are different. Therefore I would like to do some tests from location A over location B.

Hey,
I've connected two sites over a Wireguard VPN connection. There's an OPNsense at site A and a OpenWRT router at site B.
Everything works fine. Each site can reach the internal addresses of the other site over the WG tunnel.

Now I want to route a hugh amount of external addresses from site A over the Internet connection of site B.

I've tested it already successfully with an single IP address. I've added the single IP address to the WG peer configuration on the OPNsense as allowed IP, created a FW rule for the traffic to this address and choosed the WG gateway of site B as gateway.

But it's not very conveniant for more than a few addresses, because I've to add them all to the WG peer configuration as allowed addresses.
So I added 0.0.0.0/0 instead, but then I've two default routes on the OPNsense and the WG route is prefered. Every external traffic is routed to site B then.

Next try: I disabled the routes in the local WG configuration of the OPNsense to prevent routing entries for the WG tunnel allowed IP addresses. That works perfect at site A. But in this case, site B is not able to reach the clients at site A anymore. The packet arrives at the host at site A, but the answer packets are routed back over the default route of site A, means to the Internet.

Did I missed something? Or what would be the preffered solution for my needs?

Thanks.
Jas Man

Hey sy,
The bypass mode didn't changed the behaviour.

I could imagine that Zenarmor is having a problem when the WG interfaces disappear during the service restart.

I've found the reason for the WG service reloads: I'd configured a Monit task which restarted the service after some ping timeouts.  ::)

Not sure if anybody was able to reproduce the issue. But I'm asking myself where I've to report this issue? Is it an OPNsense, Wireguard or Zenarmor issue?

I've to correct my first findings.
It's not Suricata only which causing the issue. The Wireguard interfaces causing the high load too, when the WG service stops or reloads.
Each WG IF causes a high load of an eastpect instance. If I remove all WG IFs from the Zenarmor configuration, the high load doesn't appear anymore.

I can't find a connection between the IPS rule update and the restart of the WG service. It seems that Suricata doesn't restart the WG service when the rules are updated.


Any guesses?

Hi,
I noticed a recuring issue with the two processes "eastpect: Eastpect Instance 1 (eastpect){Eastpect Main Event}" and "eastpect: Eastpect Instance 2 (eastpect){Eastpect Main Event}". After some time both processes are producing 100% CPU load on two of four processors on my OPNsense appliance. I can only solve this by restarting Zensei or the whole appliance.

I've already found out, that this occures when Suricata has downloaded and installed the ET ruleset (https://rules.emergingthreats.net/open/suricata-6.0/emerging.rules.tar.gz), which is updated daily.
But the issue occures not every day. Sometimes it works for four or five days, and sometimes it happens on two or three consecutive days.

Has anybody noticed the same issue?
I've already opened a ticket at Sensei, but they don't know this issue.

Jas

EDIT: Changed the subject due to new findings

Yep, I can confirm this.

After the update Sensei said that it needs to start "Elasticsearch DB" to show the reports. When I clicked "Yes" to confirm, the process "eastpect" consumed permanently 100% CPU, and the "There was an error" message on the dashboard appeared, which showed me the "PHP deprecated" message.

Same here!
Thank you  :)