Messages

Actual state:
- Its isolated to Q35 chipset, i440x works fine
- 24.1 has multiple issues (restarts not working, high cpu usage)
- 23.7 seems to only have high cpu usage but reboots working fine

Actually its unknown if its q35 in general or if its just the machines thats getting provided by our hosting provider.

What I hadn't really noticed is that the system is permanently running at over 50% CPU utilization, while an identical older system is somewhere between 1-10%.

Code Select Expand


root@gw:~ # ps aux | sort -nrk 3,3 | head -n 10

USER      PID  %CPU %MEM   VSZ   RSS TT  STAT STARTED       TIME COMMAND
root        6 100.0  0.0     0    16  -  RL   14:40   1187:40.25 [rand_harvestq]
root       11  96.0  0.0     0    32  -  RNL  14:40   1155:22.87 [idle]
unbound 81072   0.0  1.0 71624 41988  -  Is   10:04      0:00.20 /usr/local/sbin/unbound -c /var/unbound/unbound.conf
root    96080   0.0  0.1 13572  2924  -  S    10:04      0:00.18 /bin/sh /var/db/rrd/updaterrd.sh
root    95968   0.0  0.1 12728  2128  -  Is   10:04      0:00.00 daemon: /var/db/rrd/updaterrd.sh[96080] (daemon)
root    95531   0.0  0.1 12748  2180 v7  Is+  14:40      0:00.00 /usr/libexec/getty Pc ttyv7
root    94636   0.0  0.1 12748  2180 v6  Is+  14:40      0:00.00 /usr/libexec/getty Pc ttyv6
root    94055   0.0  0.1 12748  2172 v5  Is+  14:40      0:00.00 /usr/libexec/getty Pc ttyv5
root    93937   0.0  0.1 12748  2180 v4  Is+  14:40      0:00.00 /usr/libexec/getty Pc ttyv4
root    93515   0.0  0.1 12748  2176 v3  Is+  14:40      0:00.00 /usr/libexec/getty Pc ttyv3


Code Select Expand


root@gw:~ # vmstat 2 5


procs    memory    page                      disks     faults       cpu
r  b  w  avm  fre  flt  re  pi  po   fr   sr da0 cd0   in   sy   cs us sy id
1  0  0 838M 1.9G 1.2K   0   0   0 1.1K   11   0   0    4  583  115  1 50 49
0  0  0 853M 1.9G  16K   0   0  10  17K   21   1   0    3 6.5K  335  4 54 42
0  0  0 838M 1.9G 7.2K   0   0   7 7.9K   20  27   0   18 1.4K  205  1 51 48
0  0  0 838M 1.9G   82   0   0   0  105   20   0   0    2   93  101  0 50 50
0  0  0 841M 1.9G  13K   0   0   0  14K   20   0   0    2 4.4K  257  2 53 45


Code Select Expand


root@gw:~ # top -b -o cpu -n 10 | head -n 20
last pid: 55468;  load averages:  1.17,  1.10,  1.05  up 0+19:51:44    10:32:12
45 processes:  1 running, 44 sleeping
CPU:  1.0% user,  0.0% nice, 50.4% system,  0.0% interrupt, 48.6% idle
Mem: 59M Active, 1228M Inact, 634M Wired, 386M Buf, 1901M Free
Swap: 8192M Total, 8192M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
18456 root          1  20    0    26M    14M select   1   0:13   0.00% python3.11
9919 root          3  20    0    40M    13M kqread   1   0:12   0.00% syslog-ng
  224 root          1  20    0    86M    39M accept   1   0:10   0.00% python3.11
53977 root          1  20    0    22M    11M kqread   1   0:07   0.00% lighttpd
55066 root          1  52    0    63M    32M accept   1   0:06   0.00% php-cgi
85232 root          1  20    0    27M    15M select   1   0:03   0.00% python3.11
19307 root          1  52    0    61M    30M accept   1   0:02   0.00% php-cgi
40421 root          1  52    0    59M    30M accept   1   0:01   0.00% php-cgi
56995 root          1  52    0    13M  2512K nanslp   1   0:01   0.00% cron
62065 root          1  52    0    57M    30M accept   1   0:01   0.00% php-cgi




I also noticed something today when setting up the OpenVPN, although both systems - the defective one and the productive working one - have the same patch level:

OPNsense 24.1.9_4-amd64
FreeBSD 13.2-RELEASE-p11
OpenSSL 3.0.14

A different client configuration section in VPN -> OpenVPN -> Servers - have a look at the screenshots. productive.png is the working system - malfunctional.png is the defective system.

I've tested it now on another new system with the same ISO, just to be sure:
- same behavior

Took a new download of the iso, got it uploaded by support, tested from the new ISO instead of Hetzners ISO:
- same behavior

I have another production system which works fine on the same version - but its already month old.

//Edit
I have compared the two config.xml files - from the productive system and the new system. Apart from a few deliberate changes in the production system, such as other NTP servers, an OpenVPN instance and so on, there are no differences. In terms of configuration, the systems are basically the same.

The production system is also located on the same server_type from the cloud, so the processor etc. is also identical.

I used both the ISO from Hetzner and one I pulled myself from the mirror. Both have the same problem. (The hash is identical anyway)

In the attachments the result of "ps axl" on a running system. As the system is freezed on the issue, i cant get the result of the command in the malfunctional state.

Hey everyone,

got a new instance up and installed OPNsense - on the final stage where i have to change root password and/or reboot, i choosed reboot but the system cant gracefull restart - it freezes on the message "some processes would not die; ps axl advised".

Reproduce:
- Fresh instance with WAN DHCPv4, no LAGGs,VLANs,LAN - right after installation on the completing installation step:
- - choosed Reboot - dont changed rootPW
- - The freezing console output and malfunction starts right after stopping the lighttpd
- - on sending a STRG+ALT+ENTF it starts with "Invoking stop script 'beep'" till "Invoking stop script 'config'
- - after some seconds "some processes would not die; ps axl advised"
- - console tty1 is frozen
- - webui is already down

- manual restart of instance
- server boots up
- Login UI -> Wizard
- - WAN DHCPv4, no LAN, Europe/Berlin timezone, changed root PW
- Restart from WebUI
- same behavior as when the install was done and it was time to reboot
- - ui shows "The system is rebooting now, please wait... "

So i manually restarted again - the version is:
OPNsense 24.1-amd64
FreeBSD 13.2-RELEASE-p9
OpenSSL 3.0.12

So i had run an "update from console". After the update, the machine wanted to restart - and yeah - the reboot worked fine.

We are now on version:
OPNsense 24.1.9_4-amd64
FreeBSD 13.2-RELEASE-p11
OpenSSL 3.0.14

So i checked a last time rebooting from ui after the successfull update and restart, but issue still persists with a reboot from UI.

So i rebooted again manually and tested a reboot from console (8 - Shell -> "reboot" command):
Same behavior: freezes, sending STRG+ALT+ENTF - Then the invoke scripts till config, some seconds after config again "some processes would not die; ps axl advised".

The instance is located in the Hetzner Cloud, where I already have several OPNsense systems running fine - same version, same instance type, same networking.

Would be nice if someone can help me find the issue.

Best regards

Hey everybody,

just wanna make sure i do not miss anything - is it correct, that DNAT actually isnt supported via API? SNAT is supported by the os-firewall which is already implemented to the core - but cant find any related to DNAT.

Just found this from January:
https://forum.opnsense.org/index.php?topic=38085.0

Seems like in January os-firewall was not in the core and thats just 6 months - perhaps SNAT has already found its way into the API and i'm just missing something?

Best regards

solved - Virtual IP was added, but not applied.

Hey everyone,

i'm having issues in understanding of Outbound NAT rule creation over Source NAT.

actually using:
OPNsense 24.1.9_4-amd64
FreeBSD 13.2-RELEASE-p11
OpenSSL 3.0.14

In the past i created my Outbound NAT rules in Firewall -> NAT -> Outbound. Actually we are on the way to automate things and further the rule creation after server creation. From the OPNsense API, as I see it so far, I have no possibility to create rules in FW -> NAT -> Outbound, but only in FW -> Automation -> Source NAT.

I've created a Source NAT Rule on WAN interface for translating the source (private ip) to the target (public ip). But it seems, that this rule isnt above the rules in the FW -> NAT -> Outbound, cause the general Outbound Rule in FW -> NAT -> Outbound matches.

I'm a bit scared of moving the general Outbound Rule to Source NAT, which translates everything from LAN net to FW public IP which wasnt already translated so the internal systems without a own NAT to Public IP have access to the www, cause i would lost access to the site if it does not work and would need to drive to it (Site is still in work and not connected actually via Site-to-Site to our main site).

Maybe someone can give me hint?

Best regards

[I would appreciate it if someone could give me a hand - I'm running out of ideas.]

Drove to Site B....Isolated the allow all rule until i found whats missing.

How i analyzed:
- Created Rule on Site B for all interfaces, any direction, all Networks & IPs to every destination
- Eliminated Network and IPs from that rule, until i found the Site B WAN address
- Eliminated Interfaces until i found its Site B Net A
- Ping from Site-A-OpenVPN-net to Site-B-A-net
- Checked Live log: Matched to: LAN Interface, Direction out, Source Site B WAN address, destination pinged server

I dont think that we need such rule - do you?

We have on Site B NAT Outbound Rule for LAN interface which translates to the Site B WAN address. Sounds more like a NAT issue?

I would appreciate it if someone could give me a hand - I'm running out of ideas.  :-[

Tiny update:
We added the Phase 2 for the OpenVPN network on Network B - it was missing. But dont changes the problem, that we actually need a floating rule on all Interfaces which allows as source all Networks from Network A & B.

There are connections, which use that rule, instead of the correct created ones. As i said i checked the Live View Logs from the firewall and there are connections, which are already allowed by the correct Rules from IPSec interface but only match to the Floating Rule. Without the floating rule we cant reach Network B from Network A.

Example:

I try to connect to a remote terminal server (windows server) via rdp, the following entry in the live view:
Interface:IPsec
Dir: in
Protocol: TCP
Source: 10.242.0.4 (OpenVPN Network - Client IP of my computer)
Destination: 10.0.0.33 (Internal IP of terminal server)

And in the same second there is a outbound:
Interface: LAN
Dir: out
Protocol: TCP
Source: 4.3.2.1 (WAN address of Site B)
Destination: 10.0.0.33 (Internal IP of terminal server)

So i would need the following rules on Network B:
Type: Pass
Interface: IPsec
Dir: In
Protocol: TCP
Source: OVPN net
Destination: LAN net

and

Type: Pass
Interface: LAN
Dir: out
Protocol: TCP
Source: WAN address
Destination: LAN net


Both these rules exist....

----------------------------

I may have found the solution:
We didn't create rules in the IPSec interface that had their own networks as destination and any as source, as in the instructions, but always used network against network so that a source was always defined.

However, since the IPSec interface is used for more than just network against network, I have now changed the rules so that the source is any.

However, I cannot test this (remove the floating allow all rule) as long as I am not present in Site B, as otherwise Site B may not be accessible in the worst case.

I will report back with my findings at the beginning of next week when I am on site B.

There's a dedicated forum for VPN questions

I wasn't sure if the problem was specific enough to IPsec Issue for the forum section.

What OPNsense version(s) are you using ?

OPNsense 24.1.8-amd64
FreeBSD 13.2-RELEASE-p11
OpenSSL 3.0.13

Are you using Policy or Route based IPsec ?

https://docs.opnsense.org/manual/how-tos/ipsec-s2s.html

How does your topology look like ?

In what way - is that not clear enough from the description above?

- 2 sites, OPNsense, with Public IP

Site A networks: 5 private networks incl. OpenVPN network
Site B networks: 2 private networks

Both sites connected using this IPSec method:
https://docs.opnsense.org/manual/how-tos/ipsec-s2s.html

Does Router A allows both UDP 500 and ESP traffic from destination B on WAN interface ?

Both allowing each other ESP, UDP 500, UDP 4500.

Hey folks,
long story short:

Problem:
Network A does not receive the responses from network B

RCA:
- The networks are connected via IPSec IKEv2 site-to-site
- Phase 1 proposal and authentication are the same here
- Both have a static public IPv4 address
- Network A has 5 local networks in phase 2
- Network B has 2 local networks in phase 2
- Network A has 10 entries in phase 2: Each network from A (local) to each network in network B (local)
- There are 8 entries on network B in phase 2: Each network from B (local) to each network in network A (local)
- Network A operates the OpenVPN. This has its own additional network, which is why there are also 2 more phases so that they come to both local network B networks
- Network A IPSec interface rules from the networks of A and B each as source and destination
- Network A IPSec interface rules from the networks of B and A each as source and destination
- IPSec tunnels are available on both sides according to the status overview
- Network A ping to network B IPs not possible
- Network B ping to network A IPs possible

Found discrepancies - does an allow all rule fix them?
- An allow all rule works - is it because some rules are missing?

Drill deeper:
- I have created a description in the allow all rule that I can filter in Live View
- I created a ping that was not possible before and filtered it out in Live View with allow all rule

- I have analyzed this and found that the following type of rule is needed:
- - Interface: IPSec
- - Type: Pass
- - Direction: in
- - Source: net1 Network A
- - Destination: net1 Network B

Oh wonder of wonders - exactly this rule exists...

After lengthy research, we can't understand why the IPSec interface rules in network B don't seem to work. They are all enabled and having no special configurations - just interface, dir, source, destination.

Any ideas?

Solved:

Had to extend the client configuration with "allow-pull-fqdn" and add the following to the Advanced settings:

Code Select Expand

push "route DOMAIN.TLD 255.255.255.255 VPN_GATEWAY_IP"

Hey folks,

we have a customer which development site is hosted on azure and the ip address changes often. We are only able to access this development site if we are coming from our VPN ip.

Is there a way to create a outbound rule for a url instead of a ip/host, so when a vpn connected employee visits the site, his Outbound IP is the VPN IP?

I thought it will may work to create a URL Alias (Firewall -> Aliases) which has the url as content and then create a Outbound NAT rule where source is the VPN net, destination is the URL alias and translation is the VPN ip.

Is that something that could work?

Best regards

Hey folks,

actually i'm setting up OPNsense for my private projects (not the one for business) and i got a bit confused on dns setup. I want to use the dns resolver (unbound) as dns nameservers for the servers in lan.

OPNsense 24.1.8-amd64
FreeBSD 13.2-RELEASE-p11
OpenSSL 3.0.13

Unbound is activated and i created the LAN firewall rule so that the server on lan (10.0.1.2) can access the opnsense as dns-nameservers (10.0.1.1) on destination port 53. Further i configured the server on lan (10.0.1.2) to use the opnsense (10.0.1.1) as dns-nameservers. I can see within the firewall live log, that this access works.

But i can see further, that the server on the lan (10.0.1.2) tries to access remote dns servers of my cloud provider. Shouldnt it just access the opnsense and the opnsense tries to resolve over Outbound NAT the dns servers of my cloud provider and ping back the resolution of this to the server on the lan?

Maybe I'm just misunderstanding something or have configured something incorrectly somewhere?

Best regards

Seems that any update broke syslog-ng package - reinstalled it and logs working again.