Messages

On Tuesday the WAN IP address changed at one site. My test that time was to just change the IP address in the PSK and the Connection. That worked fine. The VPN link came back up.


Today, Thursday, two of the remote sites (sites C-to-E)stopped talking. In addition my remote connection to the remote site (sites D-to-E) has been down. Granted my drop was more like a week ago. I just haven't needed access to that site over that time frame. A third connection remote to remote has been up the entire time (sites B-to-E).  I tried a few things, nope. Then I deleted and rebuilt the VPN links (C-to-E and D-to-E). Everything is working again.

More questions...

• I'm using PSK - is that reliable? should I be using key pairs?
• In the Legacy style connection you set a Phase1 and Phase2 lifetime, after that time expired a new IPSEC connection/key would be generated. With the new connection I see no place to set the life. Can the key's life expire but since the connection is established it continues to work until a reboot on one side?
• I don't think I am doing anything wrong when building the connection. It works immediately when it is brand new. So, any ideas?

Today's testing was one of the remote sites dropped their IPSEC VPN

• I tried going into IPSEC> Connections and de-selected the Enable option; I gave it 5-7 mins, hoping the a new timed key would regenerate, enabled both sides; nope, still bad
• I tried changing just the PSK (just the key, not the IDs) on both sides; nope, still bad
• restarted the IPSEC service on the remote site's side a couple times; nope, still bad
• gave up, deleted the PSK and the Connection; rebuilt on both sides; back up

Currently I'm using the 'default' in proposal. Someone said if you have an Intel nic you can select the whole AES256GCM16 type of encryption and it will offload the processing to the nic, not the CPU. Does anyone know if this it true? When AI'd if the i210AT supports this mega-mind said NO, then I rephrased the question and it said YES. I don't see this support in Intel's doc so I'm guessing no.

Sure enough, yesterday was a maintenance day. I took the main router down for 20mins. When I got home my VPN to the main site was offline. Tried a couple changes, nothing. Completely deleted the PSK and the Connection on both sides, restarted the VPN/IPSEC service on my side, rebuilt on both sides, and it connected.

Sadly, my home office also has a link to a remote site (not main) which has been up and working for weeks is now down.

I believe I built the VPN properly, it seems to work for a few weeks or so without issue. Then I reboot a router on one side of the VPN and I cannot get the IPSEC back up. I've kinda resolved to deleting both sides, restarting the IPSEC service on both routers, and then rebuilding the VPN again.

Big picture, I'm trying to convert the Legacy VPNs over to the new Connection type. This VPN issue is my big hold-up. I can't get a VPN that survives a reboot.

Mainly it is 6 routers connecting to a main office. Some of the remotes also connect to each other. The multi-connected routers can have one new Connection style VPN that works while another does not.

The log file shows

• generating IKE_AUTH response 1 [ N(AUTH_FAILED) ]
• received AUTHENTICATION_FAILED notify error
• received 1 cert requests for an unknown ca

The setup is rather basic.

• build a PSK on each side with matching key
• build a new VPN> IPSEC> Connection
• the Parent/phase1 is only updated with a description
• the Child/phase2 has the local segment IPs and encryption scheme, all other info is default/empty
• some VPNs (Child/phase2) connect to multiple segments on one side, most are single segment
• some public IPs are static with a DNS entry somewhere vpn.acme.com or vpn.roadrunner.com
• other public IPs are DHCP from ISP, these are linked to DDNS like coyote.duckdns.org

The hardware is old - AMD 5350 CPUs and HP NC364T quad LAN cards (Intel 8253?) and APU-2E4 AMD Embedded GX-412TC i210AT.

• is the issue the encryption scheme doesn't work with these devices
• with 'Legacy' VPN configuration they work fine
• most OPNsense are 25.7_9
• my home router is 26.1_10

A bunch of questions that have come up from my troubleshooting

Should LocalID be the same as LocalIP?

• I've read some posts which say it doesn't matter
• others that say they need to be different
• I haven't been able to get the VPN to work using a FQDN like vpn.acme.com

Should PSK be PSK or EAP?

• one post said the failure (AUTH_FAILED) is tied to no address info in a PSK and to try EAP
• I did change one link to EAP on both sides, no change, still failed

Do I need to import a cert or ca into Trusts then select it in the Parent/phase1?

Again the big frustration is everything works, data flows back and forth until a reboot.

Yup, Memtest is on the list but I can't take the server offline until this weekend. I'm just looking for as many things to test in my maintenance window as possible.

I will also check BIOS installed vs BIOS available.

And, yes, I'm completely in the wrong forum. I meant to post in a Proxmox forum but I was online chasing down an OPNsense VPN setup question. Wrong tab, my brain hurts.

Besides extreme heat on the X570D4U chipset, I've been rather happy with the Asrock boards - X470D4U and X570D4U. I've been running the X470D4U with AMD 3700X Ryzens. Most boxes are running 6-8 vms.

Please review this excerpt from my Proxmox log. Prox is 8.4.19, the mobo is Asrock X470D4U, the RAM is 128GB (4x32GB) Nemix.

I have 5 other servers similar to this one working fine. The exception is one other server with the same HARDWARE ERROR that pops up on the console. I believe that one also has Nemix RAM. I believe the other four servers have Kingston KSM26ED8/16ME RAM but no errors.

12:37 Hardware Error
13:21 Hardware Error
14:53 Hardware Error
15:45 server reboot

I also see a SMART thermal message but that seems like more of a 'notice'.

Under the Prox log is an output from 'dmidecode -t 17'. While it lists specs it doesn't list actual manufacturer part number. I believe the RAM is actually 3200 speed but running at a lower 2666 speed. I tried an 'lshw -C memory' but lshw is not installed.


------------------------------------------ start some Proxmox log dump
Jun 30 12:37:21 virt09b kernel: mce: [Hardware Error]: Machine check events logged
Jun 30 12:37:21 virt09b kernel: [Hardware Error]: Corrected error, no action required.
Jun 30 12:37:21 virt09b kernel: [Hardware Error]: CPU:0 (17:71:0) MC17_STATUS[-|CE|MiscV|AddrV|-|-|SyndV|CECC|-|-|Scrub]: 0x9c2041000000011b
Jun 30 12:37:21 virt09b kernel: [Hardware Error]: Error Addr: 0x0000000bbf588300
Jun 30 12:37:21 virt09b kernel: [Hardware Error]: IPID: 0x0000009600050f00, Syndrome: 0x000000040a801101
Jun 30 12:37:21 virt09b kernel: [Hardware Error]: Unified Memory Controller Ext. Error Code: 0
Jun 30 12:37:21 virt09b kernel: EDAC MC0: 1 CE Cannot decode normalized address on mc#0csrow#1channel#0 (csrow:1 channel:0 page:0x0 offset:0x0 grain:64 syndrome:0x4)
Jun 30 12:37:21 virt09b kernel: [Hardware Error]: cache level: L3/GEN, tx: GEN, mem-tx: RD
Jun 30 12:43:12 virt09b smartd[1543]: Device: /dev/sdb [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 67 to 66
Jun 30 13:04:37 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 13:08:53 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 13:17:01 virt09b CRON[151843]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jun 30 13:17:01 virt09b CRON[151844]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jun 30 13:17:01 virt09b CRON[151843]: pam_unix(cron:session): session closed for user root
Jun 30 13:19:41 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 13:21:03 virt09b kernel: mce: [Hardware Error]: Machine check events logged
Jun 30 13:21:03 virt09b kernel: [Hardware Error]: Corrected error, no action required.
Jun 30 13:21:03 virt09b kernel: [Hardware Error]: CPU:0 (17:71:0) MC17_STATUS[-|CE|MiscV|AddrV|-|-|SyndV|CECC|-|-|Scrub]: 0x9c2041000000011b
Jun 30 13:21:03 virt09b kernel: [Hardware Error]: Error Addr: 0x0000000bbf588300
Jun 30 13:21:03 virt09b kernel: [Hardware Error]: IPID: 0x0000009600050f00, Syndrome: 0x000000040a801101
Jun 30 13:21:03 virt09b kernel: [Hardware Error]: Unified Memory Controller Ext. Error Code: 0
Jun 30 13:21:03 virt09b kernel: EDAC MC0: 1 CE Cannot decode normalized address on mc#0csrow#1channel#0 (csrow:1 channel:0 page:0x0 offset:0x0 grain:64 syndrome:0x4)
Jun 30 13:21:03 virt09b kernel: [Hardware Error]: cache level: L3/GEN, tx: GEN, mem-tx: RD
Jun 30 13:43:12 virt09b smartd[1543]: Device: /dev/sda [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 67 to 66
Jun 30 13:45:16 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 13:47:15 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 13:50:08 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 13:54:31 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 13:57:41 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 14:05:23 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 14:06:44 virt09b systemd[1]: Starting systemd-tmpfiles-clean.service - Cleanup of Temporary Directories...
Jun 30 14:06:44 virt09b systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Jun 30 14:06:44 virt09b systemd[1]: Finished systemd-tmpfiles-clean.service - Cleanup of Temporary Directories.
Jun 30 14:06:44 virt09b systemd[1]: run-credentials-systemd\x2dtmpfiles\x2dclean.service.mount: Deactivated successfully.
Jun 30 14:17:01 virt09b CRON[172546]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jun 30 14:17:01 virt09b CRON[172547]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jun 30 14:17:01 virt09b CRON[172546]: pam_unix(cron:session): session closed for user root
Jun 30 14:27:29 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 14:28:37 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 14:35:25 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 14:43:12 virt09b smartd[1543]: Device: /dev/sda [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 66 to 67
Jun 30 14:44:19 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 14:45:10 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 14:53:09 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 14:53:53 virt09b kernel: mce: [Hardware Error]: Machine check events logged
Jun 30 14:53:53 virt09b kernel: [Hardware Error]: Corrected error, no action required.
Jun 30 14:53:53 virt09b kernel: [Hardware Error]: CPU:0 (17:71:0) MC17_STATUS[-|CE|MiscV|AddrV|-|-|SyndV|CECC|-|-|Scrub]: 0x9c2041000000011b
Jun 30 14:53:53 virt09b kernel: [Hardware Error]: Error Addr: 0x0000000bbf520300
Jun 30 14:53:53 virt09b kernel: [Hardware Error]: IPID: 0x0000009600050f00, Syndrome: 0x000000040a801101
Jun 30 14:53:53 virt09b kernel: [Hardware Error]: Unified Memory Controller Ext. Error Code: 0
Jun 30 14:53:53 virt09b kernel: EDAC MC0: 1 CE Cannot decode normalized address on mc#0csrow#1channel#0 (csrow:1 channel:0 page:0x0 offset:0x0 grain:64 syndrome:0x4)
Jun 30 14:53:53 virt09b kernel: [Hardware Error]: cache level: L3/GEN, tx: GEN, mem-tx: RD
Jun 30 14:56:46 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 15:06:35 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 15:07:25 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 15:08:03 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 15:13:12 virt09b smartd[1543]: Device: /dev/sda [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 67 to 66
Jun 30 15:15:21 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 15:17:01 virt09b CRON[193198]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jun 30 15:17:01 virt09b CRON[193199]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jun 30 15:17:01 virt09b CRON[193198]: pam_unix(cron:session): session closed for user root
Jun 30 15:38:08 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 15:45:39 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
-- Reboot --
Jun 30 15:47:33 virt09b kernel: Linux version 6.8.12-30-pve (build@proxmox) (gcc (Debian 12.2.0-14+deb12u1) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-30 (2026-06-11T10:10Z) ()
Jun 30 15:47:33 virt09b kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-6.8.12-30-pve root=/dev/mapper/pve-root ro quiet
Jun 30 15:47:33 virt09b kernel: KERNEL supported cpus:
Jun 30 15:47:33 virt09b kernel:   Intel GenuineIntel



------------------------------------------ end some Proxmox log dump








------------------------------------------
dmidecode -t 17 show

Memory Device
        Array Handle: 0x0014
        Error Information Handle: 0x0021
        Total Width: 72 bits
        Data Width: 64 bits
        Size: 32 GB
        Form Factor: DIMM
        Set: None
        Locator: DIMM 0
        Bank Locator: P0 CHANNEL B
        Type: DDR4
        Type Detail: Synchronous Unbuffered (Unregistered)
        Speed: 2666 MT/s
        Manufacturer: Unknown
        Serial Number: 5D270016
        Asset Tag: Not Specified
        Part Number: Unknown
        Rank: 2
        Configured Memory Speed: 2666 MT/s
        Minimum Voltage: 1.2 V
        Maximum Voltage: 1.2 V
        Configured Voltage: 1.2 V
        Memory Technology: DRAM
        Memory Operating Mode Capability: Volatile memory
        Firmware Version: Unknown
        Module Manufacturer ID: Unknown
        Module Product ID: Unknown
        Memory Subsystem Controller Manufacturer ID: Unknown
        Memory Subsystem Controller Product ID: Unknown
        Non-Volatile Size: None
        Volatile Size: 32 GB
        Cache Size: None
        Logical Size: None

------------------------------------------

Hey,

I'm not clear if you've rebooted just the router or the router and the cablemodem. For Cox in NV, you MUST power cycle the cablemodem if there is a hardware change (I'm guessing an update to the router's MAC address in the cablemodem?). Did you change any hardware or the assigned interface?

This is a broad question. I'm trying to figure out where to migrate my connections.

My IPSEC site-to-site are now labeled as legacy. There is a new connection methodology for that tech. There is also Wireguard as a methodology. But if you add the wireguard plugin there are notes against it.

My use cases are mainly single site-to-site VPNs. Half of the time one side has a static IP. Half the time DuckDNS for both. Also a fair amount of road warriors doing an OpenVPN connection.

One pro of WireGuard is that it works fine with one side static and one side dynamic for site-to-site. From what I've read, the dynamic site is the side that always kicks off the connection. It could also be dual used for road warrior connections.

For road warriors I've had to use only the OpenVPN client bundled in the OPNsense package. New OpenVPN clients don't seem to work with the generated package/key. For me, that'd be another plus for WireGuard. But, the whole

<code>
=====
Message from wireguard-kmod-0.0.20220615_1:

--
At this time this code is new, unvetted, possibly buggy, and should be
considered "experimental". It might contain security issues. We gladly
welcome your testing and bug reports, but do keep in mind that this code
is new, so some caution should be exercised at the moment for using it
in mission critical environments.
--
===>   NOTICE:

This port is deprecated; you may wish to reconsider installing it:

Only useful for FreeBSD 12 which is EoL soon.

It is scheduled to be removed on or after 2023-12-31.
Checking integrity... done (0 conflicting)
Nothing to do.
***DONE***
</code>

has me confused. This is from a plug-in install.

Thanks for guidance and opinions!

Hello,

There is probably no firm 'real' answer to this question. If there is please let me know. In general, is your router more secure if you do not have an external domain associated with it's public IP? For example is router.walmart.com less secure than only using it's static IP of 12.34.56.78?

In this scenario, it is an OPNsense router with out of the box security enabled. There are IPSEC VPN connections to the box - both tunnel links and OpenVPN temporary connections. No other WAN ports open. I like the idea of setting a DNS name to it. I don't see how it would be less secure.

... I meant for the plugin only.

Blah! I looked all about except for the obvious. Thanks.

Hello,

I've been looking but not finding, any help?

We have a couple dozen pfSense routers we support. Some of the sites have IPSEC VPNs, some use OpenVPN for remote access. Most sites are basic single segment networks - DHCP, DNS, NTP. The plan is to migrate them to OPNsense. Is there a tool/service that we can manage all of the routers from a single dashboard?

It can be a limited overview/manager. Something that takes care of 80% of the jobs. For all of the features of OPNsense we only use 10% of them. Items we're looking for are
* sends the remote router's firmware/version
* allows remote kickoff of updates (yes, I know this can be dangerous)
* check status and restart of IPSEC VPNs
* alerts on gateway failures and packet loss alerts (pull sided vs remote pushing?)
* pull config backups from remote routers so we have a backup locally for future jobs/recovery
* since this is a wish list, a feature that we could semi prep the router (WAN static or DHCP), a public/private SSH key exchange (?), when the user plugs it in, we could then attach and do a little deeper config
* this tool would send out email/sms notifications

I imagine you could do a lot of the reporting with Zabbix. Is the plugin current?. There is Monit on the boxes but I haven't looked into its features yet.

And, yes, we'd happily pay for a quality product, anything that saves us work.

Thanks for ideas and input!

Here's my working setup...

* I'm running Incredible PBX on a Raspberry Pi4
* I use SIPstation as my SIP provider
* I have an OPNsense router
* I have a static WAN (public) IP
* these are the settings for OPNsense and Incredible PBX

In OPNsense
NAT> Port Forward
* this auto-creates the Firewall> Rules entries
* IncredPBX is an OPNsense Alias pointing to my PBX which uses a static LAN IP

                                Source    Destination                  NAT    
Interface   Proto    Address    Ports    Address          Ports          IP                 Ports             Description    
LAN    TCP    *            *        LAN address     80, 443          *                 *                Anti-Lockout Rule    
WAN    UDP    *            *        WAN address    5060 - 5061     IncredPBX     5060 - 5061   IncredPBX 1.1    
WAN    UDP    *            *       WAN address    10000 - 20000  IncredPBX     10000 - 20000    IncredPBX 1.2


NAT> Outbound
* set to Hybrid then add the following rule
* the rule could probably be tightened up a bit

                                Source                       Destination      NAT                 NAT      Static       
Interface  Source     Port      Destination      Port              Address            Port      Port       Description    
WAN    LAN net      *                *                    *              WAN address     *         YES       IncredPBX 1.4



In Incredible PBX
Settings> Asterisk SIP Settings> Nat Settings
* make sure your External Address is accurate
* make sure your Local Networks is accurate

Connectivity> SIPstation
* obviously only if your are using SIPstation
* make sure your Primary SIPstation Server is talking, at times you may need to refresh
* make sure your Secondary SIPstation Server
* test External Connectivity
** the Firewall Status will Fail; lots of reason for this - you are not using FreePBX's firewall/you're using OPNsense/your PBX is not directly on the Internet>>> don't worry about it
** External IP should be accurate



The fix was one of those FM (fricking magic) fixes. It just started working. The real fix was one or more of these...
* I was making changes one at at time but not resetting my States
* my Outbound Static Port = Yes was one of my last changes
* Asterisk SIP Settings> NAT was not accurate there although the SIPstation said things were good

Thanks, but no other router. My OPNsense connects to my cable modem.

And I've set

Firewall> Settings> Advanced> Firewall Optimization to Conservative