Messages

Please review this excerpt from my Proxmox log. Prox is 8.4.19, the mobo is Asrock X470D4U, the RAM is 128GB (4x32GB) Nemix.

I have 5 other servers similar to this one working fine. The exception is one other server with the same HARDWARE ERROR that pops up on the console. I believe that one also has Nemix RAM. I believe the other four servers have Kingston KSM26ED8/16ME RAM but no errors.

12:37 Hardware Error
13:21 Hardware Error
14:53 Hardware Error
15:45 server reboot

I also see a SMART thermal message but that seems like more of a 'notice'.

Under the Prox log is an output from 'dmidecode -t 17'. While it lists specs it doesn't list actual manufacturer part number. I believe the RAM is actually 3200 speed but running at a lower 2666 speed. I tried an 'lshw -C memory' but lshw is not installed.


------------------------------------------ start some Proxmox log dump
Jun 30 12:37:21 virt09b kernel: mce: [Hardware Error]: Machine check events logged
Jun 30 12:37:21 virt09b kernel: [Hardware Error]: Corrected error, no action required.
Jun 30 12:37:21 virt09b kernel: [Hardware Error]: CPU:0 (17:71:0) MC17_STATUS[-|CE|MiscV|AddrV|-|-|SyndV|CECC|-|-|Scrub]: 0x9c2041000000011b
Jun 30 12:37:21 virt09b kernel: [Hardware Error]: Error Addr: 0x0000000bbf588300
Jun 30 12:37:21 virt09b kernel: [Hardware Error]: IPID: 0x0000009600050f00, Syndrome: 0x000000040a801101
Jun 30 12:37:21 virt09b kernel: [Hardware Error]: Unified Memory Controller Ext. Error Code: 0
Jun 30 12:37:21 virt09b kernel: EDAC MC0: 1 CE Cannot decode normalized address on mc#0csrow#1channel#0 (csrow:1 channel:0 page:0x0 offset:0x0 grain:64 syndrome:0x4)
Jun 30 12:37:21 virt09b kernel: [Hardware Error]: cache level: L3/GEN, tx: GEN, mem-tx: RD
Jun 30 12:43:12 virt09b smartd[1543]: Device: /dev/sdb [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 67 to 66
Jun 30 13:04:37 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 13:08:53 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 13:17:01 virt09b CRON[151843]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jun 30 13:17:01 virt09b CRON[151844]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jun 30 13:17:01 virt09b CRON[151843]: pam_unix(cron:session): session closed for user root
Jun 30 13:19:41 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 13:21:03 virt09b kernel: mce: [Hardware Error]: Machine check events logged
Jun 30 13:21:03 virt09b kernel: [Hardware Error]: Corrected error, no action required.
Jun 30 13:21:03 virt09b kernel: [Hardware Error]: CPU:0 (17:71:0) MC17_STATUS[-|CE|MiscV|AddrV|-|-|SyndV|CECC|-|-|Scrub]: 0x9c2041000000011b
Jun 30 13:21:03 virt09b kernel: [Hardware Error]: Error Addr: 0x0000000bbf588300
Jun 30 13:21:03 virt09b kernel: [Hardware Error]: IPID: 0x0000009600050f00, Syndrome: 0x000000040a801101
Jun 30 13:21:03 virt09b kernel: [Hardware Error]: Unified Memory Controller Ext. Error Code: 0
Jun 30 13:21:03 virt09b kernel: EDAC MC0: 1 CE Cannot decode normalized address on mc#0csrow#1channel#0 (csrow:1 channel:0 page:0x0 offset:0x0 grain:64 syndrome:0x4)
Jun 30 13:21:03 virt09b kernel: [Hardware Error]: cache level: L3/GEN, tx: GEN, mem-tx: RD
Jun 30 13:43:12 virt09b smartd[1543]: Device: /dev/sda [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 67 to 66
Jun 30 13:45:16 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 13:47:15 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 13:50:08 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 13:54:31 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 13:57:41 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 14:05:23 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 14:06:44 virt09b systemd[1]: Starting systemd-tmpfiles-clean.service - Cleanup of Temporary Directories...
Jun 30 14:06:44 virt09b systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Jun 30 14:06:44 virt09b systemd[1]: Finished systemd-tmpfiles-clean.service - Cleanup of Temporary Directories.
Jun 30 14:06:44 virt09b systemd[1]: run-credentials-systemd\x2dtmpfiles\x2dclean.service.mount: Deactivated successfully.
Jun 30 14:17:01 virt09b CRON[172546]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jun 30 14:17:01 virt09b CRON[172547]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jun 30 14:17:01 virt09b CRON[172546]: pam_unix(cron:session): session closed for user root
Jun 30 14:27:29 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 14:28:37 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 14:35:25 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 14:43:12 virt09b smartd[1543]: Device: /dev/sda [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 66 to 67
Jun 30 14:44:19 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 14:45:10 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 14:53:09 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 14:53:53 virt09b kernel: mce: [Hardware Error]: Machine check events logged
Jun 30 14:53:53 virt09b kernel: [Hardware Error]: Corrected error, no action required.
Jun 30 14:53:53 virt09b kernel: [Hardware Error]: CPU:0 (17:71:0) MC17_STATUS[-|CE|MiscV|AddrV|-|-|SyndV|CECC|-|-|Scrub]: 0x9c2041000000011b
Jun 30 14:53:53 virt09b kernel: [Hardware Error]: Error Addr: 0x0000000bbf520300
Jun 30 14:53:53 virt09b kernel: [Hardware Error]: IPID: 0x0000009600050f00, Syndrome: 0x000000040a801101
Jun 30 14:53:53 virt09b kernel: [Hardware Error]: Unified Memory Controller Ext. Error Code: 0
Jun 30 14:53:53 virt09b kernel: EDAC MC0: 1 CE Cannot decode normalized address on mc#0csrow#1channel#0 (csrow:1 channel:0 page:0x0 offset:0x0 grain:64 syndrome:0x4)
Jun 30 14:53:53 virt09b kernel: [Hardware Error]: cache level: L3/GEN, tx: GEN, mem-tx: RD
Jun 30 14:56:46 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 15:06:35 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 15:07:25 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 15:08:03 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 15:13:12 virt09b smartd[1543]: Device: /dev/sda [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 67 to 66
Jun 30 15:15:21 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 15:17:01 virt09b CRON[193198]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jun 30 15:17:01 virt09b CRON[193199]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jun 30 15:17:01 virt09b CRON[193198]: pam_unix(cron:session): session closed for user root
Jun 30 15:38:08 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
Jun 30 15:45:39 virt09b kernel: AMD-Vi: Completion-Wait loop timed out
-- Reboot --
Jun 30 15:47:33 virt09b kernel: Linux version 6.8.12-30-pve (build@proxmox) (gcc (Debian 12.2.0-14+deb12u1) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-30 (2026-06-11T10:10Z) ()
Jun 30 15:47:33 virt09b kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-6.8.12-30-pve root=/dev/mapper/pve-root ro quiet
Jun 30 15:47:33 virt09b kernel: KERNEL supported cpus:
Jun 30 15:47:33 virt09b kernel:   Intel GenuineIntel



------------------------------------------ end some Proxmox log dump








------------------------------------------
dmidecode -t 17 show

Memory Device
        Array Handle: 0x0014
        Error Information Handle: 0x0021
        Total Width: 72 bits
        Data Width: 64 bits
        Size: 32 GB
        Form Factor: DIMM
        Set: None
        Locator: DIMM 0
        Bank Locator: P0 CHANNEL B
        Type: DDR4
        Type Detail: Synchronous Unbuffered (Unregistered)
        Speed: 2666 MT/s
        Manufacturer: Unknown
        Serial Number: 5D270016
        Asset Tag: Not Specified
        Part Number: Unknown
        Rank: 2
        Configured Memory Speed: 2666 MT/s
        Minimum Voltage: 1.2 V
        Maximum Voltage: 1.2 V
        Configured Voltage: 1.2 V
        Memory Technology: DRAM
        Memory Operating Mode Capability: Volatile memory
        Firmware Version: Unknown
        Module Manufacturer ID: Unknown
        Module Product ID: Unknown
        Memory Subsystem Controller Manufacturer ID: Unknown
        Memory Subsystem Controller Product ID: Unknown
        Non-Volatile Size: None
        Volatile Size: 32 GB
        Cache Size: None
        Logical Size: None

------------------------------------------

Hey,

I'm not clear if you've rebooted just the router or the router and the cablemodem. For Cox in NV, you MUST power cycle the cablemodem if there is a hardware change (I'm guessing an update to the router's MAC address in the cablemodem?). Did you change any hardware or the assigned interface?

This is a broad question. I'm trying to figure out where to migrate my connections.

My IPSEC site-to-site are now labeled as legacy. There is a new connection methodology for that tech. There is also Wireguard as a methodology. But if you add the wireguard plugin there are notes against it.

My use cases are mainly single site-to-site VPNs. Half of the time one side has a static IP. Half the time DuckDNS for both. Also a fair amount of road warriors doing an OpenVPN connection.

One pro of WireGuard is that it works fine with one side static and one side dynamic for site-to-site. From what I've read, the dynamic site is the side that always kicks off the connection. It could also be dual used for road warrior connections.

For road warriors I've had to use only the OpenVPN client bundled in the OPNsense package. New OpenVPN clients don't seem to work with the generated package/key. For me, that'd be another plus for WireGuard. But, the whole

<code>
=====
Message from wireguard-kmod-0.0.20220615_1:

--
At this time this code is new, unvetted, possibly buggy, and should be
considered "experimental". It might contain security issues. We gladly
welcome your testing and bug reports, but do keep in mind that this code
is new, so some caution should be exercised at the moment for using it
in mission critical environments.
--
===>   NOTICE:

This port is deprecated; you may wish to reconsider installing it:

Only useful for FreeBSD 12 which is EoL soon.

It is scheduled to be removed on or after 2023-12-31.
Checking integrity... done (0 conflicting)
Nothing to do.
***DONE***
</code>

has me confused. This is from a plug-in install.

Thanks for guidance and opinions!

Hello,

There is probably no firm 'real' answer to this question. If there is please let me know. In general, is your router more secure if you do not have an external domain associated with it's public IP? For example is router.walmart.com less secure than only using it's static IP of 12.34.56.78?

In this scenario, it is an OPNsense router with out of the box security enabled. There are IPSEC VPN connections to the box - both tunnel links and OpenVPN temporary connections. No other WAN ports open. I like the idea of setting a DNS name to it. I don't see how it would be less secure.

... I meant for the plugin only.

Blah! I looked all about except for the obvious. Thanks.

Hello,

I've been looking but not finding, any help?

We have a couple dozen pfSense routers we support. Some of the sites have IPSEC VPNs, some use OpenVPN for remote access. Most sites are basic single segment networks - DHCP, DNS, NTP. The plan is to migrate them to OPNsense. Is there a tool/service that we can manage all of the routers from a single dashboard?

It can be a limited overview/manager. Something that takes care of 80% of the jobs. For all of the features of OPNsense we only use 10% of them. Items we're looking for are
* sends the remote router's firmware/version
* allows remote kickoff of updates (yes, I know this can be dangerous)
* check status and restart of IPSEC VPNs
* alerts on gateway failures and packet loss alerts (pull sided vs remote pushing?)
* pull config backups from remote routers so we have a backup locally for future jobs/recovery
* since this is a wish list, a feature that we could semi prep the router (WAN static or DHCP), a public/private SSH key exchange (?), when the user plugs it in, we could then attach and do a little deeper config
* this tool would send out email/sms notifications

I imagine you could do a lot of the reporting with Zabbix. Is the plugin current?. There is Monit on the boxes but I haven't looked into its features yet.

And, yes, we'd happily pay for a quality product, anything that saves us work.

Thanks for ideas and input!

Here's my working setup...

* I'm running Incredible PBX on a Raspberry Pi4
* I use SIPstation as my SIP provider
* I have an OPNsense router
* I have a static WAN (public) IP
* these are the settings for OPNsense and Incredible PBX

In OPNsense
NAT> Port Forward
* this auto-creates the Firewall> Rules entries
* IncredPBX is an OPNsense Alias pointing to my PBX which uses a static LAN IP

                                Source    Destination                  NAT    
Interface   Proto    Address    Ports    Address          Ports          IP                 Ports             Description    
LAN    TCP    *            *        LAN address     80, 443          *                 *                Anti-Lockout Rule    
WAN    UDP    *            *        WAN address    5060 - 5061     IncredPBX     5060 - 5061   IncredPBX 1.1    
WAN    UDP    *            *       WAN address    10000 - 20000  IncredPBX     10000 - 20000    IncredPBX 1.2


NAT> Outbound
* set to Hybrid then add the following rule
* the rule could probably be tightened up a bit

                                Source                       Destination      NAT                 NAT      Static       
Interface  Source     Port      Destination      Port              Address            Port      Port       Description    
WAN    LAN net      *                *                    *              WAN address     *         YES       IncredPBX 1.4



In Incredible PBX
Settings> Asterisk SIP Settings> Nat Settings
* make sure your External Address is accurate
* make sure your Local Networks is accurate

Connectivity> SIPstation
* obviously only if your are using SIPstation
* make sure your Primary SIPstation Server is talking, at times you may need to refresh
* make sure your Secondary SIPstation Server
* test External Connectivity
** the Firewall Status will Fail; lots of reason for this - you are not using FreePBX's firewall/you're using OPNsense/your PBX is not directly on the Internet>>> don't worry about it
** External IP should be accurate



The fix was one of those FM (fricking magic) fixes. It just started working. The real fix was one or more of these...
* I was making changes one at at time but not resetting my States
* my Outbound Static Port = Yes was one of my last changes
* Asterisk SIP Settings> NAT was not accurate there although the SIPstation said things were good

Thanks, but no other router. My OPNsense connects to my cable modem.

And I've set

Firewall> Settings> Advanced> Firewall Optimization to Conservative

I also found these instructions in pfSense docs


Manual Outbound NAT

For Manual Outbound NAT, navigate to Firewall > NAT, Outbound tab, switch from Automatic Outbound NAT to Manual Outbound NAT and press Save. Then at the top of the list, create a rule that looks like so:

*    Interface: WAN
*    Protocol: UDP
*    Source: Network, PBX
*    Source Port: [blank]
*    Destination: Network, SIP_Trunks – Or Any for the type if the SIP trunk IP addresses are not known
*    Destination Port: PBX_Ports (or leave blank)
*    Translation: Interface address if using the WAN IP address, or the external VIP for the PBX
*    Port: [blank]
*    Static Port: CHECKED

Which I interpreted this way

Interface   Source    Source Port   Destination   Destination Port   NAT Address   NAT Port   Static Port  Description    
WAN     10.10.20.20/24  udp/ *    *               udp/ *                   11.22.33.44    *              YES    IncredPBX 1.4 

I've tried a number of variations - FreePBX vs Incredible PBX and pfSense vs OPNsense. Since the end result is always the same (calls ring+pick but no audio), I figure it has to be me.

Hello all,

I'm having a devil of the time trying to get my PBX to talk through the router. My belief is that the root cause is my lack of understanding NAT. Any help would be appreciated.

* PBX (10.10.20.20/24) is on the LAN network
* phone (10.10.20.30/24) is on the LAN network
* external/Internet SIP service provider (SIPstation) appears to see/talk to the PBX
* calls ring from my cell (outside network) to PBX phone (inside network)
* calls ring from PBX phone (inside network) to cell (outside network)
* no audio either way
* I've added NAT port forward; in this 11.22.33.44 is my WAN address
Interface  Proto    Address    Ports    Address             Ports             IP                     Ports    Description    
LAN    TCP    *             *            LAN address    80, 443              *                   *             Anti-Lockout Rule    
WAN    UDP    *             *           11.22.33.44    5060 - 5061       10.10.20.20      5060 - 5061    IncredPBX
WAN    UDP    *             *           11.22.33.44    10000 - 20000   10.10.20.20  10000 - 20000  IncredPBX


* doing that auto-added the Firewall Rules
Protocol    Source    Port    Destination    Port    Gateway    Schedule    Description    
IPv4 UDP    *    *    10.10.20.20    5060 - 5061       *         *    IncredPBX 1.1    
IPv4 UDP    *    *    10.10.20.200    10000 - 20000    *         *    IncredPBX 1.2


* I've read some that suggest set NAT Outbound to Hybrid then build a manual rule; I built this but I'm not sure it's valid
- Destination = SIPstation which is an alias to trunk.freepbx.com + trunk1.freepbx.com + trunk2.freepbx.com
- Destination Port = SIPports which is an alias to UDP 5060:5061 + UDP 10000:20000
Interface   Source    Source Port  Destination  Destination Port       NAT Address  NAT Port  Static Port  Description    
WAN     LAN net    udp/ *    SIPstation     udp/ SIPports    Interface address    *      NO        IncredPBX