Messages

I got my CWWK N6005 working in Proxmox for 6 days without any crashing so far.

Opt-In Kernel 6.1

Code Select Expand


apt update
apt install pve-kernel-6.1


Update Intel Processor Microcode
Edit /etc/apt/sources.list add non-free to the following

Code Select Expand


deb http://ftp.debian.org/debian bullseye main contrib non-free
deb http://ftp.debian.org/debian bullseye-updates main contrib non-free
deb http://security.debian.org bullseye-security main contrib non-free

then

Code Select Expand


apt update
apt install intel-microcode

You can choose to remove non-free from the sources.list file, it doesn't really matter.

Disable CSTATE in GRUB (I am using UEFI with PCIe Passthru) do it according to your install

Code Select Expand


GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt intel_idle.max_cstate=1 processor.max_cstate=1"

then

Code Select Expand


update-grub


I do not know what combination of the above is helping, I will see when I have time to re-test it if the VM doesn't crash after at least 10 days.

Thanks @freejack,

Yes, the systems, N5105 and N6005, are separate ones, they have their own RAM, both sets are brand new and tested with the new memtest86+, as a added precaution, I also do a Prime95 burn-in, making sure no threads fails, before tinkering with them.

Anyways, updating the Intel Microcode method seems to be working, I have absolutely 0 weird quirks that I noticed before and dismissed, and VM (OPNsense) in Proxmox, has been running roughly around 48 hrs without crashing. If it works for a week, I will try to get someone to pass the message to CWWK so hopefully, they will update it in the BIOS and release it.

The above method does not work!
After stress testing connection for 2 hours, the VM (OPNsense) crashed.

Currently testing with the following combination:

1. Opt-in Kernel 6.1

Code Select Expand

apt update
apt install pve-kernel-6.1
2. Add in grub (I am using XFS with UEFI)

Code Select Expand

intel_idle.max_cstate=1 processor.max_cstate=1

Code Select Expand

update-grub
3. Update Intel-Microcode to 3.20220510.1
edit /etc/apt/sources.list add non-free to

Code Select Expand

deb http://ftp.debian.org/debian bullseye main contrib non-free
deb http://ftp.debian.org/debian bullseye-updates main contrib non-free
deb http://security.debian.org bullseye-security main contrib non-free
update by

Code Select Expand

apt update
apt install intel-microcode
Then remove the non-free from sources.list because you don't need it anymore, and reboot.

Currently VM (OPNsense) has been running for 19 hours without any crashes, in which during this duration, I have been stressing it with high internet activity.

I do not know what might be working, it could be one, to the combination of all 3. I will revise it if the VM doesn't crash after a week with a clean install.

Currently I have the following miniPC boxes in testing with the following processors.
N6005 (6x i226) CW-NW11v2
N5105 (4x i226) CW-FMI01v5
J4125 (4x i226) J4125-4L-i226

J4125 works flawlessly. So this is the last I will be mentioning it on this post.

N5105 and N6005 both suffers frequent VM (OPNsense) reboots or crashes.
My BIOS settings for all the miniPCs are default.

While people on Proxmox suggest to disable C STATE in BIOS, mine are disabled in BIOS by default.
Turbo Boost by default in BIOS for N6005 is 3300MHz and N5105 is 2900MHz.
Both N6005/N5105 base frequency is 2000MHz.

When in Proxmox shell using cli command

Code Select Expand

watch "lscpu | grep MHz

N6005 will be constantly at 3300MHz
N5105 will be constantly at 2900MHz

This should not be the "correct" states as they are Boost Frequencies and should be at this frequency for short periods of time.

Solution 1:
Goto BIOS disable MAX Turbo Boost
Both N6005/N5105 will be constantly at base max frequency 2000MHz instead of their boost frequencies.
The CPU will never use their Turbo Boost frequencies

Solution 2:
This method is not recommended by the people at Proxmox which prefers to have CPU Frequency at the maximum state at all times (Something about stability or another)
I am currently testing this method.

Change governor to "ondemand" instead of "performance"

In Proxmox shell use the cli command

Code Select Expand

echo "ondemand" | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

You can see the frequency of the CPU with 2 second intervals using command

Code Select Expand

watch "lscpu | grep MHz"
or

Code Select Expand

cat /proc/cpuinfo |grep "cpu MH"

To set a cron to change governor to "ondemand" every reboot

Code Select Expand

crontab -e
If this is your first time editing your crontab, choose your editor and add

Code Select Expand

@reboot echo "ondemand" | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

The reason I choose ondemand instead of powersave is because powersave seems to stick at 800MHz and does not increase upon load.

Again, I am just starting to test this, and so far, have no issues. I will continue to monitor this and post if there is any further crashes.

@Heiko910

Hi, and welcome to OPNsense.

The "usual method" of multiple NIC on pfSense or OPNsense if you have 4 NICs is:
1x WAN - Your Internet Connection
1x LAN - You LAN port, usually if you need more ports, you connect this to a switch
2x OPT1, OPT2 - You can use this for VLAN, DMZ, LAN2 subnet, etc, and if you need more ports, you connect them to a switch.

However, the alternative which you are asking for is probably this:
1x WAN
3x LAN (Switch)

The guide is here...
https://docs.opnsense.org/manual/how-tos/lan_bridge.html

Sadly, this is my solution for a N5105/N6005 regarding stability issues. (See attached picture)
Only the J4125 units does not require a fan over it.

CPU set to host
Machine set to q35
Memory allocated 8gb (Ballooning = Off)
PCI Device passthrough for LAN1 ~ LAN4 (6 port units)
PCI Device passthrough for LAN1 ~ LAN3 (4 port units)

The units for 6 ports are using Intel i226
The units for 4 ports are either using Intel i226 or RealTEK RTL8125

It was running for weeks on 22.7.8 then I upgraded.

OPNsense 22.7.10_2 seems to have issues with all my units, J4125 / N5105 / N6005 tested, but only as a VM in Proxmox. I have not tested bare metal. The N5105 / N6005 has stability issues outside OPNsense, when stress testing with Prime95. Memtest86 passes multiple times. I put a USB 140mm fan over the unit, and stability issues with stress testing goes away. OPNsense 22.7.10_2 still has issues crashing (VM in Proxmox), although it takes days to crash. No issues yet after 2 weeks, with pfSesnse 2.7 but that is a  development version.

Sorry to bring up pfSense if that annoys some people. I don't really care what software solutions is being used, as long as it is suitable for the use case.

@RAL9005
Is it the one where you have to set to 12500 to get 12.5W for PL2 - Turbo Performance?
25W for PL2 is pretty much correct for Intel N6005, to boost to 3.3GHz.
You will see the same setting on Intel NUC 11 Essential Kit NUC11ATKPE, except the NUC has a more aggressive setting of 15W at PL1 - Sustained State and 25W at PL2 - Burst State. (See attachment)

What probably "helped" your issues was disabling Turbo Boost mode, where the CPU is limited to 2GHz and unable to boost to 3.3GHz

Just wondering, on your N6005 Topton box, did you put a fan over the case or are you using it without any active cooling.

@chenganir
You can find the settings:
Advanced -> Power & Performance -> CPU - Power Management Control -> View/Configure Turbo Options -> Power Limit 2 Override

I placed a 140mm USB FAN over the unit and that solved the overheating stability issues. There are some raised concerns of overheating on the Chinese forums about these Changwang (CWWK) boxes that uses N5105 and N6005, and most recommend only using the units with a active cooler (USB / 12V FAN) over the unit. The last known unit that doesn't require active cooling are the J4125 units, IMO.

It however does not solve the issues of OPNsense  22.7.10_2, I recommend using 22.7.8, the last known "more stable" version, again IMO.

Hope you find out what is wrong with it, for me it was ballooning memory. OPNsense and pfSense would both hang after a few hours, I don't use unbound so I have that disabled. I am using either pihole or adguard as my DNS.

After disabling, didn't need to downgrade Suricata and/or ntopng. So I am lucky on that. I do have that annoying constant non-stop

"/usr/local/opnsense/scripts/routes/gateway_status.php: plugins_run return_gateways_status (execute task : dpinger_status())"

thing in the logs in OPNsense. Other than being annoying, it doesn't have anything bad happening yet.

Yes it's on Proxmox. Ballooning was always off, this is not it.

AND I spoke too soon: suricata 6.0.8_1 and OPNsense 22.7.10_2 after a bit more than a day had the same issue: memory trippled and DNS no longer resolving. All services running fine from what I could see.
So there seems to be another issue here apart from suricata. Unbound (also has a bigger update)?

I have reverted to a snapshot from Dec 2nd, were all was fine for ages. Running 22.7.9 with the plugins from that release and see how it goes.

Are you using Proxmox and running OPNsense in a VM?
If you are, please disable memory ballooning.
You can find the settings in:
OPNsense VM
Hardware -> Memory
Tick Advanced
Untick Ballooning Device

It seems that FreeBSD 13.1 and MangoDB does not like Proxmox using Ballooning memory and will exhibit modules in OPNsense failing than crashing the whole VM.

@franco

UPDATE:

I have been running 22.7.10_2 and the previous update with the Suricata 6.0.9_1

I have found that for the last few days I get to a time when I see memory usage going up, swap space being used (I have 12GB assigned and pratically never use swap) and then suddenly some domains start not resolving.
Then even more stop resolving. A reboot fixes this.
I have switched from Unbound to DNSmasq but the same happens after a day or so.
I have reverted to OPNsense 22.7.8 have locked the suricata at 6.0.8_1 and updated again to 22.7.10_2

Running fine now again, with normal memory usage and all domains resolving.

So Suricata 6.0.9_1 DOES have issues still........