Opensense hangs on Proxmox cli filterlog flood fall conection! Web UI not resp..

[John Wick]

Hi Guys!

Have a problem with Opensense.
Newly instaled virtual Opensense appliance on kvm hypervisor and its works not stable, sudenly hangs.
Router instaled on my hypervisor proxmox node with minimal plugins mode with two interfaces WAN/LAN
and disabled IDS.
2 virt cores 2048mb ram. System resours have reserv 70%

OPNsense set up in DCHP mode with 1 static ip. Primary DNS on gate 1.1.1.1 - 1.0.0.1/
LAN port conected to WAN port of my home phisycal linksys router with dhcp on 192.168.2.1 adress. Dns auto.
First time when node with gate boot up all work stable after some time internet connection sudenly falls, cant acces in router web GUI. Console works and shows filterlog messages with calcru runtime error!

Topic of this problem on Github - https://github.com/opnsense/core/issues/29

Sometimemes i have calcru error maybe that because speedstep tecnology enabled for cpu on host (when enable cpu threads downclock to 1.6 ghz in standby) or problem with NTP server sync

[guest19757]

Hello,

I have Opensense 18.7.9 running on Proxmox myself and I'm curious about something. What is your hardware settings? /etc/pve/qem-server/XXX.conf?

Regards

[John Wick]

Hi there is config.
How to find a way to beat this problem?

agent: 1
bootdisk: virtio0
cores: 1
cpu: host,flags=+pcid;+spec-ctrl
cpuunits: 100
ide2: none,media=cdrom
memory: 2048
name: VPS-X100
net0: virtio=A2:6A:85:EA:12:9A,bridge=vmbr1
net1: virtio=CE:9B:27:A8:87:50,bridge=vmbr2
numa: 1
onboot: 1
ostype: l26
parent: dec_25_18
protection: 1
scsihw: virtio-scsi-pci
smbios1: uuid=dcff8f45-afff-4630-bc6d-52b1e65db11b
sockets: 2
unused0: data-x0:vm-100-disk-0
vcpus: 2
virtio0: data-x0:vm-100-disk-1,cache=writethrough,iothread=1,size=5G
vmgenid: 5685a1e8-890e-4601-8452-c2ff166b19dc

[dec_25_18]
agent: 1
bootdisk: virtio0
cores: 1
cpu: host,flags=+pcid;+spec-ctrl
cpuunits: 100
ide2: none,media=cdrom
memory: 2048
name: VPS-X100
net0: virtio=A2:6A:85:EA:12:9A,bridge=vmbr1
net1: virtio=CE:9B:27:A8:87:50,bridge=vmbr2
numa: 1
onboot: 1
ostype: l26
protection: 1
runningmachine: pc-i440fx-2.12
scsihw: virtio-scsi-pci
smbios1: uuid=dcff8f45-afff-4630-bc6d-52b1e65db11b
snaptime: 1546018922
sockets: 2
vcpus: 2
virtio0: data-x0:vm-100-disk-1,cache=writethrough,iothread=1,size=5G
vmgenid: 5685a1e8-890e-4601-8452-c2ff166b19dc
vmstate: data-x0:vm-100-state-dec_25_18

[guest19757]

Thanks for sharing your configuration, here's mine if you are interested:

Code Select Expand


#OPNsense Firewall
#hostpci0%3A 03%3A06.0,pcie=on
agent: 1
balloon: 512
bios: seabios
bootdisk: virtio0
cores: 4
cpu: Opteron_G3
efidisk0: local:112/vm-112-disk-1.qcow2,size=128K
ide2: none,media=cdrom
machine: q35
memory: 1536
name: firewall.lhprojects.int
net0: e1000=9E:0C:E8:FB:F2:98,bridge=vmbr1
net1: e1000=66:9F:2D:02:F8:34,bridge=vmbr0
numa: 0
onboot: 1
ostype: other
scsihw: virtio-scsi-pci
smbios1: uuid=fb4c14e3-698d-47b0-954a-7b330a797f96
sockets: 1
startup: order=0
virtio0: local:112/vm-112-disk-0.qcow2,size=50G
vmgenid: 772c8306-82b0-4daf-9a03-003247f1e3e2


I ran into problems booting Opensense on host cpu, AMD here, and that was inadvertently due to a bug in freebsb kernel.

[John Wick]

Regrettable
probably will have to switch to another gate  :-[

[guest19757]

Well, it works fine here, just had to switch from host to Opteron_G3. Technically speaking, I was on *-p11 and now on *-p12 and I haven't tested if the patch that fixes the issue was included. I just wanted to share my experience, nothing definitive here. I get 800-900Mbps, with IPS enabled, throughput here. That's impressive on emulated nic cards. That bug affects pfsense more than it affects opensense since 19.1 is on the horizon that brings in a lot fixes.

[John Wick]

Nothing effect its problem with NTP

[guest19757]

Have you tried emulating everything, ie the CPU/nic cards? See https://www.berrange.com/posts/2018/06/29/cpu-model-configuration-for-qemu-kvm-on-x86-hosts/ for recommended CPU types. Note, there are pending issues with Suricata IPS and virio drivers, it's recommended, at least for now, to use e1000 nic emulation. Changing BIOS, switch to q35 machine?

ostype: l26

What ostype is that?

[John Wick]

l26  Linux 2.6/3.X Kernel
Ok need to try
change OS type
set the virtualized cpu type
set e1000
but this reduce performance

Q35 is this chipset mode? Where i can set this option in PVE?

Thanks for the reply! Need to try some variations.
I have never tried before run bsd based disros on kvm. I think my problem can be  nearly ntp server synchro clock settings on PVE or OPNsense

[guest19757]

First, I would try to narrow down the problem, try just changing ostype to other and test. Lastly, change CPU Type to something that matches your HOST CPU closely, if you have VT enabled, the performance impact are relatively negligible, AFAIK, KVM will passthrough CPU Feature instructions.

[John Wick]

Hi

Allready set up CPU and OS type also set a time zone in web interface.

Thanks! Now its more stable! But sometimes bug relapse (when install modules/updates or without load trafic) not Unfortunately this is not suitable for production scenario. Temporarily install VyOS or Simplewall until I solve the problem.

I found interesting topic https://forum.proxmox.com/threads/anyone-successfully-running-pfsense.45079/

Maybe best way switch to model (Baremetal + OPNsense) or (Baremetal + vSphere + OPNsense appliance)

[guest19757]

That's great to hear, however in my experience, see uptime below, been running relatively stable for me. Another trick would probably be switching nic cards to e1000. Especially if you are using IPS as that will incur disruptions to traffic due to buggy 'netmap' implementation in virtio.  Nevertheless, if you plan to service a lot clients, ideally you would want to run Opsense on a appliance (baremetal), see https://www.applianceshop.eu/opnsense.

Code Select Expand


4:15PM  up 5 days, 15:11, 4 users, load averages: 0.26, 0.24, 0.25


[John Wick]

Need to try this manual for OPNsense https://www.netgate.com/docs/pfsense/virtualization/virtualizing-pfsense-with-proxmox.html (kvm64 cpu recomended)

I tryed e1000 nics but its can detect it! maybe need assing interfaces or reinstall OPNsense.
In my case OPNsense needs in gate role + reverse proxy  for 2-3 webservers VMs in PVE cluster
vith virtio i have 10Gbit virtual buss interface but e1000 not bad variant in my case.

You recomended e1000 if use IDS/IPS pakages like Suricata or you means IPs Guest agent?

For production hi load scenario with IDS + Geoip + SYN-flood protection needed dedicated 2 soket baremetal server instance

[guest19757]

Ahh yeah, with 10Gbit, you'll likely to hit CPU bottleneck with emulated nic cards. Assuming you don't have a fast enough CPU.

I tryed e1000 nics but its can detect it!

So they work?

You recomended e1000 if use IDS/IPS pakages like Suricata?

The caveat here is, you can still implement legacy PCAP filtering with virtio. Read more about the differences here https://forum.netgate.com/topic/96482/suricata-true-inline-ips-mode-coming-with-pfsense-2-3-here-is-a-preview. If that's possible or not with Opensense, at a glance, it seems to default to 'inline ips'. For the moment, e1000 is recommended till Qemu/KVM fixes their 'netmap' implementation in virtio.

[guest19757]

I have a relatively marginal 2010 AMD CPU here, for 1Gbits traffic, mind you that the same box is running a Media center that also doing Video/Audio transcoding, Webserver, Windows 2019. It has hold up pretty well in my use case for 1Gbs traffic. Although, I'll admit, at a high cost of latency in terms of responsiveness.