OPNsense Forum
Archive => 18.7 Legacy Series => Topic started by: John Wick on December 29, 2018, 11:24:53 pm
-
Hi Guys!
Have a problem with Opensense.
Newly instaled virtual Opensense appliance on kvm hypervisor and its works not stable, sudenly hangs.
Router instaled on my hypervisor proxmox node with minimal plugins mode with two interfaces WAN/LAN
and disabled IDS.
2 virt cores 2048mb ram. System resours have reserv 70%
OPNsense set up in DCHP mode with 1 static ip. Primary DNS on gate 1.1.1.1 - 1.0.0.1/
LAN port conected to WAN port of my home phisycal linksys router with dhcp on 192.168.2.1 adress. Dns auto.
First time when node with gate boot up all work stable after some time internet connection sudenly falls, cant acces in router web GUI. Console works and shows filterlog messages with calcru runtime error!
Topic of this problem on Github - https://github.com/opnsense/core/issues/29
Sometimemes i have calcru error maybe that because speedstep tecnology enabled for cpu on host (when enable cpu threads downclock to 1.6 ghz in standby) or problem with NTP server sync (http://)
-
Hello,
I have Opensense 18.7.9 running on Proxmox myself and I'm curious about something. What is your hardware settings? /etc/pve/qem-server/XXX.conf?
Regards
-
Hi there is config.
How to find a way to beat this problem?
agent: 1
bootdisk: virtio0
cores: 1
cpu: host,flags=+pcid;+spec-ctrl
cpuunits: 100
ide2: none,media=cdrom
memory: 2048
name: VPS-X100
net0: virtio=A2:6A:85:EA:12:9A,bridge=vmbr1
net1: virtio=CE:9B:27:A8:87:50,bridge=vmbr2
numa: 1
onboot: 1
ostype: l26
parent: dec_25_18
protection: 1
scsihw: virtio-scsi-pci
smbios1: uuid=dcff8f45-afff-4630-bc6d-52b1e65db11b
sockets: 2
unused0: data-x0:vm-100-disk-0
vcpus: 2
virtio0: data-x0:vm-100-disk-1,cache=writethrough,iothread=1,size=5G
vmgenid: 5685a1e8-890e-4601-8452-c2ff166b19dc
[dec_25_18]
agent: 1
bootdisk: virtio0
cores: 1
cpu: host,flags=+pcid;+spec-ctrl
cpuunits: 100
ide2: none,media=cdrom
memory: 2048
name: VPS-X100
net0: virtio=A2:6A:85:EA:12:9A,bridge=vmbr1
net1: virtio=CE:9B:27:A8:87:50,bridge=vmbr2
numa: 1
onboot: 1
ostype: l26
protection: 1
runningmachine: pc-i440fx-2.12
scsihw: virtio-scsi-pci
smbios1: uuid=dcff8f45-afff-4630-bc6d-52b1e65db11b
snaptime: 1546018922
sockets: 2
vcpus: 2
virtio0: data-x0:vm-100-disk-1,cache=writethrough,iothread=1,size=5G
vmgenid: 5685a1e8-890e-4601-8452-c2ff166b19dc
vmstate: data-x0:vm-100-state-dec_25_18
-
Thanks for sharing your configuration, here's mine if you are interested:
#OPNsense Firewall
#hostpci0%3A 03%3A06.0,pcie=on
agent: 1
balloon: 512
bios: seabios
bootdisk: virtio0
cores: 4
cpu: Opteron_G3
efidisk0: local:112/vm-112-disk-1.qcow2,size=128K
ide2: none,media=cdrom
machine: q35
memory: 1536
name: firewall.lhprojects.int
net0: e1000=9E:0C:E8:FB:F2:98,bridge=vmbr1
net1: e1000=66:9F:2D:02:F8:34,bridge=vmbr0
numa: 0
onboot: 1
ostype: other
scsihw: virtio-scsi-pci
smbios1: uuid=fb4c14e3-698d-47b0-954a-7b330a797f96
sockets: 1
startup: order=0
virtio0: local:112/vm-112-disk-0.qcow2,size=50G
vmgenid: 772c8306-82b0-4daf-9a03-003247f1e3e2
I ran into problems booting Opensense on host cpu, AMD here, and that was inadvertently due to a bug in freebsb kernel.
-
Regrettable
probably will have to switch to another gate :-[
-
Well, it works fine here, just had to switch from host to Opteron_G3. Technically speaking, I was on *-p11 and now on *-p12 and I haven't tested if the patch that fixes the issue was included. I just wanted to share my experience, nothing definitive here. I get 800-900Mbps, with IPS enabled, throughput here. That's impressive on emulated nic cards. That bug affects pfsense more than it affects opensense since 19.1 is on the horizon that brings in a lot fixes.
-
Nothing effect its problem with NTP
-
Have you tried emulating everything, ie the CPU/nic cards? See https://www.berrange.com/posts/2018/06/29/cpu-model-configuration-for-qemu-kvm-on-x86-hosts/ for recommended CPU types. Note, there are pending issues with Suricata IPS and virio drivers, it's recommended, at least for now, to use e1000 nic emulation. Changing BIOS, switch to q35 machine?
ostype: l26
What ostype is that?
-
l26 Linux 2.6/3.X Kernel
Ok need to try
change OS type
set the virtualized cpu type
set e1000
but this reduce performance
Q35 is this chipset mode? Where i can set this option in PVE?
Thanks for the reply! Need to try some variations.
I have never tried before run bsd based disros on kvm. I think my problem can be nearly ntp server synchro clock settings on PVE or OPNsense
-
First, I would try to narrow down the problem, try just changing ostype to other and test. Lastly, change CPU Type to something that matches your HOST CPU closely, if you have VT enabled, the performance impact are relatively negligible, AFAIK, KVM will passthrough CPU Feature instructions.
-
Hi
Allready set up CPU and OS type also set a time zone in web interface.
Thanks! Now its more stable! But sometimes bug relapse (when install modules/updates or without load trafic) not Unfortunately this is not suitable for production scenario. Temporarily install VyOS or Simplewall until I solve the problem.
I found interesting topic https://forum.proxmox.com/threads/anyone-successfully-running-pfsense.45079/
Maybe best way switch to model (Baremetal + OPNsense) or (Baremetal + vSphere + OPNsense appliance)
-
That's great to hear, however in my experience, see uptime below, been running relatively stable for me. Another trick would probably be switching nic cards to e1000. Especially if you are using IPS as that will incur disruptions to traffic due to buggy 'netmap' implementation in virtio. Nevertheless, if you plan to service a lot clients, ideally you would want to run Opsense on a appliance (baremetal), see https://www.applianceshop.eu/opnsense.
4:15PM up 5 days, 15:11, 4 users, load averages: 0.26, 0.24, 0.25
-
Need to try this manual for OPNsense https://www.netgate.com/docs/pfsense/virtualization/virtualizing-pfsense-with-proxmox.html (kvm64 cpu recomended)
I tryed e1000 nics but its can detect it! maybe need assing interfaces or reinstall OPNsense.
In my case OPNsense needs in gate role + reverse proxy for 2-3 webservers VMs in PVE cluster
vith virtio i have 10Gbit virtual buss interface but e1000 not bad variant in my case.
You recomended e1000 if use IDS/IPS pakages like Suricata or you means IPs Guest agent?
For production hi load scenario with IDS + Geoip + SYN-flood protection needed dedicated 2 soket baremetal server instance
-
Ahh yeah, with 10Gbit, you'll likely to hit CPU bottleneck with emulated nic cards. Assuming you don't have a fast enough CPU.
I tryed e1000 nics but its can detect it!
So they work?
You recomended e1000 if use IDS/IPS pakages like Suricata?
The caveat here is, you can still implement legacy PCAP filtering with virtio. Read more about the differences here https://forum.netgate.com/topic/96482/suricata-true-inline-ips-mode-coming-with-pfsense-2-3-here-is-a-preview. If that's possible or not with Opensense, at a glance, it seems to default to 'inline ips'. For the moment, e1000 is recommended till Qemu/KVM fixes their 'netmap' implementation in virtio.
-
I have a relatively marginal 2010 AMD CPU here, for 1Gbits traffic, mind you that the same box is running a Media center that also doing Video/Audio transcoding, Webserver, Windows 2019. It has hold up pretty well in my use case for 1Gbs traffic. Although, I'll admit, at a high cost of latency in terms of responsiveness.
-
Need to create a topic Proxmox + OPNsense success stories + config share.
There is a similar thread on the Natgate forum.
I will continue to try to change the settings of cpu qemu64 / kvm64 / default kvm64 + I will try to run on e1000 + disable memory balooning. I also requested the config from a forum member who successfully works with OPNsense in Proxmox.
The problem is that this error has to wait, it is not clear until the end that causes an error.
I have several old Proliant G7 servers in the Proxmox cluster. I consider in my case it too overhead to allocate the Gate a whole baremetal server.
-
I'm really disappointed that error come back again at night in standby mode.
1. Need to try e1000
2. Need to try disable speed step technology
-
It's probably time to pinpoint the cause the problem. Did you look at dmesg? Logs?
-
How to watch this logs in bsd systems or you means proxmox logs?
with e1000 drivers i have cathch te same error
I think the best way install OPNsense on baremetal server 1/2U with directed hardware nics 1 or 10 Gbe
-
Probably a better idea, I'm not aware of Freebsb/KVM bugs that would contribute to freezes, though seem to be more common on this forum.
Regards
H
-
I had the same this morning. Als a AMD machine, bare metal.
After commanding to clean op log files, the machine went all wierd. Wan connection droped for a moment but came back up.
SSH promting for username but hangs in verifying password.
Console shows firewall logging but cant be stopt by CTRL+C, Quit, Reboot, Restart, CTRL+X etc..
Last resort option was to press the reset button on the machine itself.
Happy to help troubelshoot if this is nessesarry.
-
I finde the topic with same problem on netgate forum https://forum.netgate.com/topic/130467/solved-pfsense-2-4-3-halts-with-calcru-messages-in-console
Its conflict of NTP how to set local NTP in the gate?
I steel cant resolve the problem ***sense works but cli canot work wihout pause/stop vm if hours no traffic router get down.
After pause stop vm have calcru error in console.
-
Maybe problem with timeconters :( Need to try need add sysctl kern.timecounter=ACPI-fast in tunables who know?