Home
Help
Search
Login
Register
OPNsense Forum
»
Archive
»
17.1 Legacy Series
»
17.1.b & Suricata fails on ESXi
« previous
next »
Print
Pages:
1
[
2
]
3
Author
Topic: 17.1.b & Suricata fails on ESXi (Read 15398 times)
phoenix
Hero Member
Posts: 541
Karma: 58
Re: 17.1.b & Suricata fails on ESXi
«
Reply #15 on:
December 30, 2016, 07:12:16 pm »
Hi Franco
Thanks for that prod, I'd forgotten about testing the E1000 NIC - the obvious sometimes escapes me. I did try the VMXNET2 NIC as well and that also failed to allow IDS enabling but I guess that's to be expected.
I should point out to anyone else that tries this, you can't leave the VMXNETx in the system, it has to be a removal and change to the E1000 NIC then a clean install of 17.1.b and then it works a treat with IDS up and running smoothly.
Thanks for your help and I wish you and the OPNsense team (and the other forum members) a happy and prosperous New Year, have a great week-end.
Logged
Regards
Bill
lattera
Full Member
Posts: 207
Karma: 82
Re: 17.1.b & Suricata fails on ESXi
«
Reply #16 on:
December 30, 2016, 07:14:31 pm »
Great to hear you've gotten it working with the emulated Intel driver. That confirms that it's the same issue that I saw and should be fixed with the patch Franco linked to.
Logged
phoenix
Hero Member
Posts: 541
Karma: 58
Re: 17.1.b & Suricata fails on ESXi
«
Reply #17 on:
December 31, 2016, 03:20:37 pm »
There's an unfortunate side effect of this, the CPU usage goes up to 100% and the Load is 1.3%. Using the VMXNET3 driver on 16.7 the Load was about the same with CPU usage around the 12% mark. This is a VM on a lightly loaded server so I'll leave it as it is for now and keep an eye on it.
Would it be worth mentioning this problem in the Release Notes for 17.1 (and the RCs?) just in case anyone else hits this problem.
Logged
Regards
Bill
franco
Administrator
Hero Member
Posts: 16009
Karma: 1419
Re: 17.1.b & Suricata fails on ESXi
«
Reply #18 on:
December 31, 2016, 03:57:06 pm »
Will do.
Logged
franco
Administrator
Hero Member
Posts: 16009
Karma: 1419
Re: 17.1.b & Suricata fails on ESXi
«
Reply #19 on:
January 03, 2017, 05:38:17 pm »
I ran into this with the intel-em-kmod driver we maintain, it surprisingly (but not unjustly) uses the netmap(4) emulation mode as opposed to its native support, which made it possible to easily run into the same panic. First test with the new netmap(4) changes in 12-CURRENT had no conclusive results. We're definitely not going to solve this for the initial 17.1 release, but I will work with the authors to see if we can resolve this ASAP to port it over.
Cheers,
Franco
--
775.468651 [ 268] generic_find_num_desc called, in tx 1024 rx 1024
775.476185 [ 276] generic_find_num_queues called, in txq 0 rxq 0
775.483286 [ 801] generic_netmap_dtor Restored native NA 0
775.496255 [ 268] generic_find_num_desc called, in tx 1024 rx 1024
775.503779 [ 276] generic_find_num_queues called, in txq 0 rxq 0
775.511347 [ 801] generic_netmap_dtor Restored native NA 0
775.527056 [ 276] generic_find_num_queues called, in txq 0 rxq 0
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x1
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff80732c2a
stack pointer = 0x28:0xfffffe00a17cb300
frame pointer = 0x28:0xfffffe00a17cb350
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 80820 (W#01-em1+)
[ thread pid 80820 tid 100213 ]
Stopped at generic_xmit_frame+0x2a: movl (%rax),%eax
Logged
franco
Administrator
Hero Member
Posts: 16009
Karma: 1419
Re: 17.1.b & Suricata fails on ESXi
«
Reply #20 on:
January 07, 2017, 05:12:18 pm »
Bill,
I looked into this all the way up to involving FreeBSD/netmap people.
The good news is: the panic is gone in code in 12-CURRENT and we have a working backport.
The bad news for now: neither 12-CURRENT nor the backport for 17.1 work in our inline IPS setup with Suricata.
I'll drop by again when we have more info.
Cheers,
Franco
Logged
phoenix
Hero Member
Posts: 541
Karma: 58
Re: 17.1.b & Suricata fails on ESXi
«
Reply #21 on:
January 07, 2017, 05:48:56 pm »
Hi Franco
Thanks for both of those updates, I seem to have missed the post on Jan 3rd. It's not an urgent problem for me so I reverted to using the VMXNET3 NICs so I could drop the cpu usage and stay on the 17.1 beta. I'm quite happy to leave Suricata disabled for now and I'll wait for any updates you get on this, I'll also be willing to be a guinea pig if you need it tested.
Thanks for all you hard work on this and a Happy new Year to you and all the team.
Logged
Regards
Bill
franco
Administrator
Hero Member
Posts: 16009
Karma: 1419
Re: 17.1.b & Suricata fails on ESXi
«
Reply #22 on:
January 07, 2017, 06:17:45 pm »
Hi Bill,
A happy new year to you too!
The issue is a bit problematic as it is largely present FreeBSD 11.0 but was working in 10.3 just fine. It unfortunately points to "us" being a major provider/user of the functionality, actually only a small subset or niche feature of what others are *not* directly using, not even the developers themselves. This comes with mixed implications of having to make sure the features we use are not being deleted as unused or silently broken months before they are released.
I don't know how we can pull this off, but hopefully with the current discussions we will find a way in the next weeks.
Cheers,
Franco
Logged
franco
Administrator
Hero Member
Posts: 16009
Karma: 1419
Re: 17.1.b & Suricata fails on ESXi
«
Reply #23 on:
January 15, 2017, 05:43:24 pm »
How about this kernel then? Make sure to snapshot.
# opnsense-update -kr 17.1.b-netmap-fix
Cheers,
Franco
Logged
phoenix
Hero Member
Posts: 541
Karma: 58
Re: 17.1.b & Suricata fails on ESXi
«
Reply #24 on:
January 15, 2017, 06:12:33 pm »
Gosh, that was quick.
I (almost) always take a snapshot and I did today. Just done the update and after enabling IPS/IDS and updating the rules all seems to be quite calm with a normal relatively low CPU usage - I also have this on a VM with the VMXNET3 NICs installed. If there's anything that breaks or looks out of place I'll post here.
Logged
Regards
Bill
franco
Administrator
Hero Member
Posts: 16009
Karma: 1419
Re: 17.1.b & Suricata fails on ESXi
«
Reply #25 on:
January 15, 2017, 06:16:28 pm »
Quick? Took me a couple of days to dig through 2 years of netmap commit history to find it.
That's a good sign. If the guys at Deciso and the netmap peeps are ok with it I shall add the fix just in time for 17.1-RC1.
Cheers,
Franco
Logged
phoenix
Hero Member
Posts: 541
Karma: 58
Re: 17.1.b & Suricata fails on ESXi
«
Reply #26 on:
January 15, 2017, 06:24:47 pm »
Sounds good to me, I'll keep a close eye on it for the moment and see what happens. Without IDS enabled it's been running at about 2-3% cpu usage and with it it seems to be hovering around 7-8% and obviously there was a larger spike to 10-12% as the rules were downloaded but that dropped after a few minutes.
Thanks for all your hard work on this and enjoy the rest of the evening.
Logged
Regards
Bill
franco
Administrator
Hero Member
Posts: 16009
Karma: 1419
Re: 17.1.b & Suricata fails on ESXi
«
Reply #27 on:
January 15, 2017, 06:40:32 pm »
Thank you Bill, you too!
Cheers,
Franco
Logged
phoenix
Hero Member
Posts: 541
Karma: 58
Re: 17.1.b & Suricata fails on ESXi
«
Reply #28 on:
January 16, 2017, 08:03:09 am »
Good morning Franco
Bad news 'm afraid. A short while after updating the install yesterday the CPU usage went up to 100%. I didn't notice this yesterday evening as internet access was still OK but this morning I saw the cpu usage was up and internet access was almost impossible.
A reboot also had problems with various timeouts and I had to reset the VM to get it to boot correctly, that worked but CPU usage was straight up to 100%. - disabling IDS/IPS and resetting the VM doesn't resolve the 100% CPU problem and it runs like that all the time.
I've taken a snapshot of this current system so if you need me to do anything on that to get you some logs then let me know.
«
Last Edit: January 16, 2017, 03:37:49 pm by phoenix
»
Logged
Regards
Bill
phoenix
Hero Member
Posts: 541
Karma: 58
Re: 17.1.b & Suricata fails on ESXi
«
Reply #29 on:
January 16, 2017, 09:24:33 pm »
I've just been doing some testing with this and the high CPU use may not be a problem with IPS/IDS. I've enabled IPS/IDS again with the updated kernel/drivers and I'll leave it for tonight and do some more test in the morning, I'll post the results later tomorrow.
Logged
Regards
Bill
Print
Pages:
1
[
2
]
3
« previous
next »
OPNsense Forum
»
Archive
»
17.1 Legacy Series
»
17.1.b & Suricata fails on ESXi