OPNsense Forum
English Forums => Intrusion Detection and Prevention => Topic started by: dcol on November 19, 2017, 09:23:37 pm
-
I am using the IPS with the LAN interface and everything seems to work normally.
When I use the WAN Interface with IPS, with or without the LAN, I get constant up and down of the WAN link in 3-5 second intervals. When IPS is disabled the WAN link is stable. Tried restarts and even a reinstall of OPNsense.
Both WAN and LAN are on the same quad Intel NIC. Even tried to isolate the WAN on its own Intel i210T1 with same results.
Only custom setting is tunables - kern.ipc.nbmclusters=1000000
Any ideas?
-
Hi dcol,
Is the WAN on a PPP device or VLAN or LAGG? These have been know to be more flappy, netmap for IPS mode generates link events during configure so that breaks connectivity and worst case up/down loops.
Cheers,
Franco
-
The WAN and GW are static IP's.
I have been running without issue for two days with IPS on the LAN Interface only. No errors in the system log.
-
Is there a switch connected to WAN or a (bridged) modem? If the link goes down on the ISP/modem side (it may be there all the time), I can see that Suricata/Netmap would not like that very much as the reconfigure cycle is a lot longer than link up/downs without the overhead and that causes the visible disruption. But it is only a guess.
:/
Cheers,
Franco
-
No switch, but I may be narrowing down the possibilities. I just setup a totally new box with 16GB, i5-3570 and Intel i340T4. Did all the update and loaded thousands of drop rules and enabled IPS on the WAN. It seems to work with no link issues.
The Wan connection is identical to the box that has the WAN link constantly going up and down.
The software is setup identical on the two boxes so I have to assume the first box can't handle IPS on the WAN but works fine on the LAN, but there is much less traffic on the LAN.
Maybe it's a NIC driver issue. Both use the igb driver. I wouldn't think that the first box wasn't fast enough, but I would like to hear opinions on this.
-
Maybe https://forum.opnsense.org/index.php?topic=5511.msg23591#msg23591 ?
From what I heard from Werner, this is fine on 18.1-BETA (FreeBSD 11.1).
-
Thanks for that link, I will test it tomorrow. The first box is an embedded system so maybe that's the issue.
It would be nice to have a link that shows a list of optional tunables and their effect. The only one I did add from my pfsense days was kern.ipc.nmbclusters=1000000 which is also needed for igb NIC's
-
Actually, for some reason, kern.ipc.nmbclusters does not work. see pic below
-
Well, tried the tunables and it didn't make any difference. Here is a look at the system.log with IPS enabled. You can see the link go up and down every few seconds. If you see anything in there weird, let me know
Nov 21 16:12:59 firewall opnsense: /usr/local/etc/rc.linkup: ROUTING: setting IPv4 default route to 10.10.10.113
Nov 21 16:13:02 firewall configd.py: [c35e7e75-68f7-4c75-b918-3974d5260666] request pfctl byte/packet counters
Nov 21 16:13:03 firewall kernel: igb0: link state changed to UP
Nov 21 16:13:03 firewall configd.py: [bde38537-aa86-42df-a3b2-8d982344379c] request pfctl byte/packet counters
Nov 21 16:13:04 firewall configd.py: [a068734d-5fe8-460b-be1d-94b0f31438e4] updating dyndns wan
Nov 21 16:13:04 firewall configd.py: [5e068c96-f6dd-468d-af85-0be952afbac3] Linkup stopping igb0
Nov 21 16:13:05 firewall opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet detached event for wan
Nov 21 16:13:05 firewall configd.py: [1a7dda7c-a65e-42eb-82e5-430e70b90c5e] Linkup starting igb0
Nov 21 16:13:05 firewall opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet attached event for wan
Nov 21 16:13:05 firewall opnsense: /usr/local/etc/rc.linkup: HOTPLUG: Configuring interface wan
Nov 21 16:13:05 firewall opnsense: /usr/local/etc/rc.linkup: ROUTING: setting IPv4 default route to 10.10.10.113
Nov 21 16:13:05 firewall kernel: igb0: link state changed to DOWN
Nov 21 16:13:05 firewall configd.py: [a22355f7-0d2e-4c76-b851-773d5ef71b47] request pfctl byte/packet counters
Nov 21 16:13:09 firewall kernel: igb0: link state changed to UP
Nov 21 16:13:10 firewall configd.py: [b216176e-4375-4bc0-8694-4fdb8239689d] updating dyndns wan
Nov 21 16:13:10 firewall configd.py: [7e26fb9d-9640-40cc-8a9d-ae10eb710ef3] Linkup stopping igb0
Nov 21 16:13:10 firewall opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet detached event for wan
Nov 21 16:13:10 firewall configd.py: [64cd09d7-952a-4f41-8768-d95f9e7bb77f] Linkup starting igb0
Nov 21 16:13:10 firewall opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet attached event for wan
Nov 21 16:13:10 firewall opnsense: /usr/local/etc/rc.linkup: HOTPLUG: Configuring interface wan
Nov 21 16:13:11 firewall kernel: igb0: link state changed to DOWN
Nov 21 16:13:11 firewall opnsense: /usr/local/etc/rc.linkup: ROUTING: setting IPv4 default route to 10.10.10.113
Nov 21 16:13:11 firewall configd.py: [9b4ecadb-b291-4593-ae0e-a91b21970395] request pfctl byte/packet counters
Nov 21 16:13:14 firewall kernel: igb0: link state changed to UP
Nov 21 16:13:16 firewall configd.py: [63239c4f-7e46-4846-9eec-184b9f637223] updating dyndns wan
Nov 21 16:13:16 firewall configd.py: [d5690e83-d2ed-4b65-86f2-4658cc9750e4] Linkup stopping igb0
Nov 21 16:13:16 firewall opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet detached event for wan
Nov 21 16:13:16 firewall configd.py: [0d6b455f-01c1-43d8-9ef5-59f99436869b] Linkup starting igb0
Nov 21 16:13:17 firewall opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet attached event for wan
Nov 21 16:13:17 firewall opnsense: /usr/local/etc/rc.linkup: HOTPLUG: Configuring interface wan
Nov 21 16:13:17 firewall kernel: igb0: link state changed to DOWN
Nov 21 16:13:17 firewall opnsense: /usr/local/etc/rc.linkup: ROUTING: setting IPv4 default route to 10.10.10.113
Nov 21 16:13:17 firewall configd.py: [c3aa21a5-f5b6-404e-a2e8-cca5bfbec204] request pfctl byte/packet counters
Nov 21 16:13:21 firewall kernel: igb0: link state changed to UP
Nov 21 16:13:22 firewall configd.py: [554f7473-135d-4d98-a52e-1b41cf0d43d6] updating dyndns wan
Nov 21 16:13:22 firewall configd.py: [40cde200-1283-4710-a593-2f8f28394d60] Linkup stopping igb0
Nov 21 16:13:22 firewall opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet detached event for wan
Nov 21 16:13:22 firewall configd.py: [7d74672b-f3a8-49b3-9c4f-ea441e315364] Linkup starting igb0
Nov 21 16:13:23 firewall opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet attached event for wan
Nov 21 16:13:23 firewall opnsense: /usr/local/etc/rc.linkup: HOTPLUG: Configuring interface wan
Nov 21 16:13:23 firewall opnsense: /usr/local/etc/rc.linkup: ROUTING: setting IPv4 default route to 10.10.10.113
Nov 21 16:13:23 firewall kernel: igb0: link state changed to DOWN
Nov 21 16:13:23 firewall configd.py: [22cd9751-6c77-4844-ae55-d627879171ec] request pfctl byte/packet counters
Nov 21 16:13:27 firewall kernel: igb0: link state changed to UP
Nov 21 16:13:28 firewall configd.py: [6a19a1ae-c312-4e84-8395-ac7f7fa49a29] updating dyndns wan
Nov 21 16:13:28 firewall configd.py: [f75beaa2-d641-405c-8eb6-0bc114138175] Linkup stopping igb0
Nov 21 16:13:28 firewall opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet detached event for wan
Nov 21 16:13:29 firewall configd.py: [1119b31c-4ac0-42a8-af84-523fb788c67d] Linkup starting igb0
Nov 21 16:13:29 firewall opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet attached event for wan
Nov 21 16:13:29 firewall opnsense: /usr/local/etc/rc.linkup: HOTPLUG: Configuring interface wan
Nov 21 16:13:29 firewall opnsense: /usr/local/etc/rc.linkup: ROUTING: setting IPv4 default route to 10.10.10.113
Nov 21 16:13:29 firewall kernel: igb0: link state changed to DOWN
Nov 21 16:13:29 firewall configd.py: [0933707f-913b-473e-90e2-6d6c9c2db18a] request pfctl byte/packet counters
Nov 21 16:13:33 firewall kernel: igb0: link state changed to UP
Nov 21 16:13:34 firewall configd.py: [86c33c3e-6142-4e1f-95fd-9f04edbf36ff] updating dyndns wan
Nov 21 16:13:34 firewall configd.py: [1fb8d832-21af-4f3d-82ab-135e312880a6] Linkup stopping igb0
Nov 21 16:13:34 firewall opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet detached event for wan
Nov 21 16:13:34 firewall configd.py: [9783f110-588c-4d33-8528-9f21e33f079c] Linkup starting igb0
Nov 21 16:13:34 firewall opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet attached event for wan
Nov 21 16:13:34 firewall opnsense: /usr/local/etc/rc.linkup: HOTPLUG: Configuring interface wan
Nov 21 16:13:35 firewall kernel: igb0: link state changed to DOWN
Nov 21 16:13:35 firewall opnsense: /usr/local/etc/rc.linkup: ROUTING: setting IPv4 default route to 10.10.10.113
Nov 21 16:13:35 firewall configd.py: [353f1a1d-a48a-44bd-bdbd-a7d165a8d91b] request pfctl byte/packet counters
Nov 21 16:13:36 firewall configd.py: [ac6d6e60-0497-46d4-a121-e8bf8db8151c] request pfctl byte/packet counters
Nov 21 16:13:39 firewall kernel: igb0: link state changed to UP
-
Just finished trying the i340T4 NIC that was working in the 2nd box and I still have the link issues. The only thing left is the computer itself. I will try to alter some BIOS settings, but if that doesn't do it then it must be that using IPS requires a lot of processing beef. An 8 core atom C2458 @ 2.4Ghz is not fast enough but the Intel i5 was.
As stated before, if I run IPS on the LAN and not the WAN, it works fine. Now maybe running the IPS on the LAN is good enough since everything I need protecting is on the LAN. What are everyone's thoughts on this? Is every system on the LAN subnet fully protected via the rules I have in place with IPS on the LAN interface?
-
I would say LAN is our default and acts as a good quality settings. Nobody ever came to us and said "this doesn't work" and Suricon last week would have been a good time for the experts to say that. :)
From your logs, the problem seems to be a physical drop. You could try two things:
1. Add a small plastic switch to WAN so that the link to the NIC does not go down. Maybe the drop is coming from the device in front of your NIC, which can't cope with the full traffic.
2. Change the WAN to a different NIC port, worst case where your LAN resides so that you can make sure the port is not damaged. Do this with and without 1. to see if that changes things.
Cheers,
Franco
-
WAN comes up as the default. So you say it is good protection to have IPS on LAN and not WAN?
I did try different NIC's and WAN was plugged directly into the modem. No switch. The 2nd box had no issues. I still think it is a performance issue.
-
Can you be sure the WAN NIC disconnects? If it's the modem the WAN port only follows the link state...
Cheers,
Franco
-
PS: Or it could be the cable itself.
-
Well, the thing is the 2nd box I put together is using the same modem/cable/NIC and does not go down. I purposely kept all the same network components. It has to do with the computer itself. Which is a Supermicro 5018A-FTN4. The second computer, which works, is a Dell Optiplex 7010 with an Intel i5-3570.
-
Ok, just wanted to be sure. Could be what you said is true, at least some level of quality in equipment is required for flawless IPS, maybe we should assemble a list of requirements and put that in the docs as well.
Cheers,
Franco
-
Yes, I believe that achieving a stable IPS requires a certain level of hardware. We already know that only certain NIC's are capable of running netmap I will start testing other hardware and processors over the next few weeks. When I have some results, I will post a new thread in this category.
In the meantime for other users using OPNsense with IPS on the WAN interface, post your hardware and experience in this thread. ie. CPU, memory, chipset, NIC.
-
I stand corrected, sort of. I do suggest at least 4 cores and 2Ghz processing power as well as a minimum 8GB memory. The sticky in this topic has more to do with the issue.
-
I saw this discussion and I have exactly the same problem:
IDS/IPS enabled on LAN working fine.
IDS/IPS enabled on WAN (without router for firewall) wan interface is going down and up (see log below)
IDS/IPS enabled on WAN (with apple airport expres before)wan interface everything is working fine.
I have a Jetway NF9HB with 4x NIC Intel i211AT Gigabit Ethernet
Processor Intel Celeron N2930 SoC, 1.83GHz – 2.16GHz Burst, Quad-Core
is there already a solution for running IDS/IPS on the WAN without a extra router?
what triggers the wan connection to go down? If I can test something on my firewall to solve the problem please let me know.
Problem Log:
Jun 22 14:57:42 kernel: igb0: link state changed to UP
Jun 22 14:57:38 kernel: igb0: link state changed to DOWN
Jun 22 14:57:37 opnsense: /usr/local/etc/rc.linkup: HOTPLUG: Configuring interface wan
Jun 22 14:57:37 opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet attached event for wan
Jun 22 14:57:37 opnsense: /usr/local/etc/rc.linkup: Clearing states to old gateway 84.28.94.1.
Jun 22 14:57:37 opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet detached event for wan
Jun 22 14:57:33 opnsense: /usr/local/etc/rc.linkup: ROUTING: skipping IPv6 default route
Jun 22 14:57:33 opnsense: /usr/local/etc/rc.linkup: ROUTING: keeping current default gateway '84.28.94.1'
Jun 22 14:57:33 opnsense: /usr/local/etc/rc.linkup: ROUTING: setting IPv4 default route to 84.28.94.1
Jun 22 14:57:33 opnsense: /usr/local/etc/rc.linkup: ROUTING: no IPv6 default gateway set, assuming wan
Jun 22 14:57:33 opnsense: /usr/local/etc/rc.linkup: ROUTING: IPv4 default gateway set to wan
Jun 22 14:57:33 opnsense: /usr/local/etc/rc.linkup: ROUTING: entering configure using 'wan'
Jun 22 14:57:32 opnsense: /usr/local/etc/rc.newwanip: ROUTING: skipping IPv6 default route
Jun 22 14:57:32 opnsense: /usr/local/etc/rc.newwanip: ROUTING: keeping current default gateway '84.28.94.1'
Jun 22 14:57:32 opnsense: /usr/local/etc/rc.newwanip: ROUTING: setting IPv4 default route to 84.28.94.1
Jun 22 14:57:32 opnsense: /usr/local/etc/rc.newwanip: ROUTING: no IPv6 default gateway set, assuming wan
Jun 22 14:57:32 opnsense: /usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to wan
Jun 22 14:57:32 opnsense: /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'wan'
Jun 22 14:57:32 opnsense: /usr/local/etc/rc.newwanip: On (IP address: 84.28.94.25) (interface: WANzigo[wan]) (real interface: igb0).
Jun 22 14:57:32 opnsense: /usr/local/etc/rc.newwanip: IP renewal is starting on 'igb0'
Jun 22 14:57:31 kernel: igb0: link state changed to UP
Jun 22 14:57:27 kernel: igb0: link state changed to DOWN
-
I stand corrected, sort of. I do suggest at least 4 cores and 2Ghz processing power as well as a minimum 8GB memory. The sticky in this topic has more to do with the issue.
Wonder if it's anything to do with the Meltdown and Spectre mitigation fixes?
To opt out of one or both features, the following values can now be persistently set under System: Settings: Tunables:
Disable PTI via "vm.pmap.pti" to "0" and a reboot, and
Disable IBRS via "hw.ibrs_disable" to "1" with a simple "Apply".
Here are the full patch notes:
Edit: Thinking again, I don't see how it could be though... just a thought.
-
> Intel i211AT Gigabit Ethernet
Should wait and see if the newer drivers from 11.2 are working better. We will have a call for testing for 18.7 out soon. (We will use FreeBSD 11.1 with several driver updates.)
Cheers,
Franco
-
The tunables fixed it for me long ago. Interested to test the new Intel drivers.