link goes down on WAN using IDS/IPS

[dcol]

I am using the IPS with the LAN interface and everything seems to work normally.
When I use the WAN Interface with IPS, with or without the LAN, I get constant up and down of the WAN link in 3-5 second intervals. When IPS is disabled the WAN link is stable. Tried restarts and even a reinstall of OPNsense.
Both WAN and LAN are on the same quad Intel NIC. Even tried to isolate the WAN on its own Intel i210T1 with same results.
Only custom setting is tunables - kern.ipc.nbmclusters=1000000
Any ideas?

[franco]

Hi dcol,

Is the WAN on a PPP device or VLAN or LAGG? These have been know to be more flappy, netmap for IPS mode generates link events during configure so that breaks connectivity and worst case up/down loops.


Cheers,
Franco

[dcol]

The WAN and GW are static IP's.
I have been running without issue for two days with IPS on the LAN Interface only. No errors in the system log.

[franco]

Is there a switch connected to WAN or a (bridged) modem? If the link goes down on the ISP/modem side (it may be there all the time), I can see that Suricata/Netmap would not like that very much as the reconfigure cycle is a lot longer than link up/downs without the overhead and that causes the visible disruption. But it is only a guess.
 :/

Cheers,
Franco

[dcol]

No switch, but I may be narrowing down the possibilities.  I just setup a totally new box with 16GB, i5-3570 and Intel i340T4. Did all the update and loaded thousands of drop rules and enabled IPS on the WAN. It seems to work with no link issues.
The Wan connection is identical to the box that has the WAN link constantly going up and down.

The software is setup identical on the two boxes so I have to assume the first box can't handle IPS on the WAN but works fine on the LAN, but there is much less traffic on the LAN.

Maybe it's a NIC driver issue. Both use the igb driver. I wouldn't think that the first box wasn't fast enough, but I would like to hear opinions on this.

[franco]

Maybe https://forum.opnsense.org/index.php?topic=5511.msg23591#msg23591 ?

From what I heard from Werner, this is fine on 18.1-BETA (FreeBSD 11.1).

[dcol]

Thanks for that link, I will test it tomorrow. The first box is an embedded system so maybe that's the issue.
It would be nice to have a link that shows a list of optional tunables and their effect. The only one I did add from my pfsense days was kern.ipc.nmbclusters=1000000 which is also needed for igb NIC's

[dcol]

Actually, for some reason, kern.ipc.nmbclusters does not work. see pic below

[dcol]

Well, tried the tunables and it didn't make any difference. Here is a look at the system.log with IPS enabled. You can see the link go up and down every few seconds. If you see anything in there weird, let me know

Nov 21 16:12:59 firewall opnsense: /usr/local/etc/rc.linkup: ROUTING: setting IPv4 default route to 10.10.10.113
Nov 21 16:13:02 firewall configd.py: [c35e7e75-68f7-4c75-b918-3974d5260666] request pfctl byte/packet counters
Nov 21 16:13:03 firewall kernel: igb0: link state changed to UP
Nov 21 16:13:03 firewall configd.py: [bde38537-aa86-42df-a3b2-8d982344379c] request pfctl byte/packet counters
Nov 21 16:13:04 firewall configd.py: [a068734d-5fe8-460b-be1d-94b0f31438e4] updating dyndns wan
Nov 21 16:13:04 firewall configd.py: [5e068c96-f6dd-468d-af85-0be952afbac3] Linkup stopping igb0
Nov 21 16:13:05 firewall opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet detached event for wan
Nov 21 16:13:05 firewall configd.py: [1a7dda7c-a65e-42eb-82e5-430e70b90c5e] Linkup starting igb0
Nov 21 16:13:05 firewall opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet attached event for wan
Nov 21 16:13:05 firewall opnsense: /usr/local/etc/rc.linkup: HOTPLUG: Configuring interface wan
Nov 21 16:13:05 firewall opnsense: /usr/local/etc/rc.linkup: ROUTING: setting IPv4 default route to 10.10.10.113
Nov 21 16:13:05 firewall kernel: igb0: link state changed to DOWN
Nov 21 16:13:05 firewall configd.py: [a22355f7-0d2e-4c76-b851-773d5ef71b47] request pfctl byte/packet counters
Nov 21 16:13:09 firewall kernel: igb0: link state changed to UP
Nov 21 16:13:10 firewall configd.py: [b216176e-4375-4bc0-8694-4fdb8239689d] updating dyndns wan
Nov 21 16:13:10 firewall configd.py: [7e26fb9d-9640-40cc-8a9d-ae10eb710ef3] Linkup stopping igb0
Nov 21 16:13:10 firewall opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet detached event for wan
Nov 21 16:13:10 firewall configd.py: [64cd09d7-952a-4f41-8768-d95f9e7bb77f] Linkup starting igb0
Nov 21 16:13:10 firewall opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet attached event for wan
Nov 21 16:13:10 firewall opnsense: /usr/local/etc/rc.linkup: HOTPLUG: Configuring interface wan
Nov 21 16:13:11 firewall kernel: igb0: link state changed to DOWN
Nov 21 16:13:11 firewall opnsense: /usr/local/etc/rc.linkup: ROUTING: setting IPv4 default route to 10.10.10.113
Nov 21 16:13:11 firewall configd.py: [9b4ecadb-b291-4593-ae0e-a91b21970395] request pfctl byte/packet counters
Nov 21 16:13:14 firewall kernel: igb0: link state changed to UP
Nov 21 16:13:16 firewall configd.py: [63239c4f-7e46-4846-9eec-184b9f637223] updating dyndns wan
Nov 21 16:13:16 firewall configd.py: [d5690e83-d2ed-4b65-86f2-4658cc9750e4] Linkup stopping igb0
Nov 21 16:13:16 firewall opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet detached event for wan
Nov 21 16:13:16 firewall configd.py: [0d6b455f-01c1-43d8-9ef5-59f99436869b] Linkup starting igb0
Nov 21 16:13:17 firewall opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet attached event for wan
Nov 21 16:13:17 firewall opnsense: /usr/local/etc/rc.linkup: HOTPLUG: Configuring interface wan
Nov 21 16:13:17 firewall kernel: igb0: link state changed to DOWN
Nov 21 16:13:17 firewall opnsense: /usr/local/etc/rc.linkup: ROUTING: setting IPv4 default route to 10.10.10.113
Nov 21 16:13:17 firewall configd.py: [c3aa21a5-f5b6-404e-a2e8-cca5bfbec204] request pfctl byte/packet counters
Nov 21 16:13:21 firewall kernel: igb0: link state changed to UP
Nov 21 16:13:22 firewall configd.py: [554f7473-135d-4d98-a52e-1b41cf0d43d6] updating dyndns wan
Nov 21 16:13:22 firewall configd.py: [40cde200-1283-4710-a593-2f8f28394d60] Linkup stopping igb0
Nov 21 16:13:22 firewall opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet detached event for wan
Nov 21 16:13:22 firewall configd.py: [7d74672b-f3a8-49b3-9c4f-ea441e315364] Linkup starting igb0
Nov 21 16:13:23 firewall opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet attached event for wan
Nov 21 16:13:23 firewall opnsense: /usr/local/etc/rc.linkup: HOTPLUG: Configuring interface wan
Nov 21 16:13:23 firewall opnsense: /usr/local/etc/rc.linkup: ROUTING: setting IPv4 default route to 10.10.10.113
Nov 21 16:13:23 firewall kernel: igb0: link state changed to DOWN
Nov 21 16:13:23 firewall configd.py: [22cd9751-6c77-4844-ae55-d627879171ec] request pfctl byte/packet counters
Nov 21 16:13:27 firewall kernel: igb0: link state changed to UP
Nov 21 16:13:28 firewall configd.py: [6a19a1ae-c312-4e84-8395-ac7f7fa49a29] updating dyndns wan
Nov 21 16:13:28 firewall configd.py: [f75beaa2-d641-405c-8eb6-0bc114138175] Linkup stopping igb0
Nov 21 16:13:28 firewall opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet detached event for wan
Nov 21 16:13:29 firewall configd.py: [1119b31c-4ac0-42a8-af84-523fb788c67d] Linkup starting igb0
Nov 21 16:13:29 firewall opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet attached event for wan
Nov 21 16:13:29 firewall opnsense: /usr/local/etc/rc.linkup: HOTPLUG: Configuring interface wan
Nov 21 16:13:29 firewall opnsense: /usr/local/etc/rc.linkup: ROUTING: setting IPv4 default route to 10.10.10.113
Nov 21 16:13:29 firewall kernel: igb0: link state changed to DOWN
Nov 21 16:13:29 firewall configd.py: [0933707f-913b-473e-90e2-6d6c9c2db18a] request pfctl byte/packet counters
Nov 21 16:13:33 firewall kernel: igb0: link state changed to UP
Nov 21 16:13:34 firewall configd.py: [86c33c3e-6142-4e1f-95fd-9f04edbf36ff] updating dyndns wan
Nov 21 16:13:34 firewall configd.py: [1fb8d832-21af-4f3d-82ab-135e312880a6] Linkup stopping igb0
Nov 21 16:13:34 firewall opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet detached event for wan
Nov 21 16:13:34 firewall configd.py: [9783f110-588c-4d33-8528-9f21e33f079c] Linkup starting igb0
Nov 21 16:13:34 firewall opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet attached event for wan
Nov 21 16:13:34 firewall opnsense: /usr/local/etc/rc.linkup: HOTPLUG: Configuring interface wan
Nov 21 16:13:35 firewall kernel: igb0: link state changed to DOWN
Nov 21 16:13:35 firewall opnsense: /usr/local/etc/rc.linkup: ROUTING: setting IPv4 default route to 10.10.10.113
Nov 21 16:13:35 firewall configd.py: [353f1a1d-a48a-44bd-bdbd-a7d165a8d91b] request pfctl byte/packet counters
Nov 21 16:13:36 firewall configd.py: [ac6d6e60-0497-46d4-a121-e8bf8db8151c] request pfctl byte/packet counters
Nov 21 16:13:39 firewall kernel: igb0: link state changed to UP

[dcol]

Just finished trying the i340T4 NIC that was working in the 2nd box and I still have the link issues. The only thing left is the computer itself. I will try to alter some BIOS settings, but if that doesn't do it then it must be that using IPS requires a lot of processing beef. An 8 core atom C2458 @ 2.4Ghz is not fast enough but the Intel i5 was.

As stated before, if I run IPS on the LAN and not the WAN, it works fine. Now maybe running the IPS on the LAN is good enough since everything I need protecting is on the LAN. What are everyone's thoughts on this? Is every system on the LAN subnet fully protected via the rules I have in place with IPS on the LAN interface?

[franco]

I would say LAN is our default and acts as a good quality settings. Nobody ever came to us and said "this doesn't work" and Suricon last week would have been a good time for the experts to say that.

From your logs, the problem seems to be a physical drop. You could try two things:

1. Add a small plastic switch to WAN so that the link to the NIC does not go down. Maybe the drop is coming from the device in front of your NIC, which can't cope with the full traffic.

2. Change the WAN to a different NIC port, worst case where your LAN resides so that you can make sure the port is not damaged. Do this with and without 1. to see if that changes things.


Cheers,
Franco

[dcol]

WAN comes up as the default. So you say it is good protection to have IPS on LAN and not WAN?

I did try different NIC's and WAN was plugged directly into the modem. No switch. The 2nd box had no issues. I still think it is a performance issue.

[franco]

Can you be sure the WAN NIC disconnects? If it's the modem the WAN port only follows the link state...


Cheers,
Franco

[franco]

PS: Or it could be the cable itself.

[dcol]

Well, the thing is the 2nd box I put together is using the same modem/cable/NIC and does not go down. I purposely kept all the same network components. It has to do with the computer itself. Which is a Supermicro 5018A-FTN4. The second computer, which works, is a Dell Optiplex 7010 with an Intel i5-3570.