How do I detect "bad" packets and prevent them from hitting the WAN?

[RNHurt]

I'm using OPNSense to protect a small elementary school's network and we have been having frequent outages for the past 4 months.  We've investigated everything we can think of and so far haven't been able to fix the problem. 

Our ISP has suggested that our problem looks similar to other issues they've seen where there is a bad or failing component on the network that is sending out malformed packets and disrupting our service.  The cable modem sees these "bad" packets and starts dropping legitimate packets.

To me, our problem feels like a lack-of-bandwidth type of issue but everything we can see tells us that we are not oversubscribed.  When the outage happens, OPNSense is not reporting any type of bandwidth spike, our WAN traffic isn't topped out, even the firewall hardware looks fine (CPU, RAM, temp, etc.)

My question then is how do I track down the device that is causing the issue?  Is it possible to do a tcpdump on OPNsense and export that data for later evaluation?  I've installed the VnStat plugin but I don't think that is going to help.  Are there any other plugins that I should try?

[bartjsmit]

Hi Richard,

You can capture packets on the interface and/or configure ntopng to show your most prolific talkers.

Interfaces, diagnosis, packet capture

Another option is to push netflow data to an external host for analysis. Solarwinds is a common option.

Bart...

[ArrayElement]

Seems like you're going to have a ton of packets to deal with, and tcpdump will very likely fill up your hard drive if you are using smallre equipment.

If that is the case, you can do this:

1. SSH into your box
2. Run this command:

tcpdump -s 0 -U -n -i reX -w - | nc [ip you want to send to] [port number]

This will fire up tcpdump on your specified interface (re0? re1?), and pipe the output to netcat, which will send that data to an ip and port of your choosing.

Let's assume you have a (linux) workstation at 192.168.1.100, you would do:

tcpdump -s 0 -U -n -i re0 -w - | nc 192.168.1.100 8888

No, on that workstation, you need to have netcat start listening to that traffic, and allow it to be read:

First, we have to make a named pipe:

mkfifo /tmp/fwdump

Next, we need to have netcat listen to the traffic and run it to that pipe:

nc -l 8888 /tmp/fwdump

Lastly, start wireshark to listen to that pipe:

wireshark -k -i /tmp/fwdump

Now, you can watch the traffic in real-time on your local workstation, and dump the *giant* file to disk for perusal.

[xames]

Wow, array solution seems advanced mode.

[RNHurt]

You can capture packets on the interface and/or configure ntopng to show your most prolific talkers.

Interfaces, diagnosis, packet capture

Another option is to push netflow data to an external host for analysis. Solarwinds is a common option.

I don't think the amount of traffic is a problem, it seems to be the type of traffic.  I'm using the built-in flow logs to watch things happen and at no point does our bandwidth spike and no individual IP address has a lot of activity.

I do like your idea about SolarWinds, unfortunately they seem to be focused on Windows and we're more of a Unix/macOS shop.  Any other, possibly open source, suggestions for similar network monitoring?

Thanx!
Richard

[bartjsmit]

There is a netflow add-on for Splunk: https://splunkbase.splunk.com/app/1658/

Bart...

[RNHurt]

I installed and enabled the ntopng plugin and it has the ability to capture and download the most recent packets.  I'm guessing it captures them in a round-robin sort of fashion and allows you to download the last 5 seconds --> 10 minutes of packets captured with pcap.

I think what I'll do is wait until the problem happens and then grab the last 10 minutes of packets.  I should be able to load that up into Wireshark and see what's going on.

Thanx!