Messages

Replying to myself again.

I see this in the documentation (https://docs.opnsense.org/manual/reporting_traffic.html):

When an interface doesn't report traffic and you are certain there should be any, make sure to check if you have any services enabled that use netmap (zero copy) support on the selected interface (such as IPS and Sensei). When zero copy is used, packets won't by copied in the kernel in which case bpf can't read from the usual in memory buffer.

I have Suricata enabled. So I guess that's what's going on? When using IPS, the traffic disappears from these reporting graphs just as well as they disappear from the exporter into Prometheus...?

Bit disappointing if I'm forced to choose between one or the other.

Actually I just noticed something else. When I go into Reporting->Traffic, those graphs at the top also do not reflect reality and in fact seem to show pretty much the same thing that the Prometheus node exporter does. HOWEVER the tables below, which breaks down traffic by IP, seem to reflect reality. E.g. when a computer on the LAN ramps up an internet download it immediately shows up in the table graphs, however the summarizing graphs above at interface level does not reflect anything.

Also, the graphs in Reporting->Health->Traffic show the right thing.

Plot thickens...? What am I missing? Starting to wonder if I'm missing something obvious here  ;-)

Hi - does someone use this combination successfully for monitoring and graphing? I can't get it to behave properly but I don't know if I'm misinterpreting metrics or if the FreeBSD implementation of node exporter is just quirky.

So... I was happy enough to find the os-node_exporter plugin. This installs the Prometheus exporter and it runs fine. Prometheus picks up the metrics and all good... except that the node_network_[receive|transmit]_bytes_total metric doesn't actually do seem to reflect anything that goes in and out. No matter what I pump through the firewall, the metrics maintain a very slow, linear increment across all devices and seemingly unrelated to what actually goes on. I can't figure out why - and by comparison, my Linux systems behave exactly as expected from Prometheus documentation and tutorials, so I don't think the problem lies with my queries.

Anyone else managed to get this to work?

Depending on where you are you may be able to get it cheaper than that. I got the i5 delivered straight from China for just over $300. With 8GB RAM and 4x Intel nic. Very hard to beat that performance per dollar I think, unless buying used of course in which case anything is possible. Mine has been working well (except when it overheated and I had to put fans on it) so in a way I'm unfair, but as I mentioned in another thread I've decided I don't trust it so will look to replace it anyway. Partly after I read some speculation elsewhere of the CPUs having been extracted from 2nd hand hardware.

There seem to be a lot of people recommending I5 or higher for gigabit throughput with IPS/IDS.  Given my current CPU I'm questioning if it's required or if I've just not noticed that it's slowing some traffic down.  I'd be interested in your experience on that.

My Qotom has got an i5-5250U laptop CPU. According to passmark it's roughly equivalent to your E8400 in single thread and overall performance BUT at 15W as opposed to 65W... For what it's worth it saturates my 1Gb WAN, with Suricata enabled. I can't get VPN (OpenVPN) above ~300Mb but I don't know if the bottleneck is my router or at the other end.

I'm still leaning towards a mini itx SuperMicro m/board with AMD Epyc 3201 embedded. And a generic (non-SuperMicro) case. However holding back right now to see if they'll update the lineup now that Zen3 is out in the wild. The 3201 is almost 2 years old now but still costs as much as it did new.

PS. I'm still on 20.1...

This is most likely due to FreeBSD 12. However since this was introduced in July, it's seems a "bug" reducing WAN throughput from 1gb to 518 mbps would be one of your top priorities.  ;)

Sooner or later people will realise that we do not write FreeBSD code and if FreeBSD barely makes this a priority this top priority has nobody who will quickly deal with the issue despite all the friendly nudging in a sub forum of a related downstream project.

I can see that it must be very frustrating to field constant questions on issues that are not related to OPNsense as such but rather the upstream OS. And particularly if the interest from FreeBSD to fix them is only lukewarm at best, as you imply above.

However I think one must also realise that for most users, myself included, OPNsense is not so much a "downstream project" to FreeBSD, as it is an appliance with a very specific purpose. It is actually of no consequence to me whether it is based off FreeBSD, Linux, or something else as long as it delivers on the features, configurability, performance and level of security that I need. I spend 98% of the time that I manage OPNsense in the web GUI. I would hazard a guess that vast majority of OPNsense users are the same.

Given what you say about FreeBSD above, and how long this has been going on, I think it actually raises questions on the longevity and viability of FreeBSD as base OS? Hopefully I'm wrong..!

I'll reply to myself as I've done some more research... It seems the SuperMicro mITX cases can be pretty loud and particularly the SuperChassis 101F which would otherwise have been the ideal choice.

So leaning towards either Optiplex and taking a gamble on finding a quad Ethernet card that works, or a SuperMicro mITX m/board in a non-SuperMicro mITX case.

Hi all

So in previous threads I've recommended Qotom. Still can't say anything bad about them. However one of my (managed) switches spontaneously broke the other day and it was a pain to replace (and created bad karma in the household)... So I dread the same happening to my router and have decided to look for something a bit more robust. The Qotom can be repurposed into an esxi node, easy to replace if it dies...

The router is such a critical component that I'm prepared to throw a little bit of money at it if it gives me the peace of mind that it'll then just chug along for 5-10 years (assuming I won't botch it up with software upgrades...!).

So help me decide.
Requirements: silent or whisper quiet (bedroom cupboard install). Enough grunt to drive 1Gb fiber wan with intrusion detection, vpn, and 20+ clients. 4 NICs.

Considered options:

SuperMicro AMD Epyc 3101 (M11SDV-4CT-LN4F) or 3201 with 4 built-in NICs. SuperMicro mITX case (SuperChassis E300).
Pros: purpose built hardware. Enterprise grade stuff. IPMI. Cons: is the case quiet? AMD compatibility for FreeBSD? Single core performance?

Dell Optiplex 3070 SFF (i3 gen 9) with Intel i350-T4 quad NIC in PCI-e.
Pros: guaranteed quiet. Well-renowned reliability (although no ECC ram). Proper desktop cpu (with fast single core performance). Cons: some reports on the internet of Intel i350-T4 built for server hardware and not working with Dell desktops.

Help me decide!

I've used this: https://www.supermicro.com/en/products/motherboard/X11SCL-iF along with a Xeon-E 2126G. I have gigabit up and down and run traffic shaping + IPS/IDS as well. No slowdowns (that I can observe)

Nice solution. NB this version https://www.supermicro.com/en/products/motherboard/X11SCL-LN4F has 4 gigabit ports but otherwise looks similar except larger form factor (microATX). It's about €70 more expensive here (Sweden).

Another variant is this: https://www.supermicro.com/en/products/motherboard/M11SDV-4CT-LN4F
Bit slower than the Xenon above but fanless, still Mini-ITX and more energy efficient (35W). Also a bit cheaper than above m/board + CPU, at least in Sweden.

8 core version: https://www.supermicro.com/en/products/motherboard/M11SDV-8CT-LN4F
Still low power (30W). Similarly priced to X11SCL-LN4F + Xeon CPU.

The Qotoms are very nice in that they are passively cooled and therefore completely quiet. I've got an i5 myself, in the cupboard in the bedroom as that's where the fiber comes into the house...! However, the Dell small form-factor business PCs are pretty darn quiet as well and if you put a quad Intel gigabit card in one of those (or or or two dual, according to your requirements) you have a higher spec machine with more up-to-date CPUs. At that level hardware you would really struggle to max out the CPUs even if you used IDS, antivirus and VPNs etc. And they are very reliable.

My Qotom has been rock-solid as well so far, but I do wonder if it suddenly is going to die on me one day.

Yeah I've done it as I understand it you're supposed to... a scheduled allow rule above a permanent block rule. Description at the top. So when the allow rule expires (and should kill all its states with it) then the block rule immediately below ensures nothing new gets through until the schedule revives the allow rule again.

Except the states are left intact when the allow rule expires.

Sounds like I'm in the same place as you were...
I think it would be nice to get this fixed in opnsense.

Thanks. Yes I understand that a workaround is to clear states yourself through pfctl commands and cron scripts, however I would rather see the functionality works properly in the product itself. And to your particular solution - I don't want to clear *all* states but only those affected. I.e. I would need to write scripts that only clear the relevant states, according to their tags, and I believe this is exactly the functionality that is already supposedly built-in but is not working.

PS listing all the states using pfctl you can see they are tagged according to the rule that created them. There could be several reasons for this but I think at least one of them is to find all the relevant states and kill them when a given rule expires. Alas this part is not working. As mentioned I could probably hack around it with cron and my own pfctl commands but would prefer to avoid that.

I think the documentation is pretty clear. Re the wording "schedules clear the states of existing connections when the expiration time is reached" - is your point that "expiration time" could refer to something else than expiration of said schedule? I'm not aware of any other type of expiration time that it could possibly refer to including on connections or whatever. I'm pretty sure the intention here is to automatically pfctl kill all states created by the scheduled rule as soon as it expires, but this is not working and as mentioned there are many reports on the Internet of people having had problems with this in pfsense too in the past.

I'll create a bug report for it.