Topics

Hi all,
I'm itching to upgrade to 22.1 particularly for FreeBSD 13 and the potential performance improvements that come with it. Curious - has anyone measured and done this comparison in OpnSense?

My setup is 10Gb WAN and mix 10Gb/1Gb LAN, Chelsio T520 dual 10Gb and Intel I350-T4 passthrough in Proxmox VM, Ryzen 3700x 8 cores on AMD X470, not much else running on the same box. My throughput currently tops out around 4-5Gb/s but I should be able to get more.

Hi,

Just upgraded WAN to 10Gb so trying to get OPNSense up to 10Gb too. The hardware is powerful but performance is poor... I need some help troubleshooting!

Hardware:
ASRock X470-D4U motherboard with Ryzen 3700x CPU (8c/16t, 3.6GHz up to 4.4GHz turbo)
32GB RAM
Intel i350-T4 quad gigabit NIC
Chelsio T420-CR dual 10Gb SFP+
MikroTik CRS328-24P-4S+RM switch

Software environment:
Proxmox 7.0
OPNSense running virtualised with the Intel i350-T4 and Chelsio T420-CR in PCIe passthrough to the VM

And before you say anything... yes I also suspect that it's the virtualisation that somehow causes my performance issues... but I want to be sure before I migrate to bare metal as the virtualisation provides many benefits including easy snapshot backups etc.

Description of symptoms:
Noticed that WAN speed were poor (approx 1.4Gb/s) with Suricata still only consuming approx 50% of total CPU. No improvement with Suricata disabled, and in that case with CPU mostly idle according to top.

So I moved on to test my internal network with iPerf3. I verified that my NAS (TrueNAS, Chelsio T420-CR) and another Proxmox node (Ryzen 5950x, Mellanox ConnectX-4 Lx) saturate 10Gb/s no problem via iPerf3. Both machines however, via the same switch and network cards, into the Chelsio T420-CR in OPNSense, hardly manage to break 1Gb/s:

Code Select Expand

Connecting to host 192.168.200.1, port 5201
[  5] local 192.168.200.10 port 22966 connected to 192.168.200.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   124 MBytes  1.04 Gbits/sec    0    757 KBytes       
[  5]   1.00-2.00   sec   128 MBytes  1.07 Gbits/sec    0   1.30 MBytes       
[  5]   2.00-3.00   sec   132 MBytes  1.11 Gbits/sec   30    810 KBytes       
[  5]   3.00-4.00   sec   124 MBytes  1.04 Gbits/sec    0    937 KBytes       
[  5]   4.00-5.00   sec   133 MBytes  1.12 Gbits/sec    0   1.04 MBytes       
[  5]   5.00-6.00   sec   128 MBytes  1.07 Gbits/sec    0   1.16 MBytes       
[  5]   6.00-7.00   sec   144 MBytes  1.21 Gbits/sec    0   1.28 MBytes       
[  5]   7.00-8.00   sec   138 MBytes  1.15 Gbits/sec    0   1.31 MBytes       
[  5]   8.00-9.00   sec   133 MBytes  1.11 Gbits/sec    0   1.31 MBytes       
[  5]   9.00-10.00  sec   123 MBytes  1.03 Gbits/sec    0   1.31 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.28 GBytes  1.10 Gbits/sec   30             sender
[  5]   0.00-10.01  sec  1.27 GBytes  1.09 Gbits/sec                  receiver

CPU is approx 65% idle during the test so hardly the bottleneck.

Here is the output of "ifconfig -v cxgbe1":

Code Select Expand

cxgbe1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=28c00b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,HWRXTSTMP>
ether 00:07:43:11:2b:18
inet6 fe80::207:43ff:fe11:2b18%cxgbe1 prefixlen 64 scopeid 0x2
inet 192.168.200.1 netmask 0xffffff00 broadcast 192.168.200.255
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
plugged: SFP/SFP+/SFP28 1X Copper Passive (Copper pigtail)
vendor: FS PN: SFPP-PC015 SN: S2108004672-1 DATE: 2021-08-11

/boot/loader.conf.local contains:

Code Select Expand

t4fw_cfg_load="YES"
if_cxgbe_load="YES"

# Disabling cxgbe caps
hw.cxgbe.toecaps_allowed="0"
hw.cxgbe.rdmacaps_allowed="0"
hw.cxgbe.iscsicaps_allowed="0"
hw.cxgbe.fcoecaps_allowed="0"

I have no idea where to go from here. Any ideas on how to troubleshoot to find the bottleneck?

Hi all,
I'll be getting 10Gbit wan soon so will be looking at upgrading my current Qotom i5 (which has worked well!). I've been on these forums before discussing options, including Epyc embedded etc.

I've got one specific question regarding Dell Optiplex SFF, as these seem a popular choice. They actually have two PCIe slots, one x16 and one x4. From what I can tell this is unusual, as many consumer/business PCs only have 1x16 for graphics and then x1 slots for other stuff. I am planning to put a Chelsio 10Gb SFP+ in one slot and Intel i350-T4 in the other, so I need both slots.

Does anyone know if this would work, or if the Dell is hardwired to assume that the x16 card is a GPU? I've seen some reports like that for earlier versions.

So I've been looking at the AMD Epyc embedded for a while, considering one of the SuperMicro M11SDV motherboards with its 4 on-board Intel NICs. In truth the humble 3101 would probably do me fine (domestic use, 1Gb fiber wan, Suricata, VPN) but it bothers me that the embedded series are still 1st generation Epyc when the regular series are now on 3rd, with large performance and efficiency gains. And prices for the M11SDV haven't even adjusted, they are the same as when they were released a few years back.

I don't see much if any information online on whether a refresh is expected. Any thoughts or clues from anyone on this forum?

Otherwise I'll probably bite the bullet and get desktop hardware and in that case a quiet business PC with intel CPU and PCIe quad NIC. However I prefer server hardware if poss... for IPMI if nothing else.

Hi guys,
My OPNSense has started to freeze up completely on any config change I make through the GUI. I am not sure what has changed other than I installed the node exporter plugin.

The symptom is a stalled GUI and complete system freeze; it doesn't even respond to ping.

Log file around the time shows

Code Select Expand

[...]
2021-01-24T21:33:01 opnsense[53206] /usr/local/etc/rc.linkup: DEVD Ethernet attached event for lan
2021-01-24T21:33:01 kernel igb1: link state changed to UP
2021-01-24T21:32:58 kernel pflog0: promiscuous mode enabled
2021-01-24T21:32:58 kernel pflog0: promiscuous mode disabled
2021-01-24T21:32:57 opnsense[63288] /usr/local/etc/rc.linkup: DEVD Ethernet detached event for lan
2021-01-24T21:32:57 kernel igb1: link state changed to DOWN
2021-01-24T21:32:55 configctl[1695] event @ 1611520374.79 exec: system event config_changed
2021-01-24T21:32:55 configctl[1695] event @ 1611520374.79 msg: Jan 24 21:32:54 xxx.yyy.org.uk config[79201]: config-event: new_config /conf/backup/config-1611520374.7843.xml

I.e. it enter some re-initialization of all network interfaces after my config change? In any case it seems to render OPNsense dead and unresponsive afterwards. On a soft reset (using hardware reset button) it shuts down ok and when it come back up it has taken my config change and works as normal again.

Edit: should have said I am running 20.7.8 and upgraded recently. I saw the same behaviour on 20.7.7 just before I upgraded. Also, I notice now the freeze doesn't always happen. I am not sure yet what the pattern is though.

Where should I start to troubleshoot...?

Hi - does someone use this combination successfully for monitoring and graphing? I can't get it to behave properly but I don't know if I'm misinterpreting metrics or if the FreeBSD implementation of node exporter is just quirky.

So... I was happy enough to find the os-node_exporter plugin. This installs the Prometheus exporter and it runs fine. Prometheus picks up the metrics and all good... except that the node_network_[receive|transmit]_bytes_total metric doesn't actually do seem to reflect anything that goes in and out. No matter what I pump through the firewall, the metrics maintain a very slow, linear increment across all devices and seemingly unrelated to what actually goes on. I can't figure out why - and by comparison, my Linux systems behave exactly as expected from Prometheus documentation and tutorials, so I don't think the problem lies with my queries.

Anyone else managed to get this to work?

Hi all

So in previous threads I've recommended Qotom. Still can't say anything bad about them. However one of my (managed) switches spontaneously broke the other day and it was a pain to replace (and created bad karma in the household)... So I dread the same happening to my router and have decided to look for something a bit more robust. The Qotom can be repurposed into an esxi node, easy to replace if it dies...

The router is such a critical component that I'm prepared to throw a little bit of money at it if it gives me the peace of mind that it'll then just chug along for 5-10 years (assuming I won't botch it up with software upgrades...!).

So help me decide.
Requirements: silent or whisper quiet (bedroom cupboard install). Enough grunt to drive 1Gb fiber wan with intrusion detection, vpn, and 20+ clients. 4 NICs.

Considered options:

SuperMicro AMD Epyc 3101 (M11SDV-4CT-LN4F) or 3201 with 4 built-in NICs. SuperMicro mITX case (SuperChassis E300).
Pros: purpose built hardware. Enterprise grade stuff. IPMI. Cons: is the case quiet? AMD compatibility for FreeBSD? Single core performance?

Dell Optiplex 3070 SFF (i3 gen 9) with Intel i350-T4 quad NIC in PCI-e.
Pros: guaranteed quiet. Well-renowned reliability (although no ECC ram). Proper desktop cpu (with fast single core performance). Cons: some reports on the internet of Intel i350-T4 built for server hardware and not working with Dell desktops.

Help me decide!

Hi,

This seems to be a long standing issue in pfsense as well and since several years back:
https://forum.netgate.com/topic/69331/scheduled-blocks-won-t-work-without-manual-states-reset/2

Long and the short of it; in order to restrict kids' access to Internet at certain times of the day I've got the following rules:

Alias: kids' devices (KD)

Enable kids' devices to any, source KD, on schedule
Block kids' devices to any (source KD)
Default allow any other LAN to any (source !KD)

This almost works... except that states are not killed when the scheduled pass rule expires. So any new connection is blocked as expected, but already open ones are kept alive, which means the kids continue to play... until they have to switch page or whatever and suddenly find themselves locked out.

Firewall -> Advanced -> Schedule States is UNchecked (somewhat non-intuitively, but that's what everyone says)

As mentioned and as per the link above this seems to be an old issue and inherited into opnsense..?

Any ideas...?

Hi all,

I've been trying to get auto discovery of my web proxy to work on my LAN with limited success and I'm not sure how to debug it so wondering if there are success stories out there or if this is just intrinsically flaky with a diverse set of clients...?

My test client is Mac OS X 10.14.3. I have followed the guide, although not managing to make it work, experimented a bit further. My current config in OPNSense generates this file:

Code Select Expand


/*
  PAC file created via OPNsense
  To use this file you have to enter its URL into your browsers network settings.
*/
function FindProxyForURL(url, host) {

if (!((isPlainHostName(host)) || (shExpMatch(host, "*.mydomain.com")))) {
return "PROXY 192.168.200.1:3128";
}

   // If no rule exists - use a direct connection
   return "DIRECT";
}


... which I think should work. I have added an option to the DHCP server to send the URL on field 252 according to the instructions.

Indeed, on the client:

Code Select Expand


$ scutil --proxy
<dictionary> {
  ExceptionsList : <array> {
    0 : *.local
    1 : 169.254/16
  }
  FTPPassive : 1
  HTTPEnable : 0
  HTTPSEnable : 0
  ProxyAutoConfigEnable : 1
  ProxyAutoConfigURLString : http://192.168.200.1/wpad.dat
  ProxyAutoDiscoveryEnable : 1
}


... looks promising (the URL is picked up from DHCP).

HOWEVER. Safari completely ignores the proxy setting and just loads pages direct instead. Chrome on the other hand actually honours the proxy setting... unless in "incognity mode" in which case it also ignores the proxy setting and hits the pages directly! I can see this by following Squid's access log while loading up different pages on the client using various browsers.

Do I need to block direct access in order to get the browsers to actually use the proxy settings specified?

Very odd... Anyone got experience from this?

Hi! I've got a weird problem and can't figure out what I'm doing wrong.

Setup:
LAN: 192.168.200.1/24
DMZ: 192.168.1.1/24

When I'm done I'm going to let LAN access DMZ but not the other way around. For now I've left the default LAN "pass everything" and added the equivalent for DMZ (interface DMZ, source DMZ net, dest any, action Pass).

However when I access DMZ from LAN I get dropouts and timeouts (e.g. when setting up NFS connection). Debugging this, accessing a web management GUI in the DMZ from my laptop on the LAN, some packets are let through on the basis of the default LAN rule but then some dropped on the default deny, and I can't see the reason for this at all.

See attached opn1.jpg screenshot from the Live View firewall log and opn2.jpg from the drill down of one of the blocked connections - how can that not be caught by the same default pass rule?

Also, I don't understand why "default allow LAN to any" comes in pairs with DMZ "let anything out from firewall itself".

Totally stumped on this so would appreciate pointers!

Hi! First post on the forum. Just installed opnsense on a Qotom i5 mini pc at home and got it up and running fine (as router + firewall). I also managed to configure IPSec for road warriors as described here: https://wiki.opnsense.org/manual/how-tos/ipsec-road.html. All good so far and very happy!

There is however one thing I cannot figure out. When I connect to my opnsense firewall via IPSec (as road warrior) I would like to access my LAN (works) but also use my opnsense as Internet gateway (i.e. NAT through the WAN as if I would have been on the internal lan already). I cannot figure this out.

My setup:
VPN -> IPSec -> Mobile Clients; Virtual Address Pool = 10.10.0.0/24, DNS Servers = 192.168.1.1
Firewall -> Rules -> IPSec; single rule allowing anything to anything on the IPSEC interface
Firewall -> NAT -> Outbound; Hybrid outbound NAT rule generation, added a rule Interface = WAN, Source address=10.10.0.0/24, Translation / target = Interface address.

With this, I can connect to the VPN from an iOS client and successful access the LAN. When I try to load external pages from iOS Safari, I can see the connections being let through in Firewall -> Log Files -> Log view however iOS does not get any response and eventually times out.

I cannot figure this out... Would appreciate some pointers!

I'm running OPNsense 18.1.4-amd64 by the way...