Messages

Does this also happen with 20.7.8?

i didn't test it on 20.7.8
i tried to downgrade to 20.7.8 but it didn't succeed:

Code Select Expand

opnsense-update -r 20.7.8[/s]
Fetching base-20.7.8-amd64.txz: .. failed, no signature found


edit:
did a clean install of 20.7, upgraded it to 20.7.8_4
same results...
not sure what the problem is, guess i have to investigate if its not OPNsense but unRAID or the switch.

a view days ago i did upgrade OPNsense and my server to 10Gbit NICs

hardware:
Intel Ethernet Converged Network Adapter X540-T2  (OPNsense)
Mellanox ConnectX-3 CX311A (unRAID server)
MikroTik Cloud Smart Switch 326-24G-2S+RM (switch)


Iperf3 results:

suricata OFF = cpu usage 40% / 51%

Code Select Expand

iperf3 -c 10.0.3.1 -t 60 -i 10
Connecting to host 10.0.3.1, port 5201
[  5] local 10.0.3.2 port 35558 connected to 10.0.3.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-10.00  sec  3.03 GBytes  2.60 Gbits/sec    0    252 KBytes       
[  5]  10.00-20.00  sec  2.99 GBytes  2.57 Gbits/sec    0    246 KBytes       
[  5]  20.00-30.00  sec  2.98 GBytes  2.56 Gbits/sec    0    243 KBytes       
[  5]  30.00-40.00  sec  2.96 GBytes  2.54 Gbits/sec    0    209 KBytes       
[  5]  40.00-50.00  sec  2.93 GBytes  2.52 Gbits/sec    0    277 KBytes       
[  5]  50.00-60.00  sec  2.97 GBytes  2.55 Gbits/sec    0    260 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec  17.9 GBytes  2.56 Gbits/sec    0             sender
[  5]   0.00-60.00  sec  17.9 GBytes  2.56 Gbits/sec                  receiver

iperf Done.

iperf3 -c 10.0.3.1 -t 60 -i 10 -R
Connecting to host 10.0.3.1, port 5201
Reverse mode, remote host 10.0.3.1 is sending
[  5] local 10.0.3.2 port 36642 connected to 10.0.3.1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  3.82 GBytes  3.28 Gbits/sec                 
[  5]  10.00-20.00  sec  3.89 GBytes  3.35 Gbits/sec                 
[  5]  20.00-30.00  sec  3.82 GBytes  3.28 Gbits/sec                 
[  5]  30.00-40.00  sec  3.75 GBytes  3.22 Gbits/sec                 
[  5]  40.00-50.00  sec  3.60 GBytes  3.09 Gbits/sec                 
[  5]  50.00-60.00  sec  3.76 GBytes  3.23 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec  22.6 GBytes  3.24 Gbits/sec  8384             sender
[  5]   0.00-60.00  sec  22.6 GBytes  3.24 Gbits/sec                  receiver

iperf Done.


suricata ON = cpu usage 59% / 76%

Code Select Expand

iperf3 -c 10.0.3.1 -t 60 -i 10
Connecting to host 10.0.3.1, port 5201
[  5] local 10.0.3.2 port 43546 connected to 10.0.3.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-10.00  sec   753 MBytes   632 Mbits/sec    2   5.66 KBytes       
[  5]  10.00-20.00  sec   748 MBytes   627 Mbits/sec    8    219 KBytes       
[  5]  20.00-30.00  sec   745 MBytes   625 Mbits/sec    5    209 KBytes       
[  5]  30.00-40.00  sec   774 MBytes   649 Mbits/sec   12    188 KBytes       
[  5]  40.00-50.00  sec   744 MBytes   624 Mbits/sec    5    218 KBytes       
[  5]  50.00-60.00  sec   795 MBytes   667 Mbits/sec    7    215 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec  4.45 GBytes   637 Mbits/sec   39             sender
[  5]   0.00-60.00  sec  4.45 GBytes   637 Mbits/sec                  receiver

iperf Done.

iperf3 -c 10.0.3.1 -t 60 -i 10 -R
Connecting to host 10.0.3.1, port 5201
Reverse mode, remote host 10.0.3.1 is sending
[  5] local 10.0.3.2 port 38420 connected to 10.0.3.1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.40 GBytes  1.21 Gbits/sec                 
[  5]  10.00-20.00  sec  1.37 GBytes  1.17 Gbits/sec                 
[  5]  20.00-30.00  sec  1.40 GBytes  1.20 Gbits/sec                 
[  5]  30.00-40.00  sec  1.39 GBytes  1.19 Gbits/sec                 
[  5]  40.00-50.00  sec  1.40 GBytes  1.20 Gbits/sec                 
[  5]  50.00-60.00  sec  1.41 GBytes  1.21 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec  8.37 GBytes  1.20 Gbits/sec   18             sender
[  5]   0.00-60.00  sec  8.37 GBytes  1.20 Gbits/sec                  receiver

iperf Done.

UDP:

Code Select Expand

iperf3 -c 10.0.3.1 -u -t 60 -i 10 -b 10000M
Connecting to host 10.0.3.1, port 5201
[  5] local 10.0.3.2 port 59369 connected to 10.0.3.1 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-10.00  sec  2.88 GBytes  2.48 Gbits/sec  2138663 
[  5]  10.00-20.00  sec  2.89 GBytes  2.48 Gbits/sec  2143473 
[  5]  20.00-30.00  sec  2.85 GBytes  2.45 Gbits/sec  2110755 
[  5]  30.00-40.00  sec  2.81 GBytes  2.41 Gbits/sec  2081894 
[  5]  40.00-50.00  sec  2.87 GBytes  2.46 Gbits/sec  2126508 
[  5]  50.00-60.00  sec  2.92 GBytes  2.51 Gbits/sec  2167670 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-60.00  sec  17.2 GBytes  2.47 Gbits/sec  0.000 ms  0/12768963 (0%)  sender
[  5]   0.00-60.01  sec  12.5 GBytes  1.79 Gbits/sec  0.001 ms  3471092/12768963 (27%)  receiver

iperf Done.

i know that there are some posts going on that on OPNsense 21.1 there is slowdown...

its looks like i am not the only one who doesn't get 10Gb speeds...

i tried a view tuneables but it didn't do anything:

Code Select Expand

kern.ipc.maxsockbuf:  16777216
net.inet.ip.intr_queue_maxlen:  2048
net.inet.tcp.recvspace:  4194304
net.inet.tcp.sendspace:  2097152
net.inet.tcp.recvbuf_max:  16777216
net.inet.tcp.recvbuf_inc:  524288
net.inet.tcp.sendbuf_max:  16777216
net.inet.tcp.sendbuf_inc:  32768
net.route.netisr_maxqlen:  2048
net.link.ifqmaxlen:  2048

need to do more investigation why it won't do 10Gb, maybe its the switch who has wrong settings (it using the defaults settings) or maybe it's the unRAID server...

today i did upgrade OPNsense end my server to 10Gbit NICs

hardware:
Intel Ethernet Converged Network Adapter X540-T2  (OPNsense)
Mellanox ConnectX-3 CX311A (unRAID server)
MikroTik Cloud Smart Switch 326-24G-2S+RM (switch)


Iperf results:

suricata OFF = cpu usage 40% / 51%

Code Select Expand

iperf3 -c 10.0.3.1 -t 60 -i 10
Connecting to host 10.0.3.1, port 5201
[  5] local 10.0.3.2 port 35558 connected to 10.0.3.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-10.00  sec  3.03 GBytes  2.60 Gbits/sec    0    252 KBytes       
[  5]  10.00-20.00  sec  2.99 GBytes  2.57 Gbits/sec    0    246 KBytes       
[  5]  20.00-30.00  sec  2.98 GBytes  2.56 Gbits/sec    0    243 KBytes       
[  5]  30.00-40.00  sec  2.96 GBytes  2.54 Gbits/sec    0    209 KBytes       
[  5]  40.00-50.00  sec  2.93 GBytes  2.52 Gbits/sec    0    277 KBytes       
[  5]  50.00-60.00  sec  2.97 GBytes  2.55 Gbits/sec    0    260 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec  17.9 GBytes  2.56 Gbits/sec    0             sender
[  5]   0.00-60.00  sec  17.9 GBytes  2.56 Gbits/sec                  receiver

iperf Done.

iperf3 -c 10.0.3.1 -t 60 -i 10 -R
Connecting to host 10.0.3.1, port 5201
Reverse mode, remote host 10.0.3.1 is sending
[  5] local 10.0.3.2 port 36642 connected to 10.0.3.1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  3.82 GBytes  3.28 Gbits/sec                 
[  5]  10.00-20.00  sec  3.89 GBytes  3.35 Gbits/sec                 
[  5]  20.00-30.00  sec  3.82 GBytes  3.28 Gbits/sec                 
[  5]  30.00-40.00  sec  3.75 GBytes  3.22 Gbits/sec                 
[  5]  40.00-50.00  sec  3.60 GBytes  3.09 Gbits/sec                 
[  5]  50.00-60.00  sec  3.76 GBytes  3.23 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec  22.6 GBytes  3.24 Gbits/sec  8384             sender
[  5]   0.00-60.00  sec  22.6 GBytes  3.24 Gbits/sec                  receiver

iperf Done.


suricata ON = cpu usage 59% / 76%

Code Select Expand

iperf3 -c 10.0.3.1 -t 60 -i 10
Connecting to host 10.0.3.1, port 5201
[  5] local 10.0.3.2 port 37868 connected to 10.0.3.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-10.00  sec  2.80 GBytes  2.40 Gbits/sec    0   5.66 KBytes       
[  5]  10.00-20.00  sec  2.81 GBytes  2.42 Gbits/sec    0    272 KBytes       
[  5]  20.00-30.00  sec  2.78 GBytes  2.38 Gbits/sec    0    223 KBytes       
[  5]  30.00-40.00  sec  2.79 GBytes  2.40 Gbits/sec    0    240 KBytes       
[  5]  40.00-50.00  sec  1.53 GBytes  1.32 Gbits/sec    4   1.41 KBytes       
[  5]  50.00-60.01  sec  0.00 Bytes  0.00 bits/sec    2   1.41 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.01  sec  12.7 GBytes  1.82 Gbits/sec    6             sender
[  5]   0.00-61.65  sec  12.7 GBytes  1.77 Gbits/sec                  receiver

iperf Done.

iperf3 -c 10.0.3.1 -t 60 -i 10 -R
Connecting to host 10.0.3.1, port 5201
Reverse mode, remote host 10.0.3.1 is sending
[  5] local 10.0.3.2 port 38420 connected to 10.0.3.1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.40 GBytes  1.21 Gbits/sec                 
[  5]  10.00-20.00  sec  1.37 GBytes  1.17 Gbits/sec                 
[  5]  20.00-30.00  sec  1.40 GBytes  1.20 Gbits/sec                 
[  5]  30.00-40.00  sec  1.39 GBytes  1.19 Gbits/sec                 
[  5]  40.00-50.00  sec  1.40 GBytes  1.20 Gbits/sec                 
[  5]  50.00-60.00  sec  1.41 GBytes  1.21 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec  8.37 GBytes  1.20 Gbits/sec   18             sender
[  5]   0.00-60.00  sec  8.37 GBytes  1.20 Gbits/sec                  receiver

iperf Done.

@annoniempjuh you tested iperf3 with UDP. using udp i get simila numbers.
My result [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  44.5 MBytes   373 Mbits/sec   47    626 KBytes       
[  5]   1.00-2.00   sec  78.7 MBytes   660 Mbits/sec    0    711 KBytes       
[  5]   2.00-3.00   sec  77.4 MBytes   649 Mbits/sec    1    559 KBytes       
[  5]   3.00-4.00   sec  78.7 MBytes   660 Mbits/sec    0    656 KBytes       
[  5]   4.00-5.00   sec  77.4 MBytes   650 Mbits/sec    0    741 KBytes       
[  5]   5.00-6.00   sec  74.9 MBytes   628 Mbits/sec    5    585 KBytes       
[  5]   6.00-7.00   sec  78.7 MBytes   660 Mbits/sec    0    680 KBytes       
[  5]   7.00-8.00   sec  78.7 MBytes   660 Mbits/sec    0    764 KBytes       
[  5]   8.00-9.00   sec  78.6 MBytes   660 Mbits/sec    8    618 KBytes       
[  5]   9.00-10.00  sec  78.7 MBytes   660 Mbits/sec    0    710 KBytes 


was with plain settings: iperf3 -c <serverip>

What i ment with "What decreased the performance between 20.7.5 and 20.7.8?" was refering to klamath post.
Still this question remains unanswered. Maybe franco can shine a little light on this.

didn't notice i was using UDP...

Code Select Expand

iperf3 -c 10.0.3.1
Connecting to host 10.0.3.1, port 5201
[  5] local 10.0.3.2 port 44238 connected to 10.0.3.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  80.9 MBytes   679 Mbits/sec    0    243 KBytes       
[  5]   1.00-2.00   sec  63.4 MBytes   532 Mbits/sec    0    243 KBytes       
[  5]   2.00-3.00   sec  39.6 MBytes   332 Mbits/sec    0    243 KBytes       
[  5]   3.00-4.00   sec  49.5 MBytes   416 Mbits/sec    1    243 KBytes       
[  5]   4.00-5.00   sec  56.8 MBytes   476 Mbits/sec    0    243 KBytes       
[  5]   5.00-6.00   sec  54.5 MBytes   457 Mbits/sec    0    246 KBytes       
[  5]   6.00-7.00   sec  48.3 MBytes   405 Mbits/sec    1    246 KBytes       
[  5]   7.00-8.00   sec  44.4 MBytes   372 Mbits/sec    0    243 KBytes       
[  5]   8.00-9.00   sec  74.6 MBytes   626 Mbits/sec    0    246 KBytes       
[  5]   9.00-10.00  sec  35.9 MBytes   301 Mbits/sec    0   5.66 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   548 MBytes   460 Mbits/sec    2             sender
[  5]   0.00-10.00  sec   546 MBytes   458 Mbits/sec                  receiver

iperf Done.

its indeed slower then i expected ::)

Suricata is in IPS mode ;)
i only tested v20.1.8_1 and v21.1

i am on OPNsense 21.1 and i don't have any problem?

Code Select Expand

iperf3 -c 10.0.3.1 -u -t 60 -i 10 -b 1000M
Connecting to host 10.0.3.1, port 5201
[  5] local 10.0.3.2 port 60596 connected to 10.0.3.1 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-10.00  sec  1.10 GBytes   943 Mbits/sec  813711 
[  5]  10.00-20.00  sec  1.10 GBytes   943 Mbits/sec  813645 
[  5]  20.00-30.00  sec  1.10 GBytes   943 Mbits/sec  813746 
[  5]  30.00-40.00  sec  1.10 GBytes   943 Mbits/sec  813787 
[  5]  40.00-50.00  sec  1.10 GBytes   943 Mbits/sec  813730 
[  5]  50.00-60.00  sec  1.10 GBytes   943 Mbits/sec  813777 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-60.00  sec  6.58 GBytes   943 Mbits/sec  0.000 ms  0/4882396 (0%)  sender
[  5]   0.00-60.00  sec  6.56 GBytes   939 Mbits/sec  0.011 ms  20901/4882368 (0.43%)  receiver

iperf Done.

Hardware:
AMD Ryzen 3 2200G with Radeon Vega Graphics (4 cores)
8GB RAM
Intel PRO/1000 PT Dual Port Server Adapter (PCI-e 4x) (driver: EM)

when i was on OPNsense 20.1.8_1 it was:

Code Select Expand

iperf3 -c 10.0.3.31 -u -t 60 -i 10 -b 1000M
Connecting to host 10.0.3.31, port 5201
[  5] local 10.0.3.1 port 44924 connected to 10.0.3.31 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-10.00  sec  1.16 GBytes  1000 Mbits/sec  856118
[  5]  10.00-20.00  sec  1.16 GBytes  1.00 Gbits/sec  856870
[  5]  20.00-30.00  sec  1.16 GBytes  1000 Mbits/sec  857061
[  5]  30.00-40.00  sec  1.16 GBytes  1.00 Gbits/sec  856166
[  5]  40.00-50.00  sec  1.16 GBytes  1000 Mbits/sec  857113
[  5]  50.00-60.00  sec  1.16 GBytes  1.00 Gbits/sec  857192
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-60.00  sec  6.98 GBytes  1000 Mbits/sec  0.000 ms  0/5140520 (0%)  sender
[  5]   0.00-60.00  sec  3.34 GBytes   479 Mbits/sec  0.046 ms  2680818/5140353 (52%)  receiver

iperf Done.

next week i'm going to upgrade to 10Gbe nic and fiber, will test if there will be a decrease of performance...

can confirm, editing the config file and remove some old code will reset suricata etc. after this try i did a full reset and put the edited config back, works fine, but WAN still drops offline. mostly while whatsing some youtube video on my kodi machine...

more info, because its not fixed.

My ISP is Ziggo (dutch) its using only IPV4 on DHCP, the modem is an UBEE 1318ZG in Bridge modus.
OPNsense version: 20.7.3
hardware:
AMD ryzen 3 2200g
ASRock Fatal1ty B450 Gaming-ITX/ac
Crucial Ballistix Sport LT BLS8G4D32AESBK
Intel PRO/1000 PT Dual Port Server Adapter (PCI-e 4x) (e1000)

WAN drops offline, dpinger is showing it. it wont come back automatically, have to do a manual restart of suricata (it work always!)
the isp modem logbooks are fine, nothing what can cause this problem.
Logbooks are showing NOTHING!!!!!!! very annoying...

hope someone has some tip/tricks to try...

at 08:11:02 WAN drops offline
LOGBOOKS:
General:

Code Select Expand


Date                                 Process                   Line
2020-10-20T08:13:12 syslog-ng[29923] Destination timeout has elapsed, closing connection; fd='31'
2020-10-20T08:13:07 syslog-ng[29923] Destination timeout has elapsed, closing connection; fd='33'
2020-10-20T08:13:07 syslog-ng[29923] Destination timeout has elapsed, closing connection; fd='32'
2020-10-20T08:13:03 syslog-ng[29923] Destination timeout has elapsed, closing connection; fd='25'
2020-10-20T08:12:12 opnsense[54388] plugins_configure hosts (execute task : unbound_hosts_generate())
2020-10-20T08:12:12 opnsense[54388] plugins_configure hosts (execute task : dnsmasq_hosts_generate())
2020-10-20T08:12:12 opnsense[54388] plugins_configure hosts ()
2020-10-20T08:12:12 opnsense[54388] /usr/local/etc/rc.newwanip: On (IP address: 10.0.37.1) (interface: Pihole[opt5]) (real interface: em1_vlan1337).
2020-10-20T08:12:12 opnsense[54388] /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'em1_vlan1337'
2020-10-20T08:12:12 opnsense[6369] /usr/local/etc/rc.linkup: Hotplug event detected for Pihole(opt5) but ignoring since interface is configured with static IP (10.0.37.1 ::)
2020-10-20T08:12:12 opnsense[60071] plugins_configure hosts (execute task : unbound_hosts_generate())
2020-10-20T08:12:12 opnsense[60071] plugins_configure hosts (execute task : dnsmasq_hosts_generate())
2020-10-20T08:12:12 opnsense[60071] plugins_configure hosts ()
2020-10-20T08:12:12 opnsense[60071] /usr/local/etc/rc.newwanip: On (IP address: 10.0.13.1) (interface: GuestLan[opt3]) (real interface: em1_vlan13).
2020-10-20T08:12:12 opnsense[60071] /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'em1_vlan13'
2020-10-20T08:12:12 opnsense[51800] /usr/local/etc/rc.linkup: Hotplug event detected for GuestLan(opt3) but ignoring since interface is configured with static IP (10.0.13.1 ::)
2020-10-20T08:12:12 opnsense[12331] plugins_configure hosts (execute task : unbound_hosts_generate())
2020-10-20T08:12:12 opnsense[12331] plugins_configure hosts (execute task : dnsmasq_hosts_generate())
2020-10-20T08:12:12 opnsense[12331] plugins_configure hosts ()
2020-10-20T08:12:12 opnsense[12331] /usr/local/etc/rc.newwanip: On (IP address: 10.0.12.1) (interface: IOTlan[opt2]) (real interface: em1_vlan12).
2020-10-20T08:12:12 opnsense[12331] /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'em1_vlan12'
2020-10-20T08:12:12 opnsense[38844] /usr/local/etc/rc.linkup: Hotplug event detected for IOTlan(opt2) but ignoring since interface is configured with static IP (10.0.12.1 ::)
2020-10-20T08:12:11 opnsense[18637] plugins_configure hosts (execute task : unbound_hosts_generate())
2020-10-20T08:12:11 opnsense[18637] plugins_configure hosts (execute task : dnsmasq_hosts_generate())
2020-10-20T08:12:11 opnsense[18637] plugins_configure hosts ()
2020-10-20T08:12:11 opnsense[18637] /usr/local/etc/rc.newwanip: On (IP address: 10.0.11.1) (interface: NVRlan[opt1]) (real interface: em1_vlan11).
2020-10-20T08:12:11 opnsense[18637] /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'em1_vlan11'
2020-10-20T08:12:11 opnsense[48008] /usr/local/etc/rc.linkup: Hotplug event detected for NVRlan(opt1) but ignoring since interface is configured with static IP (10.0.11.1 ::)
2020-10-20T08:12:11 opnsense[7364] plugins_configure hosts (execute task : unbound_hosts_generate())
2020-10-20T08:12:11 opnsense[7364] plugins_configure hosts (execute task : dnsmasq_hosts_generate())
2020-10-20T08:12:11 opnsense[7364] plugins_configure hosts ()
2020-10-20T08:12:11 opnsense[7364] /usr/local/etc/rc.newwanip: On (IP address: 10.0.3.1) (interface: LAN[lan]) (real interface: em1).
2020-10-20T08:12:11 opnsense[7364] /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'em1'
2020-10-20T08:12:11 opnsense[21211] /usr/local/etc/rc.linkup: Hotplug event detected for LAN(lan) but ignoring since interface is configured with static IP (10.0.3.1 ::)
2020-10-20T08:12:09 opnsense[56743] /usr/local/etc/rc.dyndns: Dynamic DNS: (Success) IP Address Updated Successfully!
2020-10-20T08:12:09 opnsense[56743] /usr/local/etc/rc.dyndns: Dynamic DNS: updating cache file /var/cache/dyndns_wan_myddomain.duckdns.org_0.cache: XX.xx.xx.XX
2020-10-20T08:12:09 opnsense[56743] /usr/local/etc/rc.dyndns: Dynamic DNS (myddomain.duckdns.org): XX.xx.xx.XX extracted
2020-10-20T08:12:09 opnsense[56743] /usr/local/etc/rc.dyndns: Dynamic DNS (myddomain.duckdns.org): Current Service: custom
2020-10-20T08:12:09 opnsense[56743] /usr/local/etc/rc.dyndns: Dynamic DNS (myddomain.duckdns.org): _checkStatus() starting.
2020-10-20T08:12:09 upsmon[80213] Communications with UPS ups@10.0.3.2 established
2020-10-20T08:12:08 opnsense[56743] /usr/local/etc/rc.dyndns: Dynamic DNS (myddomain.duckdns.org via Custom): _update() starting.
2020-10-20T08:12:08 opnsense[56743] /usr/local/etc/rc.dyndns: Dynamic DNS (myddomain.duckdns.orgg): running dyndns_failover_interface for wan. found em0
2020-10-20T08:12:08 opnsense[56743] /usr/local/etc/rc.dyndns: Dynamic DNS (myddomain.duckdns.org): XX.xx.xx.XX extracted
2020-10-20T08:12:08 opnsense[56743] /usr/local/etc/rc.dyndns: Dynamic DNS: updatedns() starting
2020-10-20T08:12:07 opnsense[28380] plugins_configure dns (execute task : unbound_configure_do())
2020-10-20T08:12:07 opnsense[28380] plugins_configure dns (execute task : dnsmasq_configure_do())
2020-10-20T08:12:07 opnsense[28380] plugins_configure dns ()
2020-10-20T08:12:07 opnsense[28380] plugins_configure dhcp (execute task : dhcpd_dhcp_configure())
2020-10-20T08:12:07 opnsense[28380] plugins_configure dhcp ()
2020-10-20T08:12:07 opnsense[28380] plugins_configure ipsec (execute task : ipsec_configure_do(,wan))
2020-10-20T08:12:07 opnsense[28380] plugins_configure ipsec (,wan)
2020-10-20T08:12:07 opnsense[28380] /usr/local/etc/rc.linkup: ROUTING: keeping current default gateway 'XX.xx.xx.XX'
2020-10-20T08:12:07 opnsense[28380] /usr/local/etc/rc.linkup: ROUTING: setting IPv4 default route to XX.xx.xx.XX
2020-10-20T08:12:07 opnsense[28380] /usr/local/etc/rc.linkup: ROUTING: IPv4 default gateway set to wan
2020-10-20T08:12:07 opnsense[28380] /usr/local/etc/rc.linkup: ROUTING: entering configure using 'wan'
2020-10-20T08:12:07 opnsense[98376] plugins_configure newwanip (execute task : webgui_configure_do(,wan))
2020-10-20T08:12:07 opnsense[98376] plugins_configure newwanip (execute task : vxlan_configure_interface())
2020-10-20T08:12:07 opnsense[98376] plugins_configure newwanip (execute task : unbound_configure_do(,wan))
2020-10-20T08:12:07 opnsense[98376] plugins_configure newwanip (execute task : openssh_configure_do(,wan))
2020-10-20T08:12:07 opnsense[98376] plugins_configure newwanip (execute task : opendns_configure_do())
2020-10-20T08:12:07 opnsense[98376] plugins_configure newwanip (execute task : ntpd_configure_defer())
2020-10-20T08:12:06 opnsense[98376] /usr/local/etc/rc.newwanip: Dynamic DNS: (Success) IP Address Updated Successfully!
2020-10-20T08:12:06 opnsense[98376] /usr/local/etc/rc.newwanip: Dynamic DNS: updating cache file /var/cache/dyndns_wan_myddomain.duckdns.org_0.cache: XX.xx.xx.XX
2020-10-20T08:12:06 opnsense[98376] /usr/local/etc/rc.newwanip: Dynamic DNS (myddomain.duckdns.org): XX.xx.xx.XX extracted
2020-10-20T08:12:06 opnsense[98376] /usr/local/etc/rc.newwanip: Dynamic DNS (myddomain.duckdns.org): Current Service: custom
2020-10-20T08:12:06 opnsense[98376] /usr/local/etc/rc.newwanip: Dynamic DNS (myddomain.duckdns.org): _checkStatus() starting.
2020-10-20T08:12:04 upsmon[80213] Communications with UPS ups@10.0.3.2 lost
2020-10-20T08:12:04 upsmon[80213] Poll UPS [ups@10.0.3.2] failed - Server disconnected
2020-10-20T08:12:04 opnsense[98376] /usr/local/etc/rc.newwanip: Dynamic DNS (myddomain.duckdns.org via Custom): _update() starting.
2020-10-20T08:12:04 opnsense[98376] /usr/local/etc/rc.newwanip: Dynamic DNS (myddomain.duckdns.org): running dyndns_failover_interface for wan. found em0
2020-10-20T08:12:04 opnsense[98376] /usr/local/etc/rc.newwanip: Dynamic DNS (myddomain.duckdns.org): XX.xx.xx.XX extracted
2020-10-20T08:12:04 opnsense[98376] /usr/local/etc/rc.newwanip: Dynamic DNS: updatedns() starting
2020-10-20T08:12:04 opnsense[98376] plugins_configure newwanip (execute task : dyndns_configure_do(,wan))
2020-10-20T08:12:04 opnsense[98376] plugins_configure newwanip (,wan)
2020-10-20T08:12:04 opnsense[98376] /usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface WAN.
2020-10-20T08:12:04 opnsense[98376] plugins_configure vpn (execute task : openvpn_configure_do(,wan))
2020-10-20T08:12:04 opnsense[98376] plugins_configure vpn (execute task : ipsec_configure_do(,wan))
2020-10-20T08:12:04 opnsense[98376] plugins_configure vpn (,wan)
2020-10-20T08:12:03 opnsense[98376] plugins_configure monitor (execute task : dpinger_configure_do())
2020-10-20T08:12:03 opnsense[98376] plugins_configure monitor ()
2020-10-20T08:12:03 opnsense[98376] /usr/local/etc/rc.newwanip: ROUTING: keeping current default gateway 'XX.xx.xx.XX'
2020-10-20T08:12:03 opnsense[98376] /usr/local/etc/rc.newwanip: ROUTING: setting IPv4 default route to XX.xx.xx.XX
2020-10-20T08:12:03 opnsense[98376] /usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to wan
2020-10-20T08:12:03 opnsense[98376] /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'wan'
2020-10-20T08:12:03 opnsense[98376] plugins_configure hosts (execute task : unbound_hosts_generate())
2020-10-20T08:12:03 opnsense[98376] plugins_configure hosts (execute task : dnsmasq_hosts_generate())
2020-10-20T08:12:03 opnsense[98376] plugins_configure hosts ()
2020-10-20T08:12:03 opnsense[98376] /usr/local/etc/rc.newwanip: On (IP address: XX.xx.xx.XX) (interface: WAN[wan]) (real interface: em0).
2020-10-20T08:12:03 opnsense[98376] /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'em0'
2020-10-20T08:12:03 dhclient[28531] Creating resolv.conf
2020-10-20T08:12:03 dhclient[58696] route add default XX.xx.xx.XX
2020-10-20T08:12:03 dhclient[49439] New Routers (em0): XX.xx.xx.XX
2020-10-20T08:12:03 dhclient[69234] New Broadcast Address (em0): 255.255.255.255
2020-10-20T08:12:03 dhclient[91537] New Subnet Mask (em0): 255.255.255.0
2020-10-20T08:12:03 dhclient[15365] New IP Address (em0): XX.xx.xx.XX
2020-10-20T08:12:03 dhclient[80651] Comparing IPs: Old: "same as new" New: XX.xx.xx.XX
2020-10-20T08:12:03 dhclient[44443] Starting delete_old_states()
2020-10-20T08:12:03 dhclient[92185] Removing states from old IP 'XX.xx.xx.XX' (new IP '')
2020-10-20T08:12:03 dhclient[54838] Comparing IPs: Old: XX.xx.xx.XX New:
2020-10-20T08:12:03 dhclient[44874] Starting delete_old_states()
2020-10-20T08:12:03 opnsense[28380] /usr/local/etc/rc.linkup: HOTPLUG: Configuring interface wan
2020-10-20T08:12:03 opnsense[28380] /usr/local/etc/rc.linkup: DEVD Ethernet attached event for wan
2020-10-20T08:12:02 opnsense[63404] /usr/local/etc/rc.linkup: Hotplug event detected for Pihole(opt5) but ignoring since interface is configured with static IP (10.0.37.1 ::)
2020-10-20T08:12:02 opnsense[45932] /usr/local/etc/rc.linkup: Hotplug event detected for GuestLan(opt3) but ignoring since interface is configured with static IP (10.0.13.1 ::)
2020-10-20T08:12:02 opnsense[70505] /usr/local/etc/rc.linkup: Hotplug event detected for IOTlan(opt2) but ignoring since interface is configured with static IP (10.0.12.1 ::)
2020-10-20T08:12:01 opnsense[58830] /usr/local/etc/rc.linkup: Hotplug event detected for NVRlan(opt1) but ignoring since interface is configured with static IP (10.0.11.1 ::)
2020-10-20T08:12:01 opnsense[28598] /usr/local/etc/rc.linkup: Hotplug event detected for LAN(lan) but ignoring since interface is configured with static IP (10.0.3.1 ::)
2020-10-20T08:12:00 opnsense[64257] /usr/local/etc/rc.dyndns: Curl error occurred: Resolving timed out after 15003 milliseconds
2020-10-20T08:12:00 opnsense[64257] /usr/local/etc/rc.dyndns: Dynamic DNS (myddomain.duckdns.org): Current Service: custom
2020-10-20T08:12:00 opnsense[64257] /usr/local/etc/rc.dyndns: Dynamic DNS (myddomain.duckdns.org): _checkStatus() starting.
2020-10-20T08:11:59 opnsense[40532] /usr/local/etc/rc.linkup: Clearing states for stale wan route on em0
2020-10-20T08:11:59 dhclient[66399] exiting.
2020-10-20T08:11:59 dhclient[66399] connection closed
2020-10-20T08:11:59 opnsense[40532] /usr/local/etc/rc.linkup: DEVD Ethernet detached event for wan
2020-10-20T08:11:52 webgui[11298] /index.php: Successful login for user 'root' from: 10.0.3.40
2020-10-20T08:11:02 opnsense[64257] /usr/local/etc/rc.dyndns: Dynamic DNS (myddomain.duckdns.org via Custom): _update() starting.
2020-10-20T08:11:02 opnsense[64257] /usr/local/etc/rc.dyndns: Dynamic DNS (myddomain.duckdns.org): running dyndns_failover_interface for wan. found em0
2020-10-20T08:11:02 opnsense[64257] /usr/local/etc/rc.dyndns: Dynamic DNS (myddomain.duckdns.org): XX.xx.xx.XX extracted
2020-10-20T08:11:02 opnsense[64257] /usr/local/etc/rc.dyndns: Dynamic DNS: updatedns() starting
2020-10-20T08:10:50 syslog-ng[29923] Destination timeout has elapsed, closing connection; fd='5'
2020-10-20T08:09:50 syslog-ng[29923] Destination timeout has elapsed, closing connection; fd='26'
2020-10-20T08:09:46 syslog-ng[29923] Destination timeout has elapsed, closing connection; fd='5'
2020-10-20T08:08:46 syslog-ng[29923] Destination timeout has elapsed, closing connection; fd='5'
2020-10-20T08:07:46 syslog-ng[29923] Destination timeout has elapsed, closing connection; fd='5'
2020-10-20T08:06:46 syslog-ng[29923] Destination timeout has elapsed, closing connection; fd='5'
at the moment WAN drops offline, DynDNS is started but will fail...

dpinger:

Code Select Expand

2020-10-20T08:13:39 dpinger[47604] send_interval 1000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr xx.xx.xx.xx bind_addr xx.xx.xx.xx identifier "WAN "
2020-10-20T08:13:39 dpinger[36894] WAN xx.xx.xx.xx: sendto error: 65
2020-10-20T08:13:38 dpinger[36894] WAN xx.xx.xx.xx: sendto error: 65
2020-10-20T08:13:37 dpinger[36894] WAN xx.xx.xx.xx: sendto error: 65
2020-10-20T08:13:36 dpinger[36894] WAN xx.xx.xx.xx: sendto error: 65
2020-10-20T08:13:35 dpinger[36894] WAN xx.xx.xx.xx: sendto error: 65
2020-10-20T08:13:34 dpinger[36894] WAN xx.xx.xx.xx: sendto error: 65
2020-10-20T08:13:33 dpinger[36894] WAN xx.xx.xx.xx: sendto error: 65
2020-10-20T08:13:32 dpinger[36894] WAN xx.xx.xx.xx: sendto error: 65
2020-10-20T08:13:31 dpinger[36894] WAN xx.xx.xx.xx: sendto error: 65
2020-10-20T08:13:30 dpinger[36894] WAN xx.xx.xx.xx: sendto error: 50
2020-10-20T08:12:03 dpinger[36894] send_interval 1000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr xx.xx.xx.xx bind_addr xx.xx.xx.xx identifier "WAN "
2020-10-20T08:12:02 dpinger[43866] WAN xx.xx.xx.xx: sendto error: 65
2020-10-20T08:12:01 dpinger[43866] WAN xx.xx.xx.xx: sendto error: 65
2020-10-20T08:12:00 dpinger[43866] WAN xx.xx.xx.xx: sendto error: 65
2020-10-20T08:11:59 dpinger[43866] WAN xx.xx.xx.xx: sendto error: 50
2020-10-20T08:11:01 dpinger[4152] GATEWAY ALARM: WAN (Addr: xx.xx.xx.xx Alarm: 1 RTT: 12038ms RTTd: 8903ms Loss: 22%)
2020-10-20T08:11:01 dpinger[43866] WAN xx.xx.xx.xx: Alarm latency 12038us stddev 8903us loss 22%
2020-10-17T22:43:10 dpinger[43290] GATEWAY ALARM: WAN (Addr: xx.xx.xx.xx Alarm: 0 RTT: 9130ms RTTd: 1408ms Loss: 0%)

i struggle for months to get this issue tackled, but i cant found it.
biggest issue: NO LOGBOOKS! cant found anything in it what resulting in a dropping WAN, no idea why.
but the solution that is always working is restart suricata, restarting the wan interface only doesn't always work.
it doens't come back online by itself, i have to manually restart suricata.
i did try a full reset and put back the config, reinstalling the system... no luck...

its only doing this while watching Youtube video's, hammering the network with steam/sabnzdb/Plex/ or other things, doesn't tricker the drop out.
i tried a couple of days with suricata disabled, problem is still here...

is there a way to get uplift the logbook registration? its also spamming it with the logbook with syslog-ng messages:

Code Select Expand

syslog-ng[9873] Destination timeout has elapsed, closing connection; fd='5'
it bloody annoying!!!

Can i edit the config.xml and remove some of it without making it corrupt? like the IDS rule, i want the reset suricata...

i have this exact same problem, started since 20.7. every minute a logbook registration from syslog-ng:

Code Select Expand

2020-09-16T20:06:08 syslog-ng[41410] Destination timeout has elapsed, closing connection; fd='5'
2020-09-16T20:05:08 syslog-ng[41410] Destination timeout has elapsed, closing connection; fd='27'
2020-09-16T20:04:49 syslog-ng[41410] Destination timeout has elapsed, closing connection; fd='26'
2020-09-16T20:04:01 syslog-ng[41410] Destination timeout has elapsed, closing connection; fd='25'
2020-09-16T20:03:10 syslog-ng[41410] Destination timeout has elapsed, closing connection; fd='26'
2020-09-16T20:02:35 syslog-ng[41410] Destination timeout has elapsed, closing connection; fd='5'
2020-09-16T20:01:35 syslog-ng[41410] Destination timeout has elapsed, closing connection; fd='5'
2020-09-16T20:00:35 syslog-ng[41410] Destination timeout has elapsed, closing connection; fd='5'
2020-09-16T19:59:35 syslog-ng[41410] Destination timeout has elapsed, closing connection; fd='5'
2020-09-16T19:58:35 syslog-ng[41410] Destination timeout has elapsed, closing connection; fd='26'
2020-09-16T19:58:01 syslog-ng[41410] Destination timeout has elapsed, closing connection; fd='23'
2020-09-16T19:57:34 syslog-ng[41410] Destination timeout has elapsed, closing connection; fd='27'
2020-09-16T19:56:41 syslog-ng[41410] Destination timeout has elapsed, closing connection; fd='23'

it's pretty annoying while trouble shouting problem with rondom WAN drop offline isseu...

Thank you! Do you have destination/invert checked for the port 53 redirect NAT rules? You are explicitly blocking all traffic from the pihole to the LANs with no exceptions, so I assume that the NAT destination/invert rule takes care of that and that it is is higher up in the order that the rules get processed. Otherwise, I can't see how the clients get their DNS responses back.
Did I get that right?

if you checked the attachment you did see that sources/invert and destination/invert are checked ;)
Not only pi-hole vlan can't talk to other interfaces, they also can't talk to pi-hole (expect "lan" this use the default "allow to all" rule)
if i disable this rule, no one can access Pihole dns (expect "lan")

see under this message a attachment of the firewall rules from "Guest Lan":

i did the same as mg82, created a vlan pure for pi-hole.
those vlans cant talk to each other, only the redirect allow dns traffic to pihole: (see attachment)
then a create under NAT: port forward, for every interface that need to use pi-hole a rule (see attachment)
The rule it self is elso in the attachment.
i didn't touch anything in NAT and DHCP, those are Default.
Pi-hole itself use unbound.

i was thinking of some performance tuning, did disabled:
- Hardware CRC
- Hardware TCO
- Hardware LRO
- VLAN Hardware Filtering
changed the Pattern matcher to 'hyperscan'
enabled  IPS mode and Promiscuous mode.
i didn't change anything else.

iperf3:

Code Select Expand

iperf3 -c 10.0.3.31 -u -t 60 -i 10 -b 1000M
Connecting to host 10.0.3.31, port 5201
[  5] local 10.0.3.1 port 44924 connected to 10.0.3.31 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-10.00  sec  1.16 GBytes  1000 Mbits/sec  856118 
[  5]  10.00-20.00  sec  1.16 GBytes  1.00 Gbits/sec  856870 
[  5]  20.00-30.00  sec  1.16 GBytes  1000 Mbits/sec  857061 
[  5]  30.00-40.00  sec  1.16 GBytes  1.00 Gbits/sec  856166 
[  5]  40.00-50.00  sec  1.16 GBytes  1000 Mbits/sec  857113 
[  5]  50.00-60.00  sec  1.16 GBytes  1.00 Gbits/sec  857192 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-60.00  sec  6.98 GBytes  1000 Mbits/sec  0.000 ms  0/5140520 (0%)  sender
[  5]   0.00-60.00  sec  3.34 GBytes   479 Mbits/sec  0.046 ms  2680818/5140353 (52%)  receiver

iperf Done.

server statics say: 962Mbit/sec.

well.... i don't need any tuning?  ::)

Suricata is active on WAN and LAN, tested iperf on Lan.
if i change the pattern match to aho-corasick its around the 450Mbit.

rules: 56019
is this command the right one?:

Code Select Expand

root@OPNsense:/usr/local/etc/suricata/rules # cat *.rules | sed 's/^ *#.*//' | sed '/^ *$/d' | wc -l
Hardware:
AMD Ryzen 3 2200G with Radeon Vega Graphics (4 cores)
8GB RAM
Intel PRO/1000 PT Dual Port Server Adapter (PCI-e 4x) (driver: EM)
OPNsense 20.1.8_1