Messages

(i do not see this behave, when doing a ssh connetion from another server to server1)

(the huge wait appears always after the message "debug3: verify_host_key_dns")

Versions    OPNsense 22.1.1_3-amd64
FreeBSD 13.0-STABLE
OpenSSL 1.1.1m 14 Dec 2021

Hi all

i've got a strange behave on Opnsense  - when ssh to another server (which is configured in UNBOUND DNS),
ssh connection takes sometimes up to 30 seconds - but not always.

Most of the time the ssh connects very quickly, but sometimes there is a huge wait.

I.e. opnsense# ssh server1

Did run the command (ssh server1) with debug option, i see these messages:

debug3: verify_host_key_dns
DNS lookup error: general failure

The "DNS lookup error: general failure" appears also, when the ssh connection is quick, so i think this not the culprit here.

nslookup of server1 is always very fast  (means, the server can be resolved by DNS).

cannot find any relevant information into the logs ..

(reboot did not solve the issue, that's why the only option available was the full recovery)

[22.1.1_3]

7.3.2022: Did an upgrade from:OPNsense 22.1.1_3 TO:OPNsense 22.1.2.

After that, UNBOUND got permanent CPU of 100% & was most of time unresponsive.

Had to do a FULL ZFS-Recovery to the previous version (22.1.1_3).
Now system is running stable again.

ZFS ist das weltweit beste FileSystem auf der Welt.
Es gibt bislang kein einziges FileSystem, das VolumeManagement, FileSystem, (Software)Raid-Controller und Snaphots so homogen in Perfektion vereint wie ZFS. Ich betreibe viele ZFS-Systeme und keines bietet Datenschutzt auf Enterprise Niveau wie ZFS. Keines!. Ich betreibe auch Systeme mit btrfs, das ZFS nahe kommt und auch sehr gut/ähnlich ist. Es ist natürlich empfehlenswert ECC für ZFS zu verwenden (gerade weil ZFS professionell und Enterprise ist, schlägt es das vor), aber man kann auch ZFS ohne ECC verwenden.

Wenn man dies tut, dann sollte man die Option verwenden:
zfs_flags is set to 0x10 (important for NON-ECC Memory - die einen annährenden Schutz bietet wie ECC-Memory, weil diese Optine zusätzliche Checks ausführt, die mit ECC nicht nötig wären.

Wie schon erwähnt wurde - kein FileSystem auf der Welt verträgt Cold-Power Offs so gut wie ZFS.
Ist Deduplikation nicht eingeschaltet, geht auch ZFS mit dem Memory sparsam um. ... Ich könnte Euch noch viel, viel mehr über ZFS erzählen. Ich war lange Zeit Storage Engineer auf Netapp Systemen, 3PAR, XP, EMC ..etc. Es gibt nur ein Storage-Filesystem, das ZFS nahe kommt und das ist Netapp.

just did the change from a APU2 Board to new TopTon Hardware today.

* Installed new OPNsense 21.7.3_3-amd64 on ZFS.
* backup config from APU2 Board
* change this backup xml config from
    <primaryconsole>serial</primaryconsole>
                               to
    <primaryconsole>video</primaryconsole>

    (because TopTon HW is has got a HD-/Video Port)
* Interface-names are the same.

So finally .. just power off the APU2, re-cabling, restore xml-backup-config from old HW to the new hardware,
reboot & and it works like a charm

Thx a lot oneplane for your support.

I think, too - make not really sense, what TP suggested ..I tried to draw something simple
(i hope it's not too simple :-) )

Code Select Expand

         ┌───────────────────────────────────────┐
         │                                       │    ┌──────────┐
         │    TP-Link Switch  TL-SG1024DE  Port12│────│ Windows │
         │                                       │    └──────────┘
         │                                 Port 5│──── ┌────────────────┐
         │                                       │     │         │   ┌───────┐
         │  Port 6                Port 9   Port 8│     │VoIP/Wifi Router│───│  WiFi │
         └───┬──────────────────────┬─────────┬──┘     │         │   │       │
             │                      │       │        └────────────────┘   └───────┘
             │     │ ┌─────────┐
             │ Ethernet cable       │   │WiFi Ext.│
             │     │   └─────────┘
             │     │
┌───────────────────────────┐ ┌────────────────────────┐
│   igb0 LAN (static)       │ │ igb0 LAN (static)      │ 
│   igb1 WAN          ​      │ │ igb1 WAN (dhcp)        │
│ (igb1:not configured yet) ​│ │        │
│             │ │ OpnSense PROD(running) │
│ OpnSense NEW              │ └────────────────────────┘
│                           │
└───────────────────────────┘

Port 10,11,13,14,15,19,21-24 ==> Linux based Machines
Port 2,4,7,16-18,20 (disabled)

All Ports: Speed/Duplex:auto, Flowcontrol:off

- What hardware are you using (type/part/brand):

Port 9: apu Board,Model number GX-412TC (PROD running)

Network: igb0,igb1 - auto 1Gbit NIC
configured NICS: igb0,ibg1

Port 6: Intel Core i5 8250U (new PROD - not running yet)
2400Mhz DDR4 RAM Slot Max 32GB Firewall Router
Processor 6 Lans AES-NI VPN Pc

Network: igb0-igb5 - auto 1Gbit NIC
configured NICS: igb0

i did setup a new Device with "OPNsense 21.7.3_3".

After completing the setup i did connect the device to another port on my TP-LINK Switch.
After doing this, the device is not reachable anymore from network.
(i used the same ethernet cable).

After contacting the TP-Link support, they told me, the OpnSense has enabled "802.1X-Authentication" per default - and the opnsense will allow connections to this specific switch port (port:6) only - now and in the future.

The TP-Link support told me - when i would change this behave, then i need to buy a Switch, which will have the following characteristics:

* Level 2+ or Level 3 Switch, which supports 802.1x
* Switch Dynamic VLAN Assignment

Does anyone have got some experience, advices on that ?

(may be there is configuration i missed to setup and i don't need to buy another switch)

Thanks all for any answer. I do appreciate.
Mike

did a reboot this morning and now memory-/swap/-cpu consumption is normal.

So - reboot fixes this issue definitively

yes, thanks .. i already mentioned that in the other board  :D

My system is near to crash - did the upgrade this morning to 21.7.3_1

CPU most of the time 100%, Memory increasing and increasing ..
Think it's a phython 3.8 issue together with syslog-ng ?!

last pid: 13263;  load averages:  3.97,  3.91,  3.46                                                                                                                 up 9+13:55:29  18:08:53
53 processes:  3 running, 48 sleeping, 2 zombie
CPU: 54.5% user,  0.0% nice, 40.1% system,  1.1% interrupt,  4.3% idle
Mem: 851M Active, 503M Inact, 1384M Laundry, 1047M Wired, 392M Buf, 123M Free
Swap: 5120M Total, 3212M Used, 1908M Free, 62% Inuse, 128K In, 6016K Out

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
74757 root          6  28    0    33M  4388K kqread   0  91:32  93.87% syslog-ng
90861 root          1 101    0    11M  1564K RUN      2  87:22  87.99% syslogd
15412 root          1 101    0  4261M  2115M CPU3     3 842:39  85.44% python3.8


Name    opnadm9.opn9.9opn
Versions    OPNsense 21.7.3_1-amd64
FreeBSD 12.1-RELEASE-p20-HBSD
OpenSSL 1.1.1l 24 Aug 2021
Updates    Click to check for updates.
CPU type    AMD GX-412TC SOC (4 cores)
CPU usage    
Load average    3.69, 3.72, 3.33
Uptime    9 days 13:53:29
Current date/time    Tue Sep 28 18:06:53 CEST 2021
Last config change    Tue Sep 28 11:16:26 CEST 2021
CPU usage    
100 %
State table size    
0 % ( 890/403000 )
MBUF usage    
0 % ( 1806/250690 )
Memory usage    
88 % ( 3563/4035 MB )
SWAP usage    
59 % ( 3041/5120 MB )
Disk usage    
75% / [ufs] (9.3G/13G)

same to me did the upgrade yesterday to  OPNsense 21.7.3_1 - CPU and Memory are close to full (100%).

Syslog-ng Daemon agent was not started.

Check top .. saying .. that sylog-ng and python 3.8 eats all the ressources on the system.

PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
----------------------------------------------------------------------------------------------
74757 root          6  29    0    33M    10M kqread   0   9:57 109.57% syslog-ng
15412 root          1 101    0  1435M  1221M CPU0     0 429:56  97.99% python3.8
90861 root          1 101    0    11M  1916K CPU1     1  15:39  97.89% syslogd

Did not do a reboot till now ..

Seems this firmware release was not properly tested before populate it.

Dear all

i can confirm - there is a big memory issue with Maltrail ..

Just one week ago - i did increase the Swap Swap space of my opnsense system to almost 6GB and my memor is 4GB .. that means .. the system has got about 10GB of memory space for use.

But maltrail eats this all - every day - more than 1GB of additional memory is needed - at the end i had to reboot my machine two time a week !

Now - after disabling maltrail - the swap usage is 0 (ZERO !!!) for days .. and the consumption is only 24%.

Fact is .. there must be a big memory bug into this software.