Messages

This problem isn't isolated to just the switch or cable. Anything I plug into this interface card has this issue. My 2 onboard interface ports are the only ones that are working correctly.

Can you change the interface card too?
Just to be sure  :)

Hi Oxy,

ich kenne mich nicht so gut mit TP aus aber was verstehst Du genau unter einem Trunk?
Es gibt je nach Hersteller unterschiedliche Bezeichnungen für so etwas:
Cisco: Ether Channel
Avaya: Trunk
Linux: Bond
BSD: LAG

Hast Du irgendeine Link Aggregation eingerichtet?

Kann denn der Switch die 192.168.1.7 auch nicht erreichen?
Wie sieht es mit den Routen auf den Clients aus?


Viele Grüße
Frank

Hi Dario,

you could try the os-monit plugin. It monitors your system and sends messages on various events.

Unfortunately due to a bug in monit 5.22 it cannot check filesystems.
But you can revert to monit version 5.20 with

Code Select Expand

opnsense-revert -r 17.1.5 monit


Cheers,
Frank

@abel408
You have more Ierrs than Ipkts!
This could be a hardware problem. Check the cables.
I had recently a series of bad cables that causes this number of errors.
Or maybe the switch has a problem. Any errors on the switch?

@brononius

Code Select Expand

Device: /dev/bus/4 [megaraid_disk_22] [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 78 to 79
Device: /dev/bus/4 [megaraid_disk_22] [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 72 to 73
Quite hot. BBQ?
You should check your air con. And the disks. And the switch. Any errors/discards, temperature?
And can you check the errors from a ifconfig on the Proxmox host?

To understand you right. You have one physical interface on the Proxmox blade and put 4 VLANs (btw: What is your understanding of a trunk?) on it and press a backup through this single NIC in and out?
I'd try the virtio nic. 6GByte RAM are good for some performance tweaks.  ;)
FreeBSD uses resources very sparingly with the default settings.

One important thing:
When the problem occurs again, before you reboot the OPNsense VM please do a
netstat -m
netstat -s
sysctl dev.em
etc.
and post it here.

Is the backup traffic going through the DMZ interface?

@abel408
Please post the output of
sysctl dev.igb.0
sysctl hw.igb
netstat -idb -I igb0

You have many Ierrs on em1 (vmbr777). Is this the DMZ interface? Can't see a public IP address.
2 discards because of memory problems and 5% of the packets are out of order.
Any errors from dmesg?

Did you check the Proxmox logs?
Do you have bonding on the Proxmox host configured?

Hi,

as soon as I put a little bit more load on my firewall cluster it looses packets and the TCP connections get closed.
The nodes are ProLiant DL380 G7 with 32GB RAM, two Quad-Core Xeons X5660 and three Quad-Port Intel 82580 NICs. So I assume the hardware is not the problem. It has link aggregation with loadbalance mode on all interfaces.
The system is not under stress. It has approx. 10k sessions. 1% CPU load. Lots of mbufs, no errors, no drops neither on the NICs nor on the switch ports.

At some indefinite point the firewall looses packets.
The trouble starts after acknowledging number 291137. The database server sends packages until the TCP window gets full. But these packages didn't reach the other site as well as the ACK's from the webserver didn't reach the database. And after retransmission timed out the connection is reset from the database server.

The traces were made on the firewall. I've made them on the physical and the lagg interfaces with no difference.

Any ideas where to look further?
And why do I see ICMP packages from the firewall on this TCP connection?


Many thanks
Frank

lagg0 - 192.168.19.0/24

Code Select Expand

330 299.939233  172.16.6.69 -> 192.168.19.4   TCP 54 55353 > ms-sql-s [ACK] Seq=12642 Ack=283137 Win=45312 Len=0
331 299.939238   192.168.19.4 -> 172.16.6.69  TDS 1514 Unknown Packet Type: 0
332 299.939252  172.16.6.69 -> 192.168.19.4   TCP 54 55353 > ms-sql-s [ACK] Seq=12642 Ack=291137 Win=37376 Len=0
333 299.939397   192.168.19.4 -> 172.16.6.69  TDS 1514 Unknown Packet Type: 0
334 299.939572   192.168.19.4 -> 172.16.6.69  TDS 1514 Unknown Packet Type: 0 (Not last buffer)
335 299.939576   192.168.19.4 -> 172.16.6.69  TDS 1514 Unknown Packet Type: 0
336 299.939579   192.168.19.4 -> 172.16.6.69  TDS 1514 Unknown Packet Type: 0 (Not last buffer)
337 299.939582   192.168.19.4 -> 172.16.6.69  TDS 1514 Unknown Packet Type: 0 (Not last buffer)
338 299.939585   192.168.19.4 -> 172.16.6.69  TDS 1514 Unknown Packet Type: 0 (Not last buffer)
339 299.939588   192.168.19.4 -> 172.16.6.69  TDS 1514 Unknown Packet Type: 0 (Not last buffer)
340 299.939591   192.168.19.4 -> 172.16.6.69  TDS 1514 Unknown Packet Type: 0
341 299.939595   192.168.19.4 -> 172.16.6.69  TDS 1514 Unknown Packet Type: 0
342 299.939599   192.168.19.4 -> 172.16.6.69  TDS 1514 Unknown Packet Type: 0 (Not last buffer)
343 299.939602   192.168.19.4 -> 172.16.6.69  TDS 1514 Unknown Packet Type: 0
344 299.939605   192.168.19.4 -> 172.16.6.69  TDS 1514 Unknown Packet Type: 0
345 299.939608   192.168.19.4 -> 172.16.6.69  TCP 1514 ms-sql-s > 55353 [PSH, ACK] Seq=324657 Ack=12642 Win=65536 Len=1460
346 299.939610   192.168.19.4 -> 172.16.6.69  TDS 1514 Unknown Packet Type: 0
347 300.239719   192.168.19.4 -> 172.16.6.69  TCP 1514 [TCP Retransmission] ms-sql-s > 55353 [ACK] Seq=291137 Ack=12642 Win=65536 Len=1460
348 300.239743    192.168.19.31 -> 192.168.19.4   ICMP 82 Destination unreachable (Host unreachable)
349 300.838833   192.168.19.4 -> 172.16.6.69  TCP 1514 [TCP Retransmission] ms-sql-s > 55353 [ACK] Seq=291137 Ack=12642 Win=65536 Len=1460
350 300.838859    192.168.19.31 -> 192.168.19.4   ICMP 82 Destination unreachable (Host unreachable)
351 302.041479   192.168.19.4 -> 172.16.6.69  TCP 1514 [TCP Retransmission] ms-sql-s > 55353 [ACK] Seq=291137 Ack=12642 Win=65536 Len=1460
352 302.041502    192.168.19.31 -> 192.168.19.4   ICMP 82 Destination unreachable (Host unreachable)
353 304.438934   192.168.19.4 -> 172.16.6.69  TCP 1514 [TCP Retransmission] ms-sql-s > 55353 [ACK] Seq=291137 Ack=12642 Win=65536 Len=1460
354 304.438957    192.168.19.31 -> 192.168.19.4   ICMP 82 Destination unreachable (Host unreachable)
355 309.239126   192.168.19.4 -> 172.16.6.69  TCP 1514 [TCP Retransmission] ms-sql-s > 55353 [ACK] Seq=291137 Ack=12642 Win=65536 Len=1460
356 309.239148    192.168.19.31 -> 192.168.19.4   ICMP 82 Destination unreachable (Host unreachable)
357 318.839481   192.168.19.4 -> 172.16.6.69  TCP 60 ms-sql-s > 55353 [RST, ACK] Seq=292597 Ack=12642 Win=0 Len=0
358 329.939143  172.16.6.69 -> 192.168.19.4   TCP 55 [TCP Keep-Alive] [TCP Window Full] 55353 > ms-sql-s [ACK] Seq=12641 Ack=307137 Win=131328 Len=1
359 329.939261   192.168.19.4 -> 172.16.6.69  TCP 60 ms-sql-s > 55353 [RST] Seq=307137 Win=0 Len=0

lagg1 - 172.16.6.0/24

Code Select Expand

329 299.939251  172.16.6.69 -> 192.168.19.4   TCP 60 55353 > ms-sql-s [ACK] Seq=12642 Ack=283137 Win=45312 Len=0
330 299.939261   192.168.19.4 -> 172.16.6.69  TDS 1514 Unknown Packet Type: 0
331 299.939271  172.16.6.69 -> 192.168.19.4   TCP 60 55353 > ms-sql-s [ACK] Seq=12642 Ack=291137 Win=37376 Len=0
332 299.939273   192.168.19.4 -> 172.16.6.69  TDS 1514 Unknown Packet Type: 0
333 299.939321  172.16.6.69 -> 192.168.19.4   TCP 60 55353 > ms-sql-s [ACK] Seq=12642 Ack=299137 Win=29440 Len=0
334 299.939492  172.16.6.69 -> 192.168.19.4   TCP 60 55353 > ms-sql-s [ACK] Seq=12642 Ack=307137 Win=21504 Len=0
335 299.939636  172.16.6.69 -> 192.168.19.4   TCP 60 [TCP Window Update] 55353 > ms-sql-s [ACK] Seq=12642 Ack=307137 Win=69376 Len=0
336 299.940190  172.16.6.69 -> 192.168.19.4   TCP 60 [TCP Window Update] 55353 > ms-sql-s [ACK] Seq=12642 Ack=307137 Win=131328 Len=0
337 318.839520   192.168.19.4 -> 172.16.6.69  TCP 54 ms-sql-s > 55353 [RST, ACK] Seq=292597 Ack=12642 Win=0 Len=0
338 329.939156  172.16.6.69 -> 192.168.19.4   TCP 60 [TCP Keep-Alive] [TCP Window Full] 55353 > ms-sql-s [ACK] Seq=12641 Ack=307137 Win=131328 Len=1
339 329.939300   192.168.19.4 -> 172.16.6.69  TCP 54 ms-sql-s > 55353 [RST] Seq=307137 Win=0 Len=0

Hi Werner,

you can do that with the new os-monit plugin and a little script.
The plugin is available since 17.1.5. It's a monitoring software that can execute scripts and generate alarms according to the return value.

Create a script e.g. /usr/local/bin/check_gmirror.sh and make it executable.

Code Select Expand

#!/bin/sh
ReturnString=$(/sbin/gmirror status)
ReturnStatus=$?
echo $ReturnString
if [ $ReturnStatus -ne 0 ]; then
   exit 1
fi
exit 0

Then create a Monit Service Test (Services -> Monit -> Settings -> Service Tests Settings)

Code Select Expand

Name: ExitStatus
Condition: status notequal 0
Action: Alert

and a Service Setting

Code Select Expand

Name: GMirrorTest
Type: Custom
Path: /usr/local/bin/check_gmirror.sh
Tests: ExecStatus

You can see the output of the script at the Monit status page.


Regards
Frank

[Port Forward]

Hi,

it is possible since OPNsense Version 17.1.3. Many thanks franco!

o firewall: port forwarding enhancements for tag, pool options and target subnet

To create a simple port rewriting add a Port Forward from network A to network B Destination port range: 1 with  Redirect Target IP: 0.0.0.0/0, Redirect Target Port: 2 and Pool Options: Bitmask.
To restrict it tag it with Set local tag and use this value in a rule as Match local tag.

... or you can use SELinux or whatever.

But then I wonder for what reason the firewall was installed?
If you do all these security measures why not simply put the ftp server to a DMZ?

Yes, this would work but it's unsafe.
You need to open a port range to your ftp server without a relation to an existing ftp command connection.
These ports are always open regardless if a ftp connection exists or not or what services listen on these ports on the ftp server.

Unfortunately TLS won't work because the encryption happens between the client and the server.
But the ftp proxy needs to know the port commands to create the appropriate firewall rules.

Consider using SFTP instead.


Regards,
Frank

[reverse]

Ok, the ftp server cannot create an active data connection.
Possible reasons:
- a firewall or selinux prevent the vsftpd from making the data connection.
- an explicit rule on the OPNsense blocks traffic from LAN to WAN
- the client on WAN site has a firewall blocking the data connection

Did you made your tests always with active ftp?

The clients tested are Debian's default FTP and MS Windows, both can connect to ftp.debian.org for example eliminating the local firewall.

This has nothing to do with a reverse proxy. Connecting from LAN to WAN requires a forward ftp proxy.

You cannot use the same proxy as forward (LAN -> WAN) and reverse (WAN -> LAN) proxy.

[System->Firmware->Plugins]

OK, maybe the documentation doesn't explain it clear enough.

• Make sure you have installed the os-ftp-proxy plugin. See System->Firmware->Plugins.
  Reload the page if the FTP Proxy item doesn't appear in the Services Menu after install.
• In Services->FTP Proxy click the + button at bottom right, underneath the table to add a new ftp-proxy.
  
• Now the Edit Proxy dialog should open and you can create the proxy as described in the documentation section Reverse FTP Proxy.

Does this procedure work?
If not which step fails?


Frank