OPNsense Forum
Archive => 17.1 Legacy Series => Topic started by: abel408 on March 21, 2017, 07:54:50 pm
-
I have an Intel 82580 Network controller that I've been having issues with.
I just have a very simple network on it.
OPNSense via igb0 82580 port (10.134.0.1) ---> TP-Link Switch (10.134.0.2)
When the OPNSense box starts up, everything works fine. I can connect to the switch and all devices attached to the switch. Then, for no apparent reason, the connection between the switch and OPNSense box breaks. Both devices still see an active connection, but the OPNSense box loses the ARP entries it had for the switch and devices on the switch. They only way to get the connection back is to reboot the OPNSense box. Sometimes it will go hours without losing the ARP entries, sometimes only minutes.
I had a VyOS router in place of the OPNSense box before and never had these issues so I know there is nothing wrong with the switch.
-
I just upgraded to the latest version and I'm still seeing this issue.
I have attached other devices to the Intel 82580 network controller and they also lose their ARP entry. The onboard network controller works fine.
-
I'm having the same issue, with a slighty different setup.
I'm using an old cisco blade server with the latest proxmox 5.0.9. This server is connected with a trunk to a cisco switch in 1GB.
And everything is working fine, and suddently, after approx 24 à 48 hours, I'm losing a virtual interface.
LAN-WAN-GUESTS are still working, DMZ not.
Once I restart the virtual interface (disable - enable), the connection is back.
I've now fix all network card to 1000baseT. Fingers crossed that it stays stable now...
My opnsense version is: OPNsense 17.1.6-amd64.
My Promox config, with network card Intel E1000
(https://forum.opnsense.org/index.php?action=dlattach;topic=4850.0;attach=2079)
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=2088<VLAN_MTU,VLAN_HWCSUM,WOL_MAGIC>
ether 96:7e:28:e9:06:1c
inet6 fe80::947e:28ff:fee9:61c%em0 prefixlen 64 scopeid 0x1
inet 192.168.111.254 netmask 0xffffff00 broadcast 192.168.111.255
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet 1000baseT (1000baseT <full-duplex>)
status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=2088<VLAN_MTU,VLAN_HWCSUM,WOL_MAGIC>
ether 1e:2a:a7:75:12:28
inet6 fe80::1c2a:a7ff:fe75:1228%em1 prefixlen 64 scopeid 0x2
inet 192.168.222.254 netmask 0xffffff00 broadcast 192.168.222.255
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet 1000baseT (1000baseT <full-duplex>)
status: active
em2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=2088<VLAN_MTU,VLAN_HWCSUM,WOL_MAGIC>
ether fa:13:81:16:cb:08
inet6 fe80::f813:81ff:fe16:cb08%em2 prefixlen 64 scopeid 0x3
inet 192.168.1.254 netmask 0xffffff00 broadcast 192.168.1.255
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet 1000baseT (1000baseT <full-duplex>)
status: active
em3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=2088<VLAN_MTU,VLAN_HWCSUM,WOL_MAGIC>
ether 22:98:02:e4:30:b6
inet6 fe80::2098:2ff:fee4:30b6%em3 prefixlen 64 scopeid 0x4
inet 192.168.0.226 netmask 0xffffff00 broadcast 192.168.0.255
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet 1000base
netstat -i
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll
em0 1500 <Link#1> 96:7e:28:e9:06:1c 183889088 0 0 122338557 0 0
em0 - fe80::%em0/64 fe80::947e:28ff:f 1764 - - 30831 - -
em0 - 192.168.111.0 securitus 888 - - 1883 - -
em1 1500 <Link#2> 1e:2a:a7:75:12:28 138311520 6339 0 218558256 0 0
em1 - fe80::%em1/64 fe80::1c2a:a7ff:f 0 - - 0 - -
em1 - 192.168.222.0 securitus 2389 - - 2624 - -
em2 1500 <Link#3> fa:13:81:16:cb:08 0 0 0 13 0 0
em2 - fe80::%em2/64 fe80::f813:81ff:f 0 - - 1 - -
em2 - 192.168.1.0/2 192.168.1.254 51 - - 0 - -
em3 1500 <Link#4> 22:98:02:e4:30:b6 38157986 0 0 19099481 0 0
em3 - fe80::%em3/64 fe80::2098:2ff:fe 0 - - 1 - -
em3 - 192.168.0.0/2 192.168.0.226 563 - - 0 - -
enc0* 1536 <Link#5> enc0 0 0 0 0 0 0
lo0 16384 <Link#6> lo0 7341 0 0 7341 0 0
lo0 - localhost localhost 0 - - 0 - -
lo0 - fe80::%lo0/64 fe80::1%lo0 0 - - 0 - -
lo0 - your-net localhost 7341 - - 7341 - -
pflog 33160 <Link#7> pflog0 0 0 0 710086 0 0
pfsyn 1500 <Link#8> pfsync0 0 0 0 0 0 0
ovpns 1500 <Link#9> ovpns1 0 0 0 10 0 0
ovpns - fe80::%ovpns1 fe80::4c76:46ba:6 0 - - 1 - -
ovpns - 192.168.112.1 192.168.112.1 0 - - 0 - -
em0_v 1500 <Link#10> 96:7e:28:e9:06:1c 0 0 0 5 0 0
em0_v - fe80::%em0_vl fe80::947e:28ff:f 0 - - 1 - -
em1_v 1500 <Link#11> 1e:2a:a7:75:12:28 0 0 0 3 0 0
em1_v - fe80::%em1_vl fe80::1c2a:a7ff:f 0 - - 1 - -
em2_v 1500 <Link#12> fa:13:81:16:cb:08 0 0 0 5 0 0
em2_v - fe80::%em2_vl fe80::f813:81ff:f 0 - - 1 - -
em3_v 1500 <Link#13> 22:98:02:e4:30:b6 0 0 0 4 1 0
em3_v - fe80::%em3_vl fe80::2098:2ff:fe 0 - - 1 - -
tcp:
522658 packets sent
57058 data packets (80334229 bytes)
680 data packets (849291 bytes) retransmitted
5 data packets unnecessarily retransmitted
0 resends initiated by MTU discovery
463766 ack-only packets (0 delayed)
0 URG only packets
0 window probe packets
554 window update packets
600 control packets
498363 packets received
31457 acks (for 80336075 bytes)
1273 duplicate acks
0 acks for unsent data
439680 packets (634087515 bytes) received in-sequence
99 completely duplicate packets (85432 bytes)
12 old duplicate packets
0 packets with some dup. data (0 bytes duped)
23419 out-of-order packets (33905296 bytes)
0 packets (0 bytes) of data after window
0 window probes
7 window update packets
53 packets received after close
0 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
2 discarded due to memory problems
299 connection requests
35 connection accepts
0 bad connection attempts
0 listen queue overflows
0 ignored RSTs in the windows
334 connections established (including accepts)
6036 connections closed (including 0 drops)
145 connections updated cached RTT on close
145 connections updated cached RTT variance on close
106 connections updated cached ssthresh on close
0 embryonic connections dropped
31111 segments updated rtt (of 28921 attempts)
22 retransmit timeouts
0 connections dropped by rexmit timeout
0 persist timeouts
0 connections dropped by persist timeout
0 Connections (fin_wait_2) dropped because of timeout
0 keepalive timeouts
0 keepalive probes sent
0 connections dropped by keepalive
22336 correct ACK header predictions
439265 correct data packet header predictions
35 syncache entries added
0 retransmitted
0 dupsyn
0 dropped
35 completed
0 bucket overflow
0 cache overflow
0 reset
0 stale
0 aborted
0 badack
0 unreach
0 zone failures
35 cookies sent
0 cookies received
12 hostcache entries added
0 bucket overflow
94 SACK recovery episodes
568 segment rexmits in SACK recovery episodes
691091 byte rexmits in SACK recovery episodes
3406 SACK options (SACK blocks) received
23041 SACK options (SACK blocks) sent
0 SACK scoreboard overflow
0 packets with ECN CE bit set
0 packets with ECN ECT(0) bit set
0 packets with ECN ECT(1) bit set
0 successful ECN handshakes
0 times ECN reduced the congestion window
TCP connection count by state:
0 connections in CLOSED state
6 connections in LISTEN state
0 connections in SYN_SENT state
0 connections in SYN_RCVD state
3 connections in ESTABLISHED state
0 connections in CLOSE_WAIT state
0 connections in FIN_WAIT_1 state
0 connections in CLOSING state
0 connections in LAST_ACK state
0 connections in FIN_WAIT_2 state
0 connections in TIME_WAIT state
udp:
585116 datagrams received
0 with incomplete header
0 with bad data length field
0 with bad checksum
1892 with no checksum
34032 dropped due to no socket
146301 broadcast/multicast datagrams undelivered
0 dropped due to full socket buffers
0 not for hashed pcb
404783 delivered
439680 datagrams output
0 times multicast source filter matche
Let me know if I can help in any way to troubleshoot it a bit more...
-
No luck, the port just went down again. :-[
A reboot and eveyrthing is back online. Don't have the time now to go through the log files...
So fixing the speed didn't resolve the issue.
-
You have many Ierrs on em1 (vmbr777). Is this the DMZ interface? Can't see a public IP address.
2 discards because of memory problems and 5% of the packets are out of order.
Any errors from dmesg?
Did you check the Proxmox logs?
Do you have bonding on the Proxmox host configured?
-
@abel408
Please post the output of
sysctl dev.igb.0
sysctl hw.igb
netstat -idb -I igb0
-
You have many Ierrs on em1 (vmbr777). Is this the DMZ interface? Can't see a public IP address.
2 discards because of memory problems and 5% of the packets are out of order.
Any errors from dmesg?
Did you check the Proxmox logs?
Do you have bonding on the Proxmox host configured?
vmbr777 is my server DMZ. The public IP/side is on vmbr999, but it's hidden after a router.
No bonding. Just 1 interface in use, with 4 trunks (666 - 777 - 888 - 999). This is the part that bothers me, if it was an issue with the hardware (cisco blade), all vlan's should have issues, not only a virutal vlan, no? Or even on the proxmox side?
I've got the feeling that problem is linked with heavy data transfer on opnsense and my 'virtual drivers'.
Fe when my backupserver starts, a big download... But this is just a feeling.
In att some lines from my syslog-server...
-
@abel408
Please post the output of
sysctl dev.igb.0
sysctl hw.igb
netstat -idb -I igb0
I've switched ports to igb2
sysctl dev.igb.2
dev.igb.2.host.header_redir_missed: 0
dev.igb.2.host.serdes_violation_pkt: 0
dev.igb.2.host.length_errors: 0
dev.igb.2.host.tx_good_bytes: 2584813
dev.igb.2.host.rx_good_bytes: 9629714
dev.igb.2.host.breaker_tx_pkt_drop: 0
dev.igb.2.host.tx_good_pkt: 0
dev.igb.2.host.breaker_rx_pkt_drop: 0
dev.igb.2.host.breaker_rx_pkts: 0
dev.igb.2.host.rx_pkt: 2
dev.igb.2.host.host_tx_pkt_discard: 0
dev.igb.2.host.breaker_tx_pkt: 0
dev.igb.2.interrupts.rx_overrun: 0
dev.igb.2.interrupts.rx_desc_min_thresh: 0
dev.igb.2.interrupts.tx_queue_min_thresh: 1626039
dev.igb.2.interrupts.tx_queue_empty: 23780
dev.igb.2.interrupts.tx_abs_timer: 0
dev.igb.2.interrupts.tx_pkt_timer: 0
dev.igb.2.interrupts.rx_abs_timer: 0
dev.igb.2.interrupts.rx_pkt_timer: 115268
dev.igb.2.interrupts.asserts: 3561305
dev.igb.2.mac_stats.tso_ctx_fail: 0
dev.igb.2.mac_stats.tso_txd: 0
dev.igb.2.mac_stats.tx_frames_1024_1522: 278
dev.igb.2.mac_stats.tx_frames_512_1023: 588
dev.igb.2.mac_stats.tx_frames_256_511: 285
dev.igb.2.mac_stats.tx_frames_128_255: 144
dev.igb.2.mac_stats.tx_frames_65_127: 16075
dev.igb.2.mac_stats.tx_frames_64: 6410
dev.igb.2.mac_stats.mcast_pkts_txd: 5
dev.igb.2.mac_stats.bcast_pkts_txd: 26
dev.igb.2.mac_stats.good_pkts_txd: 23780
dev.igb.2.mac_stats.total_pkts_txd: 116515
dev.igb.2.mac_stats.total_octets_txd: 8519853
dev.igb.2.mac_stats.good_octets_txd: 2584813
dev.igb.2.mac_stats.total_octets_recvd: 11198418
dev.igb.2.mac_stats.good_octets_recvd: 9629714
dev.igb.2.mac_stats.rx_frames_1024_1522: 1114
dev.igb.2.mac_stats.rx_frames_512_1023: 803
dev.igb.2.mac_stats.rx_frames_256_511: 218
dev.igb.2.mac_stats.rx_frames_128_255: 193
dev.igb.2.mac_stats.rx_frames_65_127: 1006
dev.igb.2.mac_stats.rx_frames_64: 111936
dev.igb.2.mac_stats.mcast_pkts_recvd: 0
dev.igb.2.mac_stats.bcast_pkts_recvd: 111044
dev.igb.2.mac_stats.good_pkts_recvd: 115270
dev.igb.2.mac_stats.total_pkts_recvd: 139781
dev.igb.2.mac_stats.mgmt_pkts_txd: 0
dev.igb.2.mac_stats.mgmt_pkts_drop: 0
dev.igb.2.mac_stats.mgmt_pkts_recvd: 0
dev.igb.2.mac_stats.unsupported_fc_recvd: 0
dev.igb.2.mac_stats.xoff_txd: 58621
dev.igb.2.mac_stats.xoff_recvd: 0
dev.igb.2.mac_stats.xon_txd: 34114
dev.igb.2.mac_stats.xon_recvd: 0
dev.igb.2.mac_stats.coll_ext_errs: 0
dev.igb.2.mac_stats.tx_no_crs: 0
dev.igb.2.mac_stats.alignment_errs: 0
dev.igb.2.mac_stats.crc_errs: 0
dev.igb.2.mac_stats.recv_errs: 0
dev.igb.2.mac_stats.recv_jabber: 0
dev.igb.2.mac_stats.recv_oversize: 0
dev.igb.2.mac_stats.recv_fragmented: 0
dev.igb.2.mac_stats.recv_undersize: 0
dev.igb.2.mac_stats.recv_no_buff: 0
dev.igb.2.mac_stats.recv_length_errors: 0
dev.igb.2.mac_stats.missed_packets: 24511
dev.igb.2.mac_stats.defer_count: 0
dev.igb.2.mac_stats.sequence_errors: 0
dev.igb.2.mac_stats.symbol_errors: 0
dev.igb.2.mac_stats.collision_count: 0
dev.igb.2.mac_stats.late_coll: 0
dev.igb.2.mac_stats.multiple_coll: 0
dev.igb.2.mac_stats.single_coll: 0
dev.igb.2.mac_stats.excess_coll: 0
dev.igb.2.queue7.lro_flushed: 0
dev.igb.2.queue7.lro_queued: 0
dev.igb.2.queue7.rx_bytes: 0
dev.igb.2.queue7.rx_packets: 182577
dev.igb.2.queue7.rxd_tail: 1023
dev.igb.2.queue7.rxd_head: 0
dev.igb.2.queue7.tx_packets: 4035
dev.igb.2.queue7.no_desc_avail: 0
dev.igb.2.queue7.txd_tail: 93
dev.igb.2.queue7.txd_head: 0
dev.igb.2.queue7.interrupt_rate: 8000
dev.igb.2.queue6.lro_flushed: 0
dev.igb.2.queue6.lro_queued: 0
dev.igb.2.queue6.rx_bytes: 0
dev.igb.2.queue6.rx_packets: 114128
dev.igb.2.queue6.rxd_tail: 1023
dev.igb.2.queue6.rxd_head: 0
dev.igb.2.queue6.tx_packets: 1087
dev.igb.2.queue6.no_desc_avail: 0
dev.igb.2.queue6.txd_tail: 82
dev.igb.2.queue6.txd_head: 0
dev.igb.2.queue6.interrupt_rate: 8000
dev.igb.2.queue5.lro_flushed: 0
dev.igb.2.queue5.lro_queued: 0
dev.igb.2.queue5.rx_bytes: 0
dev.igb.2.queue5.rx_packets: 11567
dev.igb.2.queue5.rxd_tail: 1023
dev.igb.2.queue5.rxd_head: 0
dev.igb.2.queue5.tx_packets: 561
dev.igb.2.queue5.no_desc_avail: 0
dev.igb.2.queue5.txd_tail: 22
dev.igb.2.queue5.txd_head: 0
dev.igb.2.queue5.interrupt_rate: 8000
dev.igb.2.queue4.lro_flushed: 0
dev.igb.2.queue4.lro_queued: 0
dev.igb.2.queue4.rx_bytes: 0
dev.igb.2.queue4.rx_packets: 136909
dev.igb.2.queue4.rxd_tail: 1023
dev.igb.2.queue4.rxd_head: 0
dev.igb.2.queue4.tx_packets: 11072
dev.igb.2.queue4.no_desc_avail: 4095
dev.igb.2.queue4.txd_tail: 0
dev.igb.2.queue4.txd_head: 0
dev.igb.2.queue4.interrupt_rate: 8000
dev.igb.2.queue3.lro_flushed: 0
dev.igb.2.queue3.lro_queued: 0
dev.igb.2.queue3.rx_bytes: 0
dev.igb.2.queue3.rx_packets: 328
dev.igb.2.queue3.rxd_tail: 1023
dev.igb.2.queue3.rxd_head: 0
dev.igb.2.queue3.tx_packets: 5116
dev.igb.2.queue3.no_desc_avail: 5103
dev.igb.2.queue3.txd_tail: 1022
dev.igb.2.queue3.txd_head: 0
dev.igb.2.queue3.interrupt_rate: 8000
dev.igb.2.queue2.lro_flushed: 0
dev.igb.2.queue2.lro_queued: 0
dev.igb.2.queue2.rx_bytes: 0
dev.igb.2.queue2.rx_packets: 239406
dev.igb.2.queue2.rxd_tail: 1023
dev.igb.2.queue2.rxd_head: 0
dev.igb.2.queue2.tx_packets: 3577
dev.igb.2.queue2.no_desc_avail: 5095
dev.igb.2.queue2.txd_tail: 1022
dev.igb.2.queue2.txd_head: 0
dev.igb.2.queue2.interrupt_rate: 8000
dev.igb.2.queue1.lro_flushed: 0
dev.igb.2.queue1.lro_queued: 0
dev.igb.2.queue1.rx_bytes: 0
dev.igb.2.queue1.rx_packets: 342474
dev.igb.2.queue1.rxd_tail: 1023
dev.igb.2.queue1.rxd_head: 0
dev.igb.2.queue1.tx_packets: 2683
dev.igb.2.queue1.no_desc_avail: 5103
dev.igb.2.queue1.txd_tail: 1022
dev.igb.2.queue1.txd_head: 0
dev.igb.2.queue1.interrupt_rate: 8000
dev.igb.2.queue0.lro_flushed: 0
dev.igb.2.queue0.lro_queued: 0
dev.igb.2.queue0.rx_bytes: 0
dev.igb.2.queue0.rx_packets: 68432
dev.igb.2.queue0.rxd_tail: 1023
dev.igb.2.queue0.rxd_head: 0
dev.igb.2.queue0.tx_packets: 4274
dev.igb.2.queue0.no_desc_avail: 4095
dev.igb.2.queue0.txd_tail: 0
dev.igb.2.queue0.txd_head: 0
dev.igb.2.queue0.interrupt_rate: 8000
dev.igb.2.fc_low_water: 33152
dev.igb.2.fc_high_water: 33168
dev.igb.2.rx_buf_alloc: 0
dev.igb.2.tx_buf_alloc: 0
dev.igb.2.extended_int_mask: 2147483648
dev.igb.2.interrupt_mask: 4
dev.igb.2.rx_control: 67141634
dev.igb.2.device_control: 1490027073
dev.igb.2.watchdog_timeouts: 0
dev.igb.2.rx_overruns: 0
dev.igb.2.tx_dma_fail: 0
dev.igb.2.mbuf_defrag_fail: 0
dev.igb.2.link_irq: 2
dev.igb.2.dropped: 135555
dev.igb.2.tx_processing_limit: -1
dev.igb.2.rx_processing_limit: 100
dev.igb.2.fc: 3
dev.igb.2.enable_aim: 1
dev.igb.2.nvm: -1
dev.igb.2.%parent: pci1
dev.igb.2.%pnpinfo: vendor=0x8086 device=0x150e subvendor=0x8086 subdevice=0x12a1 class=0x020000
dev.igb.2.%location: slot=0 function=2 dbsf=pci0:1:0:2
dev.igb.2.%driver: igb
dev.igb.2.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k
sysctl hw.igb
hw.igb.tx_process_limit: -1
hw.igb.rx_process_limit: 100
hw.igb.num_queues: 0
hw.igb.header_split: 0
hw.igb.buf_ring_size: 4096
hw.igb.max_interrupt_rate: 8000
hw.igb.enable_msix: 1
hw.igb.enable_aim: 1
hw.igb.txd: 1024
hw.igb.rxd: 1024
netstat -idb -I igb2
Name Mtu Network Address Ipkts Ierrs Idrop Ibytes Opkts Oerrs Obytes Coll Drop
igb2 1500 <Link#3> 00:1b:21:a6:c5:06 115293 135555 24511 9631186 23780 0 2584813 0 109068
igb2 - fe80::%igb2/6 fe80::21b:21ff:fe 0 - - 0 1 - 96 - -
igb2 - x.x.x.x.0/2 x-x-x-x.tvc-i 17976 - - 1500557 0 - 0 - -
-
@brononius
Device: /dev/bus/4 [megaraid_disk_22] [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 78 to 79
Device: /dev/bus/4 [megaraid_disk_22] [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 72 to 73Quite hot. BBQ?
You should check your air con. And the disks. And the switch. Any errors/discards, temperature?
And can you check the errors from a ifconfig on the Proxmox host?
To understand you right. You have one physical interface on the Proxmox blade and put 4 VLANs (btw: What is your understanding of a trunk?) on it and press a backup through this single NIC in and out?
I'd try the virtio nic. 6GByte RAM are good for some performance tweaks. ;)
FreeBSD uses resources very sparingly with the default settings.
One important thing:
When the problem occurs again, before you reboot the OPNsense VM please do a
netstat -m
netstat -s
sysctl dev.em
etc.
and post it here.
Is the backup traffic going through the DMZ interface?
-
@abel408
You have more Ierrs than Ipkts!
This could be a hardware problem. Check the cables.
I had recently a series of bad cables that causes this number of errors.
Or maybe the switch has a problem. Any errors on the switch?
-
This problem isn't isolated to just the switch or cable. Anything I plug into this interface card has this issue. My 2 onboard interface ports are the only ones that are working correctly.
-
This problem isn't isolated to just the switch or cable. Anything I plug into this interface card has this issue. My 2 onboard interface ports are the only ones that are working correctly.
Can you change the interface card too?
Just to be sure :)
-
I was trying to avoid that, but it sounds like that's my only option at this point...
-
Yes, the high error rate in contrast to the low traffic let suggest it.
I don't think it's a driver problem.
I use the igb driver too.
Have an eye on hw.igb.buf_ring_size.
Cheers,
Frank
-
Quite hot. BBQ? You should check your air con. And the disks. And the switch. Any errors/discards, temperature?
No, just an old server running at my house, so not in a cold room or so...
To understand you right. You have one physical interface on the Proxmox blade and put 4 VLANs (btw: What is your understanding of a trunk?) on it and press a backup through this single NIC in and out?.
My understanding of a trunk: multiple VLANs through 1 cable/port...
fe the config on my switch:
interface GigabitEthernet0/27
description Proxmox
switchport access vlan 777
switchport trunk encapsulation dot1q
switchport mode trunk
switchport nonegotiate
speed 1000
duplex full
spanning-tree portfast
spanning-tree bpduguard enable
And for proxmox:
root@proxmoxus:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback
iface ens1f0 inet manual
iface ens1f1 inet manual
auto bond0
iface bond0 inet manual
slaves ens1f0
bond_miimon 100
bond_mode 802.3ad
auto bond0.666
iface bond0.666 inet manual
vlan-raw-device bond0
auto bond0.777
iface bond0.777 inet manual
vlan-raw-device bond0
auto bond0.888
iface bond0.888 inet manual
vlan-raw-device bond0
auto bond0.999
iface bond0.999 inet manual
vlan-raw-device bond0
auto vmbr666
iface vmbr666 inet manual
bridge_ports bond0.666
bridge_stp off
bridge_fd 0
auto vmbr777
iface vmbr777 inet static
address 192.168.222.253
netmask 255.255.255.0
gateway 192.168.222.254
bridge_ports bond0.777
bridge_stp off
bridge_fd 0
auto vmbr888
iface vmbr888 inet manual
bridge_ports bond0.888
bridge_stp off
bridge_fd 0
auto vmbr999
iface vmbr999 inet manual
bridge_ports bond0.999
bridge_stp off
bridge_fd 0
One important thing:
When the problem occurs again, before you reboot the OPNsense VM please do a
netstat -m
netstat -s
sysctl dev.em
And can you check the errors from a ifconfig on the Proxmox host?
For the moment, no more crashes, but I've written it down, so once I see it again, I'll check it.
Is the backup traffic going through the DMZ interface?
For me it does, not sure how proxmox handles traffic, but the backupserver and others are on the same vlan, with the same gateway. But since they're all on the same KVM, I guess it all passes the network interface?
ps @abel408, sorry to post this in your topic. I thought we were having the same issue. If any moderators reads, you can split it if wanted.
-
My understanding of a trunk: multiple VLANs through 1 cable/port...
No, its a bundling of physical links. A link aggregation.
VLAN's doesn't have trunks. They are virtual network segments that use the same physical segment.
The term "VLAN trunk" is used by Cisco and means that the port/trunk accepts tagged packages for different VLANs.
And for proxmox:
root@proxmoxus:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback
iface ens1f0 inet manual
iface ens1f1 inet manual
auto bond0
iface bond0 inet manual
slaves ens1f0
bond_miimon 100
bond_mode 802.3ad
Only one interface for a bond, aka trunk ;) makes no sense.
Consider using either both interfaces in a bond or a single interface without bonding.
Have a look at https://pve.proxmox.com/wiki/Network_Model
In your case I'd configure 2 ports as layer 4 load balanced "Ether Channel" with a "VLAN trunk" on the Switch.
Then on the Proxmox host add ens1f1 to the bond slaves and change the bond_mode to balance-xor.
And consider using virtio interfaces instead of E1000.
-
In your case I'd configure 2 ports as layer 4 load balanced "Ether Channel" with a "VLAN trunk" on the Switch.
Then on the Proxmox host add ens1f1 to the bond slaves and change the bond_mode to balance-xor.
And consider using virtio interfaces instead of E1000.
The idea was that in the future, I would configure both server interfaces in bonding. For the moment, I don't have a spare port on my switch. I need to free up my old vmware server before I can do this.
Nevetheless is the idea of bonding more to have an extra connection in case the first one fails. So it should work also with 1 interface, no?
But can this explain why opnsense just shutdown 1 vlan?
-
Yes it should work with one interface. But I'd use the active-backup mode then.
I have no explanation why only vlan 1 goes down.
But I bet the problem comes from the high ambient temperature.
70 degree Celsius are by far to much for a disk and other things like network interface cards ;)
-
When the problem occurs again, before you reboot the OPNsense VM please do a
netstat -m
netstat -s
sysctl dev.em
etc.
Had the issue again when today when I came home. Seems it was down since a couple of houres.
And again, it was the same virtual interface? The others seems to stay stable. So if it's caused by the temperature, it's a hardware issue, and the other interfaces should also suffer, no?
Some debugs (i've removed lines with '0' since else the topic-reply was to long :$ ):
netstat -m
1028/6817/7845 mbufs in use (current/cache/total)
1027/3549/4576/379234 mbuf clusters in use (current/cache/total/max)
1027/3527 mbuf+clusters out of packet secondary zone in use (current/cache)
0/62/62/189617 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/56182 9k jumbo clusters in use (current/cache/total/max)
0/0/0/31602 16k jumbo clusters in use (current/cache/total/max)
2311K/9050K/11361K bytes allocated to network (current/cache/total)
netstat -s
tcp:
22994 packets sent
18533 data packets (25117361 bytes)
25 data packets (33645 bytes) retransmitted
1 data packet unnecessarily retransmitted
4237 ack-only packets (0 delayed)
1 window update packet
198 control packets
21234 packets received
17209 acks (for 25106463 bytes)
56 duplicate acks
3947 packets (2856419 bytes) received in-sequence
10 completely duplicate packets (1452 bytes)
8 out-of-order packets (10903 bytes)
21 connection requests
236 connection accepts
8 ignored RSTs in the windows
257 connections established (including accepts)
16451 connections closed (including 1 drop)
238 connections updated cached RTT on close
238 connections updated cached RTT variance on close
4 connections updated cached ssthresh on close
17122 segments updated rtt (of 7597 attempts)
1 retransmit timeout
3523 correct ACK header predictions
3096 correct data packet header predictions
236 syncache entries added
236 completed
236 cookies sent
19 hostcache entries added
0 bucket overflow
1 SACK recovery episode
24 segment rexmits in SACK recovery episodes
33606 byte rexmits in SACK recovery episodes
124 SACK options (SACK blocks) received
6 SACK options (SACK blocks) sent
TCP connection count by state:
6 connections in LISTEN state
1 connection in ESTABLISHED state
udp:
1793582 datagrams received
5953 with no checksum
109794 dropped due to no socket
358136 broadcast/multicast datagrams undelivered
1325652 delivered
1437215 datagrams output
ip:
1654426635 total packets received
9 with data size < data length
1825071 packets for this host
1651951898 packets forwarded (0 packets fast forwarded)
57445 packets not forwardable
1526352 packets sent from this host
280 packets sent with fabricated ip header
icmp:
56056 calls to icmp_error
Output histogram:
echo reply: 10073
destination unreachable: 55855
time exceeded: 201
Input histogram:
echo reply: 3
destination unreachable: 178
echo: 10073
10073 message responses generated
ICMP address mask responses are disabled
arp:
111092 ARP requests sent
63047 ARP replies sent
111111 ARP requests received
5343 ARP replies received
117077 ARP packets received
114881 total packets dropped due to no ARP entry
21749 ARP entrys timed out
0 Duplicate IPs seen
ip6:
3936 total packets received
37 packets sent from this host
19 output packets discarded due to no route
Input histogram:
UDP: 68
ICMP6: 3868
Mbuf statistics:
0 one mbuf
3936 one ext mbuf
0 two or more ext mbuf
sysctl dev.em
dev.em.3.mac_stats.tso_ctx_fail: 0
dev.em.3.mac_stats.tso_txd: 0
dev.em.3.mac_stats.tx_frames_1024_1522: 20922846
dev.em.3.mac_stats.tx_frames_512_1023: 132219
dev.em.3.mac_stats.tx_frames_256_511: 262934
dev.em.3.mac_stats.tx_frames_128_255: 408116
dev.em.3.mac_stats.tx_frames_65_127: 58215365
dev.em.3.mac_stats.tx_frames_64: 46
dev.em.3.mac_stats.mcast_pkts_txd: 4
dev.em.3.mac_stats.bcast_pkts_txd: 2515
dev.em.3.mac_stats.good_pkts_txd: 85927270
dev.em.3.mac_stats.total_pkts_txd: 85949019
dev.em.3.mac_stats.good_octets_txd: 36226734433
dev.em.3.mac_stats.good_octets_recvd: 247276921066
dev.em.3.mac_stats.rx_frames_1024_1522: 162410701
dev.em.3.mac_stats.rx_frames_512_1023: 300853
dev.em.3.mac_stats.rx_frames_256_511: 389427
dev.em.3.mac_stats.rx_frames_128_255: 469147
dev.em.3.mac_stats.rx_frames_65_127: 13040706
dev.em.3.mac_stats.rx_frames_64: 1807739
dev.em.3.mac_stats.mcast_pkts_recvd: 7515
dev.em.3.mac_stats.bcast_pkts_recvd: 754
dev.em.3.mac_stats.good_pkts_recvd: 178361149
dev.em.3.mac_stats.total_pkts_recvd: 178418573
dev.em.3.rxd_tail: 143
dev.em.3.rxd_head: 144
dev.em.3.txd_tail: 82
dev.em.3.txd_head: 82
dev.em.3.fifo_reset: 0
dev.em.3.fc_low_water: 45604
dev.em.3.fc_high_water: 47104
dev.em.3.rx_control: 32770
dev.em.3.device_control: 1075053120
dev.em.3.rx_processing_limit: 100
dev.em.3.itr: 488
dev.em.3.tx_abs_int_delay: 66
dev.em.3.rx_abs_int_delay: 66
dev.em.3.tx_int_delay: 66
dev.em.3.rx_int_delay: 0
dev.em.3.nvm: -1
dev.em.3.%parent: pci0
dev.em.3.%pnpinfo: vendor=0x8086 device=0x100e subvendor=0x1af4 subdevice=0x1100 class=0x020000
dev.em.3.%location: slot=21 function=0 dbsf=pci0:0:21:0 handle=\_SB_.PCI0.SA8_
dev.em.3.%driver: em
dev.em.3.%desc: Intel(R) PRO/1000 Legacy Network Connection 1.1.0
dev.em.2.mac_stats.tso_ctx_fail: 0
dev.em.2.mac_stats.tso_txd: 0
dev.em.2.mac_stats.tx_frames_1024_1522: 0
dev.em.2.mac_stats.tx_frames_512_1023: 0
dev.em.2.mac_stats.tx_frames_256_511: 0
dev.em.2.mac_stats.tx_frames_128_255: 2
dev.em.2.mac_stats.tx_frames_65_127: 4
dev.em.2.mac_stats.tx_frames_64: 0
dev.em.2.mac_stats.mcast_pkts_txd: 6
dev.em.2.mac_stats.bcast_pkts_txd: 1
dev.em.2.mac_stats.good_pkts_txd: 7
dev.em.2.mac_stats.total_pkts_txd: 7
dev.em.2.mac_stats.good_octets_txd: 686
dev.em.2.mac_stats.good_octets_recvd: 0
dev.em.2.mac_stats.rx_frames_1024_1522: 0
dev.em.2.mac_stats.rx_frames_512_1023: 0
dev.em.2.mac_stats.rx_frames_256_511: 0
dev.em.2.mac_stats.rx_frames_128_255: 0
dev.em.2.mac_stats.rx_frames_65_127: 0
dev.em.2.mac_stats.rx_frames_64: 0
dev.em.2.rxd_tail: 255
dev.em.2.txd_tail: 25
dev.em.2.txd_head: 25
dev.em.2.fc_low_water: 45604
dev.em.2.fc_high_water: 47104
dev.em.2.rx_control: 32770
dev.em.2.device_control: 1075053120
dev.em.2.flow_control: 3
dev.em.2.rx_processing_limit: 100
dev.em.2.itr: 488
dev.em.2.tx_abs_int_delay: 66
dev.em.2.rx_abs_int_delay: 66
dev.em.2.tx_int_delay: 66
dev.em.2.nvm: -1
dev.em.2.%parent: pci0
dev.em.2.%pnpinfo: vendor=0x8086 device=0x100e subvendor=0x1af4 subdevice=0x1100 class=0x020000
dev.em.2.%location: slot=20 function=0 dbsf=pci0:0:20:0 handle=\_SB_.PCI0.SA0_
dev.em.2.%driver: em
dev.em.2.%desc: Intel(R) PRO/1000 Legacy Network Connection 1.1.0
dev.em.1.mac_stats.tso_ctx_fail: 0
dev.em.1.mac_stats.tso_txd: 0
dev.em.1.mac_stats.tx_frames_1024_1522: 807177625
dev.em.1.mac_stats.tx_frames_512_1023: 61499407
dev.em.1.mac_stats.tx_frames_256_511: 10051401
dev.em.1.mac_stats.tx_frames_128_255: 10380894
dev.em.1.mac_stats.tx_frames_65_127: 168395840
dev.em.1.mac_stats.tx_frames_64: 124
dev.em.1.mac_stats.mcast_pkts_txd: 5
dev.em.1.mac_stats.bcast_pkts_txd: 30975
dev.em.1.mac_stats.good_pkts_txd: 1058410184
dev.em.1.mac_stats.total_pkts_txd: 1058578569
dev.em.1.mac_stats.good_octets_txd: 1274381690472
dev.em.1.mac_stats.good_octets_recvd: 210095791832
dev.em.1.mac_stats.rx_frames_1024_1522: 114080043
dev.em.1.mac_stats.rx_frames_512_1023: 2387087
dev.em.1.mac_stats.rx_frames_256_511: 3081951
dev.em.1.mac_stats.rx_frames_128_255: 21197508
dev.em.1.mac_stats.rx_frames_65_127: 433582216
dev.em.1.mac_stats.rx_frames_64: 5409878
dev.em.1.mac_stats.mcast_pkts_recvd: 3
dev.em.1.mac_stats.bcast_pkts_recvd: 5567
dev.em.1.mac_stats.good_pkts_recvd: 579625210
dev.em.1.mac_stats.total_pkts_recvd: 579738683
dev.em.1.mac_stats.recv_undersize: 823828
dev.em.1.rxd_tail: 59
dev.em.1.rxd_head: 59
dev.em.1.txd_tail: 185
dev.em.1.txd_head: 185
dev.em.1.fc_low_water: 45604
dev.em.1.fc_high_water: 47104
dev.em.1.rx_control: 32770
dev.em.1.device_control: 1075053120
dev.em.1.flow_control: 3
dev.em.1.rx_processing_limit: 100
dev.em.1.itr: 488
dev.em.1.tx_abs_int_delay: 66
dev.em.1.rx_abs_int_delay: 66
dev.em.1.tx_int_delay: 66
dev.em.1.rx_int_delay: 0
dev.em.1.nvm: -1
dev.em.1.%parent: pci0
dev.em.1.%pnpinfo: vendor=0x8086 device=0x100e subvendor=0x1af4 subdevice=0x1100 class=0x020000
dev.em.1.%location: slot=19 function=0 dbsf=pci0:0:19:0 handle=\_SB_.PCI0.S98_
dev.em.1.%driver: em
dev.em.1.%desc: Intel(R) PRO/1000 Legacy Network Connection 1.1.0
dev.em.0.mac_stats.tso_ctx_fail: 0
dev.em.0.mac_stats.tso_txd: 0
dev.em.0.mac_stats.tx_frames_1024_1522: 101369123
dev.em.0.mac_stats.tx_frames_512_1023: 2562105
dev.em.0.mac_stats.tx_frames_256_511: 3160012
dev.em.0.mac_stats.tx_frames_128_255: 21456364
dev.em.0.mac_stats.tx_frames_65_127: 380160410
dev.em.0.mac_stats.tx_frames_64: 57
dev.em.0.mac_stats.mcast_pkts_txd: 1
dev.em.0.mac_stats.bcast_pkts_txd: 77649
dev.em.0.mac_stats.good_pkts_txd: 508954398
dev.em.0.mac_stats.total_pkts_txd: 509045855
dev.em.0.mac_stats.good_octets_txd: 184144261120
dev.em.0.mac_stats.good_octets_recvd: 1044119307954
dev.em.0.mac_stats.rx_frames_1024_1522: 653001254
dev.em.0.mac_stats.rx_frames_512_1023: 61543647
dev.em.0.mac_stats.rx_frames_256_511: 10548611
dev.em.0.mac_stats.rx_frames_128_255: 10566114
dev.em.0.mac_stats.rx_frames_65_127: 160011699
dev.em.0.mac_stats.rx_frames_64: 699332
dev.em.0.mac_stats.mcast_pkts_recvd: 96
dev.em.0.mac_stats.bcast_pkts_recvd: 413446
dev.em.0.mac_stats.good_pkts_recvd: 896238495
dev.em.0.mac_stats.total_pkts_recvd: 896370657
dev.em.0.rxd_tail: 227
dev.em.0.rxd_head: 228
dev.em.0.txd_tail: 250
dev.em.0.txd_head: 250
dev.em.0.fc_low_water: 45604
dev.em.0.fc_high_water: 47104
dev.em.0.rx_control: 32770
dev.em.0.device_control: 1075053120
dev.em.0.flow_control: 3
dev.em.0.rx_processing_limit: 100
dev.em.0.itr: 488
dev.em.0.tx_abs_int_delay: 66
dev.em.0.rx_abs_int_delay: 66
dev.em.0.tx_int_delay: 66
dev.em.0.nvm: -1
dev.em.0.%parent: pci0
dev.em.0.%pnpinfo: vendor=0x8086 device=0x100e subvendor=0x1af4 subdevice=0x1100 class=0x020000
dev.em.0.%location: slot=18 function=0 dbsf=pci0:0:18:0 handle=\_SB_.PCI0.S90_
dev.em.0.%driver: em
dev.em.0.%desc: Intel(R) PRO/1000 Legacy Network Connection 1.1.0
dev.em.%parent:
ifconfig
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=2088<VLAN_MTU,VLAN_HWCSUM,WOL_MAGIC>
ether 96:7e:28:e9:06:1c
inet6 fe80::947e:28ff:fee9:61c%em0 prefixlen 64 scopeid 0x1
inet 192.168.111.254 netmask 0xffffff00 broadcast 192.168.111.255
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet 1000baseT (1000baseT <full-duplex>)
status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=2088<VLAN_MTU,VLAN_HWCSUM,WOL_MAGIC>
ether 1e:2a:a7:75:12:28
inet6 fe80::1c2a:a7ff:fe75:1228%em1 prefixlen 64 scopeid 0x2
inet 192.168.222.254 netmask 0xffffff00 broadcast 192.168.222.255
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
em2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=2088<VLAN_MTU,VLAN_HWCSUM,WOL_MAGIC>
ether fa:13:81:16:cb:08
inet6 fe80::f813:81ff:fe16:cb08%em2 prefixlen 64 scopeid 0x3
inet 192.168.1.254 netmask 0xffffff00 broadcast 192.168.1.255
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet 1000baseT (1000baseT <full-duplex>)
status: active
em3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=2088<VLAN_MTU,VLAN_HWCSUM,WOL_MAGIC>
ether 22:98:02:e4:30:b6
inet6 fe80::2098:2ff:fee4:30b6%em3 prefixlen 64 scopeid 0x4
inet 192.168.0.226 netmask 0xffffff00 broadcast 192.168.0.255
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet 1000baseT (1000baseT <full-duplex>)
status: active
enc0: flags=0<> metric 0 mtu 1536
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: enc
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
pflog0: flags=100<PROMISC> metric 0 mtu 33160
groups: pflog
pfsync0: flags=0<> metric 0 mtu 1500
groups: pfsync
syncpeer: 0.0.0.0 maxupd: 128 defer: off
ovpns1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
options=80000<LINKSTATE>
inet6 fe80::4c76:46ba:6695:f5bc%ovpns1 prefixlen 64 scopeid 0x9
inet 192.168.112.1 --> 192.168.112.2 netmask 0xffffffff
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: tun openvpn
Opened by PID 57520
em0_vlan666: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 96:7e:28:e9:06:1c
inet6 fe80::947e:28ff:fee9:61c%em0_vlan666 prefixlen 64 scopeid 0xa
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet 1000baseT (1000baseT <full-duplex>)
status: active
vlan: 666 vlanpcp: 0 parent interface: em0
groups: vlan
em1_vlan777: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 1e:2a:a7:75:12:28
inet6 fe80::1c2a:a7ff:fe75:1228%em1_vlan777 prefixlen 64 scopeid 0xb
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
vlan: 777 vlanpcp: 0 parent interface: em1
groups: vlan
em2_vlan888: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether fa:13:81:16:cb:08
inet6 fe80::f813:81ff:fe16:cb08%em2_vlan888 prefixlen 64 scopeid 0xc
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet 1000baseT (1000baseT <full-duplex>)
status: active
vlan: 888 vlanpcp: 0 parent interface: em2
groups: vlan
em3_vlan999: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 22:98:02:e4:30:b6
inet6 fe80::2098:2ff:fee4:30b6%em3_vlan999 prefixlen 64 scopeid 0xd
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet 1000baseT (1000baseT <full-duplex>)
status: active
vlan: 999 vlanpcp: 0 parent interface: em3
groups: vlan
To solve it, I just did a shut, no-shut of the failing virtual interface...
ping 192.168.222.22
PING 192.168.222.22 (192.168.222.22): 56 data bytes
ping: sendto: Host is down
ping: sendto: Host is down
ifconfig em1 down
ifconfig em1 up
ping 192.168.222.22
PING 192.168.222.22 (192.168.222.22): 56 data bytes
64 bytes from 192.168.222.22: icmp_seq=0 ttl=64 time=1.279 ms
64 bytes from 192.168.222.22: icmp_seq=1 ttl=64 time=0.883 ms
So guess I could write a small script to check and re-enable the interface. But of course, would be better to solve the issue instead of bypassing it. So if anybody has an idea where it goes wrong...
-
Hi brononius,
nothing suspicious to see here. Not even much traffic.
Have you disabled the 'Energy Efficient Ethernet' setting yet?
If not add hw.em.eee_setting="0" to your /boot/loader.conf.local and reboot.
Regards,
Frank
-
Have you disabled the 'Energy Efficient Ethernet' setting yet?
I've tried that, but gave me the same results.
For now, I've migrated the proxmox/opnsense to other hardware, and since 1 week, it's stable.
So somewhere, at first sight, it was a link between the cisco hardware blade and proxmox/opnsense.