Messages

I'd check if the Broadcom card is supported at al https://www.freebsd.org/releases/11.1R/hardware.html#ethernet

Code Select Expand

[i386,amd64] The bce(4) driver provides support for various NICs based on the QLogic NetXtreme II family of Gigabit Ethernet controllers, including the following:
QLogic NetXtreme II BCM5706 1000Base-SX
QLogic NetXtreme II BCM5706 1000Base-T
QLogic NetXtreme II BCM5708 1000Base-SX
QLogic NetXtreme II BCM5708 1000Base-T
QLogic NetXtreme II BCM5709 1000Base-SX
QLogic NetXtreme II BCM5709 1000Base-T
QLogic NetXtreme II BCM5716 1000Base-T
Dell PowerEdge 1950 integrated BCM5708 NIC
Dell PowerEdge 2950 integrated BCM5708 NIC
Dell PowerEdge R710 integrated BCM5709 NIC
HP NC370F Multifunction Gigabit Server Adapter
HP NC370T Multifunction Gigabit Server Adapter
HP NC370i Multifunction Gigabit Server Adapter
HP NC371i Multifunction Gigabit Server Adapter
HP NC373F PCIe Multifunc Giga Server Adapter
HP NC373T PCIe Multifunction Gig Server Adapter
HP NC373i Multifunction Gigabit Server Adapter
HP NC373m Multifunction Gigabit Server Adapter
HP NC374m PCIe Multifunction Adapter
HP NC380T PCIe DP Multifunc Gig Server Adapter
HP NC382T PCIe DP Multifunction Gigabit Server Adapter
HP NC382i DP Multifunction Gigabit Server Adapter
HP NC382m DP 1GbE Multifunction BL-c Adapter

Then check if the card has errors (Interfaces -> Overview or netstat -idb -I bce0).

And read the bce(4) manaual page. Maybe increasing the sysctl's hw.bce.rx_pages and hw.bce.tx_pages could help.

No worries.
If it works I'll put it into the How-To section.  :)

[Ping]

Ok, first we need to know how to monitor the clients.
The easiest way would be ICMP. Make sure you can ping the remote side.

The Monit plugin installer should have already created a Ping test.
Name: Ping
Condition: failed ping
Action: Alert

Then create a Service of type 'Remote Host'.
Name: Client1
Type: Remote Host
Address: <REMOTE_IP>
Tests: Ping

Restart Monit and check the status page. If the VPN is up the status of Client1 should be OK.


Now comes the difficult part: restarting the client if the ping fails.

The os-monit plugin is only a GUI for the monit(1) service configuration.
And therefore we need a shell command to start/stop the clients.
Unfortunately OPNsense has no way to restart the VPNs from command line. https://github.com/opnsense/core/issues/412

You need to create your own start/stop script.
Connect the firewall via SSH and open a Shell.
The client configuration naming is client1, client2 and so on.
The folder /var/etc/openvpn/ holds the configs. Have a look at clientX.conf to see which client for which Monit config you need.

The openvpn rc.d script expects its config files in /usr/local/etc/openvpn/.
Therefore create a link to it

Code Select Expand

# cd /usr/local/etc/
# ln -s /var/etc/openvpn

Then create a link to the openvpn start/stop script for e.g. client1.
The name of the link must match the prefix of the config file in /var/etc/openvpn/.

Code Select Expand

# cd /usr/local/etc/rc.d
# ln -s openvpn client1

Then you can check the status of client1:

Code Select Expand

# /usr/local/etc/rc.d/client1 onestatus
openvpn_client1 is running as pid 55319.

Try to stop and start it:

Code Select Expand

# /usr/local/etc/rc.d/client1 stop
Stopping client1.
Waiting for PIDS: 55319.
# /usr/local/etc/rc.d/client1 onestart
Starting client1.

Now edit the Monit service 'Client1' and add the
start/stop scripts
Start: /usr/local/etc/rc.d/client1 onestart
Stop: /usr/local/etc/rc.d/client1 stop

Edit the 'Ping' Service Test, change the action from 'Alert' to 'Restart' and restart Monit.

Additionally I'd limit the restarts.
Add a service test:
Name: RestartLimit5
Condition: 5 restarts within 5 cycles
Action: unmonitor

And add it to the 'Client1' service:
Test: Ping, RestartLimit5

Hi ezra55,

At random they all stop and not get back up. Does anyone has some experience setting this up in Monit to get fixed?
I use monit CLI but this GUI stuff is hard to figure out.

what exactly is your problem?
Do you have a Monit config and don't know how to do the same with the GUI?
Or do you need a more basic advice how to monitor your VPNs?


Cheers,
Frank

Hi Wayne,

Isn't it more logical to make the proxy listening on for example 192.168.1.1 f.e., if this is considered the internal gateway on the LAN side ?

no.

Connecting the proxy without a redirection doesn't work. Therefore you don't need it to listen on LAN/OPT.

If you let the proxy listen on the LAN/OPT interface you need a proxy for every LAN/OPT interface.
A proxy on 127.0.0.1 can serve all FTP connections.
In any case you still need the port forwarding rules.


Cheers,
Frank

Hi car_gli,

schuld daran ist Monit 5.22.
Du kannst Monit 5.23 aus den Ports installieren oder auf OPNsense 17.1.9 warten.


Viele Grüße
Frank

[Services->Monit->Settings]

Monit is a software that can check your filesystems, disks, processes, system and many more.
It runs on the firewall host and sends messages or executes actions on various events.
With the os-monit plugin you can configure the Monit daemon.

Installation
To use Monit install the os-monit plugin first. It installs the monit package as a dependency.
After the installation has finished reload the GUI and navigate to Services->Monit->Settings.

Configuration
The first step is to check if the plugin installer has imported your System->Notification settings correctly. Then have a look at the other tabs. To help you get acquainted the installer has added some standard entries.

The order to configure a monitoring is to create Service Tests first, then Services to check and in the end Alerts.

Let's start with the Service Test Settings. A test contains a condition and an action. It can be assigned to one or more services. The Monit documentation shows possible tests. You can simply adopt it by omitting the IF and THEN statements.

The next step is to configure service checks. Depending on the service type we need to set a path or start/stop scripts etc. and assign tests we have defined before. You can assign the same tests to different service checks.

At the Alert Settings tab you can configure to whom sent alerts to on which events and to whom not.
Furthermore you can format the mail text. E.g. Subject: $SERVICE failed on $HOST at $DATE

NOTE: For cluster setups you can synchronize the configuration to the other node if you check Monit System Monitoring
      under System->High Availabilty->Settings. Then go to Firewall->Diagnostics->Filter Reload and click on Force Config Sync.

Status
After starting Moint you can view detailed status information on the  Services->Monit->Status page.

System 'firewall.example.com'
  status                       Running
  monitoring status            Monitored
  monitoring mode              active
  on reboot                    start
  load average                 [0.59] [0.41] [0.35]
  cpu                          0.2%us 0.9%sy
  memory usage                 2.6 GB [8.1%]
  swap usage                   0 B [0.0%]
  uptime                       20d 4h 25m
  boot time                    Thu, 18 May 2017 13:27:11
  data collected               Wed, 07 Jun 2017 17:51:23

Filesystem 'RootFs'
  status                       Does not exist
  monitoring status            Monitored
  monitoring mode              active
  on reboot                    start
  data collected               Wed, 07 Jun 2017 17:51:23

Program 'IPMI'
  status                       Status ok
  monitoring status            Monitored
  monitoring mode              active
  on reboot                    start
  last exit value              0
  last output                  System Health: OK, Power Meter: 174 Watts, Ambient Temp: 19 degrees C, Fans: 39 percent
  data collected               Wed, 07 Jun 2017 17:51:23

Examples

FTP Proxy
Check FTP proxy and restart it. If it doesn't run after 5 restarts don't check it again.
Tests
Name: RestartLimit5
Condition: 5 restarts within 5 cycles
Action: Unmonitor
Name: FTPProxy8021
Condition: failed host 127.0.0.1 port 8021 type tcp
Action: Restart

Service
Name: FTPProxy8021
Type: Process
PID File: /var/run/osftpproxy.127_0_0_1_8021.pid
Start: /usr/local/sbin/configctl ftpproxy start 127_0_0_1_8021
Stop: /usr/local/sbin/configctl ftpproxy start 127_0_0_1_8021
Tests: FTPProxy8021, RestartLimit5


HPE ProLiant DL380 G7 via IPMI
Check HPE ProLiant DL380 G7 hardware via IPMI.
Install ipmitool, load the ipmi kernel driver and create a check script.

Code Select Expand

pkg install ipmitool
kldload ipmi
echo 'ipmi_load="YES"' >>/boot/loader.conf.local

create a script /usr/local/bin/CheckIPMI.sh

Code Select Expand

#!/bin/csh

set Status = 0
set FanCount = 0
set FanSum = 0
foreach Line (`/usr/local/bin/ipmitool sdr list | sed 's/ /\\t/g'`)
   set SensorData = `echo $Line | sed 's/\\t/ /g'`
   set SensorName = `echo $SensorData | awk -F\| '{print $1}'`
   set SensorValue = `echo $SensorData | awk -F\| '{print $2}'`
   set SensorStatus = `echo $SensorData | awk -F\| '{print $3}'`
   if ( "$SensorStatus" != "ok" && "$SensorValue" != "disabled" ) then
      echo "$SensorData"
      set Status = 1
   endif
   if ( "$SensorName" =~ "Power Supply*" ) then
      if ( "$SensorValue" == "0 Watts" ) then
         echo "$SensorData"
         set Status = 1
      endif
   endif
   if ( "$SensorName" == "Power Meter" ) set PowerMeter = "$SensorValue"
   if ( "$SensorName" == "Temp 1" ) set AmbientTemp = "$SensorValue"
   if ( "$SensorName" =~ "Fan *" ) then
     @ FanCount = ( $FanCount + 1 )
     set FanValue = `echo $SensorValue | awk '{print $1}' | awk -F. '{print $1}'`
     @ FanSum = ( $FanSum + $FanValue )
   endif
end
if ( $Status == 0 ) then
   @ FanSpeed = ( $FanSum / $FanCount )
   echo "System Health: OK, Power Meter: $PowerMeter, Ambient Temp: $AmbientTemp, Fans: $FanSpeed percent"
endif

exit $Status

Make it executable.

Code Select Expand

chmod +x /usr/local/bin/CheckIPMI.sh

Tests
Name: ExecStatus
Condition: status notequal 0
Action: Alert

Service
Name: IPMI
Type: Custom
Path: /usr/local/bin/CheckIPMI.sh
Tests: ExecStatus

Hi Andreas,

JeGr hat es bereits mehrfach erklärt. Das was Du machen willst funktioniert einfach nicht. Nie und nimmer!!

Du kannst keine IP Adresse (10.96.100.2) aus einem Adressbereich (10.96.100.0/24) über ein Interface routen das zu einem anderen Bereich (10.100.0.0/16 bzw. 193.26.188.0/22) gehört.

Desweiteren ist es nicht möglich eine private Adresse über das Internet zu routen (indirekt per VPN aber schon).
D.h. 10.96.100.2 wird auf WAN4 niemals erreichbar sein und die 'privaten' Netze werden auch nie eine Antwort erhalten.

Wenn die Firewall aus unterschiedlichen Netzen erreichbar sein soll, dann nur über unterschiedliche IP Adressen.
Wenn sie über das Internet erreichbar sein soll, dann nur über eine öffentliche Adresse oder VPN.

Du solltest Dein Vorhaben nochmal neu konzipieren.


Viele Grüße
Frank

Hi brononius,

nothing suspicious to see here. Not even much traffic.
Have you disabled the 'Energy Efficient Ethernet' setting yet?
If not add hw.em.eee_setting="0" to your /boot/loader.conf.local and reboot.


Regards,
Frank

Hi Dario,

no idea what 'password_ of_ email_ account' is.
Is this possibly the password of your email account?
If the password contains spaces it must be enclosed in single quotation marks.
But this only a guess because the monit(1) manual doesn't tell us much about special characters in passwords.
Better omit spaces.

Regards,
Frank

Correct.
These settings are the result of many tries I've made until I've got the most stability and performance.
Feel free and test it out on other hardware ;)

Solved it by increasing the undocumented igb(4) hw.igb.buf_ring_size setting.
It seems that the HPE NC365T adapter cannot push the packets fast enough out to the wire.
But that's anyone's guess.

If someone runs OPNsense on a ProLiant too here are my settings.
@franco: Could be the first settings for the network card tweak plugin.  :)

/boot/loader.conf.local

Code Select Expand

ipmi_load="YES"
net.link.ifqmaxlen="8192"
hw.igb.buf_ring_size="32768"
hw.igb.max_interrupt_rate="96000"
hw.igb.num_queues="1"
hw.igb.rx_process_limit="4096"
hw.igb.tx_process_limit="4096"
hw.igb.rxd="4096"
hw.igb.txd="4096"
net.pf.states_hashsize="16777216"

System -> Settings -> Tunables

Code Select Expand

kern.ipc.maxsockbuf 8388608
net.inet.tcp.sendbuf_max 16777216
net.inet.tcp.recvbuf_max 16777216
net.inet.tcp.sendspace 131072
net.inet.tcp.recvspace 131072
net.inet.tcp.sendbuf_inc 32768
net.inet.tcp.recvbuf_inc 65536
kern.ipc.soacceptqueue 1024

Interfaces -> Settings

Code Select Expand

uncheck 'Disable hardware CRC, TSO and LRO'

Yes it should work with one interface. But I'd use the active-backup mode then.

I have no explanation why only vlan 1 goes down.
But I bet the problem comes from the high ambient temperature.
70 degree Celsius are by far to much for a disk and other things like network interface cards  ;)

My understanding of a trunk: multiple VLANs through 1 cable/port...

No, its a bundling of physical links. A link aggregation.
VLAN's doesn't have trunks. They are virtual network segments that use the same physical segment.
The term "VLAN trunk" is used by Cisco and means that the port/trunk accepts tagged packages for different VLANs.

And for proxmox:

Code Select Expand

root@proxmoxus:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback

iface ens1f0 inet manual

iface ens1f1 inet manual

auto bond0
iface bond0 inet manual
slaves ens1f0
bond_miimon 100
bond_mode 802.3ad


Only one interface for a bond, aka trunk ;) makes no sense.
Consider using either both interfaces in a bond or a single interface without bonding.
Have a look at https://pve.proxmox.com/wiki/Network_Model

In your case I'd configure 2 ports as layer 4 load balanced "Ether Channel" with a "VLAN trunk" on the Switch.
Then on the Proxmox host add ens1f1 to the bond slaves and change the bond_mode to balance-xor.

And consider using virtio interfaces instead of E1000.

Yes, the high error rate in contrast to the low traffic let suggest it.
I don't think it's a driver problem.
I use the igb driver too.
Have an eye on hw.igb.buf_ring_size.


Cheers,
Frank