Topics

I am attempting to get monit to send me an alert when an IPsec [legacy] tunnel is down.

I am seeing some behaviours that I do not understand, I'm reluctant to apply the potential solutions I've found during my searching 'til I know better what's going on.

monit shows this in the logs when I disable the tunnel I am attempting to monitor

Code Select Expand


2023-10-29T21:13:30-07:00 Error monit Aborting event
2023-10-29T21:13:30-07:00 Error monit Mail: Delivery failed -- no mail server is available
2023-10-29T21:13:30-07:00 Error monit Cannot open a connection to the mailserver 192.168.254.9:25 -- Operation now in progress
2023-10-29T21:13:30-07:00 Error monit Cannot connect to [192.168.254.9]:25 -- Connection timed out
2023-10-29T21:13:00-07:00 Error monit 'restart_PCC_IPsec_tunnel' ping test failed
2023-10-29T21:13:00-07:00 Error monit Ping response for 192.168.78.254 5/5 timed out -- no response within 5 s
2023-10-29T21:12:55-07:00 Warning monit Ping response for 192.168.78.254 4/5 timed out -- no response within 5 s
2023-10-29T21:12:50-07:00 Warning monit Ping response for 192.168.78.254 3/5 timed out -- no response within 5 s
2023-10-29T21:12:45-07:00 Warning monit Ping response for 192.168.78.254 2/5 timed out -- no response within 5 s
2023-10-29T21:12:40-07:00 Warning monit Ping response for 192.168.78.254 1/5 timed out -- no response within 5 s
2023-10-29T21:12:35-07:00 Informational monit 'gw1.domain.tld' Monit reloaded


Additional information:
1st - the firewall running monit [OPNsense 23.7.7_3-amd64] is at 192.168.17.254
2nd - the mail server is at 192.168.254.9
3rd - the mail server at 192.168.254.9 is reachable from any other host on the 192.168.17.0/24 network

example from a normal host on the LAN:

Code Select Expand


root@ns1:~# ip addr show dev enp0s4 | grep inet
    inet 192.168.17.14/24 brd 192.168.17.255 scope global noprefixroute enp0s4
#
root@ns1:~# ping -q -c 3 192.168.254.9
PING 192.168.254.9 (192.168.254.9) 56(84) bytes of data.

--- 192.168.254.9 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2014ms
rtt min/avg/max/mdev = 14.848/16.489/18.878/1.727 ms
#
root@ns1:~# telnet 192.168.254.9 25
Trying 192.168.254.9...
Connected to 192.168.254.9.
Escape character is '^]'.
220 mx-backup.bisi.ca ESMTP Postfix (Ubuntu)


Example from the shell of the OPNsense box:

Code Select Expand


root@gw1:~ # ifconfig re0 | grep inet
inet 192.168.17.254 netmask 0xffffff00 broadcast 192.168.17.255
#
root@gw1:~ # ping -q -c 3 192.168.254.9
PING 192.168.254.9 (192.168.254.9): 56 data bytes

--- 192.168.254.9 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss
#
root@gw1:~ # telnet 192.168.254.9 25
Trying 192.168.254.9...
^C


This description from the netgate docs seems to describe the situation, as well as a fix, but is this really what's going on here?
https://docs.netgate.com/pfsense/en/latest/vpn/ipsec/access-firewall-over-ipsec.html
https://web.archive.org/web/20231030051250/https://docs.netgate.com/pfsense/en/latest/vpn/ipsec/access-firewall-over-ipsec.html

I have obtained the same result from 4 other OPNsense firewalls of various generations, both from the shell and from Interfaces -> diagnostics -> ping.

All firewalls can ping (or telnet/ssh) to) hosts outside the firewall, and hosts on their own class C LAN (192.168.17.x in this example).

So - any feedback appreciated (other than to migrate the tunnels to "Connections" I've already incurred *way* too many unbillable hours for this client trying to do just that).  Sorely hoping there's a simpler fix to this issue so I can stop hearing from these clients

[/usr/local/etc/ddclient.conf]

Well, push has finally come to shove.  I have 15 clients hosted at dyn.com (formerly and a.k.a. dyndns) for which I have repeatedly failed to get os-ddclient to work.  I no longer have the luxury of relying on the os-dyndns plugin. I am faced with a hardware replacement scenario, and I might as well bite the bullet.  I have a new protectli-hosted OPNsense 23.7.3-amd64; FreeBSD 13.2-Release-p2; OpenSSL 1.1.1v 1 Aug 2023.

Everything restored perfectly from backup, excluding of course the os-dyndns plugin. So I've added os-ddclient, and it has reliably failed out of the gate.  After several unproductive troubleshooting attempts, including a file generated by dyn.com's "upate client configurator" https://account.dyn.com/tools/clientconfig.html, I have, per some successful ddclient troubleshooting on the FreeBSD forums, reached an error message that I can't get past, and a large amount of internet searching has not provided any clues.  The basic idea that worked with native FreeBSD does not work here.  I always end up with this message:

Code Select Expand

FAILED:   updating N0T: notfqdn: A Fully-Qualified Domain Name was not provided

So I've reproduced the logs and the troubleshooting in the hope that someone here can elucidate, or suggest something to try, or at least a clue where the problem originates.

The starting conditions:
The OPNSense host is named bogus1.dyndns-at-home.com
There is a DNS record at dyndns.com of the same name, with the IP address of the machine before the hardware failure, from a different ISP

I have os-ddclient set with these parameters:

Code Select Expand

Edit Account
Enabled               < X >
Description           https://account.dyn.com/
Service               DynDNS.com
Username              <redacted>
Password              <redacted>
Wildcard              <  >
Hostname(s)           bogus1.dyndns-at-home.com
Check ip method       Interface
Interface to monitor  WAN
Check ip timeout      10
Force SSL             < X >

Which creates this config file ( /usr/local/etc/ddclient.conf ):

Code Select Expand

syslog=yes                  # log update msgs to syslog
pid=/var/run/ddclient.pid   # record PID in file.
ssl=yes

usev4=ifv4, ifv4=igb0, \
protocol=dyndns2, \
login=<redacted>, \
password=<redacted> \
bogus1.dyndns-at-home.com

Now, get shell access as root via SSH
stop the daemonized ddclient

Code Select Expand

kill -TERM `cat /var/run/ddclient.pid`
verify the .pid file is gone

Code Select Expand

cat /var/run/ddclient.pid
  cat: /var/run/ddclient.pid: No such file or directory
purge the ddclient cache

Code Select Expand

rm /var/tmp/ddclient.cache
run ddclient in debug mode, in the foreground (per FreeBSD troubleshooting)

Code Select Expand

ddclient -daemon=0 -debug -verbose -noquiet

This also fails to update the ip address at dyn.com, ending with the usual.

Code Select Expand

FAILED:   updating N0T: notfqdn: A Fully-Qualified Domain Name was not provided

I have attached two files:
ddclient_debug_output - the output from the debug screen
ddclient_syslog.log - the "all levels" output recorded by syslog during the exercise.

Any help greatly appreciated!
d.

... or at least a web link where I can check every so often?

I have been using dyn.com since they had free accounts (and were called dyndns.org), sometime around the turn of the century.  Anyway, I have quite a large number of clients supported through my account at dyn.com, and I'm getting pretty antsy at the warning that support for os-dyndns is about to be terminated.

I have tried and failed several times to switch to os-ddclient and have it work with my clients using OPNsense/dyn.com, and several browsing sessions on the forums has not enabled me to come up with a reliable recipe/hack, plus a fear that I'm going to have to migrate everyone to another service, in a hurry, when os-dyndns finally joins IE 11 in the bitbucket.

Dyn.com has support instructions for ddclient at https://help.dyn.com/ddclient/, but my I'm really not into leaving my clients with invisible hacks that nobody but me can support in an emergency. Also I don't know how to persist this through upgrades, etc. Finally, the "issues to be aware of" section of those instructions fairly drip with unbillable time.

Any suggestions welcome (including alternate dynamic DNS services with good records, and decent management interfaces for 30 or so names)!

I have a (well one of many) opnsense community edition with a particular new need.
version info

OPNsense 21.7.2_1-amd64
FreeBSD 12.1-RELEASE-p20-HBSD
OpenSSL 1.1.1l 24 Aug 2021

We have port 443 forwarded and access controlled using GeoIP.  It greatly reduced the log noise, and the slow-moving brute force attacks that every so often triggered the mail server's auto-lockout defences.  We do need to allow this access to a wide range of local IP addresses ( various ISPs, plus people do travel, and want to check their work mail while away).  This has been an acceptable compromise 'til now.

The client has now signed up for a CRM service that requires 3 addresses from the Amazon cloud to also have access to port 443.  The vendor has not been particularly impressive in their grasp of technical detail, and proposing access via a custom port (limited only to them) was met with consternation and clear lack of knowledge if it was even possible.

Is there a way to set up an alias that allows both the GeoIP and the CRM addresses?

I have been unable to figure this out from the documentation, and from just playing with my Dev firewall.

Some other method would be acceptable.  The mail server is zimbra OSE, and implementing 2FA is in the works, but until then I'm hoping for an extra layer from the firewall.

Thanks in advance!
d.

It was working pre-upgrade (21.1.5).  Essentially the widget on the lobby dashboard is now just a white box.  I have rebooted, removed and re-instantiated the widget (with reboots at each stage).

The Reporting -> Traffic page is in a similar state - two tabs (Graph, Top Talkers), and "Nothing Selected" with the pull down for the selection showing nothing.

I have attached a screenshot of the main reporting page

I have several other routers on the same hardware that all upgraded successfully

Where to go from here?

Thanks in advance!
d.

I have attached a screen shot of the captioned error message, generated when I attempt to create an alias as part of setting up GeoIP blocking.

By way of background, GeoIP was working on this firewall, and then we started getting lockouts on our mail server.  Investigation revealed the geographic blocking was no longer working.   See this discussion for some of the things I have tried.
https://forum.opnsense.org/index.php?topic=18411.0

I have several times removed all parts of the configuration for the geoIP function (including removing contents of directories in the filesystem) and attempted to re-create it, using various forms of the name for the Alias.  I have numerous other opnsense firewalls with working geo-blocking configurations, and the same configuration on this particular box does not work.

The firewall is updated to OPNsense 20.7.4-amd64 (it stopped working before this update).  I have working instances of GeoBlocking on the same code level, on both identical and completely different hardware.

Where to go with troubleshooting?  I'd like to make this fixable, rather than just configure another box and drop it in, but we have email accounts exposed to slow-moving brute force password attacks, which generate lockout for some of our users, so it's not something we can live with for very long.

Thanks in advance for any help!
d.

PS - here's my recipe for creating a geo-blocking rule.  It's in wikimedia format, so readability might be an issue (monospace font helps).

Code Select Expand

===GeoIP===
new procedure for OPNSense v20.1x
[[https://docs.opnsense.org/manual/how-tos/maxmind_geo_ip.html must sign up at Maxmind]]
:TL;DR
:have set up an account that all clients can use. Details, if you need to do it again to provide completely separate credentials for a client, are below.
: paste this URL in the entry box for ''Firewall:Aliases:GeoIP Settings''
::https://download.maxmind.com/etc/etc/etc/

:Credentials
   userID:
password:
     name:       

'''License Key'''
Your new license key ''YadaYadaYada'' has been created.
This license key is stored in hashed format for security purposes.

It may take up to five minutes for this new key to be activated.

This will be the only time this key is displayed to you in full.
Please copy the key to a safe location for your future reference.

Account/User ID:
     License key:
====set up alias====
Firewall -> aliases
   + button at bottom of alias list
     Enabled <X>
     Name    [GeoBlocking1]
     Type    [GeoIP] [ IPv4 | IPv6 | IPv4+IPv6 ] 
     Content [pick your countries to accept/block] <-- start at bottom of list and work upwards
     statistics <X>  (or not - you decide)
     Description [block incoming connections by geographic region]

====set up WAN Rule====
Firewall -> Rules -> WAN
<pre>
Action                        [Block]
Disabled                      < > Disable this rule
Quick                         <X> Apply the action immediately on match.
Interface                     [WAN]
Direction                     [in]
TCP/IP Version                [IPv4]
Protocol                      [any]
Source / Invert               < >
Source                        [Geographic_blocks] (from aliases)
Source                        [Advanced]  (not used)
Destination / Invert          < >
Destination                   [WAN address]
Destination port range from:  [any]    to:  [any]
Log                           <X> Log packets that are handled by this rule
Category                      [GeoBlock1 ]
Description                   [REMEMBER TO TURN OFF LOGGING]
Advanced features
Source OS                     [Any]
No XMLRPC Sync                < >
Schedule                      [none]
Gateway                       [default]
Advanced Options              [Show/Hide]  (not used)
</pre>
'''Position rule at top of ruleset''

Sorry - I just realized I posted this to the 19.1 thread in error.  S/B the 19.7 thread, and I don't see a way to unpost or move it to the correct place...
------------------------------

A curious thing just happened at a client site that I don't think I've seen elsewhere.

The ISP dropped in and replaced their modem, and of course something didn't work afterward, so they cycled power on the OPNSense router to troubleshoot, by pulling the power cord (didn't fix the problem). 

Once they called me and we got the real problem solved (Windows Server DNS needed a restart), I wanted to review the health data, especially the Quality metric for the the upstream gateway.

I was dismayed to see that these are all the data I have  (screen shots below -- you can see when the router was brought back up).



This router has been running for months, and has been through at least two update cycles, with at least one reboot.  There are 5 other routers as part of the system, all deployed pretty close to the same time and configured the same.  They all have the normal complement of data (example at bottom).

So, is this by design?  Or have I missed something in my configuring?  Where to start troubleshooting?

Thanks in advance!


PS I don't seem to be able to get the .png screenshots to show up in the body.  They are attached, and all fit under the various max size limits.

I have several OPNsense firewalls deployed.  I have recently noticed (as a result of troubleshooting Firefox's inability to connect to the GUI -- stalling at the TLS handshake stage) that they all have expired certificates.  This is one I just updated to 19.1.8 last night.  The expiry date of the cert is 2 days previously.  Is there an explanation for this?  A way to rectify it?

This does not really matter for any practical purpose in my situation (it's only a small factor in the Firefox issue), except that the browser developers are constantly removing the ability for a user to exercise their judgment in situations like this, and at some point I fully expect to be barred from accessing these hosts, based on an expired (or self-signed) certificate.

I've attached a screen shot as a .png