Messages

Hi all,

After searching the forum, there is not a lot of using fail2ban with OPNsense. I wanted to create a centralized fail2ban server: meaning, if a person/IP is banned on one service, it should be banned for all of them and in the first line of defence, not only deep inside my network.

So I did some digging and found the following fail2ban action on Github:
https://github.com/fail2ban/fail2ban/pull/2761/files

Which lead me to something like this:


It works like this:

• HAproxy forwarding requests to the services in the DMZ
• all services log failed attempts to the log-dataset
• one centralized fail2ban instance (in the NAS-network, which is quite tightly locked-down) has access to all logs and takes action for wrong passwords
• if an IP is banned, it updates the alias on OPNsense via API correspondingly, instantly dropping further requests from this IP

It seems to have been working out quite nicely, but since I am rather new to OPNsense, some questions came up:

• Is it a good idea security wise, or does it rather weaken my network:
  
  
  • I am a bit anxious of using the API, and storing such powerful keys outside the firewall. Is there a way of storing them encrypted yet still usable in the f2b action? Can I limit the access of this key? It only needs to be able to update an alias, it would be nice if I could prohibit it from doing more.
  • Is the route e.g Nextcloud > Logs > f2b server > Firewall a possible attack vector? Assuming Nextcloud is highjacked, could some command injection in the logs screw with fail2ban and get access to its VM and therefore API key?
    
  • Is the curl command secure for transmitting the API-key? It calls the https adress of my firewall, so I would guess yes, but I am not sure if this is sufficient? Additional VPN tunnel, or is this an overkill?
    
• I like a lot the instantaneous action in case of wrong passwords attempts: if any service is hit with the maximum number of wrong passwords, the alias is instantly updated (in contrast to regularly retrieving a file), preventing the attacker from trying other services. BUT: it does not kill the current state. Assuming I am hacking my Nextcloud, as the state is established, even after f2b updated the alias, I can still keep trying passwords and even get access when finally entering the correct one. Is there a way of killing the state by API? Or some workaround?
• 
  The ssh dummy-server is not working yet. I would like to see anybody instantly blocked who tries to reach my 22 (nobody with good intentions does). I tried creating a dummy server with IPFW, logging all ssh attempts and feeding this log to f2b. But, seems like all firewall logging (apparently also pf and iptable) in iocage jails actually write to the host systems logs, which are too cluttered for f2b.
  It would work, if I actually run an accessible SSH server on my dummy, but I would rather like to stay invisible for any SSH attempts. Any alternatives of preferably clean logging of incoming connections on certain ports?
  
• 
  I listed 22 and 23 for the "honeypot". Any other ports, that are usually a sign of bad intentions? RDP port maybe?
  And this connection is not yet established, I thought a simple port-forward to the dummy should do, or is there a better way?
  

Thanks for reading, looking forward to hear any feedback about this concept.
Cheers

Update (for completeness sake):

In the meantime I got the hint to check the static arp's in the arp table. It didn't work according to the above approach with the rc.conf file (entries still expiring) but did work, when using the option "static arp" in the dhcp server settings in OPNsense GUI.

Next idea was using tcpdump to inspect the traffic, and tcpdump -e to check, if the correct mac adresses are used. As far as I could tell, OPNsense does resolve the MAC adresses correctly.

I was still unsure of which MAC adress to use in the arp table, since also behind the WiFi bridge it is a certain chain of MAC adresses that I could use for specifying the services adress:

Code Select Expand


                    | TrueNAS internally:
ManagedSwitch   ---  physical NIC   ---   Bridge   ---   VNET epair interface

I used the last one, since it is the only one linked to the IP adress.

Anyhow, inspired by Plex, which is broadcasting all the time and works for that reason, I found a workaround, that I can live with for the time being. All services drop a little "hello there" towards opnsense:

Code Select Expand

* * * * * ping -c 1 192.168.DMZ.1

Not elegant, but does the trick (not even a firewall rule needed for this, even a blocked ping makes OPNsense realize the presence of the VMs). Hopefully, later down the road I will be able to eliminate the WiFi bridge and maybe also this weird issue.

Cheers

I thought I had it.

I found this post on server fault, where someone describes exactly my issue and this was a suggestion:

Does sound like a arp problem, if its long spanned vlan over a vpn or similar, you may need to add static arp entries, with an ether address of the local switch.

https://serverfault.com/questions/521928/cannot-ping-b-from-a-until-b-pings-a


My setup looks as follows:

EVILINTERNET    ----   Modem   ----   OpnSense    ----    Wifi AP   <--wifibridge-->   Wifi AP   ----  Managed Switch (Netgear GS108Ev2)  ---- Server

I can imagine, that the wifi-bridge has a similar effect as the described long spanned vlan over vpn. So I tried the static ARP entries, as described in this post:
https://forum.opnsense.org/index.php?topic=7273.0

I created /etc/rc.conf on OPNsense with the IPs and MAC adresses of the web servers and then restarting the static arp service. But no change, it still happens, that I cannot reach or ping some servers.

Then I saw, that the serverfault post stated, that the switch MAC adress should be entered. But still nothing.

But no matter what, when displaying arp -a, I see all these expiration times on my web servers, whereas only IPs from OPNsense itself seem to be permanent. What should be expected when setting static ARP entries? I never used this before, as anyone can guess, I am not a professional.

These are the two attempts I made in the rc.conf (above individual MAC adresses of all servers, below the managed switch):

Code Select Expand


# setting MAC of each web server
#static_arp_pairs="nc vw son rad trans plex"
#static_arp_nc="192.168.DMZ.13 MAC:df"
[...]
#static_arp_plex="192.168.DMZ.14 MAC:ab"


# setting MAC of managed switch in front of web servers
static_arp_pairs="nc vw son rad trans plex"
static_arp_nc="192.168.DMZ.13 MAC:9c"
[...]
static_arp_plex="192.168.DMZ.14 MAC:9c"



Shortened output of arp -a:

Code Select Expand


root@opnsense:~ % arp -a
? (192.168.DMZ.13) at <<MAC>>:df on vlan0.DMZ expires in 898 seconds [vlan]
? (192.168.DMZ.14) at <<MAC>>:ab on vlan0.DMZ expires in 847 seconds [vlan]
[...]
? (192.168.DMZ.1)  at <<MAC>>:e8 on vlan0.DMZ permanent [vlan]
[...]
<<ISP>> (<<GLOBALIP>>.1) at <<MAC>> on igc0 expires in 1200 seconds [ethernet]
<<IP@ISP>> (<<GLOBALIP>>.180) at <<MAC>> on igc0 permanent [ethernet]


Do you guys think the ARP thing sounds realistic and should be more investigated, or is this unlikely and a dead end?

Thanks for the information @mag, this was already very helpful, even though it didn't lead me to the solution yet.

I still don't have stable access to my services, however, I was able to narrow it down by some extent.

It seems like less of a HAproxy problem, but probably more like a general networking problem. New facts:
- when a server is not responding, OPNsense can also not ping it (DMZ rules wide open for testing purposes)
- as soon as the server pings OPNsense, the OPNsense pings to the server also start to work immediately, and a couple of moments later, HAproxy manages to redirect to this server.
- even with my DMZ rules in place, the ping from server to OPNsense is not successful but still kickstarts the connection from HAproxy to the server.

Googling this issue didn't really give me anything that seams reasonable in this case. Any smart people here that are not puzzled by this??

So, i made some tests. In the logs below, there are five connection attempts, one of them (radarr at 20:30) was successful and instantly showed me the login page.

The rest of them (trying sonarr) kind of just timed out. The "0_SNI_frontend SSL-backend/SSL_server" lines appear about 30 seconds after the connection attempt, so they basically mark the timeout of the session which is noticeable in the client browser.
And two minutes after the attempt, the GET request is logged. This is a very long time. What causes this delay? What do all the numbers mean in this detailed log?  e.g. 0/90064/-1/-1/120071 503 0


Firewall log:

Code Select Expand

WAN 2022-11-19T20:31:32 <<mobileIP>>:15612 <<homeIP>>:443 tcp allow HTTP/S traffic to HAproxy
WAN 2022-11-19T20:31:00 <<mobileIP>>:15611 <<homeIP>>:443 tcp allow HTTP/S traffic to HAproxy
WAN 2022-11-19T20:30:01 <<mobileIP>>:15610 <<homeIP>>:443 tcp allow HTTP/S traffic to HAproxy
WAN 2022-11-19T20:29:23 <<mobileIP>>:15609 <<homeIP>>:443 tcp allow HTTP/S traffic to HAproxy
WAN 2022-11-19T20:28:33 <<mobileIP>>:15608 <<homeIP>>:443 tcp allow HTTP/S traffic to HAproxy


HAproxy log:

Code Select Expand

2022-11-19T20:33:32 Informational haproxy <<mobileIP>>:15612 [19/Nov/2022:20:31:32.083] 1_HTTPS_frontend~ sonarr-backend/sonarr-real 0/90064/-1/-1/120071 503 0 - - sC-- 5/3/0/0/3 0/0 "GET https://sonarr.mydomain.dedyn.io/ HTTP/2.0"
2022-11-19T20:33:00 Informational haproxy <<mobileIP>>:15611 [19/Nov/2022:20:31:00.479] 1_HTTPS_frontend~ sonarr-backend/sonarr-real 0/90091/-1/-1/120133 503 0 - - sC-- 6/4/1/1/3 0/0 "GET https://sonarr.mydomain.dedyn.io/ HTTP/2.0"
2022-11-19T20:32:02 Informational haproxy <<mobileIP>>:15612 [19/Nov/2022:20:31:31.947] 0_SNI_frontend SSL-backend/SSL_server 1/0/30178 543 cD 7/3/2/2/0 0/0
2022-11-19T20:31:30 Informational haproxy <<mobileIP>>:15611 [19/Nov/2022:20:31:00.332] 0_SNI_frontend SSL-backend/SSL_server 1/0/30203 5164 cD 6/3/2/2/0 0/0
2022-11-19T20:31:23 Informational haproxy <<mobileIP>>:15609 [19/Nov/2022:20:29:23.273] 1_HTTPS_frontend~ sonarr-backend/sonarr-real 0/90109/-1/-1/120112 503 0 - - sC-- 7/4/1/1/3 0/0 "GET https://sonarr.mydomain.dedyn.io/ HTTP/2.0"
2022-11-19T20:30:34 Informational haproxy <<mobileIP>>:15610 [19/Nov/2022:20:30:01.462] 0_SNI_frontend SSL-backend/SSL_server 1/0/32996 1978141 CD 7/3/2/2/0 0/0
2022-11-19T20:30:34 Informational haproxy <<mobileIP>>:15608 [19/Nov/2022:20:28:34.048] 1_HTTPS_frontend~ sonarr-backend/sonarr-real 0/90132/-1/-1/120138 503 0 - - sC-- 8/5/1/1/3 0/0 "GET https://sonarr.mydomain.dedyn.io/ HTTP/2.0"
2022-11-19T20:30:02 Informational haproxy <<mobileIP>>:15610 [19/Nov/2022:20:30:02.446] 1_HTTPS_frontend~ radarr-backend/radarr-real 0/0/0/4/8 200 9781 - - ---- 8/5/0/0/0 0/0 "GET https://radarr.mydomain.dedyn.io/radarr/Content/Images/Icons/apple-touch-icon.png?h=QMW7w5FhldXR090sP7kDlQ HTTP/2.0"
2022-11-19T20:30:02 Informational haproxy <<mobileIP>>:15610 [19/Nov/2022:20:30:02.436] 1_HTTPS_frontend~ radarr-backend/radarr-real 0/0/0/7/7 200 2899 - - ---- 8/5/0/0/0 0/0 "GET https://radarr.mydomain.dedyn.io/radarr/Content/Images/Icons/favicon-32x32.png?h=bUxyGU7P9+rZb2XbRYUb6Q HTTP/2.0"
2022-11-19T20:30:02 Informational haproxy <<mobileIP>>:15610 [19/Nov/2022:20:30:02.233] 1_HTTPS_frontend~ radarr-backend/radarr-real 0/0/0/13/29 200 63671 - - ---- 8/5/0/0/0 0/0 "GET https://radarr.mydomain.dedyn.io/radarr/Content/Fonts/Roboto-Regular.woff2?v=1.3.0 HTTP/2.0"
2022-11-19T20:30:02 Informational haproxy <<mobileIP>>:15610 [19/Nov/2022:20:30:02.015] 1_HTTPS_frontend~ radarr-backend/radarr-real 0/0/7/8/15 200 582 - - ---- 8/5/0/0/0 0/0 "GET https://radarr.mydomain.dedyn.io/radarr/Content/Fonts/fonts.css?h=mBv1x1/yHnvphLclzeQIBA HTTP/2.0"
2022-11-19T20:30:02 Informational haproxy <<mobileIP>>:15610 [19/Nov/2022:20:30:02.015] 1_HTTPS_frontend~ radarr-backend/radarr-real 0/0/6/7/13 200 1574 - - ---- 8/5/1/1/0 0/0 "GET https://radarr.mydomain.dedyn.io/radarr/Content/Images/logo.svg?h=UK+1ymD8NGZtp+MiKdEQRg HTTP/2.0"
2022-11-19T20:30:02 Informational haproxy <<mobileIP>>:15610 [19/Nov/2022:20:30:02.006] 1_HTTPS_frontend~ radarr-backend/radarr-real 0/0/0/8/17 200 38162 - - ---- 8/5/2/2/0 0/0 "GET https://radarr.mydomain.dedyn.io/radarr/Content/styles.css?h=mZHtITCRtmV9tzv6R5l22A HTTP/2.0"
2022-11-19T20:30:01 Informational haproxy <<mobileIP>>:15610 [19/Nov/2022:20:30:01.785] 1_HTTPS_frontend~ radarr-backend/radarr-real 0/0/0/5/6 200 2880 - - ---- 8/5/0/0/0 0/0 "GET https://radarr.mydomain.dedyn.io/radarr/login?ReturnUrl=%2F HTTP/2.0"
2022-11-19T20:30:01 Informational haproxy <<mobileIP>>:15607 [19/Nov/2022:20:28:01.705] 1_HTTPS_frontend~ sonarr-backend/sonarr-real 0/90076/-1/-1/120079 503 0 - - sC-- 9/6/2/2/3 0/0 "GET https://sonarr.mydomain.dedyn.io/ HTTP/2.0"
2022-11-19T20:30:01 Informational haproxy <<mobileIP>>:15610 [19/Nov/2022:20:30:01.699] 1_HTTPS_frontend~ radarr-backend/radarr-real 0/0/0/5/5 302 164 - - ---- 9/6/0/0/0 0/0 "GET https://radarr.mydomain.dedyn.io/login?ReturnUrl=%2F HTTP/2.0"
2022-11-19T20:30:01 Informational haproxy <<mobileIP>>:15610 [19/Nov/2022:20:30:01.616] 1_HTTPS_frontend~ radarr-backend/radarr-real 0/0/7/9/16 302 157 - - ---- 9/6/0/0/0 0/0 "GET https://radarr.mydomain.dedyn.io/ HTTP/2.0"
2022-11-19T20:29:53 Informational haproxy <<mobileIP>>:15609 [19/Nov/2022:20:29:23.167] 0_SNI_frontend SSL-backend/SSL_server 1/0/30150 543 sD 8/3/2/2/0 0/0
2022-11-19T20:29:04 Informational haproxy <<mobileIP>>:15608 [19/Nov/2022:20:28:33.913] 0_SNI_frontend SSL-backend/SSL_server 1/0/30196 543 cD 7/3/2/2/0 0/0




I attached also a log when sonarr was working again (2hrs later).

Hey, thanks for the suggestion. I've activated it and will check the logs and give an update soon.

I stand corrected. It can also happen to Bitwarden. Only Plex and Transmission seem perfectly stable.

Hi all,

I am quite new to OPNsense, but Im confident I made the right choice, and it has everything (and more) I need. I run a Truenas server in my network, with several services that I (plus friends and family) need to access remotely.

I set up HAproxy according to this how-to: https://forum.opnsense.org/index.php?topic=23339.0 , which is absolutely awesome and almost hassle free :)

It works mostly flawlessly, but I have this weird effect, where some of my services are just not reachable. If I check in back later, they respond again.
- I see the pass action in the firewall, I see also that HAproxy did its thing in the logs
- It does not matter, if accessed from within the network, or from externally
- If I access the service directly by IP, it works without issues or delays.
- Its always Nextcloud, Sonarr or Radarr that don't respond (not always at the same time), but Plex, Transmission and Bitwarden always work.

I attached the redacted HAproxy conf and the HAproxy logs (a working and a failing one). I hope anyone can help me or point me towards the right direction, I really struggle on how to debug this.

Thanks in advance, cheers