Messages

Hi guys, thanks for taking the time to read through it (at least in part ;) ) and answering me.

The downloading config thing is indeed a good idea and could have probably also brought me closer to the solution.

With help of chatGPT I was zeroing in to NAT. And I found a one-to-one NAT entry, which I couldnt explain. Maybe creating a gateway and setting it on the WAN caused this? Maybe a remnant from before the network switch? No idea. This entry was NATing 192.168.1.1/24 to 192.168.5.1 and was the culprit for both of my problems (creating the weird 192.168.5.254 IP and screwing up my outbound traffic from ..5.0 subnet).

Anyways, thanks again.

Hi all,

Recently I switched to fiber and got a new router/modem from my ISP. That damn thing does not have a bridge mode and on top of that the subnet cannot be changed (192.168.1.0/24). Unfortunately, I have a contract period of two years, and it looks like I have to accept that.

Like many of those routers, it offers a DMZ feature, which as I understand is just port forwarding of everything to the DMZ host and placing the host in a /32 network for isolation (192.168.1.254/32 in my case). Generally, this worked, but I have some issues with that.

A bit of background info:
- I run HAproxy for different self-hosted services (originally based on the Hellsite tutorial: https://forum.opnsense.org/index.php?topic=23339.0)
- As per that tutorial, I created a virtual IP (IP Alias 127.4.4.3/32 on Loopback interface), which is set as a listen address on the public server in HAproxy. Alongside the SNI listening on 0.0.0.0:443
- the services are in different DMZ vlans
- i have an MGMT network for Ubiquity gear and some other admin webUIs. Before, this was 192.168.1.0/24, which i moved to 192.168.5.0/24 in order to prevent overlap with the ISP subnet.
- I have a WAN firewall rule, that allows IPv4 TCP 443/80 to "WAN address", which was allowing for remote access on HAproxy, when OPNsense was having a public IP at the WAN interface (remember, now it has 192.168.1.254).
- Gateway for 192.168.1.1 was created (Upstream and Far checked) and specified in the WAN interface.

Now the strange thing happens, when I am trying to connect to one of my services from remote:
I see the attempt being routed to 192.168.5.254 and therefore be default-denied. What is this IP? I have never typed it anywhere. Of course it could be a typo, but the only place, where I have typed 1.254 is the static IP at the WAN interface, and this is correct (otherwise the external connection attempt would not even be registered).

Since it is not "WAN address", it does not trigger the existing rule.
Trying to circumvent this, I added the 192.168.5.254 IP alongside "WAN Address", but HAproxy is not catching those requests and lists nothing in the log.


Another issue (maybe related, thats why I shortly mention it here) is the lack of internet access in the MGMT network, and that network alone. All other networks and VLANs under OPNsense have internet access.
ping -S 192.168.10.1 8.8.8.8  is successful
ping -S 192.168.5.1 8.8.8.8  fails.
Despite having a temporary allow all protocols from any to any on all ports, in/out as top rule in MGMT rules.


Can anybody make some sense of this? did I forget anything due to the fact that OPNsense has a private IP on the WAN?

Thankful for all help and pointers
Cheers,
Untoasted

Hope its ok to jump into this slightly older thread, but my issue is related to that one. I switched to fiber, and my crappy ISP only provides modems/router with no bridge mode, meaning I also need to rely on placing my OPNsense into this pseudo-DMZ.

If OPNsense has the DMZ-host (private) IP on its WAN interface, what does that mean in terms of fire-walling? Do I lose all information about originating IPs? Will all inbound connections to OPNsense just show the ISP router IP 192.168.1.1? I rely heavily on geo-blocking and additional blocklists, which I cannot afford to lose.

Sorry if this is a bad place to ask, I can start a new thread if this seems like a hijack of that thread.

Hi all,

After searching the forum, there is not a lot of using fail2ban with OPNsense. I wanted to create a centralized fail2ban server: meaning, if a person/IP is banned on one service, it should be banned for all of them and in the first line of defence, not only deep inside my network.

So I did some digging and found the following fail2ban action on Github:
https://github.com/fail2ban/fail2ban/pull/2761/files

Which lead me to something like this:


It works like this:

• HAproxy forwarding requests to the services in the DMZ
• all services log failed attempts to the log-dataset
• one centralized fail2ban instance (in the NAS-network, which is quite tightly locked-down) has access to all logs and takes action for wrong passwords
• if an IP is banned, it updates the alias on OPNsense via API correspondingly, instantly dropping further requests from this IP

It seems to have been working out quite nicely, but since I am rather new to OPNsense, some questions came up:

• Is it a good idea security wise, or does it rather weaken my network:
  
  
  • I am a bit anxious of using the API, and storing such powerful keys outside the firewall. Is there a way of storing them encrypted yet still usable in the f2b action? Can I limit the access of this key? It only needs to be able to update an alias, it would be nice if I could prohibit it from doing more.
  • Is the route e.g Nextcloud > Logs > f2b server > Firewall a possible attack vector? Assuming Nextcloud is highjacked, could some command injection in the logs screw with fail2ban and get access to its VM and therefore API key?
    
  • Is the curl command secure for transmitting the API-key? It calls the https adress of my firewall, so I would guess yes, but I am not sure if this is sufficient? Additional VPN tunnel, or is this an overkill?
    
• I like a lot the instantaneous action in case of wrong passwords attempts: if any service is hit with the maximum number of wrong passwords, the alias is instantly updated (in contrast to regularly retrieving a file), preventing the attacker from trying other services. BUT: it does not kill the current state. Assuming I am hacking my Nextcloud, as the state is established, even after f2b updated the alias, I can still keep trying passwords and even get access when finally entering the correct one. Is there a way of killing the state by API? Or some workaround?
• 
  The ssh dummy-server is not working yet. I would like to see anybody instantly blocked who tries to reach my 22 (nobody with good intentions does). I tried creating a dummy server with IPFW, logging all ssh attempts and feeding this log to f2b. But, seems like all firewall logging (apparently also pf and iptable) in iocage jails actually write to the host systems logs, which are too cluttered for f2b.
  It would work, if I actually run an accessible SSH server on my dummy, but I would rather like to stay invisible for any SSH attempts. Any alternatives of preferably clean logging of incoming connections on certain ports?
  
• 
  I listed 22 and 23 for the "honeypot". Any other ports, that are usually a sign of bad intentions? RDP port maybe?
  And this connection is not yet established, I thought a simple port-forward to the dummy should do, or is there a better way?
  

Thanks for reading, looking forward to hear any feedback about this concept.
Cheers

Update (for completeness sake):

In the meantime I got the hint to check the static arp's in the arp table. It didn't work according to the above approach with the rc.conf file (entries still expiring) but did work, when using the option "static arp" in the dhcp server settings in OPNsense GUI.

Next idea was using tcpdump to inspect the traffic, and tcpdump -e to check, if the correct mac adresses are used. As far as I could tell, OPNsense does resolve the MAC adresses correctly.

I was still unsure of which MAC adress to use in the arp table, since also behind the WiFi bridge it is a certain chain of MAC adresses that I could use for specifying the services adress:

Code Select Expand


                    | TrueNAS internally:
ManagedSwitch   ---  physical NIC   ---   Bridge   ---   VNET epair interface

I used the last one, since it is the only one linked to the IP adress.

Anyhow, inspired by Plex, which is broadcasting all the time and works for that reason, I found a workaround, that I can live with for the time being. All services drop a little "hello there" towards opnsense:

Code Select Expand

* * * * * ping -c 1 192.168.DMZ.1

Not elegant, but does the trick (not even a firewall rule needed for this, even a blocked ping makes OPNsense realize the presence of the VMs). Hopefully, later down the road I will be able to eliminate the WiFi bridge and maybe also this weird issue.

Cheers

I thought I had it.

I found this post on server fault, where someone describes exactly my issue and this was a suggestion:

Does sound like a arp problem, if its long spanned vlan over a vpn or similar, you may need to add static arp entries, with an ether address of the local switch.

https://serverfault.com/questions/521928/cannot-ping-b-from-a-until-b-pings-a


My setup looks as follows:

EVILINTERNET    ----   Modem   ----   OpnSense    ----    Wifi AP   <--wifibridge-->   Wifi AP   ----  Managed Switch (Netgear GS108Ev2)  ---- Server

I can imagine, that the wifi-bridge has a similar effect as the described long spanned vlan over vpn. So I tried the static ARP entries, as described in this post:
https://forum.opnsense.org/index.php?topic=7273.0

I created /etc/rc.conf on OPNsense with the IPs and MAC adresses of the web servers and then restarting the static arp service. But no change, it still happens, that I cannot reach or ping some servers.

Then I saw, that the serverfault post stated, that the switch MAC adress should be entered. But still nothing.

But no matter what, when displaying arp -a, I see all these expiration times on my web servers, whereas only IPs from OPNsense itself seem to be permanent. What should be expected when setting static ARP entries? I never used this before, as anyone can guess, I am not a professional.

These are the two attempts I made in the rc.conf (above individual MAC adresses of all servers, below the managed switch):

Code Select Expand


# setting MAC of each web server
#static_arp_pairs="nc vw son rad trans plex"
#static_arp_nc="192.168.DMZ.13 MAC:df"
[...]
#static_arp_plex="192.168.DMZ.14 MAC:ab"


# setting MAC of managed switch in front of web servers
static_arp_pairs="nc vw son rad trans plex"
static_arp_nc="192.168.DMZ.13 MAC:9c"
[...]
static_arp_plex="192.168.DMZ.14 MAC:9c"



Shortened output of arp -a:

Code Select Expand


root@opnsense:~ % arp -a
? (192.168.DMZ.13) at <<MAC>>:df on vlan0.DMZ expires in 898 seconds [vlan]
? (192.168.DMZ.14) at <<MAC>>:ab on vlan0.DMZ expires in 847 seconds [vlan]
[...]
? (192.168.DMZ.1)  at <<MAC>>:e8 on vlan0.DMZ permanent [vlan]
[...]
<<ISP>> (<<GLOBALIP>>.1) at <<MAC>> on igc0 expires in 1200 seconds [ethernet]
<<IP@ISP>> (<<GLOBALIP>>.180) at <<MAC>> on igc0 permanent [ethernet]


Do you guys think the ARP thing sounds realistic and should be more investigated, or is this unlikely and a dead end?

Thanks for the information @mag, this was already very helpful, even though it didn't lead me to the solution yet.

I still don't have stable access to my services, however, I was able to narrow it down by some extent.

It seems like less of a HAproxy problem, but probably more like a general networking problem. New facts:
- when a server is not responding, OPNsense can also not ping it (DMZ rules wide open for testing purposes)
- as soon as the server pings OPNsense, the OPNsense pings to the server also start to work immediately, and a couple of moments later, HAproxy manages to redirect to this server.
- even with my DMZ rules in place, the ping from server to OPNsense is not successful but still kickstarts the connection from HAproxy to the server.

Googling this issue didn't really give me anything that seams reasonable in this case. Any smart people here that are not puzzled by this??

So, i made some tests. In the logs below, there are five connection attempts, one of them (radarr at 20:30) was successful and instantly showed me the login page.

The rest of them (trying sonarr) kind of just timed out. The "0_SNI_frontend SSL-backend/SSL_server" lines appear about 30 seconds after the connection attempt, so they basically mark the timeout of the session which is noticeable in the client browser.
And two minutes after the attempt, the GET request is logged. This is a very long time. What causes this delay? What do all the numbers mean in this detailed log?  e.g. 0/90064/-1/-1/120071 503 0


Firewall log:

Code Select Expand

WAN 2022-11-19T20:31:32 <<mobileIP>>:15612 <<homeIP>>:443 tcp allow HTTP/S traffic to HAproxy
WAN 2022-11-19T20:31:00 <<mobileIP>>:15611 <<homeIP>>:443 tcp allow HTTP/S traffic to HAproxy
WAN 2022-11-19T20:30:01 <<mobileIP>>:15610 <<homeIP>>:443 tcp allow HTTP/S traffic to HAproxy
WAN 2022-11-19T20:29:23 <<mobileIP>>:15609 <<homeIP>>:443 tcp allow HTTP/S traffic to HAproxy
WAN 2022-11-19T20:28:33 <<mobileIP>>:15608 <<homeIP>>:443 tcp allow HTTP/S traffic to HAproxy


HAproxy log:

Code Select Expand

2022-11-19T20:33:32 Informational haproxy <<mobileIP>>:15612 [19/Nov/2022:20:31:32.083] 1_HTTPS_frontend~ sonarr-backend/sonarr-real 0/90064/-1/-1/120071 503 0 - - sC-- 5/3/0/0/3 0/0 "GET https://sonarr.mydomain.dedyn.io/ HTTP/2.0"
2022-11-19T20:33:00 Informational haproxy <<mobileIP>>:15611 [19/Nov/2022:20:31:00.479] 1_HTTPS_frontend~ sonarr-backend/sonarr-real 0/90091/-1/-1/120133 503 0 - - sC-- 6/4/1/1/3 0/0 "GET https://sonarr.mydomain.dedyn.io/ HTTP/2.0"
2022-11-19T20:32:02 Informational haproxy <<mobileIP>>:15612 [19/Nov/2022:20:31:31.947] 0_SNI_frontend SSL-backend/SSL_server 1/0/30178 543 cD 7/3/2/2/0 0/0
2022-11-19T20:31:30 Informational haproxy <<mobileIP>>:15611 [19/Nov/2022:20:31:00.332] 0_SNI_frontend SSL-backend/SSL_server 1/0/30203 5164 cD 6/3/2/2/0 0/0
2022-11-19T20:31:23 Informational haproxy <<mobileIP>>:15609 [19/Nov/2022:20:29:23.273] 1_HTTPS_frontend~ sonarr-backend/sonarr-real 0/90109/-1/-1/120112 503 0 - - sC-- 7/4/1/1/3 0/0 "GET https://sonarr.mydomain.dedyn.io/ HTTP/2.0"
2022-11-19T20:30:34 Informational haproxy <<mobileIP>>:15610 [19/Nov/2022:20:30:01.462] 0_SNI_frontend SSL-backend/SSL_server 1/0/32996 1978141 CD 7/3/2/2/0 0/0
2022-11-19T20:30:34 Informational haproxy <<mobileIP>>:15608 [19/Nov/2022:20:28:34.048] 1_HTTPS_frontend~ sonarr-backend/sonarr-real 0/90132/-1/-1/120138 503 0 - - sC-- 8/5/1/1/3 0/0 "GET https://sonarr.mydomain.dedyn.io/ HTTP/2.0"
2022-11-19T20:30:02 Informational haproxy <<mobileIP>>:15610 [19/Nov/2022:20:30:02.446] 1_HTTPS_frontend~ radarr-backend/radarr-real 0/0/0/4/8 200 9781 - - ---- 8/5/0/0/0 0/0 "GET https://radarr.mydomain.dedyn.io/radarr/Content/Images/Icons/apple-touch-icon.png?h=QMW7w5FhldXR090sP7kDlQ HTTP/2.0"
2022-11-19T20:30:02 Informational haproxy <<mobileIP>>:15610 [19/Nov/2022:20:30:02.436] 1_HTTPS_frontend~ radarr-backend/radarr-real 0/0/0/7/7 200 2899 - - ---- 8/5/0/0/0 0/0 "GET https://radarr.mydomain.dedyn.io/radarr/Content/Images/Icons/favicon-32x32.png?h=bUxyGU7P9+rZb2XbRYUb6Q HTTP/2.0"
2022-11-19T20:30:02 Informational haproxy <<mobileIP>>:15610 [19/Nov/2022:20:30:02.233] 1_HTTPS_frontend~ radarr-backend/radarr-real 0/0/0/13/29 200 63671 - - ---- 8/5/0/0/0 0/0 "GET https://radarr.mydomain.dedyn.io/radarr/Content/Fonts/Roboto-Regular.woff2?v=1.3.0 HTTP/2.0"
2022-11-19T20:30:02 Informational haproxy <<mobileIP>>:15610 [19/Nov/2022:20:30:02.015] 1_HTTPS_frontend~ radarr-backend/radarr-real 0/0/7/8/15 200 582 - - ---- 8/5/0/0/0 0/0 "GET https://radarr.mydomain.dedyn.io/radarr/Content/Fonts/fonts.css?h=mBv1x1/yHnvphLclzeQIBA HTTP/2.0"
2022-11-19T20:30:02 Informational haproxy <<mobileIP>>:15610 [19/Nov/2022:20:30:02.015] 1_HTTPS_frontend~ radarr-backend/radarr-real 0/0/6/7/13 200 1574 - - ---- 8/5/1/1/0 0/0 "GET https://radarr.mydomain.dedyn.io/radarr/Content/Images/logo.svg?h=UK+1ymD8NGZtp+MiKdEQRg HTTP/2.0"
2022-11-19T20:30:02 Informational haproxy <<mobileIP>>:15610 [19/Nov/2022:20:30:02.006] 1_HTTPS_frontend~ radarr-backend/radarr-real 0/0/0/8/17 200 38162 - - ---- 8/5/2/2/0 0/0 "GET https://radarr.mydomain.dedyn.io/radarr/Content/styles.css?h=mZHtITCRtmV9tzv6R5l22A HTTP/2.0"
2022-11-19T20:30:01 Informational haproxy <<mobileIP>>:15610 [19/Nov/2022:20:30:01.785] 1_HTTPS_frontend~ radarr-backend/radarr-real 0/0/0/5/6 200 2880 - - ---- 8/5/0/0/0 0/0 "GET https://radarr.mydomain.dedyn.io/radarr/login?ReturnUrl=%2F HTTP/2.0"
2022-11-19T20:30:01 Informational haproxy <<mobileIP>>:15607 [19/Nov/2022:20:28:01.705] 1_HTTPS_frontend~ sonarr-backend/sonarr-real 0/90076/-1/-1/120079 503 0 - - sC-- 9/6/2/2/3 0/0 "GET https://sonarr.mydomain.dedyn.io/ HTTP/2.0"
2022-11-19T20:30:01 Informational haproxy <<mobileIP>>:15610 [19/Nov/2022:20:30:01.699] 1_HTTPS_frontend~ radarr-backend/radarr-real 0/0/0/5/5 302 164 - - ---- 9/6/0/0/0 0/0 "GET https://radarr.mydomain.dedyn.io/login?ReturnUrl=%2F HTTP/2.0"
2022-11-19T20:30:01 Informational haproxy <<mobileIP>>:15610 [19/Nov/2022:20:30:01.616] 1_HTTPS_frontend~ radarr-backend/radarr-real 0/0/7/9/16 302 157 - - ---- 9/6/0/0/0 0/0 "GET https://radarr.mydomain.dedyn.io/ HTTP/2.0"
2022-11-19T20:29:53 Informational haproxy <<mobileIP>>:15609 [19/Nov/2022:20:29:23.167] 0_SNI_frontend SSL-backend/SSL_server 1/0/30150 543 sD 8/3/2/2/0 0/0
2022-11-19T20:29:04 Informational haproxy <<mobileIP>>:15608 [19/Nov/2022:20:28:33.913] 0_SNI_frontend SSL-backend/SSL_server 1/0/30196 543 cD 7/3/2/2/0 0/0




I attached also a log when sonarr was working again (2hrs later).

Hey, thanks for the suggestion. I've activated it and will check the logs and give an update soon.

I stand corrected. It can also happen to Bitwarden. Only Plex and Transmission seem perfectly stable.

Hi all,

I am quite new to OPNsense, but Im confident I made the right choice, and it has everything (and more) I need. I run a Truenas server in my network, with several services that I (plus friends and family) need to access remotely.

I set up HAproxy according to this how-to: https://forum.opnsense.org/index.php?topic=23339.0 , which is absolutely awesome and almost hassle free :)

It works mostly flawlessly, but I have this weird effect, where some of my services are just not reachable. If I check in back later, they respond again.
- I see the pass action in the firewall, I see also that HAproxy did its thing in the logs
- It does not matter, if accessed from within the network, or from externally
- If I access the service directly by IP, it works without issues or delays.
- Its always Nextcloud, Sonarr or Radarr that don't respond (not always at the same time), but Plex, Transmission and Bitwarden always work.

I attached the redacted HAproxy conf and the HAproxy logs (a working and a failing one). I hope anyone can help me or point me towards the right direction, I really struggle on how to debug this.

Thanks in advance, cheers