Messages

1

Virtual private networks / Re: Weird WG firewall/NAT problems, behavior differs from docs howto

« on: July 23, 2022, 11:13:01 pm »

No, unfortunately didn't find why this happens. Currently I don't have too much time to look at it, but I'll update if I figure it out.

2

Virtual private networks / Weird WG firewall/NAT problems, behavior differs from docs howto

« on: July 06, 2022, 11:32:43 am »

Hi all,

I have a WireGuard road warrior setup with two WG instances, wg0 and wg1. No interfaces are defined for WG because I don't want clients to connect to internet via the tunnel – so just aliases defined. I have set the usual pass rule on WAN for ports 51820-51821 where each of the wg instances listen on, and the further rules to allow the clients to LAN resources. In other words, the service has been configured according to WireGuard Road Warrior Setup (https://docs.opnsense.org/manual/how-tos/wireguard-client.html), but without creating WG interfaces or without outbound NAT rule.

The thing is that wg1 works as expected, but wg0 does not. When I try to connect to wg0, the logs tell me the connection has fallen through and was blocked by the "default deny / state violation" rule. Indeed, firewall logs show me a denied connection from the WG client's external address to the firewall's LAN address. This does not happen with wg1 and the same single WAN rule to allow incoming WG connections on port range 51820-51821 seems to work in wg1's case.

wg0 starts to work when I add a pass rule on WAN with the firewall's LAN address as destination. But this behavior baffles me and makes me concerned. Could anyone help to explain why this happens?

When I checked the wgX config files, they have identical structure with only minor differences. I can't come up with any meaningful difference between the two instances and why the other one works expectedly whereas the other does not.

Code: [Select]

# cat /usr/local/etc/wireguard/wg0.conf
[Interface]
PrivateKey = *
Address = 10.x.11.1/24
ListenPort = 51820

[Peer]
PublicKey = *

AllowedIPs = 10.x.11.2/32
PersistentKeepalive = 25

# cat /usr/local/etc/wireguard/wg1.conf
[Interface]
PrivateKey = *
Address = 10.x.12.1/24
ListenPort = 51821

[Peer]
PublicKey = *

AllowedIPs = 10.x.12.2/32


3

21.7 Legacy Series / Re: Multiwan load balancer has intermittent client connectivity problems

« on: August 18, 2021, 02:39:56 pm »

Hi Franco, thanks a lot for the insight! I looked at the said outbound rules (89 and 90 in the previous post) and it seems they were automatic floating rules generated the unchecked "Disable force gateway" option in Advanced settings under Firewall – I had missed that setting before.

The automatic rules vanished as they are supposed when I checked the option, but I still managed to get some timeouts. Anyway, that was a quick test and I'll need to investigate it again later when I get back to the test setup.

4

21.7 Legacy Series / Re: Multiwan load balancer has intermittent client connectivity problems

« on: August 17, 2021, 11:45:03 pm »

I looked further into this problem and actually got it reproduced on another site with entirely different setup – different OPNsense box, different LTE box, different carrier. The only thing that remains similar-ish with the two cases is that the "public IP" the interface gets from DHCP begins with 100.x. Interestingly, in this second case the address space the carrier uses for their CGNAT is 100.0.0.0/8, not 100.64/10 as the first carrier (and as RFC 6598 suggests).

So what I did to reproduce the problem was:
* add a new interface and set it to configure by DHCP,
* physically connect it to a separate LTE box set to bridge mode,
* set the LTE interface as Tier 1 route in a GW group with the default uplink, and
* created a firewall rule to route some traffic (port 80) via the GW group.

The outcome when testing with HTTP requests was exactly as described in the earlier post: the GW group -routed connections from LAN clients randomly working and failing.

But this time I got some more accurate logs. Edit: to add, the first box has the latest version of OPNsense running, this does not yet.

Code: [Select]

# opnsense-version
OPNsense 21.1.5 (amd64/OpenSSL)

 LAN (igb1)      -> v4: 10.x.x.x/22
 LTE (igb2)      -> v4/DHCP4: 100.81.238.96/8
 VLAN99 (igb1_vlan99) -> v4: 10.x.x.x/24
 WAN (igb0)      -> v4/DHCP4: [redacted public wan addr]/22

---

Info from GUI admin panel:


Firewall: Rules: LAN

pass in quick    IPv4 TCP    LAN net    *    *    80 (HTTP)    WANGRP_BAL    *    balancer port 80 (testing with one port)


System: Gateways: Group

WANGRP_BAL    Tier 1    WAN_DHCP, Online  LTE_DHCP, Online

---

### FAILED CONNECTION ATTEMPT ###

host1$ curl -vv --no-keepalive ipinfo.io/ip
*   Trying 34.117.59.81...
* TCP_NODELAY set
^C

pflog port 80:
 00:01:24.454627 rule 89/0(match): pass out on igb2: 100.81.238.96.32462 > 34.117.59.81.80: Flags [S], seq 3455205181, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1383378181 ecr 0,sackOK,eol], length 0
 00:00:00.000033 rule 12/0(match): block in on igb1: 100.81.238.96.32462 > 34.117.59.81.80: Flags [S], seq 3455205181, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1383378181 ecr 0,sackOK,eol], length 0
 00:00:01.004095 rule 12/0(match): block in on igb1: 100.81.238.96.32462 > 34.117.59.81.80: Flags [S], seq 3455205181, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1383379181 ecr 0,sackOK,eol], length 0
 00:00:01.001265 rule 12/0(match): block in on igb1: 100.81.238.96.32462 > 34.117.59.81.80: Flags [S], seq 3455205181, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1383380181 ecr 0,sackOK,eol], length 0
 00:00:01.006849 rule 12/0(match): block in on igb1: 100.81.238.96.32462 > 34.117.59.81.80: Flags [S], seq 3455205181, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1383381182 ecr 0,sackOK,eol], length 0
 00:00:01.009931 rule 12/0(match): block in on igb1: 100.81.238.96.32462 > 34.117.59.81.80: Flags [S], seq 3455205181, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1383382183 ecr 0,sackOK,eol], length 0

# pfctl -vvsr
[related rule excerpts shown]
@12 block drop in log inet all label "02f4bab031b57d1e30553ce08e0ec131"
@89 pass out log route-to (igb2 100.0.0.1) inet from (igb2:1) to ! (igb2:network:1) flags S/SA keep state allow-opts label "2353a7ed14c76ff61dfcda4957a92650"
@90 pass out log route-to (igb0 [redacted wan gw addr]) inet from (igb0:1) to ! (igb0:network:1) flags S/SA keep state allow-opts label "ef794793b2e3764b938bd04cba88e8a3"
@97 pass in quick on igb1 route-to { (igb0 [redacted wan gw addr]), (igb2 100.0.0.1) } round-robin sticky-address inet proto tcp from (igb1:network:1) to any port = http flags S/SA keep state label "d99770ba92021f7399ca68212d1a6837"

igb1 (LAN):
18:27:38.914248 IP host1.local.lan.58623 > 81.59.117.34.bc.googleusercontent.com.http: Flags [S], seq 3455205181, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1383378181 ecr 0,sackOK,eol], length 0
18:27:39.918344 IP host1.local.lan.58623 > 81.59.117.34.bc.googleusercontent.com.http: Flags [S], seq 3455205181, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1383379181 ecr 0,sackOK,eol], length 0
18:27:40.919651 IP host1.local.lan.58623 > 81.59.117.34.bc.googleusercontent.com.http: Flags [S], seq 3455205181, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1383380181 ecr 0,sackOK,eol], length 0
18:27:41.926466 IP host1.local.lan.58623 > 81.59.117.34.bc.googleusercontent.com.http: Flags [S], seq 3455205181, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1383381182 ecr 0,sackOK,eol], length 0
18:27:42.936431 IP host1.local.lan.58623 > 81.59.117.34.bc.googleusercontent.com.http: Flags [S], seq 3455205181, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1383382183 ecr 0,sackOK,eol], length 0
18:27:43.932328 IP host1.local.lan.58623 > 81.59.117.34.bc.googleusercontent.com.http: Flags [S], seq 3455205181, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1383383183 ecr 0,sackOK,eol], length 0


igb0/2:
(no activity)

The same behavior that was noted from the GUI live log before happened again, i.e. the initial outbound packet shows in pflog as allowed to pass out on WAN/LTE, and then further packets seem to get blocked on LAN interface.

But even though pflog shows allowing the packet to pass out on WAN/LTE it does not hit tcpdump on neither WAN or LTE interface. Does this mean packets after the first one get somehow internally routed from WAN/LTE to LAN and blocked before they actually enter the LAN interface? (because they do not show in WAN/LTE tcpdump but do show in pflog)

More reference logs:

Code: [Select]

other failed attempts from pflog -- similar as above: pass out igb0/2 (WAN/LTE), block on igb1 (LAN)

 00:00:00.028418 rule 90/0(match): pass out on igb0: [redacted public wan addr].53498 > 34.117.59.81.80: Flags [S], seq 3696100957, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1382040912 ecr 0,sackOK,eol], length 0
 00:00:00.000054 rule 12/0(match): block in on igb1: [redacted public wan addr].53498 > 34.117.59.81.80: Flags [S], seq 3696100957, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1382040912 ecr 0,sackOK,eol], length 0
 00:00:00.036057 rule 12/0(match): block in on igb1: [redacted public wan addr].53498 > 34.117.59.81.80: Flags [S], seq 3696100957, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1382041912 ecr 0,sackOK,eol], length 0


 00:00:04.591393 rule 90/0(match): pass out on igb0: [redacted public wan addr].34197 > 34.117.59.81.80: Flags [S], seq 1510025405, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1382022288 ecr 0,sackOK,eol], length 0
 00:00:00.000105 rule 12/0(match): block in on igb1: [redacted public wan addr].34197 > 34.117.59.81.80: Flags [S], seq 1510025405, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1382022288 ecr 0,sackOK,eol], length 0
 00:00:01.001680 rule 12/0(match): block in on igb1: [redacted public wan addr].34197 > 34.117.59.81.80: Flags [S], seq 1510025405, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1382023288 ecr 0,sackOK,eol], length 0
 00:00:01.003425 rule 12/0(match): block in on igb1: [redacted public wan addr].34197 > 34.117.59.81.80: Flags [S], seq 1510025405, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1382024288 ecr 0,sackOK,eol], length 0


 00:00:08.655713 rule 89/0(match): pass out on igb2: 100.81.238.96.61894 > 34.117.59.81.80: Flags [S], seq 2664141400, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1383290399 ecr 0,sackOK,eol], length 0
 00:00:00.000035 rule 12/0(match): block in on igb1: 100.81.238.96.61894 > 34.117.59.81.80: Flags [S], seq 2664141400, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1383290399 ecr 0,sackOK,eol], length 0
 00:00:01.245658 rule 12/0(match): block in on igb1: 100.81.238.96.61894 > 34.117.59.81.80: Flags [S], seq 2664141400, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1383291399 ecr 0,sackOK,eol], length 0
 00:00:01.236256 rule 12/0(match): block in on igb1: 100.81.238.96.61894 > 34.117.59.81.80: Flags [S], seq 2664141400, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1383292399 ecr 0,sackOK,eol], length 0


 00:00:02.496906 rule 89/0(match): pass out on igb2: 100.81.238.96.49976 > 34.117.59.81.80: Flags [S], seq 627667560, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1383280462 ecr 0,sackOK,eol], length 0
 00:00:00.000033 rule 12/0(match): block in on igb1: 100.81.238.96.49976 > 34.117.59.81.80: Flags [S], seq 627667560, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1383280462 ecr 0,sackOK,eol], length 0
 00:00:01.304970 rule 12/0(match): block in on igb1: 100.81.238.96.49976 > 34.117.59.81.80: Flags [S], seq 627667560, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1383281462 ecr 0,sackOK,eol], length 0
 00:00:01.096841 rule 12/0(match): block in on igb1: 100.81.238.96.49976 > 34.117.59.81.80: Flags [S], seq 627667560, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1383282462 ecr 0,sackOK,eol], length 0


### SUCCESSFUL CONNECTION (over igb2/LTE) ###

LAN$ curl -vv --no-keepalive ipinfo.io/ip
(returns normal output with the correct IP address)

pflog port 80:
 00:00:13.700721 rule 89/0(match): pass out on igb2: 100.81.238.96.40943 > 34.117.59.81.80: Flags [S], seq 481893042, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1382326151 ecr 0,sackOK,eol], length 0

tcpdump on igb2 (LTE):
18:09:56.012328 IP 100.81.238.96.40943 > 81.59.117.34.bc.googleusercontent.com.http: Flags [S], seq 481893042, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1382326151 ecr 0,sackOK,eol], length 0
18:09:56.166590 IP 81.59.117.34.bc.googleusercontent.com.http > 100.81.238.96.40943: Flags [S.], seq 3838886839, ack 481893043, win 65535, options [mss 1400,sackOK,TS val 3177229224 ecr 1382326151,nop,wscale 8], length 0
18:09:56.169993 IP 100.81.238.96.40943 > 81.59.117.34.bc.googleusercontent.com.http: Flags [.], ack 1, win 2060, options [nop,nop,TS val 1382326312 ecr 3177229224], length 0
18:09:56.170077 IP 100.81.238.96.40943 > 81.59.117.34.bc.googleusercontent.com.http: Flags [P.], seq 1:76, ack 1, win 2060, options [nop,nop,TS val 1382326312 ecr 3177229224], length 75: HTTP: GET /ip HTTP/1.1
18:09:56.209452 IP 81.59.117.34.bc.googleusercontent.com.http > 100.81.238.96.40943: Flags [.], ack 76, win 256, options [nop,nop,TS val 3177229311 ecr 1382326312], length 0
18:09:56.329450 IP 81.59.117.34.bc.googleusercontent.com.http > 100.81.238.96.40943: Flags [P.], seq 1:213, ack 76, win 256, options [nop,nop,TS val 3177229431 ecr 1382326312], length 212: HTTP: HTTP/1.1 200 OK
18:09:56.332411 IP 100.81.238.96.40943 > 81.59.117.34.bc.googleusercontent.com.http: Flags [.], ack 213, win 2057, options [nop,nop,TS val 1382326472 ecr 3177229431], length 0
18:09:56.332720 IP 100.81.238.96.40943 > 81.59.117.34.bc.googleusercontent.com.http: Flags [F.], seq 76, ack 213, win 2057, options [nop,nop,TS val 1382326472 ecr 3177229431], length 0
18:09:56.379562 IP 81.59.117.34.bc.googleusercontent.com.http > 100.81.238.96.40943: Flags [F.], seq 213, ack 77, win 256, options [nop,nop,TS val 3177229481 ecr 1382326472], length 0
18:09:56.382455 IP 100.81.238.96.40943 > 81.59.117.34.bc.googleusercontent.com.http: Flags [.], ack 214, win 2057, options [nop,nop,TS val 1382326521 ecr 3177229481], length 0

tcpdump on igb1 (LAN):
18:09:56.011995 IP host1.local.lan.58463 > 81.59.117.34.bc.googleusercontent.com.http: Flags [S], seq 481893042, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1382326151 ecr 0,sackOK,eol], length 0
18:09:56.166830 IP 81.59.117.34.bc.googleusercontent.com.http > host1.local.lan.58463: Flags [S.], seq 3838886839, ack 481893043, win 65535, options [mss 1400,sackOK,TS val 3177229224 ecr 1382326151,nop,wscale 8], length 0
18:09:56.169801 IP host1.local.lan.58463 > 81.59.117.34.bc.googleusercontent.com.http: Flags [.], ack 1, win 2060, options [nop,nop,TS val 1382326312 ecr 3177229224], length 0
18:09:56.170023 IP host1.local.lan.58463 > 81.59.117.34.bc.googleusercontent.com.http: Flags [P.], seq 1:76, ack 1, win 2060, options [nop,nop,TS val 1382326312 ecr 3177229224], length 75: HTTP: GET /ip HTTP/1.1
18:09:56.209628 IP 81.59.117.34.bc.googleusercontent.com.http > host1.local.lan.58463: Flags [.], ack 76, win 256, options [nop,nop,TS val 3177229311 ecr 1382326312], length 0
18:09:56.329601 IP 81.59.117.34.bc.googleusercontent.com.http > host1.local.lan.58463: Flags [P.], seq 1:213, ack 76, win 256, options [nop,nop,TS val 3177229431 ecr 1382326312], length 212: HTTP: HTTP/1.1 200 OK
18:09:56.332252 IP host1.local.lan.58463 > 81.59.117.34.bc.googleusercontent.com.http: Flags [.], ack 213, win 2057, options [nop,nop,TS val 1382326472 ecr 3177229431], length 0
18:09:56.332677 IP host1.local.lan.58463 > 81.59.117.34.bc.googleusercontent.com.http: Flags [F.], seq 76, ack 213, win 2057, options [nop,nop,TS val 1382326472 ecr 3177229431], length 0
18:09:56.379707 IP 81.59.117.34.bc.googleusercontent.com.http > host1.local.lan.58463: Flags [F.], seq 213, ack 77, win 256, options [nop,nop,TS val 3177229481 ecr 1382326472], length 0
18:09:56.382258 IP host1.local.lan.58463 > 81.59.117.34.bc.googleusercontent.com.http: Flags [.], ack 214, win 2057, options [nop,nop,TS val 1382326521 ecr 3177229481], length 0

### SUCCESSFUL CONNECTION (igb0/WAN) ###

curl similar as above

pflog port 80:
 00:00:10.231422 rule 90/0(match): pass out on igb0: [redacted public wan addr].45505 > 34.117.59.81.80: Flags [S], seq 4264699357, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1382488680 ecr 0,sackOK,eol], length 0

igb0 (WAN):
18:12:39.287238 IP [redacted resolved public wan addr].45505 > 81.59.117.34.bc.googleusercontent.com.http: Flags [S], seq 4264699357, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1382488680 ecr 0,sackOK,eol], length 0
18:12:39.297871 IP 81.59.117.34.bc.googleusercontent.com.http > [redacted resolved public wan addr].45505: Flags [S.], seq 318855234, ack 4264699358, win 65535, options [mss 1430,sackOK,TS val 3281088579 ecr 1382488680,nop,wscale 8], length 0
18:12:39.303371 IP [redacted resolved public wan addr].45505 > 81.59.117.34.bc.googleusercontent.com.http: Flags [.], ack 1, win 2060, options [nop,nop,TS val 1382488701 ecr 3281088579], length 0
18:12:39.304366 IP [redacted resolved public wan addr].45505 > 81.59.117.34.bc.googleusercontent.com.http: Flags [P.], seq 1:76, ack 1, win 2060, options [nop,nop,TS val 1382488701 ecr 3281088579], length 75: HTTP: GET /ip HTTP/1.1
18:12:39.323159 IP 81.59.117.34.bc.googleusercontent.com.http > [redacted resolved public wan addr].45505: Flags [.], ack 76, win 256, options [nop,nop,TS val 3281088596 ecr 1382488701], length 0
18:12:39.438547 IP 81.59.117.34.bc.googleusercontent.com.http > [redacted resolved public wan addr].45505: Flags [P.], seq 1:213, ack 76, win 256, options [nop,nop,TS val 3281088716 ecr 1382488701], length 212: HTTP: HTTP/1.1 200 OK
18:12:39.442593 IP [redacted resolved public wan addr].45505 > 81.59.117.34.bc.googleusercontent.com.http: Flags [.], ack 213, win 2057, options [nop,nop,TS val 1382488839 ecr 3281088716], length 0
18:12:39.442637 IP [redacted resolved public wan addr].45505 > 81.59.117.34.bc.googleusercontent.com.http: Flags [F.], seq 76, ack 213, win 2057, options [nop,nop,TS val 1382488839 ecr 3281088716], length 0
18:12:39.459904 IP 81.59.117.34.bc.googleusercontent.com.http > [redacted resolved public wan addr].45505: Flags [F.], seq 213, ack 77, win 256, options [nop,nop,TS val 3281088735 ecr 1382488839], length 0
18:12:39.463727 IP [redacted resolved public wan addr].45505 > 81.59.117.34.bc.googleusercontent.com.http: Flags [.], ack 214, win 2057, options [nop,nop,TS val 1382488860 ecr 3281088735], length 0

igb1 (LAN):
18:12:39.287028 IP host1.local.lan.58484 > 81.59.117.34.bc.googleusercontent.com.http: Flags [S], seq 4264699357, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1382488680 ecr 0,sackOK,eol], length 0
18:12:39.297921 IP 81.59.117.34.bc.googleusercontent.com.http > host1.local.lan.58484: Flags [S.], seq 318855234, ack 4264699358, win 65535, options [mss 1430,sackOK,TS val 3281088579 ecr 1382488680,nop,wscale 8], length 0
18:12:39.303273 IP host1.local.lan.58484 > 81.59.117.34.bc.googleusercontent.com.http: Flags [.], ack 1, win 2060, options [nop,nop,TS val 1382488701 ecr 3281088579], length 0
18:12:39.304333 IP host1.local.lan.58484 > 81.59.117.34.bc.googleusercontent.com.http: Flags [P.], seq 1:76, ack 1, win 2060, options [nop,nop,TS val 1382488701 ecr 3281088579], length 75: HTTP: GET /ip HTTP/1.1
18:12:39.323238 IP 81.59.117.34.bc.googleusercontent.com.http > host1.local.lan.58484: Flags [.], ack 76, win 256, options [nop,nop,TS val 3281088596 ecr 1382488701], length 0
18:12:39.438594 IP 81.59.117.34.bc.googleusercontent.com.http > host1.local.lan.58484: Flags [P.], seq 1:213, ack 76, win 256, options [nop,nop,TS val 3281088716 ecr 1382488701], length 212: HTTP: HTTP/1.1 200 OK
18:12:39.442547 IP host1.local.lan.58484 > 81.59.117.34.bc.googleusercontent.com.http: Flags [.], ack 213, win 2057, options [nop,nop,TS val 1382488839 ecr 3281088716], length 0
18:12:39.442603 IP host1.local.lan.58484 > 81.59.117.34.bc.googleusercontent.com.http: Flags [F.], seq 76, ack 213, win 2057, options [nop,nop,TS val 1382488839 ecr 3281088716], length 0
18:12:39.459943 IP 81.59.117.34.bc.googleusercontent.com.http > host1.local.lan.58484: Flags [F.], seq 213, ack 77, win 256, options [nop,nop,TS val 3281088735 ecr 1382488839], length 0
18:12:39.463683 IP host1.local.lan.58484 > 81.59.117.34.bc.googleusercontent.com.http: Flags [.], ack 214, win 2057, options [nop,nop,TS val 1382488860 ecr 3281088735], length 0
Any thoughts would still be warmly welcome.

5

21.7 Legacy Series / Multiwan load balancer has intermittent client connectivity problems

« on: August 16, 2021, 01:18:46 am »

Hi all,
I have a maddeningly elusive multiwan/routing/firewall problem.

I've set up a multiwan configuration according to the official OPNsense multiwan documentation, using a WAN group and policy based routing by firewall rules.

Both WAN interfaces are configured and confirmed to work independently, but by making the load balancing setup active by selecting both interfaces as Tier 1 in the WAN group, internet connectivity from the LAN clients seems to fail randomly. One of the interfaces (WAN2) gets a "public" IP address from the carrier NAT 100.64/10 block by DHCP, the other has a standard public static address setup.

The symptoms:

Connectivity may work seemingly perfectly fine for a couple of minutes after applying the configuration, but using only one of the interfaces. Traceroute seems normal and shows the WAN interface being used. Then, new connections just start getting timed out and traceroute gets only as far as the OPNsense box, which has led me to suspect something happens when the system tries to use the other WAN route.

Further suspect is that the firewall rules don't somehow match this traffic. While the connectivity problem is happening, I can see traffic getting blocked by the default firewall block rule. The attachments show firewall live log excerpts with traffic resulting from simple `curl ifconfig.me/ip` on a LAN workstation; one connection times out, the other works as supposed. Even though the excerpts show different interfaces working and failing, I think both interfaces have exhibited similar behavior.

Considering the 100.64/10 address block, I disabled the "Block private networks" configuration rule on WAN2 to see if it made any improvement, sadly not.

The gist of my firewall rules:

excerpts from floating auto-generated rules

   pass out IPv4+6 *   *   *   *   *   *   *   let out anything from firewall host itself
   pass out IPv4+6 *   (igb0)   *   *   *   WAN_GW   *   let out anything from firewall host itself (force gw)   
   pass out IPv4+6 *   (igb3)   *   *   *   WAN2_DHCP   *   let out anything from firewall host itself (force gw)   


excerpts from LAN ruleset

   pass in quick    IPv4 TCP/UDP   *   *   LAN address   53 (DNS)   *   *   
   pass in quick    IPv4 ICMP   *   *   LAN address   *   *   *   
   pass in quick    IPv4 *   LAN net   *   *   *   WANGRP_BAL   *   Load balancer allow LAN to any rule (cloned from default allow LAN rule)

Bogons and private addresses blocked in the statically configured interface, not blocked in the dynamically configured carrier NAT interface.

I have adjusted the WAN connections' parameters for now so that they are online and don't generate connectivity alerts (so they shouldn't affect any routing/load balancing decisions by the OS; the connections are somewhat low grade and could otherwise trigger an alert occasionally while still working).

The problem seems to be purely with routed traffic, I have never noticed the OPNsense box's own connections being affected. The configuration also works flawlessly as failover: if I put WAN2 as Tier 1, WAN as Tier 2 and then force-expire the WAN2 DHCP lease, Tier 2 takes over just as it should, without any bigger hiccups.

If anyone can come up with anything on this, be it further questions or suggestions, it would be greatly appreciated.

6

21.1 Legacy Series / Re: Running another instance of DNS resolver bound to a different interface?

« on: April 03, 2021, 07:09:13 pm »

Well, I thought I had read the manual, but to answer myself and if anyone ever stumbles here in the future: https://docs.opnsense.org/manual/unbound.html#advanced-configurations

I haven't really done it yet, though, but I guess if there is a way to configure this, it can be sorted out with the help of the manual.

So, sorry for the noise.

7

21.1 Legacy Series / Running another instance of DNS resolver bound to a different interface?

« on: April 03, 2021, 06:18:08 pm »

Hi all, hope this topic is on the correct forum.

I have two network segments, "clean" and "dirty". Currently I have a local DNS resolver (Unbound) running without any special filtration or configuration, but I would like to restrict DNS access from the dirty segment using overrides and blacklists (as well as prevent local DHCP issued hostnames from leaking from the clean segment) while leaving the clean segment unrestricted.

Based on that, I was wondering whether it was possible in OPNsense to run another resolver instance (from GUI or configured by hand) that could be configured separately. Or is this even the best way to approach this?

GUI configuration options don't seem to give the level of flexibility that would be needed in this, which led me to thinking that two separate instances with their own configurations could be the best way.

I'd say I'm quite familiar with Unix administration and could do this on a "vanilla" OS, but I'm a noob when it comes to OPNsense. So what I'm now mostly after is a way to achieve this without blowing up OPNsense's GUI or internals =) Any help is appreciated.

Edit: to add, the "dirty" segment is a separate segment in its own VLAN and the resolver would be bound to the VLAN interface, so it could be the default resolver for that segment.