Multi WAN + Captive Portal not working

[srijan]

Guys,

I need help in getting Muti-WAN to work with Captive Portal.

I have two gateway, one PPPoE and another Static. Both are working properly. I have configured two Gateway groups, GWGroup1 with WAN1(PPPoE - Tier1) + WAN2(Static - Tier2) and GWGroup2 as WAN1(PPPoE - Tier2) + WAN2(Static - Tier1). This is done because I want to pass LAN traffic using GWGroup1 and LAN 2 traffic via GWGroup2. So, I will at all times have a failover group for both my LAN's. Monitoring IP's have been set and I have followed the link https://docs.opnsense.org/manual/how-tos/multiwan.html to the hilt.

Have configured Firewall Rules to pass DNS traffic using default GW and also traffic to LAN Address on ports 8000-10000 via default GW (though I wasn't sure I needed this). My default LAN to any rule has the gateway set to GWGroup1. Please refer to the screenshot attached.

In this scenario, the Captive does not appear. As soon as I set the gateway on the Default LAN rule to Default GW, everything starts working properly and I get the Captive Portal.

The PF rules are all proper:
1. With Policy Base Routing (using GWGroup1) in the LAN rule.
pass in quick on em0 route-to (pppoe0 X.X.X.X) inet from 172.16.1.0/24 to any flags S/SA keep state label "USER_RULE: Default allow LAN to any rule"
2. With Default GW in the LAN rule.
pass in quick on em0 inet from 172.16.1.0/24 to any flags S/SA keep state label "USER_RULE: Default allow LAN to any rule"

What I see is that as soon as I change the LAN rule to Default GW, I can see traffic hitting port 8999 in the loopback address.
18:07:01.460540 IP localhost.15623 > localhost.8999(SYN)
18:07:01.460648 IP localhost.8999 > localhost.15623(SYN+ACK)
18:07:01.460769 IP localhost.15623 > localhost.8999(ACK)
After which, another connection is initiated from the machine's IP to the LAN IP of the firewall on port 8000. And I get the Captive Portal login page.

When I change the LAN rule to use GWGroup1, I do not see any traffic on the loopback interface for port 8999 nor any traffic on the LAN IP on port 8000.

As soon as I use multi-WAN, Captive Portal Fails. I was of the opinion that the architecture of Opnsense is such that the 'ipfw' always comes before 'pf'. In that case, in both the scenarios, 'ipfw' should always pass the un-authenticated traffic to 127.0.0.1, 8000. But, in real scenario, as soon as I use multi-WAN Captive portal does not appear.

[PotatoCarl]

Hi
I have a similar if not same problem.

I used this
https://docs.opnsense.org/manual/how-tos/guestnet.html

To setup a guest network. This works as it seems (I use WIFI APs in the guest network).
As soon as I activate the Captive Portal, the network connection fails "Network not reachable".
I believe there is some kind of rule missing or such.

We also use a multi-wan (PPPOe+Static).

I added access rights for the DNS Forwarder to the Guest network, although this is not intended, as the Guestnetwork should not have ANY access to the internal network and should use the Google DNS 8.8.8.8 and 8.8.4.4 (configured in the DHCP for the Guest network + additionally Firewall rules to allow access).

In addition, I do not see the interface that balances the external gateways (failover/load balancing) but can only select standard or one of the external interfaces -however, traffic should be distributed over those two external lines.

Any idea anyone?

Cheers

T

[srijan]

To see the Gateway Failover Group, you need to create one, under System --> Gateways --> All. Then you will be able to see the Gateway Failover/Load Balancing group for Policy Based Routing. The DNS Rule should be above the Default LAN rule and that too the gateway should be set to 'Default' i.e. without Failover/Load Balancing group.

After which you need to change the Default LAN rule to pass traffic via the gateway Failover/Load Balancing group.

This will work and Captive Portal will appear if the Default LAN rule is set to have the gateway as 'Default'.

As soon as you apply Failover/Load Balancing as the gateway on the Default LAN rule, it might not work as I am facing a similar issue. Here, the 'ipfw' rule which caters to Captive Portal does not come into picture and all traffic is passed by the 'pf' rule set.

Can someone point to a way to get 'ipfw' to see this traffic in both the scenarios. 'ipfw' applies when Default LAN rule's gateway is set to 'Default'.

[PotatoCarl]

Hi

thank you for the input. The Balancing Group was a bit hidden, but I could see it now (it is in use for other rules in the LAN area).

The rules are correctly placed and without Captive Portal it works well - either on the Google DNS or on the internal. HOWEVER, as soon as I activate the Captive Portal, it is not possible to access anything anymore.

I have used "default" route for all rules up to know (exactly as in the HowTo), but still, as soon as I turn on the captive portal, the network connection is lost in the Guest network. When I turn it off, it works immediately as expected. Even with the now installed correct balanced Gateway.

Is it possible that I have to install also a rule in the LAN-Netzwork back to the guest for the captive portal? It is not described in the Howto and I guessed it is not necessary.

Cheers

[srijan]

I am sure you might have done the following, however please re-check:

1. Captive Portal has the interface set to your GuestNetwork.
2. DNS Forwarder Enabled.
3. DNS Rule from GuestNetwork to GuestNetwork IP Address.
4. Rule to access Captive Portal on port 8000-10000 from GuestNetwork to Guest Address.
5. Rule to allow traffic from GuestNetwork to 'any'.

When you do not set the Gateway Group, this will allow you Load Balance the traffic out. When you set the Gateway Group, it will set let you out to the internet, but you might not see the Captive Portal.

[PotatoCarl]

Yes, I did all this. I double checked. I changed the Firewall rule (5) to the gateway load balance and to standard. Reproducibly, whenever I activate the Captive Portal, I cannot access the network anymore.

As soon as I turn it off, the Guest network works as expected.

[srijan]

Can you access the Captive Portal on http://GuestNetwork Address:8000 or on https (if you have enabled SSL Certificate), when you set the gateway to 'default' in the Firewall Rule?

Also the GuestNetwork has DHCP enabled and the machines in the GuestNetwork has their DNS set to the GuestNetwork Address.

If possible, please share a screenshot of the Firewall Rules for the GuestNetwork interface.

[srijan]

Guys,

For me the packet capture goes like this:

1. When Default LAN Rule has gateway set to 'default'.

tcpdump on LAN, with port 80 or port 8000

15:41:45.247575 IP 172.16.1.10.51497 > X.X.X.X.80 (SYN to website)
15:41:45.247741 IP X.X.X.X.80 > 172.16.1.10.51497 (SYN+ACK from website)
15:41:45.247933 IP 172.16.1.10.51497 > X.X.X.X.80 (ACK from my machine)

A second connection is on port 8000 for Captive Portal
15:41:45.252389 IP 172.16.1.10.51498 > 172.16.1.1.8000 (SYN)
15:41:45.252524 IP 172.16.1.1.8000 > 172.16.1.10.51498 (SYN+ACK)
15:41:45.252700 IP 172.16.1.10.51498 > 172.16.1.1.8000 (ACK)

tcpdump on the loopback has the following:

15:41:45.356778 IP 127.0.0.1.18902 > 127.0.0.1.8999 (SYN)
15:41:45.356885 IP 127.0.0.1.8999 > 127.0.0.1.18902 (SYN+ACK)
15:41:45.356964 IP 127.0.0.1.18902 > 127.0.0.1.8999 (ACK)

2. As soon as I change the gateway on the Default Rule to point to Gateway Group, my machine makes a direct connection with the website without giving the Captive Portal.

15:46:07.667431 IP 172.16.1.10.51542 > X.X.X.X.80 (SYN to website)
15:46:07.923599 IP X.X.X.X.80 > 172.16.1.10.51542 (SYN+ACK from website)
15:46:07.923911 IP 172.16.1.10.51542 > X.X.X.X.80 (ACK)

My question is when there is an 'ipfw' rule that states that all connections on the LAN port destined for port 80 be redirected to 127.0.0.1:8000, why does this not come into picture when I do policy based routing with Gateway Group.

add 5000 fwd 127.0.0.1,8000 tcp from any to any dst-port 80 in via em0

Can someone help me do this, please?

[PotatoCarl]

Hi All

I have found the solution or at last a solution that works for me now:

Do *not* select any gateway as default gateway.

As simple as it is, after this it works immediately. My configuration is:

- Multi-WAN (2 WANs)
- Gateway group as load balancing/failover
- Remove all "default gateway" settings at the gateways
- Activate captive portal

Then the instructions linked above work well without any problems.

It seems as if a default gateway is selected with a multi-wan configuration, the captive portal is not adressed. There are some similar problems with the load balancing not working either, this is the reason why I came to try this setting in the first place (in fact, the load does not get distributed).

Hope that helps others, too.

Cheers

[srijan]

Good that it worked for you. It still does not work for me.

As Ad Schellevis rightly said this is a FreeBSD bug. I am quoting what he said in Github.

"@srijannandi just tried it on my end and can confirm it doesn't work. Looks like a kernel issue. Found this online from some time ago https://lists.freebsd.org/pipermail/freebsd-pf/2010-July/005739.html , not sure if its related, there doesn't seem to be a quick fix."

FreeBSD needs to get this resolved at their end.

[srijan]

Guys,

I see Pfsense has some sysctl options to specify the order.

sysctl net.inet.ip.pfil

net.inet.ip.pfil.inbound=pf, ipfw*
net.inet.ip.pfil.outbound=pf, ipfw*

Can we implement this and hopefully this resolves the issue for everyone?

[franco]

No, we enforce the correct order too.

[srijan]

Hello Franco,

I agree it must be enforced. But the as soon as we use route-to in PF it breaks the pfil ordering. Somehow, ipfw is not coming into picture.

[franco]

Hi Srijan,

From what you've described and we've discussed internally, it seems like a real issue, albeit route-to not doing anything wrong per se. It probably shortcuts to packet transmission.

It's been like this for ages and despite being a bug still the status quo in FreeBSD. That means we're having a use case that nobody else has.

Unfortunately, this requires reading code, preparing and testing kernel patches and deploying the changes in the release. This is not going to be fixed short-term, but Ad said he will look into this when he is back at the office next week.


Cheers,
Franco

[franco]

https://github.com/opnsense/src/blob/master/sys/netpfil/pf/pf.c#L5434

I think that's the issue as it goes straight to the NIC.

This doesn't seem to be the case for IPv6. Could anybody confirm or deny this?

EDIT: Nope, code for IPv6 is also jammed elsewhere in the kernel. Goes straight to the NIC.