OPNsense Forum

Archive => 16.7 Legacy Series => Topic started by: Julien on August 21, 2016, 12:29:34 am

Title: [SOLVED] Multi WAN
Post by: Julien on August 21, 2016, 12:29:34 am
Hi Guys,
we have a new project to configure 5 OPNsense for our customers using Multi WAN.
the second WAN is gonna be just fail over, if the first WAN is down the second WAN will jump in.
the WAN1 is already configured and everything is working fine.
i want to avoid any difficulities .
i've seen this doc on the site https://docs.opnsense.org/manual/how-tos/multiwan.html?highlight=Multi
If anyone has done this already, can please share your experience with me ?
thank you
Title: Re: Multi WAN
Post by: wurmloch on August 21, 2016, 02:13:27 am
Hi Guy,

I followed this how-to three times - without success. Therefore I am interested in comments and exchange of experience, too.

Did you spent some time in a test setup?

Regards
Uwe
Title: Re: Multi WAN
Post by: Julien on August 21, 2016, 02:17:21 pm
Hi Guy,

I followed this how-to three times - without success. Therefore I am interested in comments and exchange of experience, too.

Did you spent some time in a test setup?

Regards
Uwe
Good day,
what are you trying to establish using Multi WAN?
Title: Re: Multi WAN
Post by: wurmloch on August 21, 2016, 02:40:59 pm
Salut Julien,

here are some pics: https://forum.opnsense.org/index.php?topic=3537.0 (https://forum.opnsense.org/index.php?topic=3537.0). Maybe helpful even if in German.

My opnsense has 2x WAN (both static public addresses /24) connected to different German carriers and 1x LAN with some client-PCs.

What kind of setup has your test scenario?
Title: Re: Multi WAN
Post by: Julien on August 21, 2016, 10:58:21 pm
Salut Julien,

here are some pics: https://forum.opnsense.org/index.php?topic=3537.0 (https://forum.opnsense.org/index.php?topic=3537.0). Maybe helpful even if in German.

My opnsense has 2x WAN (both static public addresses /24) connected to different German carriers and 1x LAN with some client-PCs.

What kind of setup has your test scenario?
are you trying to configure a failover ?
my Multi WAN is gonna be failover.
so 1 WAN is up and second WAN is backup.
can't seem to find a good tutorial about this.
sorry my german is bad. :)
Title: Re: Multi WAN
Post by: wurmloch on August 22, 2016, 12:45:15 am
Hi Julien,

yes, I tried to configure a multi wan failover, following the link that you mentioned in your first post. But it didn't work at all. Unfortunately I cannot see what my mistake is.

In the German part of this forum I described what I did and what went wrong. Unfortunately nobody jumped in to give some advise. If I find some time I will repeat it on another opnsense to get some English screen shots of the flop.

Helpless regards
Uwe
Title: Re: Multi WAN
Post by: AdSchellevis on August 22, 2016, 02:21:58 pm
Hi Julien, Uwe,

I have used the documentation to setup both fail-over and load-balancing without any issues. One of the things that some people forget is to change the firewall rules to actually use the gateway group, maybe that's Uwe's issue too (I didn't see any firewall rules in his posting).

Multi wan relies on policy based routing, without a traffic selector the default gateway from the machine will be used (don't forget to set rules for the local traffic too, as in the example for DNS).

Best regards,

Ad
Title: Re: Multi WAN
Post by: Julien on August 22, 2016, 11:38:15 pm
Hi Julien, Uwe,

I have used the documentation to setup both fail-over and load-balancing without any issues. One of the things that some people forget is to change the firewall rules to actually use the gateway group, maybe that's Uwe's issue too (I didn't see any firewall rules in his posting).

Multi wan relies on policy based routing, without a traffic selector the default gateway from the machine will be used (don't forget to set rules for the local traffic too, as in the example for DNS).

Best regards,

Ad
Hi Ad,
When you said Firewall rules , do you mean the DNS firewall as explained on the document ?
I haven't done anything yet as I am trying to do my home work before connecting the second WAN
On the documents it said the destination is 192.168.1.1/32 if I am using a 24 bit LAN subnet have to use 192.168.1.1/24 , am I correct ?
Do I have t o apply the rule on the LAN or WAN side for DNS ? Its not clear to me if I read the document.


Thank you
Title: Re: Multi WAN
Post by: AdSchellevis on August 23, 2016, 07:37:59 am
Hi Julien,

In https://docs.opnsense.org/manual/how-tos/multiwan.html (https://docs.opnsense.org/manual/how-tos/multiwan.html) it's step 4 (Policy based routing).

Best regards,

Ad
Title: Re: Multi WAN
Post by: wurmloch on August 25, 2016, 11:01:54 pm
Hi,

Today I installed a new system from scratch at home with two WAN (DHCP from my two internet routers/lines) and what can I say, it works as expected :-)

Now I have two different multi WAN setups/machines (1@home and 1@work), one working, one not and I will find out where the difference is!

Uwe
Title: Re: Multi WAN
Post by: wurmloch on August 25, 2016, 11:16:43 pm
Julien,

Quote
On the documents it said the destination is 192.168.1.1/32 if I am using a 24 bit LAN subnet have to use 192.168.1.1/24 , am I correct ?

Not correct. "/24" means the (sub)net, i.e. 192.168.1.0/24 = subnet mask 255.255.255.0 ==>
192.168.1.0 = the subnet
192.168.1.1 to 192.168.1.254 = available IP addresses
192.168.1.255 = broadcast address

You have to address ONE machine = ipaddress/32 ==> 192.168.1.1/32

Have a look at http://www.subnet-calculator.com (http://www.subnet-calculator.com)
Title: Re: Multi WAN
Post by: wurmloch on August 25, 2016, 11:28:51 pm
Quote
Do I have t o apply the rule on the LAN or WAN side for DNS ? Its not clear to me if I read the document.

On the LAN tab of the firewall rules. It instructs the firewall to accept DNS requests from your LAN in any case. I would add a second rule regarding ICMP. With that you can ping to the firewall to see if the client (on the LAN) can reach it.

As far as I understood you'll have to write an explicit rule for each service, you'd like to access (on the firewall), e.g. proxy, dns ...
The predefined "anti lock-out rule" guarantees that you'll be able to access the firewall via https and ssh.
Title: Re: Multi WAN
Post by: Julien on August 28, 2016, 11:07:24 pm
i'll expect next week to install the second WAN,
its will be a challenge configuring it . like old time using pfsense, Multi WAN was easy as drinking water :)
i hope we will get a support here to have the second WAN configured.
Title: Re: Multi WAN
Post by: wurmloch on August 29, 2016, 12:44:44 am
Hi Julien,

Today I played a bit with my multi wan failover setup at home, nothing to worry about.

The only noticeable thing was that nearly each time when I pulled out the [WAN] cable (for a failover test) and I plugged it in after a minute or so, the interface didn't came up again properly.

Opnsense [WAN] --X--> internet router                   ==> problem, because NIC down and up again
Opnsense [WAN2] --> switch --X--> internet router ==> no problem because NIC never goes down (still connected to the switch and link stays up.

"--X--" means the cable i pulled :-)

Both of the wan interfaces are configured with dhcp, maybe that caused the problem. You can have a look at the screen shots in my corresponding German thread.

Uwe
Title: Re: Multi WAN
Post by: Julien on August 29, 2016, 11:59:36 am
Hi guys,
i have configured it as mentioned and also created the DNS rule before any to any rules on the LAN side.
i have test both Packet Loss and member down.
however when i remove the cable of WAN1 the connections goes down. The WAN 2 doesn't goes up.
Both WAN are ISP WAN using the same Gateway.
Could this be related to my issue ? Because both WAN are having the same Gateway from my ISP site ?
Which options to chose for Load balancing for 1 WAN up and 2 WAN is stand by.
I am using the
Code: [Select]
Packet Loss
Triggers when the packet loss to a gateway is higher then the defined threshold.

Thank you
Title: Re: Multi WAN
Post by: wurmloch on August 29, 2016, 07:42:46 pm
Hi Julien,

I'm not sure that I understood what you described. In the doc is written to monitor an external IP (e.g. the two google dns servers) to see when one of the wan goes down. You are monitoring your gateway? Do you really have two different internet connections?

If I remember well there was a thread concerning multiwan with the same gateway, maybe to search in the forum will help.

At least have a look at my english screen shots in my German Multiwan thread, both gateways are up (green).

Title: Re: Multi WAN
Post by: Julien on August 29, 2016, 09:45:15 pm
Hi Julien,

I'm not sure that I understood what you described. In the doc is written to monitor an external IP (e.g. the two google dns servers) to see when one of the wan goes down. You are monitoring your gateway? Do you really have two different internet connections?

If I remember well there was a thread concerning multiwan with the same gateway, maybe to search in the forum will help.

At least have a look at my english screen shots in my German Multiwan thread, both gateways are up (green).
Thank you for your answer.
i managed to get it configured and working fine.
the issue i had is i've created 3 LANS as group and i have created the DNS forward on the group interfaces however it didnt work.
had to created DNS and apply step 4 on the doc on each interface.
https://docs.opnsense.org/manual/how-tos/multiwan.html

thank you guys for the support
Title: Re: [SOLVED] Multi WAN
Post by: Julien on August 31, 2016, 08:00:20 am
hi guys i am back.
i have a issue, whenever i remove the WAN2, internet remain working which is fine.
when i remove the WAN1 the connection everything goes offlie.

please advise as i am stuck here !
Title: Re: [SOLVED] Multi WAN
Post by: wurmloch on August 31, 2016, 09:49:56 am
Julien,

_please_ give more info. Interfaces, addresses, gateways, firewall rules, failover config...

Usually I find answers in my chrystal ball but it's under repair.
Title: Re: [SOLVED] Multi WAN
Post by: Julien on August 31, 2016, 10:40:55 am
Julien,

_please_ give more info. Interfaces, addresses, gateways, firewall rules, failover config...

Usually I find answers in my chrystal ball but it's under repair.
I like your crystal ball :)
I've configured the Multi WAN exactly as mentioned on the document.
Firewalls rules are exactly
https://docs.opnsense.org/manual/how-tos/multiwan.html
See the below screenshots.
The issue now is when I connect the second WAN2 the connection goes down, I can ping 8.8.8.8 but I can't ping www.google.com
As I believe its DNS issue.
As showed on the screenshot DNS rules is created on the LAN.
I havne't mentioned I am using VLANS. Should I create the same Rules on each VLAN ?
I am stuck here guys thank you

(http://i64.tinypic.com/2e1tzsz.png)
(http://i68.tinypic.com/esivdh.png)
(http://i66.tinypic.com/2r2lmxd.png)
(http://i68.tinypic.com/mrfq4n.png)
(http://i66.tinypic.com/2zi1teh.png)
(http://i65.tinypic.com/21dke4i.png)
(http://i63.tinypic.com/20fst46.png)


Thank you
Title: Re: [SOLVED] Multi WAN
Post by: Julien on September 01, 2016, 06:25:24 pm
Guys ,
any suggestions please ? i can't connect the backup line.
Title: Re: [SOLVED] Multi WAN
Post by: PotatoCarl on September 02, 2016, 02:25:44 pm
Hi
did you try to remove the "default gateway" setting on all gateways? I have configured a gateway group and as long as I have one of the WANs as "default gateway" I see the same problem here. However, wenn no gateway is default, it works fine.

I also use in all firewall rules from the LAN as gateway specifically the gateway group. Then you can pull either cable and it works well.

Cheers
Title: Re: [SOLVED] Multi WAN
Post by: Julien on September 03, 2016, 09:41:30 pm
Hi
did you try to remove the "default gateway" setting on all gateways? I have configured a gateway group and as long as I have one of the WANs as "default gateway" I see the same problem here. However, wenn no gateway is default, it works fine.

I also use in all firewall rules from the LAN as gateway specifically the gateway group. Then you can pull either cable and it works well.

Cheers

thank you for your answer.
i am not sure i understand what you mean with remove the default gateway.
do you mean go to default gateway and unselect Default Gateway ? if i do so, how does the firewall knows which waWAN is the primairy ? i have WAN1 with 80MB and WAN2 with 40MB.
WAN2 is just backup and dont want the firewall to be using WAN2 as primamy.
thank you

can you please share a screenshot of your Gateway settings ?
and also of your LAN settings. do you mean on the lan rules to specify gateway as the group gateway and not the default ?
below see mine.
(http://i64.tinypic.com/or328n.png)
(http://i64.tinypic.com/oqc8k8.png)
Title: Re: [SOLVED] Multi WAN
Post by: Julien on September 07, 2016, 12:41:38 am
Guy's any suggestions please ?
Why it's so difficult on fixing the multi wan ?
Can someone please advise what I am doing wrong ?
Title: Re: [SOLVED] Multi WAN
Post by: wurmloch on September 07, 2016, 02:17:32 am
n/a
Title: Re: [SOLVED] Multi WAN
Post by: jschellevis on September 07, 2016, 10:53:02 am
@Julien

I have looked at your screenshots and I see you have disabled gateway monitoring in one of them, that means it will not monitor the gateway at all... it will always be shown as online. Its important to follow the documentation to the letter otherwise it won't work (https://docs.opnsense.org/manual/how-tos/multiwan.html)

I have created a test setup for you with 1x LAN (191.168.1.1/24) and 2x WAN (dhcp), this configuration is from scratch and exactly as described in the docs and verified to work with the latest release of OPNsense (16.7.3).

Before importing it you can change the em0,em1 and em2 to the correct interface names of hardware network devices if needed.

Hopefully this will help you resolve the issue as Multi-WAN is really easy to setup once you know what to look for.

Best regards,

Jos




Title: Re: [SOLVED] Multi WAN
Post by: jschellevis on September 07, 2016, 11:15:11 am
@wurmloch
It sounds to me you have issues that are not related to multi-wan alone.
Perhaps hardware or driver issues.

If the web interface is not accessible then I would advise to look at the console to see if there are any errors, as without more details it is not possible to determine the cause.

We do have commercial support to help you track down the issue: https://opnsense.org/support-overview/commercial-support/
Title: Re: [SOLVED] Multi WAN
Post by: wurmloch on September 07, 2016, 08:13:32 pm
@Jos,

No errors at the console. I just started a fresh install. We'll see if I can do better than the first time.

Wormhole
Title: Re: [SOLVED] Multi WAN
Post by: Julien on September 07, 2016, 10:54:30 pm
@Julien

I have looked at your screenshots and I see you have disabled gateway monitoring in one of them, that means it will not monitor the gateway at all... it will always be shown as online. Its important to follow the documentation to the letter otherwise it won't work (https://docs.opnsense.org/manual/how-tos/multiwan.html)

I have created a test setup for you with 1x LAN (191.168.1.1/24) and 2x WAN (dhcp), this configuration is from scratch and exactly as described in the docs and verified to work with the latest release of OPNsense (16.7.3).

Before importing it you can change the em0,em1 and em2 to the correct interface names of hardware network devices if needed.

Hopefully this will help you resolve the issue as Multi-WAN is really easy to setup once you know what to look for.

Best regards,

Jos
Hi Jos,
thank you for your answer.
the disable gateway monitor was selected when i took the screen by mistake.
its unselected right now and i can't get the Multi wan up and running.

i have a pfsense now at the customer running fine with Multi WAN without any issues.
we would like to migrate about 21 pfsense to opnsense but Multi wan is critical for us.
i hope you guys can help me get this fixed.
Title: Re: [SOLVED] Multi WAN
Post by: jschellevis on September 08, 2016, 10:38:09 am
Julien,

Did you try the configuration file I attached to my previous message?
As yesterday I retested the whole thing and it works like a charm.. with the default config I created for you it takes about 20 seconds for the WAN to switch, that can be optimized if you like.

If your ethernet ports are intel with the em driver, then you may have run into a FreeBSD bug.
This can be easily fixed by applying the intel em driver instead:

Code: [Select]
pkg install intel-em-kmod
and

then add the next line to /boot/loader.conf.local

Code: [Select]
if_em_updated_load="YES"
- Jos
 
Title: Re: [SOLVED] Multi WAN
Post by: Julien on September 09, 2016, 12:06:34 am
Hi Jos
THE configuration is exactly as the document
I think on the configuration you guys miss a step for the dns
When both WAN are connected I can't ping google.com but I can ping 8.8.&. Or any dns IP
My firewall is a rack one and bought it from Applianceshop in the NL
Are you sure the issue is a intel driver and not a configuration ?
Much appreciate it your continue support
Title: Re: [SOLVED] Multi WAN
Post by: jschellevis on September 09, 2016, 01:58:28 pm
Hi Julien,

I am sure that current em driver of FreeBSD 10.3 is broken and doesn't register a link down when removing a cable.
So you really need to update the driver first.

DNS is covered in the docs in Step 3 and 5.

I suggest

1) you do a factory default,
2) update if not yet on 16.7.3,
3) install the em driver
4) load the sample config I posted
5) change the WAN IP's to your actual situation
6) Apply and retest with this configuration

I have tested this configuration at least 10 times and as I said the sample config was made especially for this topic and verified to work. There isn't much else I can do for you.. just try my suggestions and there is a good change you will find what your where missing in the first place.

When testing, please note:
a) it takes about 20 second to switch
b) when ping-ing a site, stop and try again
c) when using a browser, try another one too as sometimes the change is not picked up correctly and your browser needs time to recover.

Good luck!
Title: Re: [SOLVED] Multi WAN
Post by: Julien on September 10, 2016, 10:42:19 pm
Thank you Jose for your answer.
we are using exactly this model
https://www.applianceshop.eu/security-appliances/19-rack-appliances/opnsense-based/opnsense-a10-quad-core-ssd-rack.html
are you saying this model is affected with the intel drivers issue ? according to the HD ID of the appliance the NICS are GbE [Intel® 82574L].
so updating the Intel Drivers of the NICS would make the Multi wan working ?
i am just double checking before starting changing stuff on the productions.

thank you for your continu support.
Title: Re: [SOLVED] Multi WAN
Post by: jschellevis on September 12, 2016, 10:16:09 am
Hi Julien,

FreeBSD 10.3 has issues with the em driver, I don't know the full list of Intel chipsets that run into issues, but the 82574L is certainly one of them. That is why we made a solution available.

We will provide an easy package to install Intel's original drivers too, expect in one of the next 2 updates.

For now you will have to do the manual install and /boot/loader.conf.local update.

- Jos
Title: Re: [SOLVED] Multi WAN
Post by: Julien on October 01, 2016, 05:14:18 pm
Hi Julien,

FreeBSD 10.3 has issues with the em driver, I don't know the full list of Intel chipsets that run into issues, but the 82574L is certainly one of them. That is why we made a solution available.

We will provide an easy package to install Intel's original drivers too, expect in one of the next 2 updates.

For now you will have to do the manual install and /boot/loader.conf.local update.

- Jos
Hi Jos,
sorry for the late reply i was sick.
monday i am going to test this and report back.
Title: Re: [SOLVED] Multi WAN
Post by: Julien on October 03, 2016, 03:59:33 pm
Hi Jos,
i have finally managed to get this configured.
maybe is the steps i did to get it working or the intel drivers are.
all this time i have rebooted the firewall with 1 WAN in.
what i did i installed the drivers of the intel and add the line to the boot file " thank you Jos" .
connected both WAN1 and WAN2 and rebooted the firewall , and voila everything start working.

if i remove the WAN1 now, it will take 20 seconded to switch back on the WAN2 ?
because i just did it and been waiting for 3 minutes and no up link is back !
any suggestions why ?

thank you
thank you
Title: Re: [SOLVED] Multi WAN
Post by: jschellevis on October 05, 2016, 12:07:54 pm
Hi Julien,

If it doesn't work, try to see what is happening:

On the firewall check if the gateway actually went offline.

From a PC on the LAN
1) Try to ping 8.8.8.8 or 8.8.4.4 (stop the ping and restart it as it will try the same route until restart)
2) See whether it will try to use the second gateway: traceroute 8.8.8.8 or traceroute 8.8.4.4 second line will show the ip of the gateway used.
3) If ping works then you probably have a DNS issue, most likely missing the firewall rule for that (port 53, gw *)
4) If ping does not work but the traceroute is going to the right (online) gateway then you either have a firewall rule blocking the traffic or the gateway doesn't work.
5) If the ping doesn;t work and the traceroute shows that it isn't switching then you either haven't setup the firewall rule for the gateway group correctly (the default allow rule on LAN) or the the primary gateway is still listed as online or both are offline. Also make sure you have the monitor ip of both set to something different so wan1 to 8.8.8.8 and wan2 to 8.8.4.4 for instance.

That is all I can think of as it works very well. Only thing that does currently not work is combining multi-wan with captive portal or the traffic shaper. Hopefully we get that resolved before 17.1

Cheers,

Jos

Title: Re: [SOLVED] Multi WAN
Post by: Julien on October 11, 2016, 11:17:21 pm
hi Jos,
thank you for your answer,
I've done this like 20 times today and it did not works.
whenever the WAN1 is disconnect the ping dow not go up, when I trace route the connection it goes to the WAN2 when WAN1 is down.
firewall rules are fine exactly as the documents, DNS rule is on the top of default Rules on LAN rules.
I think it something wrong with the code .
I am not trying to be rude hopefully it will be fixed in the next release,
I've installed pfsense and configured the multi WAN everything works out of the box no struggling.
we have a different customer which will have a second WAN in the next 4 weeks, I hope by than we can get this working.
thank you so much for your continue support   
Title: Re: [SOLVED] Multi WAN
Post by: jschellevis on October 12, 2016, 08:11:01 am
Hi Julien,

I am not sure what is going wrong with your setup, however I have done several installs in the past weeks for our customers and none of them have issues with the multi-wan failover.

Also my test setup (that I provided a config from a few posts back) works fine every time.
It can still be a combination of things, but I don't know.

Perhaps you should consider commercial support so we can put more time into your specific case and figure out why its not working as expected. See: https://opnsense.org/support-overview/commercial-support/

- Jos
Title: Re: [SOLVED] Multi WAN
Post by: Julien on October 12, 2016, 01:28:05 pm
Hi Julien,

I am not sure what is going wrong with your setup, however I have done several installs in the past weeks for our customers and none of them have issues with the multi-wan failover.

Also my test setup (that I provided a config from a few posts back) works fine every time.
It can still be a combination of things, but I don't know.

Perhaps you should consider commercial support so we can put more time into your specific case and figure out why its not working as expected. See: https://opnsense.org/support-overview/commercial-support/

- Jos
thank you for your answer Jos,

i beleive the issue is related to code or hardware .
i've took a confguration of the firewall before i switch back to Pfsense.
is it possible to email it to you , to check the configuration ?
is it also possible to have a teamviewer live session to check the configuration ? atleast to advise its not configuration issue, if it i am able to pay some hrs to get it fix.

thank you

Title: Re: [SOLVED] Multi WAN
Post by: jschellevis on October 12, 2016, 01:47:53 pm
Hi Julien,

Well if the same hardware is now working with pfsense, then its for sure not hardware and since I have confirm multi-wan to work fine with OPNsense it must be a combination of things or configuration issue.

If you want to verify, just download the config I have send you quite some post ago and try that one.

Alternatively you can buy support hours and I'll be happy to check your configuration and/or remotely support you.

-Jos
Title: Re: [SOLVED] Multi WAN
Post by: Julien on October 13, 2016, 07:53:24 pm
Hi Julien,

Well if the same hardware is now working with pfsense, then its for sure not hardware and since I have confirm multi-wan to work fine with OPNsense it must be a combination of things or configuration issue.

If you want to verify, just download the config I have send you quite some post ago and try that one.

Alternatively you can buy support hours and I'll be happy to check your configuration and/or remotely support you.

-Jos
Hi Jos,
uploading your configuration file to the firewall would erase the vlans and vpn settings ?
is the below the same issue i am having ? as Franco confirmed it a kernel issue ?
https://forum.opnsense.org/index.php?topic=3791.0

thank you