Messages

I guess the answer is to see if any processes are hogging the resources.

Use something like top and/or pftop and enable/disable IPS to see ???

OOohhh nasty gotcha.

If I set up a IPSEC connection and specify WAN3 the system still creates a route via WAN1

https://github.com/opnsense/core/issues/1337

OOohhh nasty gotcha.

If I set up a IPSEC connection and specify WAN3 the system still creates a route via WAN1

Here's the routing table for it:

192.168.97.0/24    192.168.1.1        US       vtnet1

I think it should be:

192.168.97.0/24    192.168.2.1        US       vtnet3

Here's the same connection now set to use the gateway group and it auto selects WAN1:

192.168.97.0/24    192.168.1.1        US       vtnet1

With it set on WAN1 I can ping from the remote box -> opnsense after creating a Pass rule, but not vice versa ! See my previous on the subject: https://forum.opnsense.org/index.php?topic=4217.0

(bangshead)

:-)

Hi Ad,

me back playing again! I think I might have solved my original WAN issues with ISPs so testing other bits and pieces to see what I can make work

Yes, you need a gateway on both interfaces, then when your setup is functional you can begin setting up multiwan.

OK, I reset the interface IPs manually and the gateways as well. I think it is safer than auto generated gateways.

I now get the fact that a default gateway must be set, but the weighting in the Gateway affects which route packets will take.

I think there is a bug here. If you set the WAN IPs manually the system forces you to have a default gateway, but if you use DHCP for both it does not! I can add a bug if required.

It's also good to check if the gateways report the correct status in System -> Gateways -> Status.

Yup - that seems OK but as the box is in the UK, and I am not, I need my young monkey to pull cables and make sure it falls over correctly :-) However as each WAN has the correct gateway I think it should be OK now. Will test tomorrow

Policy based routing uses the gateway status to determine valid targets.

OK. I think I get that now. For the benefit of others you need to set the Tier in Gateway Groups and the Weight in Gateway/Advanced (I think I am right in saying)


For reference here is the updated routing table which is the same bar the fact that WAN1/vtnet1 is set as default by the system.

Internet:
Destination        Gateway            Flags      Netif Expire
default            192.168.1.1        UGS      vtnet1
8.8.4.4            192.168.2.1        UGHS     vtnet3
8.8.8.8            192.168.1.1        UGHS     vtnet1
10.0.0.0/24        link#1             U        vtnet0
10.0.0.251         link#1             UHS         lo0
127.0.0.1          link#8             UH          lo0
192.168.1.0/24     link#2             U        vtnet1
192.168.1.11       link#2             UHS         lo0
192.168.2.0/24     link#4             U        vtnet3
192.168.2.11       link#4             UHS         lo0


To test each route works you can traceroute the Gateway monitoring IP and see that the packet goes out the correct route

root@OPNsense:~ # traceroute -n 8.8.8.8
traceroute to 8.8.8.8 ( 8.8.8.8 ), 64 hops max, 40 byte packets
1  192.168.1.1  0.389 ms  0.562 ms  0.291 ms
2  81.139.192.1  27.124 ms  27.078 ms  27.322 ms
.....

root@OPNsense:~ # traceroute -n 8.8.4.4
traceroute to 8.8.4.4 ( 8.8.4.4 ), 64 hops max, 40 byte packets
1  192.168.2.1  0.433 ms  0.525 ms  0.298 ms
2  81.139.96.1  27.452 ms  25.104 ms  25.335 ms
.....

Beyond that I am not sure how else to tell if the balancing is working correctly and the system is using both routes ?

B. Rgds
John

Scratching my head here.

Setup Multi WAN with 2 x WAN connections as per the docs.

What I can't figure out is what to do with setting (or not) the default gateway on a WAN.

Clearly you can only set a default route on one connection or the other. But I presume (though it is not mentioned) that this should be disabled for both WAN connections in a Multi WAN setup.

OK, so no default gateway and we add a rule to forward all local traffic to the gateway group.

From Opnsense I can ping the gateway monitor IPs and they traceroute out via each WAN device correctly.

From Opnsense I can ping the DNS servers set by the WAN routers and they traceroute out via each WAN device correctly.

For the life of me what I cannot do is ping any other host. I just get "No route to host"

That kind of makes sense - there are routes set for the DNS IPs and for the gateway monitor IPs but no other traffic so the packets should be picked up by the firewall rule but nothing happens.

Nothing appears in the firewall logs.

You can see the routes set for the Googler DNS servers used as gateway monitor IPs and you can see the DNS servers set by the ADSL routers

Internet:
Destination        Gateway            Flags      Netif Expire
8.8.4.4            192.168.2.1        UGHS     vtnet3
8.8.8.8            192.168.1.1        UGHS     vtnet1
10.0.0.0/24        link#1             U        vtnet0
10.0.0.251         link#1             UHS         lo0
127.0.0.1          link#8             UH          lo0
192.168.1.0/24     link#2             U        vtnet1
192.168.1.11       link#2             UHS         lo0
192.168.2.0/24     link#4             U        vtnet3
192.168.2.11       link#4             UHS         lo0
208.67.220.220     192.168.2.1        UGHS     vtnet3
208.67.222.222     192.168.1.1        UGHS     vtnet1

A traceroute to any of those 4 IPs shows the packets go the right way. But everything else is not getting picked up by the firewall/gateway group

So what on earth have I missed ? Do I still need to set a default gateway ?

B. Rgds
John

Here is the best place but there are lots of issues with multi wan and gateway detection if you search the forums.

It may help if you post a bit more detail on your configuration.

B. Rgds
John

Hmmm. Still can't ping out across VPN. I did think at first it may be due to opnsense being on a VM, and that I then realised the VM host had the wrong gateway set (it should be pointed back to the VM itself) but I modified that with no joy.

Two things that I can see currently.

1. Cannot ping across Ipsec connection on WAN 1. I can ping from the remote host -> opnsense but nothing goes from opnsense -> ipsec remote network/host and I can't even see it being logged anywhere. I can ping any other host i.e opnsense -> rest of the world.

Routing table attached - routing.png.

WAN 1 and WAN 3 are DHCP from 2 different ADSL routers (as I am in test mode). Local network is 10.0.0.0/24 Remote ipsec is 192.168.97.0/24

WAN 1 IP 192.168.1.11 gateway 192.168.1.1
WAN 3 IP 192.168.2.11 gateway 192.168.2.1

2. Also I just noticed this post :

https://forum.opnsense.org/index.php?topic=1803.msg5647

Seems that the routing table is incorrect - I seemed to have the same issue if I try to use my second WAN - (I have WAN1 and WAN3)

If I have a ipsec tunnel on WAN 3 and I try to ping across the VPN it seems packets out are being routed via WAN 1.

Routing table attached routing2.png

I would expect to see:

192.168.97.0/24 192.168.2.1

I can also confirm that if I modify the Ipsec settings any IPSEC rules appear to disappear. If you select any other rule, Edit, and just save without modifying then the Ipsec rules reappear.

Happy to bug this if required.

B. Rgds
John

This still happens in 16.7.13

I modified an Ipsec connection, went back to rules and they are not there. If I create a new Ipsec rule, the others reappear.

Bug required ?

B. Rgds
John

Back to playing again as I have resolved some ISP/IP issues :-)

I have run up a ipsec v2 connection to Libreswan.

I can seem to ping from the Libre end to the Opnsense box, but not the other way around.

I followed the Ipsec howto (excellent resource!) to remind myself of how to do this but stuck on this - I guess because I have missed a firewall rule somewhere.

I have the 3 basic rules set for each WAN device (so I can swap around as required - though I guess a floating rule may be better for this ?)

WAN rules. As per docs:

Protocol ESP
UDP Traffic on Port 500 (ISAKMP)
UDP Traffic on Port 4500 (NAT-T)

Ipsec rules. As per docs:

Enable all, Lan address.

The firewall seems to pass the packet out:

@57 pass out log route-to (vtnet 192.168.1.1) inet from 192.168.1.13 to ! 192.168.1.0/24 flags S/SA keep state allow-opts label "let out anything from the firewall host itself"

But nothing seems to come back. I can ping the same Libre box from another ipsec connected machine and ping both ways happily.

Note that as this is in testing at the minute Opnsense is a VM on Proxmox.
The WAN IP is DHCP from a router at 192.168.1.13 rather than a public IP
The LAN IP is 10.0.0.251
The Ipsec address is 192.168.97.1

From the Libre box I can ping Opnsense:
[root@test ~]# ping 10.0.0.251
PING 10.0.0.251 (10.0.0.251) 56(84) bytes of data.
64 bytes from 10.0.0.251: icmp_seq=1 ttl=64 time=51.5 ms

But a ping from Opnsense on 10.0.0.251 (which has a Ipsec IP Local IP 192.168.1.13) to 192.168.97.1 returns nothing

Sure I fixed this when I had it running before but I just can't remember what I did !

Any assistance appreciated - just need to ping both ways to test a few things !

B. Rgds
John

I think you should allow ports 32400:32410

After that check in Plex Settings/Advanced/Server/Remote Access and see what you have in there as the detected port and IP - you may need to Manually specify the port.

Well, I tried every trick in the book that I know.... used the Vigor 120, tried the main 2820 in their 'half bridge mode' setting the DMZ IP by MAC (which is what works now with the Draytek 3300) But whatever I can get one connection up, but not the other....

The MPoA in the half bridge mode works but the PPPoE refuses to work correctly. I could put it in complete passthrough mode and let Opnsense try doing the PPPoE negotiation but I want both routers working the same way, either full bridge of half bridge.

So I am at a complete impasse and going to have to look for another solution. A shame as I noticed several more bugs and 'gotchas' on numerous issues (with both GUI and Firewall rules, particularly some of the multiwan stuff) but to pursue this I need to do it with a working system and can't afford anymore time on this right now.

I am pretty sure that the issue lies with FreeBSD/Apinger/low level networking but I don't have the knowledge to debug it. When something tells me that it can't connect as 'a port is using Jumbo frames' when I have pinned the MTU to say 1492 and can see that with ifconfig I know there is something fundamentally not right somewhere !

I'm pretty gutted as I think Opnsense is excellent, but if I can't get a basic connection working easily and as I require it, it is no use at all !

Note I was testing Endian and that that connected immediately....

Onwards and upwards.

B. Rgds
John

Been slowly digging away on this and revealing my woeful lack of knowledge of all things OSI :-)

I am pretty sure this is to do with ARP responses (or lack of) and MAC addressing.

I have a feeling that either:

The Draytek 120 is giving its own MAC address somehow rather than the Opnsense ethenet port MAC

The Draytek 120 is blocking ARP requests or responses (despite me having a go at allowing illegal source/destination addresses

Something in sysctl settings is buggering things up

I'm going to try and add the Opnsense MAC on the 120 and test that. My biggest problem is lack of knowledge on how to debug this :-(

Any help appreciated. It's really frustrating as this blocks me from switching off my venerable but ageing Draytek 3300 and moving fully to Opnsense.

B. Rgds
John

Been trying to figure this out the last few days and just can't get it right.

I have both a Draytek 2820 modem/route and a Vigor 120 modem for testing (I have been using 2 x 2820s but was going to move to using the 120s)

I am trying to bridge a MPoA connection but whichever way I twist and turn I come up against a brick wall.

The opensense router has a "HP NC380T PCIe DP Multifunc Gig Server Adapter" dual NIC for the two WAN ports

I have the router in Multiwan mode. WAN 2 is disconnected for testing. WAN3 has a DHCP backup wifi ether connection and is in a gateway group that seems to be handling failover OK :-)

The logs show this for WAN1 :

arpresolve: can't allocate llinfo for 80.59.1.1 on bce0

There are numerous posts on this, but few answers... drivers, other issues...

This is similar https://forum.pfsense.org/index.php?topic=31753.0

Tried force adding a default GW

root@OPNsense:~ # route add default 80.59.216.193
route: writing to routing socket: Network is unreachable
add net default: gateway 80.59.216.193 fib 0: Network is unreachable

I have read various items on issues with bridging on the Drayteks e.g.:
http://www.draytek.com/en/faq/faq-connectivity/connectivity.wan/what-should-i-do-when-vigor-router-is-getting-the-message-arp-address-mismatch-in-syslog/

I have tried ip arp accept 1 and 3  to enable acceptance of incorrectly addressed ARP packets but this does not seem to work

Quite simply I am stuck. Opnense works fine if it uses standard DHCP from the ADSL router but then the WAN interfaces get a Private IP and not the proper WAN IP (which makes mess for ipsec)

Any suggestion s on trying to pin this issue down and try to sure it ?

I'd try another ADSL modem if I knew there was one that would work. Unfortunately I can't use Opnsense in production until I can sort this out :-(


B. Rgds
John

Ipsec rsasigkey

Probably not as good as certs but much easier to implement/manage. Better than PSK.

There is undoubtedly a solution.... but my knowledge hits a brick wall at this juncture.

I guess the first question is what firewall rules do you have set up ?

How do you manage the two subnets on Opnsense ?