OPNsense Forum

English Forums => General Discussion => Topic started by: Perun on September 11, 2018, 07:17:41 am

Title: Multi WAN and ipv6
Post by: Perun on September 11, 2018, 07:17:41 am
Hi

I have 2 WAN uplinks (cable and vdsl). IPv4 does work as expected and cable ipv6 to... but I have problems with forwarding the IPv6 prefix on the VDSL uplink...

thats are my settings:

cable interface (german vodafone cable)
Code: [Select]
    <opt2>
      <if>igb0_vlan4</if>
      <descr>cable</descr>
      <enable>1</enable>
      <lock>1</lock>
      <spoofmac/>
      <blockbogons>1</blockbogons>
      <ipaddr>192.168.40.3</ipaddr>
      <subnet>24</subnet>
      <gateway>cable_gateway</gateway>
      <ipaddrv6>dhcp6</ipaddrv6>
      <dhcp6-ia-pd-len>1</dhcp6-ia-pd-len>
      <dhcp6prefixonly>1</dhcp6prefixonly>
      <dhcp6sendsolicit>1</dhcp6sendsolicit>
      <adv_dhcp6_debug>1</adv_dhcp6_debug>
      <adv_dhcp6_interface_statement_send_options/>
      <adv_dhcp6_interface_statement_request_options/>
      <adv_dhcp6_interface_statement_information_only_enable/>
      <adv_dhcp6_interface_statement_script/>
      <adv_dhcp6_id_assoc_statement_address_enable/>
      <adv_dhcp6_id_assoc_statement_address/>
      <adv_dhcp6_id_assoc_statement_address_id/>
      <adv_dhcp6_id_assoc_statement_address_pltime/>
      <adv_dhcp6_id_assoc_statement_address_vltime/>
      <adv_dhcp6_id_assoc_statement_prefix_enable/>
      <adv_dhcp6_id_assoc_statement_prefix/>
      <adv_dhcp6_id_assoc_statement_prefix_id/>
      <adv_dhcp6_id_assoc_statement_prefix_pltime/>
      <adv_dhcp6_id_assoc_statement_prefix_vltime/>
      <adv_dhcp6_prefix_interface_statement_sla_id/>
      <adv_dhcp6_prefix_interface_statement_sla_len/>
      <adv_dhcp6_authentication_statement_authname/>
      <adv_dhcp6_authentication_statement_protocol/>
      <adv_dhcp6_authentication_statement_algorithm/>
      <adv_dhcp6_authentication_statement_rdm/>
      <adv_dhcp6_key_info_statement_keyname/>
      <adv_dhcp6_key_info_statement_realm/>
      <adv_dhcp6_key_info_statement_keyid/>
      <adv_dhcp6_key_info_statement_secret/>
      <adv_dhcp6_key_info_statement_expire/>
      <adv_dhcp6_config_advanced/>
      <adv_dhcp6_config_file_override/>
      <adv_dhcp6_config_file_override_path/>
    </opt2>

vdsl interface (german 1&1 vdsl)
Code: [Select]
    <opt3>
      <if>igb0_vlan14</if>
      <descr>vdsl</descr>
      <enable>1</enable>
      <lock>1</lock>
      <spoofmac/>
      <blockbogons>1</blockbogons>
      <ipaddr>192.168.140.3</ipaddr>
      <subnet>24</subnet>
      <gateway>vdsl_gateway</gateway>
      <ipaddrv6>dhcp6</ipaddrv6>
      <dhcp6-ia-pd-len>4</dhcp6-ia-pd-len>
      <dhcp6sendsolicit>1</dhcp6sendsolicit>
      <adv_dhcp6_debug>1</adv_dhcp6_debug>
      <adv_dhcp6_interface_statement_send_options/>
      <adv_dhcp6_interface_statement_request_options/>
      <adv_dhcp6_interface_statement_information_only_enable/>
      <adv_dhcp6_interface_statement_script/>
      <adv_dhcp6_id_assoc_statement_address_enable/>
      <adv_dhcp6_id_assoc_statement_address/>
      <adv_dhcp6_id_assoc_statement_address_id/>
      <adv_dhcp6_id_assoc_statement_address_pltime/>
      <adv_dhcp6_id_assoc_statement_address_vltime/>
      <adv_dhcp6_id_assoc_statement_prefix_enable/>
      <adv_dhcp6_id_assoc_statement_prefix/>
      <adv_dhcp6_id_assoc_statement_prefix_id/>
      <adv_dhcp6_id_assoc_statement_prefix_pltime/>
      <adv_dhcp6_id_assoc_statement_prefix_vltime/>
      <adv_dhcp6_prefix_interface_statement_sla_id/>
      <adv_dhcp6_prefix_interface_statement_sla_len/>
      <adv_dhcp6_authentication_statement_authname/>
      <adv_dhcp6_authentication_statement_protocol/>
      <adv_dhcp6_authentication_statement_algorithm/>
      <adv_dhcp6_authentication_statement_rdm/>
      <adv_dhcp6_key_info_statement_keyname/>
      <adv_dhcp6_key_info_statement_realm/>
      <adv_dhcp6_key_info_statement_keyid/>
      <adv_dhcp6_key_info_statement_secret/>
      <adv_dhcp6_key_info_statement_expire/>
      <adv_dhcp6_config_advanced/>
      <adv_dhcp6_config_file_override/>
      <adv_dhcp6_config_file_override_path/>
    </opt3>

my first lan (should use cable for ipv6)
Code: [Select]
    <opt5>
      <if>igb1</if>
      <descr>lan</descr>
      <enable>1</enable>
      <lock>1</lock>
      <spoofmac/>
      <ipaddr>192.168.50.3</ipaddr>
      <subnet>24</subnet>
      <ipaddrv6>track6</ipaddrv6>
      <track6-interface>opt2</track6-interface>
      <track6-prefix-id>0</track6-prefix-id>
    </opt5>

my second lan (should use vdsl for ipv6)
Code: [Select]
    <opt1>
      <if>igb2</if>
      <descr>lan_media</descr>
      <enable>1</enable>
      <lock>1</lock>
      <spoofmac/>
      <ipaddr>192.168.150.3</ipaddr>
      <subnet>24</subnet>
      <ipaddrv6>track6</ipaddrv6>
      <track6-interface>opt3</track6-interface>
      <track6-prefix-id>0</track6-prefix-id>
    </opt1>

cat /var/etc/radvd.conf
Code: [Select]
# Automatically generated, do not edit
# Generated config for dhcp6 delegation from opt2 on opt5
interface igb1 {
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvLinkMTU 1500;
AdvOtherConfigFlag on;
prefix 2a02:8109:9d40:476::/64 {
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
};
RDNSS 2a02:8109:9d40:476:20d:b9ff:fe4a:7499 { };
DNSSL chao5.int { };
};
# Generated config for dhcp6 delegation from opt3 on opt1
interface igb2 {
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvLinkMTU 1500;
AdvOtherConfigFlag on;
prefix ::/64 {
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
};
DNSSL chao5.int { };
};

here I'm missing something, there is no prefix on igb2... why?

cat /var/etc/dhcp6c_opt2.conf
Code: [Select]
interface igb0_vlan4 {
  send ia-pd 0; # request prefix delegation
  request domain-name-servers;
  request domain-name;
  script "/var/etc/dhcp6c_opt2_script.sh"; # we'd like some nameservers please
};
id-assoc pd 0 {
  prefix-interface igb1 {
    sla-id 0;
    sla-len 1;
  };
};

cat /var/etc/dhcp6c_opt3.conf
Code: [Select]
interface igb0_vlan14 {
  send ia-na 0; # request stateful address
  send ia-pd 0; # request prefix delegation
  request domain-name-servers;
  request domain-name;
  script "/var/etc/dhcp6c_opt3_script.sh"; # we'd like some nameservers please
};
id-assoc na 0 { };
id-assoc pd 0 {
  prefix-interface igb2 {
    sla-id 1;
    sla-len 4;
  };
};

There are 2 running dhcpd6c processes... it is normal?

in /var/log/dhcpd.log | grep dhcp6c I see this:
Code: [Select]
Sep 11 07:14:13 cerber dhcp6c[19870]: Sending Solicit
Sep 11 07:14:13 cerber dhcp6c[19870]: set client ID (len 14)
Sep 11 07:14:13 cerber dhcp6c[19870]: set identity association
Sep 11 07:14:13 cerber dhcp6c[19870]: set elapsed time (len 2)
Sep 11 07:14:13 cerber dhcp6c[19870]: set option request (len 4)
Sep 11 07:14:13 cerber dhcp6c[19870]: set IA_PD
Sep 11 07:14:13 cerber dhcp6c[19870]: send solicit to ff02::1:2%igb0_vlan14
Sep 11 07:14:13 cerber dhcp6c[19870]: reset a timer on igb0_vlan14, state=SOLICIT, timeo=13, retrans=110376
Sep 11 07:14:13 cerber dhcp6c[15879]: unexpected interface (9)

what can be the problem? what I'm doing wrong?

TiA
Title: Re: Multi WAN and ipv6
Post by: marjohn56 on September 11, 2018, 09:21:43 am
You cannot have two dhcp6c processes, dhcp6c should handle all of them in one go. Currently Opnsense  doesn't support multiple WAN dhcp6. It was on my 'todo' list.
Title: Re: Multi WAN and ipv6
Post by: Perun on September 11, 2018, 11:06:00 am
was?
Title: Re: Multi WAN and ipv6
Post by: marjohn56 on September 11, 2018, 03:34:19 pm
Sorry... still is.
Title: Re: Multi WAN and ipv6
Post by: Perun on September 11, 2018, 05:21:08 pm
cool and for what version it is on the roadmap?
Title: Re: Multi WAN and ipv6
Post by: marjohn56 on September 11, 2018, 05:50:51 pm
It will be there when I have time, or someone else is welcome to do it.  8)
Title: Re: Multi WAN and ipv6
Post by: wget on February 14, 2020, 04:11:25 pm
It will be there when I have time, or someone else is welcome to do it.  8)

Does this restriction still apply with OPNsense 20.1?
Title: Re: Multi WAN and ipv6
Post by: franco on February 14, 2020, 04:17:04 pm
Yep.


Cheers,
Franco
Title: Re: Multi WAN and ipv6
Post by: marjohn56 on February 14, 2020, 06:02:09 pm
I'll pick this up and aim for 20.7...


Going to need some testers, they'll need dual WAN ISPs using dhcp6.
Title: Re: Multi WAN and ipv6
Post by: wget on February 14, 2020, 06:09:07 pm
I'll pick this up and aim for 20.7...


Going to need some testers, they'll need dual WAN ISPs using dhcp6.

Then, if you need testing, I'm your man :)

Btw, since I'm still figuring out how the fallback method is working I wonder how LAN devices tracking the IPv6 PD address range prefix from WAN1 (Global Unicast IPv6 address i.e. 2a02::/8) will fallback to WAN2 (the backup WAN) when WAN1 is down.

And how to address the issue of DNS? Since the prefix will change when WAN1 is down, LAN devices (some of them are servers) will get a new prefix...
Title: Re: Multi WAN and ipv6
Post by: marjohn56 on February 14, 2020, 06:20:53 pm
Indeed, that's why it's so complex. Getting dhcp6c to do its thing is relatively straight forwards, handling the firewall is a totally different ballgame, and will require the input of the wizards in that area; it's similar to a change in prefix delegation on a single WAN instance, something as yet not totally got to grips with.
Title: Re: Multi WAN and ipv6
Post by: marjohn56 on February 16, 2020, 06:45:08 am
OK, some success. @wget I'll PM you with what you need to do, but you must be running the dev path, can you confirm that before i send you anything.
Title: Re: Multi WAN and ipv6
Post by: marjohn56 on February 16, 2020, 09:03:24 am

Then, if you need testing, I'm your man :)

Btw, since I'm still figuring out how the fallback method is working I wonder how LAN devices tracking the IPv6 PD address range prefix from WAN1 (Global Unicast IPv6 address i.e. 2a02::/8) will fallback to WAN2 (the backup WAN) when WAN1 is down.

And how to address the issue of DNS? Since the prefix will change when WAN1 is down, LAN devices (some of them are servers) will get a new prefix...


It'll not work if you are using global DNS pointing using your GUAs, obviously you would need to update them as well; possibly only a  dynamic DNS would work with that. My task is make dhcp6c do its thing by creating the prefixes and addresses on the interfaces, which appears to work.


It appears that you can add multiple GUAs to an interface, the issue there is that you can only have one dhcpv6 server, RADVD may be able to handle it, but this is a long road and it's not going to get implemented overnight, lots more work to do.
Title: Re: Multi WAN and ipv6
Post by: wget on February 16, 2020, 06:37:43 pm
OK, some success. @wget I'll PM you with what you need to do, but you must be running the dev path, can you confirm that before i send you anything.

I'm a primarily a full stack dev and only a network enthusiast, so I'm ok with the dev path. :)

Since my OPNsense current apu1c4 is used for production, I'll just visualize one on KVM VM hosted on my Proxmox machine. :)

My xDSL and modem cable both being in bridge mode, my ISPs are allowing a second direct connection on them, shouldn't be a problem :)
Title: Re: Multi WAN and ipv6
Post by: wget on February 16, 2020, 06:39:08 pm
It appears that you can add multiple GUAs to an interface, the issue there is that you can only have one dhcpv6 server, RADVD may be able to handle it, but this is a long road and it's not going to get implemented overnight, lots more work to do.

" but this is a long road and it's not going to get implemented overnight," yes, much things to have in mind indeed :)
Title: Re: Multi WAN and ipv6
Post by: Yoshi-DE on April 11, 2020, 02:15:34 am
Hello,

is this now working on 20.7 dev?

I have installed the dev version, but I cannot get IPv6 running on both WAN interfaces. IPv6 only runs on WAN1 with prefix delegation. The radvd.conf only shows the IPv6 prefix from WAN1.

WAN1 -> Cable Provider with "static" IPv6. Changes only with new DUID.
WAN2 -> DSL over PPPoE with dynamic IPv6. The prefix changed on every reconnect.

Thanks,
Yoshi
Title: Re: Multi WAN and ipv6
Post by: marjohn56 on April 11, 2020, 08:28:21 am
Not yet. I have some PRs on Github but I don't think the devs are looking to take it forward at the moment.
Title: Re: Multi WAN and ipv6
Post by: Yoshi-DE on April 11, 2020, 05:14:15 pm
ok, then I'll have to keep working with the tunnelbroker on WAN2.
Title: Re: Multi WAN and ipv6
Post by: fryfrog on April 17, 2020, 12:09:40 am
It seems like there might also be issues w/ dhcpc6 on lagg links too. I just switched from a working ipv6 setup w/ a single, regular old connection to lagg and lost ipv6.
Title: Re: Multi WAN and ipv6
Post by: franco on May 11, 2020, 05:42:15 am
General idea is now in development branch minus some things relating to tracking WAN from LAN(s). Now all we need is tests+time. ;)


Cheers,
Franco
Title: Re: Multi WAN and ipv6
Post by: marjohn56 on May 11, 2020, 09:41:26 am
As per @Franco's comment, a lot of work has gone into this in the last week or so, indirectly due to Covid19... there are some upsides!


What we really need now are users who can run multi-wan dhcp6 and are prepared to test, and I mean test thoroughly the code behind it all and feed the information back to us. Whilst I can emulate multi-wan with use of VLANs it's not real world and that's what we need. In order to test this you would need to be on the 20.7 dev branch and know how to apply patches etc. There's more to this than just a change of the code, we have updated dhcp6c to give extra information. Be aware that testing may also involve a loss of V6 where we get something wrong, though that's usually fixed very quickly.


Let us know if you are prepared to take the risks.
Title: Re: Multi WAN and ipv6
Post by: Maurice on May 11, 2020, 02:18:02 pm
Great to hear that, thanks for all your work!

While I don't currently have multiple Internet connections with PD, I can do real-life testing for possible side effects with a single PD WAN.
I'm on 20.7.b_97 (FreeBSD 11.2). Would you recommend applying specific patches or just do a git pull / make upgrade from master?

Cheers

Maurice
Title: Re: Multi WAN and ipv6
Post by: marjohn56 on May 11, 2020, 02:26:33 pm
Let me check out some changes @franco has made to naming in a couple of areas, I may need to alter the patches to match frst.
Title: Re: Multi WAN and ipv6
Post by: marjohn56 on May 11, 2020, 03:53:39 pm

OK, looks good for testing.
So do this:

You'll need to build the new dhcp6c, so pull the repo make & install. Remember to kill the running dhcp6c before the install. Follow this list you should be good.

# cd /usr
# git clone https://github.com/opnsense/dhcp6c.git (https://github.com/opnsense/dhcp6c.git)
# cd /usr/dhcp6c
# ./configure
# make

kill the existing dhcp6c process

#killall -TERM dhcp6c

Now install the new one

# make install

cd /usr/core

# opnsense-code core
# make upgrade

Now the patches

# opnsense-patch c76a729
# opnsense-patch 88bb423
# opnsense-patch fa8b4c7

reboot, you should be good.
Title: Re: Multi WAN and ipv6
Post by: Maurice on May 12, 2020, 01:16:34 am
Thanks for the instructions. I did follow them, but unfortunately dhcp6c seemed to be stuck in a loop after the reboot. The WAN-tracking LAN interfaces didn't get addresses any more and the WAN address was repeatedly added and removed. I'll send you the log. I now rolled back to 20.7.b_97 to get it working again.

I also noticed that the "Prevent release" setting (moved to Settings / Interfaces) was disabled after the update. I had this enabled before. It might be a good idea to either migrate this or enable it by default.
Title: Re: Multi WAN and ipv6
Post by: Maurice on May 13, 2020, 07:43:23 pm
Everyone, I tried again and now it works just fine.

If you currently have prevent release enabled and don't want to lose your prefix, you might want to unplug your WAN before rebooting, reboot, re-enable prevent release, reboot again and finally reconnect the WAN. That worked for me.
Title: Re: Multi WAN and ipv6
Post by: marjohn56 on May 13, 2020, 09:23:09 pm
this is still a work in progress, but at least we are making some; but anyone testing please be aware things are not finalised yet and there are things still to do.
Title: Re: Multi WAN and ipv6
Post by: marjohn56 on May 13, 2020, 09:24:43 pm
oh yes, dhcp6c prevent release and debug are now in interfaces-> settings.
Title: Re: Multi WAN and ipv6
Post by: wget on May 27, 2020, 01:55:12 am
Ok. So I have tested the whole thing.

@marjohn56, I discarded the patches IDs you have sent to me in DM since the ones provided here above are more up to date.

Situation

PC Engines apu2c4 (apu2c4 = 3 i210AT LAN / AMD GX-412TC CPU / 4 GB DRAM) (https://www.pcengines.ch/apu2c4.htm)

1 LAN on opt0

2 WANs dual stack:
- Cable DOCSIS based connection (VOO Belgium 125/6.5Mbps (https://atlas.ripe.net/probes/18396/)) on opt1
- xDSL based connection (Proximus 35/10Mbps) (IPv6 tech info (https://2007.blog.dest-unreach.be/2015/11/11/native-ipv6-over-proximus-dsl/)) on opt2

Instructions

My OPNsense was never put in devel mode, so I had to adapt your commands a bit.

Code: [Select]
opnsense-update -t opnsense-devel
cd /usr
opnsense-code core
make upgrade

cd /usr
git clone https://github.com/opnsense/dhcp6c.git
cd dhcp6c
./configure
make
killall -TERM dhcp6c
make install

opnsense-patch c76a729
opnsense-patch 88bb423
opnsense-patch fa8b4c7

reboot

Observations

- On the dashboard, the DHCPv6 server is exactly like before in red. Trying to restart it is not working.
- If I enable the xDSL IPv6 as DHCPv6, the modem cable connection cannot get an IPv6
- As soon I disable IPv6 on the xDSL and I reboot, the modem cable connection gets an IPv6
- Compared to the production 20.1 OPNsense version, the LAN now receives an additional IPv6 /128 scope global 2a02:[xxxx] address. LAN machines are still receiving a global unicast /64 like before.

What kind of log do you need to debug? I'm using radvdump extensively =)
Title: Re: Multi WAN and ipv6
Post by: marjohn56 on May 27, 2020, 08:52:19 am
Work has not yet been done on a single LAN with multiple WAN dhcp6, however dhcp6c should still do its thing. Can you post your /var/etc/dhcp6c.conf and /var/dhcpd/etc/dhcpdv6.conf files.
Title: Re: Multi WAN and ipv6
Post by: wget on May 30, 2020, 05:39:01 pm
Work has not yet been done on a single LAN with multiple WAN dhcp6, however dhcp6c should still do its thing. Can you post your /var/etc/dhcp6c.conf and /var/dhcpd/etc/dhcpdv6.conf files.

Yes, I know work on single LAN for multi WAN has not been done yet. The problem I have been describing was related to have several dhcp6c :)

Here are the files requested. Some sensible content has been redacted with X.

/var/etc/dhcp6c.conf:

Quite normal this is not working, the file doesn't have config for ibg2. :/ I don't understand why the config is missing though.

Code: [Select]
interface igb1 {
  send ia-na 2; # request stateful address
  send ia-pd 2; # request prefix delegation
  request domain-name-servers;
  request domain-name;
  script "/var/etc/dhcp6c_opt1_script.sh"; # we'd like some nameservers please
};
id-assoc na 2 { };
id-assoc pd 2 {
  prefix-interface igb0 {
    sla-id 0;
    sla-len 0;
  };
};

/var/dhcpd/etc/dhcpdv6.conf:

Code: [Select]
option dhcp6.domain-search "home.XXXX.XX";

default-lease-time 7200;
max-lease-time 86400;
log-facility local7;
one-lease-per-client true;
deny duplicates;
ping-check true;
update-conflict-detection false;
authoritative;

subnet6 2a02:2788:XXX:XXX::/64 {
  range6 2a02:2788:XXX:XXX::1000 2a02:2788:XXX:XXX::2000;
  option dhcp6.name-servers 2a02:2788:XXX:XXX:XXX:XXX:XXXX:XXXX;
  prefix6 2a02:2788:XXX:8000:: 2a02:2788:XXX:ff00::/60;
}

ddns-update-style none;
Title: Re: Multi WAN and ipv6
Post by: marjohn56 on May 30, 2020, 10:55:42 pm
OK, the files look fine. Can you reboot, and then do a ps -auxw | grep dhcp6c and post the result... ta
Title: Re: Multi WAN and ipv6
Post by: marjohn56 on June 01, 2020, 10:29:43 am
Ah.. a closer look shows all is not as it should be. The script name 'script "/var/etc/dhcp6c_opt1_script.sh"; # we'd like some nameservers please' us wrong, that would suggest not everything is in place.



Title: Re: Multi WAN and ipv6
Post by: wget on June 01, 2020, 07:46:23 pm
Quote
that would suggest not everything is in place.

What do you suggest to do then? The procedure I followed and adapted from yours is described here (https://forum.opnsense.org/index.php?topic=9661.msg79207#msg79207).

Would you mind checking whether the steps I followed are correct?
Title: Re: Multi WAN and ipv6
Post by: marjohn56 on June 01, 2020, 10:31:53 pm
what does your version info say, 20.?


I'll try and recreate what you did, but I need to know where to start.

Title: Re: Multi WAN and ipv6
Post by: wget on June 02, 2020, 12:53:35 pm
what does your version info say, 20.?

It says: OPNsense 20.7.b_178-amd64


I'll try and recreate what you did, but I need to know where to start.

Where to start? Simply follow what I did from there (https://forum.opnsense.org/index.php?topic=9661.msg79207#msg79207).

Code: [Select]
opnsense-update -t opnsense-devel
cd /usr
opnsense-code core
make upgrade

cd /usr
git clone https://github.com/opnsense/dhcp6c.git
cd dhcp6c
./configure
make
killall -TERM dhcp6c
make install

opnsense-patch c76a729
opnsense-patch 88bb423
opnsense-patch fa8b4c7

reboot
Title: Re: Multi WAN and ipv6
Post by: wget on June 02, 2020, 12:54:24 pm
Also, if you know what is the procedure to go back to stock, let me now. That way, I can restart from scratch. :)
Title: Re: Multi WAN and ipv6
Post by: marjohn56 on June 02, 2020, 04:07:15 pm
I meant what revision you were on, I assume 20.1.7?
Title: Re: Multi WAN and ipv6
Post by: marjohn56 on June 02, 2020, 04:09:37 pm
Nah... looking again the files are OK. So you are seeing multiple dhcp6c instances?


Can you do a ps -auxw | grep dhcp6c and post results please.
Title: Re: Multi WAN and ipv6
Post by: wget on June 02, 2020, 04:11:01 pm
I meant what revision you were on, I assume 20.1.7?

I was indeed on 20.1.7, but followed the procedure I described above, and then it showed OPNsense 20.7.b_178-amd64
Title: Re: Multi WAN and ipv6
Post by: marjohn56 on June 02, 2020, 04:15:26 pm
OK. I'm going to install 20.1.7 and do what you did, see where it takes me.
Title: Re: Multi WAN and ipv6
Post by: wget on June 02, 2020, 04:19:34 pm
OK. I'm going to install 20.1.7 and do what you did, see where it takes me.


Thanks. I have just run
Code: [Select]
opnsense-update -t opnsense
and am now back on 20.1.7.

I'll also remove the files /var/etc/dhcp6c.conf, /var/dhcpd/etc/dhcpdv6.conf and related to start fresh and redo the procedure.

Indeed, something triggered my attention from the man pages of opnsense-patch:
Code: [Select]
opnsense-patch - OPNsense patch utility
[...]
Patches can also be reversed by reapplying them
[...]

which means since I tried several times, the patches may have been reverted.
Title: Re: Multi WAN and ipv6
Post by: wget on June 02, 2020, 04:20:28 pm
Nah... looking again the files are OK. So you are seeing multiple dhcp6c instances?


Can you do a ps -auxw | grep dhcp6c and post results please.

So I wasn't seeing multiple instances. and ps aux was always reporting one.
Title: Re: Multi WAN and ipv6
Post by: marjohn56 on June 02, 2020, 05:14:19 pm
Yup... I see. It would appear it's something to do with FreeBSD 11. I reported an issue a week or so ago and although dhcp6c is working the kernel is reporting issues. You have a choice, you can install the 20.7 beta that's available and carry out the same steps or wait for the new beta, should be with us any day now.
Title: Re: Multi WAN and ipv6
Post by: marjohn56 on June 02, 2020, 05:35:43 pm
Just re-installed the beta again, updated and installed the new dhcp6c and all is sweet. I did not even install the patches and its working.
Title: Re: Multi WAN and ipv6
Post by: wget on June 02, 2020, 05:37:19 pm
Just re-installed the beta again, updated and installed the new dhcp6c and all is sweet. I did not even install the patches and its working.

Could you write down the exact steps you followed?
Title: Re: Multi WAN and ipv6
Post by: marjohn56 on June 02, 2020, 05:42:11 pm
Download the beta image here https://pkg.opnsense.org/FreeBSD:12:amd64/snapshots/


Boot from the image and remember to import the config file when it asks you, log in as installer and install it.


Once rebooted carry out the steps for opnsense-code core and dhcp6c. that should get you up and running.


To be honest, I'd need to look at those patches to see what they were for, I think they were just improvements around dhcp6c and radvd. It should work without, mine does straight away.
Title: Re: Multi WAN and ipv6
Post by: wget on June 02, 2020, 05:52:48 pm
It should work without, mine does straight away.

Ok. I'll test this out with the link you provided to me first, without the patches.

1) Just to be sure. When exporting settings and reimporting them, is the private key of VPN exported as well? Otherwise I would be awful.

2) When testing out, are you using a VM with other connected to simulate 2 WAN, or are you using your real WAN at ZEN UK and adding another server upstream to generate a dummy WAN2? Just asking because I don't have a lab yet, and my OPNsense is kind of my prod right now =)
Title: Re: Multi WAN and ipv6
Post by: marjohn56 on June 02, 2020, 06:17:44 pm
Everything is imported. Save your config to file as well, if you don't do that already, just in case.


I have two Qotoms and an APU, my primary Qotom get's a /48 and I'm splitting that, in this case  a /56 and a /60 to VLANs. My test Qotom then has two WAN ports, one on each of the VLANs. The LAN outputs of the test Qotom have one output to my PC, and the second goes on to the APU where its yet again fed into the WAN port and the two LANs off that.... cool eh?  8)
Title: Re: Multi WAN and ipv6
Post by: marjohn56 on June 03, 2020, 08:22:28 am
OK. went back to remind myself what those patches were for. They tidy up some dhcpdv6 and radvd stuff so add them.
Title: Re: Multi WAN and ipv6
Post by: wget on June 10, 2020, 12:33:14 am
OK. went back to remind myself what those patches were for. They tidy up some dhcpdv6 and radvd stuff so add them.

As I have been trying out the previous process and debugging other issues, my 20.1 based installation what somewhat broken and I decided to reinstall everything from scratch. I was using the same install since a bunch of years now anyway and the reboot process was taking much more time than needed. So the 20.7 even in a dev variant was still a better bet:
https://forum.opnsense.org/index.php?topic=16545.msg79992#msg79992
And indeed it was. With a brand new install, the reboot process is so fast. ;)

However, exactly like my broken 20.1 build, my LAN is not receiving any IPv6 address any more. Like specified in my review of the 20.7 in the link just above, the LAN interface generates an IPv6 from the prefix delegation received from the WAN. However, the machines on the LAN are not seeing RA packets, so no IPv6 on the LAN :( Above all, the OPNsense router cannot even communicate to the outside in IPv6, a ping led either from the LAN or the WAN interface is timing out. The issue is not coming from the connection as a rapid test on a TPLink OpenWRT router is proving 20/20 on ipv6-test.com
Title: Re: Multi WAN and ipv6
Post by: marjohn56 on June 10, 2020, 09:47:56 am
So are you using manual override in the LAN interface for dhcpdv6?
Title: Re: Multi WAN and ipv6
Post by: wget on June 10, 2020, 12:26:41 pm
So are you using manual override in the LAN interface for dhcpdv6?

Nope. I'm not. Should I? cf. attachment.


As the IPv6 problem is quite huge, the purpose here is at least making it work with my main (cable modem) connection. I'm not to the point of using the second connection yet.

Please note, I haven't manually applied the radvd <-> dhcpv6 patches yet. I'm just using the 20.7 iso that I upgraded upsing the traditional update process, nothing more. :)
Title: Re: Multi WAN and ipv6
Post by: marjohn56 on June 10, 2020, 12:43:45 pm
It should still fire up..


Did you do the opnsense-code dhcp6c etc?
Title: Re: Multi WAN and ipv6
Post by: wget on June 10, 2020, 12:53:17 pm
Did you do the opnsense-code dhcp6c etc?

Nope I haven't. I just did a fresh reinstall using the 20.7 iso, updated it using the GUI update process, imported my settings and nothing more :)
Title: Re: Multi WAN and ipv6
Post by: wget on June 10, 2020, 12:58:58 pm
On my side, I'm investigating the MSS (Maximum Segment Size) and the MTU. I think I have seen differences about this between OpenWRT and OPNsense. Might be the reason why IPv6 is going out but not further in the ISP network. May be the reason of RA issues I have on LAN as well.
Title: Re: Multi WAN and ipv6
Post by: marjohn56 on June 10, 2020, 01:18:17 pm
Did you do the opnsense-code dhcp6c etc?

Nope I haven't. I just did a fresh reinstall using the 20.7 iso, updated it using the GUI update process, imported my settings and nothing more :)


Do so, because 20.7 relies on some new stuff in dhcp6c for prefixes.
Title: Re: Multi WAN and ipv6
Post by: wget on June 10, 2020, 06:04:40 pm
On my side, I'm investigating the MSS (Maximum Segment Size) and the MTU. I think I have seen differences about this between OpenWRT and OPNsense. Might be the reason why IPv6 is going out but not further in the ISP network. May be the reason of RA issues I have on LAN as well.

Ok it was the reason. It seems there has been some regression with the Intel I210 NIC FreeBSD driver with a dot release from 20.1.x and from 20.7. Indeed, I had to reset the MTU size to 1536 and force the overriding in the DHCP client otherwise this was not working. (cf. screenshot below).

Procedure of the test:

Max MSS calculation:
1476 (max MSS IPv6): 1536 (MTU of Ethernet II) - 40 (IPv6 header) - 20 (TCP header)
1496 (max MSS IPv4): 1536 (MTU of Ethernet II) - 20 (IPv4 header) - 20 (TCP header)

OpenWRT:
MSS: 1440
MSS: 1460

OPNsense:
MSS: 516 in IPv6
MSS: 536 in IPv4

OPNsense after the MTU override:
MSS: 1440 in IPv6
MSS: 1496 in IPv4

Note: the MSS is determined by the TCP protocol during the handshake, so depending on the needed payload it may be less than the 1496 and 1476 we computed above, that's the reason the values are not exactly equal to 1496 and 1476, but are quite near.
Title: Re: Multi WAN and ipv6
Post by: wget on June 10, 2020, 06:06:38 pm
Do so, because 20.7 relies on some new stuff in dhcp6c for prefixes.

Now my IPv6 connection if fixed \o/, I'll enable the second WAN (xDSL baed) and will let you know whether the additional patches were needed or not =)
Title: Re: Multi WAN and ipv6
Post by: wget on June 10, 2020, 06:57:22 pm
Now my IPv6 connection if fixed \o/, I'll enable the second WAN (xDSL baed) and will let you know whether the additional patches were needed or not =)

Ok without the patches, the xDSL is working, but no IPv6. I need to go in assignment and reload the interface to get an IPv6.

Also, for a reason I don't know curling google via pppoe0 or igb2 in IPv4 is working while in IPv6 this isn't :/

Code: [Select]
root@portal:/home/wget # curl -6 --interface 2a02:XXXX:XXXX:XXXX:XXXX:XXXX:XXXX:XXXX -k -L google.com
curl: (28) Failed to connect to google.com port 80: Operation timed out

(that IP address is corresponding to the one on igb2)

and a TCP dump on that interface is not returning any thing (pcap empty).

Let me know if you have an idea before I try to apply the additional patches which will certainly break my whole OS again ;D
Title: Re: Multi WAN and ipv6
Post by: marjohn56 on June 10, 2020, 07:05:33 pm
Can you just turn on dhcp6c debug in interfaces->settings, then filter the logs on dhcp6c and post them/.
Title: Re: Multi WAN and ipv6
Post by: wget on June 10, 2020, 07:54:25 pm
Can you just turn on dhcp6c debug in interfaces->settings, then filter the logs on dhcp6c and post them/.

Answered in DM as the logs were containing sensible pieces of info. :) I will make sure to expel the ones that are not needed afterwards, that way, this could be useful to others finding this topic in the future =)
Title: Re: Multi WAN and ipv6
Post by: fryfrog on July 17, 2020, 11:34:06 pm
Do you still need testers? I happen to have 3 ISPs and an extra APU2, I could test w/o killing my own internet via an extra cable and dsl, both that do dhcpv6 pd.
Title: Re: Multi WAN and ipv6
Post by: wget on July 17, 2020, 11:49:23 pm
Do you still need testers? I happen to have 3 ISPs and an extra APU2, I could test w/o killing my own internet via an extra cable and dsl, both that do dhcpv6 pd.

Hi there.

An additional test would be wise to have.

From my side, my xDSL and DOCIS based are both receiving an IPv6 address now.

But it appears the xDSL has issues and I need to reload the interface manually afterwards in order to be sure to have an IPv6 on it. Could you test this out? (confirming or disconfirm my issue)
Title: Re: Multi WAN and ipv6
Post by: fryfrog on July 19, 2020, 01:01:11 am
Is it weird that I've always been able to get an ipv6 ip and ipv6-pd delegated from all of my interfaces? My "problem" is that all my hosts then end up with 2-3 public ipv6 addresses and I've never figured out how to control it better. I *think* I need to use a private ipv6 range on my LAN and then ... ?nat64? them 1:1? I *suck* at ipv6, so I know I'm at least a little wrong, probably a lot.
Title: Re: Multi WAN and ipv6
Post by: marjohn56 on July 19, 2020, 04:56:24 pm
you'll usually see multiple ipv6 addresses on clients as that is a feature of ipv6, if you want to disable that then look at this.. https://www.ispcolohost.com/2013/07/06/how-to-disable-ipv6-stateless-autoconfig-on-windows-7/ (https://www.ispcolohost.com/2013/07/06/how-to-disable-ipv6-stateless-autoconfig-on-windows-7/).


you can als change your radvd settings to managed, however if you do that then any android clients you have will not get an address.
Title: Re: Multi WAN and ipv6
Post by: wget on August 11, 2020, 03:12:54 pm
I have reinstalled my system under a fresh 20.7 in order to avoid issues caused from the previous development tests I performed.

Now, I'm unable to get an IPv6 address on igb1 (modem cable based).

While radvdump reports RA paquets asking me to send an IPv6 sollicit DHCP client request (cf. M flag set to on):

Code: [Select]
interface igb1
{
        AdvSendAdvert on;
        # Note: {Min,Max}RtrAdvInterval cannot be obtained with radvdump
        AdvManagedFlag on;
        AdvOtherConfigFlag on;
        AdvReachableTime 3600000;
        AdvRetransTimer 0;
        AdvCurHopLimit 64;
        AdvDefaultLifetime 9000;
        AdvHomeAgentFlag off;
        AdvDefaultPreference medium;
        AdvSourceLLAddress on;
        AdvLinkMTU 1500;
}; # End of interface definition

The DHCP request is not getting an offer:

Code: [Select]
2020-08-11T15:07:19 dhcp6c[36699]: reset a timer on igb1, state=SOLICIT, timeo=5, retrans=35350
2020-08-11T15:07:19 dhcp6c[36699]: send solicit to ff02::1:2%igb1
2020-08-11T15:07:19 dhcp6c[36699]: set IA_PD
2020-08-11T15:07:19 dhcp6c[36699]: set IA_PD prefix
2020-08-11T15:07:19 dhcp6c[36699]: set option request (len 4)
2020-08-11T15:07:19 dhcp6c[36699]: set elapsed time (len 2)
2020-08-11T15:07:19 dhcp6c[36699]: set identity association
2020-08-11T15:07:19 dhcp6c[36699]: set client ID (len 14)
2020-08-11T15:07:19 dhcp6c[36699]: Sending Solicit
2020-08-11T15:07:01 dhcp6c[36699]: reset a timer on igb1, state=SOLICIT, timeo=4, retrans=17047
2020-08-11T15:07:01 dhcp6c[36699]: send solicit to ff02::1:2%igb1
2020-08-11T15:07:01 dhcp6c[36699]: set IA_PD
2020-08-11T15:07:01 dhcp6c[36699]: set IA_PD prefix
2020-08-11T15:07:01 dhcp6c[36699]: set option request (len 4)
2020-08-11T15:07:01 dhcp6c[36699]: set elapsed time (len 2)
2020-08-11T15:07:01 dhcp6c[36699]: set identity association
2020-08-11T15:07:01 dhcp6c[36699]: set client ID (len 14)
2020-08-11T15:07:01 dhcp6c[36699]: Sending Solicit
2020-08-11T15:06:53 dhcp6c[36699]: reset a timer on igb1, state=SOLICIT, timeo=3, retrans=8494

The stripped down version of dhcp6c.conf:

Code: [Select]
root@portal:/home/wget # cat /var/etc/dhcp6c.conf
interface igb1 {
  send ia-na 2; # request stateful address
  send ia-pd 2; # request prefix delegation
  request domain-name-servers;
  request domain-name;
  script "/var/etc/dhcp6c_opt1_script.sh"; # we'd like some nameservers please
};
id-assoc na 2 { };
id-assoc pd 2 {
  prefix ::/64 infinity;
};

This time I ensured the MTU + MSS were correct, so I don't think this comes from this side. Any direction would be great to have.
Title: Re: Multi WAN and ipv6
Post by: franco on August 11, 2020, 03:45:02 pm
May this be a problem with the previous lease still active or a MAC address mismatch?


Cheers,
Franco
Title: Re: Multi WAN and ipv6
Post by: wget on August 11, 2020, 03:55:58 pm
Well, it's true that up to now I have been spoofing the MACs from my WANs, but it has always worked like this before since I have this APU2 (end of 2017).

I have just unset the spoofing, rebooted. Even if the NIC is now using the real HW MAC address, I still don't get any IPv6 DHCP answer.

Also, for the record, previously in the 20.7.x dev config I had, the "prevent release" DHCPv6 setting was set. I unset it as well without much results :/

How can I see if this could come from a pending existing DHCPv6 lease that hasn't expired?
Title: Re: Multi WAN and ipv6
Post by: wget on August 11, 2020, 05:20:42 pm
Back. I have retested my APU2 board on OpenWRT and the problem was similar.

I then debugged the issue directly on my laptop. I contacted the core network team of my ISP. It appeared this was indeed a problem on their side. Problem fixed =)

I confirm that with the 20.7 release (not in dev mode any more) that dhcp6c is working for both of my connections (DOCSIS and xDSL PPPoE).

Retested again with my trick:

Code: [Select]
$ curl -6 --interface 2a02:[IPv6 address of the PPPoE] -k -L google.com
[...]
$ curl -6 --interface 2a02:[IPv6 address of the DOCIS modem bridge] -k -L google.com
[...]

and both replied correctly.

Enforcing the MTU override for the DOCSIS based connection at VOO (Belgium) is still required though otherwise OPNsense was still setting the MTU to 516 that breaks the IPv6 minimum requirements.
Title: Re: Multi WAN and ipv6
Post by: franco on August 11, 2020, 05:59:46 pm
Happy to hear :)

MTU should be advertised as 1280, not sure where 516 is coming from. Or is the OPNsense itself violating the MTU constraint?


Cheers,
Franco
Title: Re: Multi WAN and ipv6
Post by: wget on August 14, 2020, 12:03:44 am
Happy to hear :)

MTU should be advertised as 1280, not sure where 516 is coming from. Or is the OPNsense itself violating the MTU constraint?


Cheers,
Franco

I think this is indeed OPNsense (or at least the FreeBSD driver) violating this MTU constraint because the issue doesn't happen with OpenWRT on that same device.

How do you know this should be 1280? It was indeed the value that was sometimes displayed as ifconfig output when 516 wasn't.

Is 1280 the default value for Ethernet reported by DOSCIS cable modems? On my Ethernet link from my xDSL modem, the MTU was 1500.
Title: Re: Multi WAN and ipv6
Post by: wget on August 14, 2020, 12:06:38 am
[...]

I then debugged the issue directly on my laptop. I contacted the core network team of my ISP. It appeared this was indeed a problem on their side. Problem fixed =)

[...]

Actually nothing is 100 % correct when I said this was 100% working. After a reboot, I have a race condition and the IPv6 doesn't immediately show up on the xDSL link. I need to manually go in Interfaces > Overview and Reload the xDSL link in order to have an IPv6 address. Any idea to avoid this manual step at each reboot?
Title: Re: Multi WAN and ipv6
Post by: Zlapped24 on August 14, 2020, 12:28:19 pm
Btw, since I'm still figuring out how the fallback method is working
I wonder how LAN devices tracking the IPv6 PD address range prefix from WAN1
(Global Unicast IPv6 address i.e. 2a02::/8) ข่าวกีฬาออนไลน์ (https://footballarena88.com/)
will fallback to WAN2 (the backup WAN) when WAN1 is down.
Title: Re: Multi WAN and ipv6
Post by: wget on August 14, 2020, 12:37:36 pm
Btw, since I'm still figuring out how the fallback method is working
I wonder how LAN devices tracking the IPv6 PD address range prefix from WAN1
(Global Unicast IPv6 address i.e. 2a02::/8) ข่าวกีฬาออนไลน์ (https://footballhits98.com/)
will fallback to WAN2 (the backup WAN) when WAN1 is down.

@Zlapped24 They won't. The gateway won't be magically changing and devices won't be getting the new IPv6 address.

The current patch described in this thread was only the first step: supporting a merged dhcpv6 client config with different interfaces. That's only what the fixes (implemented in 20.7) are doing for now.

Gateway changing and IPv6 address changing will still need to be implemented.
Title: Re: Multi WAN and ipv6
Post by: marjohn56 on August 15, 2020, 09:39:33 am

Actually nothing is 100 % correct when I said this was 100% working. After a reboot, I have a race condition and the IPv6 doesn't immediately show up on the xDSL link. I need to manually go in Interfaces > Overview and Reload the xDSL link in order to have an IPv6 address. Any idea to avoid this manual step at each reboot?


Are you seeing the requests being sent in the log?


As a side note I used to get exactly this issue with Sky UK after a reboot. No release was sent at reboot ( if it was set to send no release )  or hard power off. Then when it sent a new request after coming back up their BNG would not respond as the XID was now different. The only way to get the v6 address back was to power down the modem completely and wait 30 minutes, turn it back on and hey presto; this was because the lease timed out after 30 minutes and the BNG would then listen again... another reason I changed ISP!

Title: Re: Multi WAN and ipv6
Post by: wget on August 19, 2020, 01:12:59 pm
Are you seeing the requests being sent in the log?
[...]

Nope. I'm not. As if the link wasn't ready to receive RA paquets when dhcp6c is being launched. This really sounds like a race condition.

From my side, I'm not aware of any BNG issues (and the need to wait to request IPs again) like you described with both of my ISPs, so I don't think this comes from this point. Actually my ISP are still quite open in the sense I could request several IPv4 or IPv6 prefixes and they are not complaining.
Title: Re: Multi WAN and ipv6
Post by: marjohn56 on August 19, 2020, 03:38:28 pm
Is there anything from dhcp6c in the logs when the condition occurs?