OPNsense Forum

Archive => 20.1 Legacy Series => Topic started by: Bytechanger on May 20, 2020, 09:13:57 am

Title: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: Bytechanger on May 20, 2020, 09:13:57 am: Hi,

On my opnsense haproxy is running.
Set a ipv4 Backend.

Frontend hearing on ipv4 0.0.0.0:56573 ipv6 [::1]:56573
but only ipv4 is working??
WAN ipv4 -> haproxy runs great
WAN ipv6 -> haproxy no reaction.

SSH on OPNSense:
Code: [Select]
sudo sockstat -6 | grep haproxy www haproxy 36535 22 tcp6 ::1:56573 *:*
So I think, haproxy is hearing on right ports.

Firewall is open on WAN to ipv4 and ipv6 for 56573

Where is my fault?

Greets

Byte
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: Maurice on May 20, 2020, 12:24:49 pm: The IPv6 equivalent of '0.0.0.0' is '::' (all zeros, unspecified address).

Cheers

Maurice
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: Bytechanger on May 20, 2020, 12:40:43 pm: Hm, OPNSense tells you to use

Code: [Select]
"Please provide a valid listen address, i.e. 127.0.0.1:8080, [::1]:8080 or www.example.com:443. Port range as start-end, i.e. 127.0.0.1:1220-1240."
[::]:56573 doesn´t work

but in ssh looks good
Code: [Select]
sudo sockstat -6 | grep haproxy www haproxy 42268 22 tcp6 *:56573 *:*
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: Bytechanger on May 22, 2020, 03:48:24 pm: Hi,

so internaly it works fine!
When I choose https://[2003:xx:xxx:xxxx:xxx:xxxx:xxxx:8584]:56573/ (IPv6 LAN-Adapter adress) it works fine.
But when I test and come from internet to WAN, nothing happens!?

I ssh into an IONOS-VServer and try to connect to my opnsense at home
ping6 2003:xx:xxx:xxxx:xxx:xxxx:xxxx:8583
works fine.
But

wget --no-check-certificate https://[2003:xx:xxx:xxxx:xxx:xxxx:xxxx:8583]:56573
--2020-05-22 13:46:55-- https://[2003:xx:xxx:xxxx:xxx:xxxx:xxxx:8583]:56573/
Connecting to 2003:xx:xxx:xxxx:xxx:xxxx:xxxx:8583]:56573...

ends there.....

wget --no-check-certificate https://87.xxx.xxx.16:56573 works fine also....

Any Idea??
Firewallrules are set to ipv4 and ipv6 opened on this ports...

Greets

Byte
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: Maurice on May 22, 2020, 04:32:25 pm: So, connecting to the LAN interface address from a host in the LAN works, but connecting to the WAN interface address from the Internet doesn't work, correct?

1. What about connecting to the WAN address from a host in the LAN?
2. Anything in the firewall logs when trying to connect from the Internet?
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: johnsmi on May 22, 2020, 05:25:31 pm: Quote from: Bytechanger on May 20, 2020, 12:40:43 pm
[::]:56573 doesn´t work

Tho it looks funny, the listen address:port is 0.0.0.0:443 :::443 localhost:443
Code: [Select]
:::56573
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: Bytechanger on May 22, 2020, 06:31:10 pm: OK, you mean, localy I put the IPv6 Adress of WAN-Interface?
This works too.

I´m not sure, I can handle filterlogs correct.
But when I filter my log (don´t know if it´s right) i can find some entry:
Code: [Select]
filterlog 134,,,0,pppoe0,match,pass,in,6,0x00,0xb70a5,58,tcp,6,40,2001:XX(IP from my IONOS Server),2003:(IP from my WAN),44608,56573,0,S,3312441647,,64800,,mss;sackOK;TS;nop;wscale
Can´t read all of it, but I think, it´s right and passing my firewall??

Greets

Byte
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: johnsmi on May 22, 2020, 08:11:14 pm: Seems like haproxy isn't responding on WAN-IPv6.

Listening on all addresses:port is three colons in a row followed by port
Code: [Select]
:::443 not ~~[::]:443~~, not ~~[::1]:443~~, not ~~::1:443~~.

Quote from: Bytechanger on May 20, 2020, 09:13:57 am
Frontend hearing on ipv4 0.0.0.0:56573 ipv6 [::1]:56573
Does haproxy public service now listen on
Code: [Select]
:::56573 2003:xx:xxx:xxxx:xxx:xxxx:xxxx:8583:56573?
all IPv6 and/or WAN-IP6?
Without [] and NOT ::1?
~~[::1]:56573~~
~~[::]:56573~~
:::56573

wget -O- --no-check-certificate https://[2003::LAN]:56573 from LAN is fine?
wget -O- --no-check-certificate https://[2003::WAN:8583]:56573 from LAN is fine?

wget -O- --no-check-certificate https://[2003::WAN:8583]:56573 from WAN passes firewall with datalen=0.
wget -O- --no-check-certificate https://[2003::LAN]:56573 from WAN?
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: Bytechanger on May 22, 2020, 08:28:53 pm: Hi,

:::56573 or [::]:56573 has the same result in haproxy
Especially when you ssh into opnsense and
Code: [Select]
sudo sockstat -6 | grep haproxy Password: www haproxy 2683 6 tcp6 *:56573 *:*
So sockstat tells it is listening to *:56573, and I think it´s for all interfaces.

wget -O- --no-check-certificate https://[2003::LAN]:56573 from LAN is fine? YES
wget -O- --no-check-certificate https://[2003::WAN:8583]:56573 from LAN is fine? YES

wget -O- --no-check-certificate https://[2003::WAN:8583]:56573 from WAN passes firewall with datalen=0.
wget -O- --no-check-certificate https://[2003::LAN]:56573 same as above... datalen=0

hm, crazy

Greets

Byte
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: johnsmi on May 22, 2020, 09:32:08 pm: Hi,

interesting.

How much information provides tcpdump?
Code: [Select]
sudo tcpdump -ni WAN-Interface 'tcp port 56573'
Is there any response from haproxy?

At least some TCP-stuff?

Might be something with MTU? You're using PPPoE so we expect mss 1452.
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: Bytechanger on May 22, 2020, 10:08:09 pm: Hi,

thanks for helping, here is my output for tcpdump
(I change to port 56571)

Code: [Select]
sudo tcpdump -ni pppoe0 'tcp port 56571' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on pppoe0, link-type NULL (BSD loopback), capture size 262144 bytes 21:59:31.099942 IP6 2001:(IP of my IONOS Server).52084 > 2003:(IP of WAN).56571: Flags [S], seq 2826379982, win 64800, options [mss 1440,sackOK,TS val 3003487412 ecr 0,nop,wscale 6], length 0 21:59:31.100008 IP6 2003:(IP of WAN).56571 > 2001:(IP of my IONOS Server).52084: Flags [S.], seq 983975308, ack 2826379983, win 65228, options [mss1432,nop,wscale 9,sackOK,TS val 4023452836 ecr 3003487412], length 0 21:59:32.127004 IP6 2001:(IP of my IONOS Server).52084 > 2003:(IP of WAN).56571: Flags [S], seq 2826379982, win 64800, options [mss 1440,sackOK,TS val 3003488438 ecr 0,nop,wscale 6], length 0 21:59:32.127051 IP6 2003:(IP of WAN).56571 > 2001:(IP of my IONOS Server).52084: Flags [S.], seq 983975308, ack 2826379983, win 65228, options [mss1432,nop,wscale 9,sackOK,TS val 4023452836 ecr 3003488438], length 0 21:59:34.143015 IP6 2001:(IP of my IONOS Server).52084 > 2003:(IP of WAN).56571: Flags [S], seq 2826379982, win 64800, options [mss1440,sackOK,TS val 3003490454 ecr 0,nop,wscale 6], length 0 21:59:34.143054 IP6 2003:(IP of WAN).56571 > 2001:(IP of my IONOS Server).52084: Flags [S.], seq 983975308, ack 2826379983, win 65228, options [mss 1432,nop,wscale 9,sackOK,TS val 4023452836 ecr 3003490454], length 0 21:59:37.144058 IP6 2003:(IP of WAN).56571 > 2001:(IP of my IONOS Server).52084: Flags [S.], seq 983975308, ack 2826379983, win 65228, options [mss 1432,nop,wscale 9,sackOK,TS val 4023452836 ecr 3003490454], length 0 21:59:38.303321 IP6 2001:(IP of my IONOS Server).52084 > 2003:(IP of WAN).56571: Flags [S], seq 2826379982, win 64800, options [mss1440,sackOK,TS val 3003494614 ecr 0,nop,wscale 6], length 0 21:59:38.303358 IP6 2003:(IP of WAN).56571 > 2001:(IP of my IONOS Server).52084: Flags [S.], seq 983975308, ack 2826379983, win 65228, options [mss 1432,nop,wscale 9,sackOK,TS val 4023452836 ecr 3003494614], length 0 21:59:41.303356 IP6 2003:(IP of WAN).56571 > 2001:(IP of my IONOS Server).52084: Flags [S.], seq 983975308, ack 2826379983, win 65228, options [mss 1432,nop,wscale 9,sackOK,TS val 4023452836 ecr 3003494614], length 0 21:59:44.503085 IP6 2003:(IP of WAN).56571 > 2001:(IP of my IONOS Server).52084: Flags [S.], seq 983975308, ack 2826379983, win 65228, options [mss 1432,nop,wscale 9,sackOK,TS val 4023452836 ecr 3003494614], length 0 21:59:46.494985 IP6 2001:(IP of my IONOS Server).52084 > 2003:(IP of WAN).56571: Flags [S], seq 2826379982, win 64800, options [mss1440,sackOK,TS val 3003502806 ecr 0,nop,wscale 6], length 0 21:59:46.495047 IP6 2003:(IP of WAN).56571 > 2001:(IP of my IONOS Server).52084: Flags [S.], seq 983975308, ack 2826379983, win 65228, options [mss 1432,nop,wscale 9,sackOK,TS val 4023452836 ecr 3003502806], length 0 21:59:49.496584 IP6 2003:(IP of WAN).56571 > 2001:(IP of my IONOS Server).52084: Flags [S.], seq 983975308, ack 2826379983, win 65228, options [mss 1432,nop,wscale 9,sackOK,TS val 4023452836 ecr 3003502806], length 0 21:59:52.696136 IP6 2003:(IP of WAN).56571 > 2001:(IP of my IONOS Server).52084: Flags [S.], seq 983975308, ack 2826379983, win 65228, options [mss 1432,nop,wscale 9,sackOK,TS val 4023452836 ecr 3003502806], length 0 21:59:55.896200 IP6 2003:(IP of WAN).56571 > 2001:(IP of my IONOS Server).52084: Flags [S.], seq 983975308, ack 2826379983, win 65228, options [mss 1432,nop,wscale 9,sackOK,TS val 4023452836 ecr 3003502806], length 0
I don´t see any response from haproxy, in protocol I also can´t see anything

MTU?
On Interfaces->WAN->MTU is empty, under field is shown: Calculated PPP MTU: 1492
MSS is also empty

when accessing with ipv4, working is so
Code: [Select]
listening on pppoe0, link-type NULL (BSD loopback), capture size 262144 bytes 22:17:19.118020 IP 82:(IP of my IONOS Server).40268 > 87:(IP of my WAN).56571: Flags [S], seq 1756100036, win 64240, options [mss 1452,sackOK,TS val 1030447999 ecr 0,nop,wscale 6], length 0 22:17:19.118083 IP 87:(IP of my WAN).56571 > 82:(IP of my IONOS Server).40268: Flags [S.], seq 3225430295, ack 1756100037, win 65228,options [mss 1452,nop,wscale 9,sackOK,TS val 4149591175 ecr 1030447999], length 0 22:17:19.134375 IP 82:(IP of my IONOS Server).40268 > 87:(IP of my WAN).56571: Flags [.], ack 1, win 1004, options [nop,nop,TS val 1030448016 ecr 4149591175], length 0 22:17:19.135332 IP 82:(IP of my IONOS Server).40268 > 87:(IP of my WAN).56571: Flags [P.], seq 1:319, ack 1, win 1004, options [nop,nop,TS val 1030448017 ecr 4149591175], length 318 22:17:19.135354 IP 87:(IP of my WAN).56571 > 82:(IP of my IONOS Server).40268: Flags [.], ack 319, win 126, options [nop,nop,TS val 4149591192 ecr 1030448017], length 0 22:17:19.152703 IP 87:(IP of my WAN).56571 > 82:(IP of my IONOS Server).40268: Flags [.], seq 1:1441, ack 319, win 127, options [nop,nop,TS val 4149591209 ecr 1030448017], length 1440 22:17:19.152722 IP 87:(IP of my WAN).56571 > 82:(IP of my IONOS Server).40268: Flags [.], seq 1441:2881, ack 319, win 127, options [nop,nop,TS val 4149591209 ecr 1030448017], length 1440 22:17:19.152734 IP 87:(IP of my WAN).56571 > 82:(IP of my IONOS Server).40268: Flags [P.], seq 2881:3623, ack 319, win 127, options [nop,nop,TS val 4149591209 ecr 1030448017], length 742 22:17:19.170552 IP 82:(IP of my IONOS Server).40268 > 87:(IP of my WAN).56571: Flags [.], ack 1441, win 1002, options [nop,nop,TS val1030448052 ecr 4149591209], length 0 22:17:19.171227 IP 82:(IP of my IONOS Server).40268 > 87:(IP of my WAN).56571: Flags [.], ack 2881, win 1002, options [nop,nop,TS val1030448053 ecr 4149591209], length 0 22:17:19.171929 IP 82:(IP of my IONOS Server).40268 > 87:(IP of my WAN).56571: Flags [.], ack 3623, win 1002, options [nop,nop,TS val1030448053 ecr 4149591209], length 0 22:17:19.172845 IP 82:(IP of my IONOS Server).40268 > 87:(IP of my WAN).56571: Flags [P.], seq 319:399, ack 3623, win 1002, options [nop,nop,TS val 1030448054 ecr 4149591209], length 80 22:17:19.172866 IP 87:(IP of my WAN).56571 > 82:(IP of my IONOS Server).40268: Flags [.], ack 399, win 127, options [nop,nop,TS val 4149591229 ecr 1030448054], length 0 22:17:19.173059 IP 87:(IP of my WAN).56571 > 82:(IP of my IONOS Server).40268: Flags [P.], seq 3623:3702, ack 399, win 127, options [nop,nop,TS val 4149591230 ecr 1030448054], length 79 22:17:19.173139 IP 87:(IP of my WAN).56571 > 82:(IP of my IONOS Server).40268: Flags [P.], seq 3702:3781, ack 399, win 127, options [nop,nop,TS val 4149591230 ecr 1030448054], length 79 22:17:19.189451 IP 82:(IP of my IONOS Server).40268 > 87:(IP of my WAN).56571: Flags [P.], seq 399:576, ack 3623, win 1002, options [nop,nop,TS val 1030448071 ecr 4149591229], length 177 22:17:19.189481 IP 87:(IP of my WAN).56571 > 82:(IP of my IONOS Server).40268: Flags [.], ack 576, win 127, options [nop,nop,TS val 4149591247 ecr 1030448071], length 0 22:17:19.190099 IP 82:(IP of my IONOS Server).40268 > 87:(IP of my WAN).56571: Flags [.], ack 3781, win 1002, options [nop,nop,TS val1030448071 ecr 4149591230], length 0 22:17:19.208669 IP 87:(IP of my WAN).56571 > 82:(IP of my IONOS Server).40268: Flags [.], seq 3781:5221, ack 576, win 127, options [nop,nop,TS val 4149591266 ecr 1030448071], length 1440 22:17:19.208701 IP 87:(IP of my WAN).56571 > 82:(IP of my IONOS Server).40268: Flags [.], seq 5221:6661, ack 576, win 127, options [nop,nop,TS val 4149591266 ecr 1030448071], length 1440 22:17:19.208713 IP 87:(IP of my WAN).56571 > 82:(IP of my IONOS Server).40268: Flags [.], seq 6661:8101, ack 576, win 127, options [nop,nop,TS val 4149591266 ecr 1030448071], length 1440 22:17:19.208725 IP 87:(IP of my WAN).56571 > 82:(IP of my IONOS Server).40268: Flags [P.], seq 8101:8206, ack 576, win 127, options [nop,nop,TS val 4149591266 ecr 1030448071], length 105 22:17:19.208805 IP 87:(IP of my WAN).56571 > 82:(IP of my IONOS Server).40268: Flags [P.], seq 8206:9591, ack 576, win 127, options [nop,nop,TS val 4149591266 ecr 1030448071], length 1385 22:17:19.227697 IP 82:(IP of my IONOS Server).40268 > 87:(IP of my WAN).56571: Flags [.], ack 6661, win 1002, options [nop,nop,TS val1030448109 ecr 4149591266], length 0 22:17:19.229938 IP 82:(IP of my IONOS Server).40268 > 87:(IP of my WAN).56571: Flags [.], ack 8206, win 1002, options [nop,nop,TS val1030448111 ecr 4149591266], length 0 22:17:19.231738 IP 82:(IP of my IONOS Server).40268 > 87:(IP of my WAN).56571: Flags [F.], seq 576, ack 9591, win 1002, options [nop,nop,TS val 1030448113 ecr 4149591266], length 0 22:17:19.231759 IP 87:(IP of my WAN).56571 > 82:(IP of my IONOS Server).40268: Flags [.], ack 577, win 127, options [nop,nop,TS val 4149591288 ecr 1030448113], length 0 22:17:19.231813 IP 87:(IP of my WAN).56571 > 82:(IP of my IONOS Server).40268: Flags [P.], seq 9591:9615, ack 577, win 127, options [nop,nop,TS val 4149591288 ecr 1030448113], length 24 22:17:19.231871 IP 87:(IP of my WAN).56571 > 82:(IP of my IONOS Server).40268: Flags [F.], seq 9615, ack 577, win 127, options [nop,nop,TS val 4149591288 ecr 1030448113], length 0 22:17:19.248161 IP 82:(IP of my IONOS Server).40268 > 87:(IP of my WAN).56571: Flags [R], seq 1756100613, win 0, length 0 22:17:19.248183 IP 82:(IP of my IONOS Server).40268 > 87:(IP of my WAN).56571: Flags [R], seq 1756100613, win 0, length 0 [code] Greets Byte
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: johnsmi on May 22, 2020, 11:34:04 pm: There are MSS of 1440 (default) and 1432 + 8Bytes for PPPoE.

Seems like Path MTU Discovery is firewalled.

Can you try permitting ICMP on WAN? At least IPv6-ICMP type "Packet too big".
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: Bytechanger on May 23, 2020, 08:49:22 am: If you mean a firewall rule on WAN allowing ICMP, it´s aleready there as last rule (IPv6 ICMP pass).
Because of this, ping6 is possible to WAN address from outsite.

Greets

Byte
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: Bytechanger on May 23, 2020, 02:02:10 pm: Should I set MTU to 1452 in Interface->WAN?
Or to anything else?
OR MSS to 1452 ?

Greets

Byte
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: Bytechanger on May 24, 2020, 07:45:26 am: Any idea?
Is this a haproxy problem?
How can I check this? Any other traffic to WAN without haproxy?
Need help, please

Greets

Byte
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: johnsmi on May 24, 2020, 01:40:56 pm: Sorry for late reply.

Use MTU 1492

IP4 with (MSS = 1452) + ( 20B TCP + 20B IP4 +8B PPPoE) = MTU 1500 default ethernet.
IP6-Header is 40 Bytes, thus MSS 1432.

With 1500 -8B PPPoE -40B IPv6 -8B ICMP = 1444 packetsize you can
ping6 -c1 -s 1444 example.net
wich passes, while
ping6 -c1 -s 1445 example.net
times out.

Please test which is largest packetsize for ping.

There might be some encapsulating overhad on the route (VXLAN VPN ...)
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: Bytechanger on May 24, 2020, 02:38:15 pm: Thank you for helping.

So now I set MTU on WAN to 1492. (Should this only on pppoe, and on other clear box??)

MSS empty box??

ping6 -c1 -s 1436 2003:(my WAN)

is the highest value, to get an response. Beginning at 1437 there is no answer!

With clear MTU:
I can ping to 1444, on 1445 it recieved an error
icmp_seq=1 Packet too big: mtu=1492

Somtimes packages over or at 1445 are completly loss!

Greets

Byte
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: johnsmi on May 24, 2020, 03:51:05 pm: Yes, leave MSS empty.

We expect the loss at 1445 due to PPPoE.

ok,
setting MTU to 1492 and max packet of 1436 does mean that it is correctly set automatically, i.e. pppoe subtracts its 8 Byte header automatically.
The default of 1500 -8 B pppoe results in max packet of 1444.

So it's fine having MTU blank/default/1500 on the pppoe interface.

I'm setting MTU 1492 everywhere on the LANs with traffic through WAN/PPPoE and JumboFrames on the local-only LAN. So my LAN and DMZ get 1492 and NAS-network is at 9k.

Seems like your webserver doesn't like the reduced MTU? Can you set MTU 1492 directly on its interface.
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: Bytechanger on May 24, 2020, 05:06:52 pm: OK, so what I have to set?

MTU on my OPNSense empty. fine.

I´m on an IONOS vServer on bash/ssh?
It ´s an Ubuntu (Linux version 4.15.0-99-generic (buildd@lcy01-amd64-013) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)) #100-Ubuntu SMP Wed Apr 22 20:32:56 UTC 2020)

hm,
try to change it

sudo ifconfig ens192 mtu 1492

check

ifconfig | grep -i MTU
ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1492
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536

seems to work.
But wget doesn´t work anyway....

Same thing....

Greets

Byte
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: Bytechanger on May 26, 2020, 08:37:32 am: Hm,

when I change my ISP soon with only DSLite, I want to recieve IPv4 on my IONOS Server and forward it over ipv6 to my reverse proxy on opnsense at home!
So I need this to work. I depend on it.

Do you think, it´s WAN MTU problem?
Could it be a problem of including haproxy in opnsense or something else?

I need help.

Greets

Byte
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: abraxxa on May 26, 2020, 11:03:01 am: I have haproxy listening on my PPPoE WAN interface but I have to start it manually after a reboot because the static IPv6 is DHCPv6 assigned on the WAN interface which takes a little bit. Would be nice if haproxy would be reloaded/restart/delayed in case of an interface change.
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: Bytechanger on May 26, 2020, 11:34:18 am: Hi,

can you tell me, how you configured haproxy an opnsense?
[::]:Portnumber ??
MTU-Settings?
etc.

Greets

Byte
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: abraxxa on May 26, 2020, 12:12:02 pm: Just 2001:db8::1:443 in the Listen Addresses field, haven't had to tinker with MTU at all.
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: Bytechanger on May 26, 2020, 01:33:22 pm: Is this the IPv6 Adress of you WAN-Port?

Do you enter the WAN IPv6 Adress on each haProxy-frontend?
On each change of ipv6 (on my ISP often) you have to change all?

This doesn´t work for me. It´s the same, no answer...

Greets

Byte
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: abraxxa on May 26, 2020, 01:51:56 pm: 2001:db8::/32 is the prefix reserved for documentation, but yes, the IPv6 address I've specified there is my WAN IPv6 address assigned by DHCPv6 which is always the same (also because I've added it to the interface config), depends on your ISP though.
I didn't bind to :::443 because internally the OPNsense WebUI listens on that port already.
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: Bytechanger on May 26, 2020, 02:02:08 pm: OK,

when I´m right, you set in Listen Addresses: you WAN IPv6:portnumber. (And its pppoe?)
Firewall open WAN ipv6 portnumber.

Call from outside WAN IPv6 woth portnumber, and it works?
So I don´t know, whats wrong with my setting.

Greets

Byte
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: abraxxa on May 26, 2020, 02:09:22 pm: Reading your fourth post with the firewall log I think you have a misunderstanding.
If haproxy listens on the WAN IPv6 address of your firewall, you need to connect to, not the IPv6 address of the host it reverse-proxies to. So if you have a DNS AAAA record it needs to point to the IPv6 address of your firewall.
Did you check if the IPv6 packets you want haproxy to handle arrive at the WAN interface of the firewall, for example with tcpdump?
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: Bytechanger on May 26, 2020, 02:19:25 pm: I´m testing. I enter real IPv6 and no DNS-Addresses.
I set IPv6-Address from WAN-Interface (firewall), the device, wich is "real server" on haproxy has no ipv6 address!
So I can only put ONE IPv6 Adress to test (WAN/Firewall).
I could also test with IPv6 of my LAN-Interface, but it´s the same.

I testet to call haproxy with ipv6 WAN Address from my LAN, wich works fine.
Otherwhise, when I call it from outsite to WAN, it passes firewall, after that, nothing....
On ipv4 it works fine.
On this post, you can see, there comes data to firewall (see post https://forum.opnsense.org/index.php?topic=17316.msg78835#msg78835)

Greets

Byte
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: abraxxa on May 26, 2020, 02:29:47 pm: What is 'IP of my IONOS Server'? Is this a host on the Internet?
The packet capture shows that the syn packet is answered with a syn-ack but the client sends another syn instead of ack.
Did you also do a packet capture on the client (IP of my IONOS Server)?
Did you try to stop haproxy and do another packet capture to find out if some other process listens on port 56571 of your firewall or maybe a NAT?
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: Bytechanger on May 26, 2020, 03:21:21 pm: OK, Sorry

IONOS Server is a vserver on internet with ipv6 and ipv4.
It has only ubuntu bash. I want to set it up for recieve ipv4 and forward it to ipv6 to haproxy at home.

actually my home has ipv4 and ipv6, but soon ISP change and i get a dslite with only ipv6.

Port 56571 on ipv4 WAN runs best. It works fine.
Port 56571 on ipv6 works fine from LAN, WAN not....

here is a dump from IONOS-Internet-Server when I call

Code: [Select]
wget -O --no-check-certificate https://[2003:(WAN Firewall)]:56573 sudo tcpdump -ni ens192 'tcp port 56573' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes 13:08:18.670174 IP6 2001:(IONOS InternetServer).52114 > 2003:(WAN Firewall).56573: Flags [S], seq 3221730538, win 64440, options [mss 1432,sackOK,TS val 2919176013 ecr 0,nop,wscale 6], length 0 13:08:19.698280 IP6 2001:(IONOS InternetServer).52114 > 2003:(WAN Firewall).56573: Flags [S], seq 3221730538, win 64440, options [mss 1432,sackOK,TS val 2919177042 ecr 0,nop,wscale 6], length 0 13:08:21.714260 IP6 2001:(IONOS InternetServer).52114 > 2003:(WAN Firewall).56573: Flags [S], seq 3221730538, win 64440, options [mss 1432,sackOK,TS val 2919179057 ecr 0,nop,wscale 6], length 0 13:08:25.778291 IP6 2001:(IONOS InternetServer).52114 > 2003:(WAN Firewall).56573: Flags [S], seq 3221730538, win 64440, options [mss 1432,sackOK,TS val 2919183121 ecr 0,nop,wscale 6], length 0 13:08:33.970295 IP6 2001:(IONOS InternetServer).52114 > 2003:(WAN Firewall).56573: Flags [S], seq 3221730538, win 64440, options [mss 1432,sackOK,TS val 2919191313 ecr 0,nop,wscale 6], length 0
I think listen ports is shown over sockstat
Code: [Select]
sudo sockstat -6 USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS www haproxy 76403 6 tcp6 *:56574 *:* www haproxy 76403 7 tcp6 2003:(WAN Firewall):56574 *:* www haproxy 76403 11 tcp6 *:56572 *:* www haproxy 76403 12 tcp6 *:32400 *:* www haproxy 76403 14 tcp6 *:56560 *:* www haproxy 76403 16 tcp6 *:56573 *:* www haproxy 76403 17 tcp6 2003:(WAN Firewall):56573 *:* www haproxy 76403 19 tcp6 *:56571 *:* www haproxy 76403 20 tcp6 ::1:56571 *:* www haproxy 76403 21 tcp6 2003:(WAN Firewall):56573 *:* www haproxy 76403 23 tcp6 *:56570 *:* www haproxy 76403 25 tcp6 *:80 *:* www haproxy 76403 27 tcp6 *:443 *:* www haproxy 76403 29 tcp6 *:56575 *:* root lighttpd 70335 7 tcp6 2003:xxx:4444 *:* root lighttpd 70335 9 tcp6 ::1:4444 *:* root lighttpd 70335 12 tcp6 2003:xxx:80 *:* root lighttpd 70335 14 tcp6 ::1:80 *:* root ntpd 80961 20 udp6 *:123 *:* root ntpd 80961 22 udp6 fe80::xxx:8584%re1:123 *:* root ntpd 80961 25 udp6 2003:xxx:123 *:* root ntpd 80961 26 udp6 ::1:123 *:* root ntpd 80961 28 udp6 fe80:xxx84%re1_vlan100:123 *:* root ntpd 80961 30 udp6 2003:xxx:8584:123 *:* root ntpd 80961 31 udp6 fe80::xxx:8584%re1_vlan50:123 *:* root ntpd 80961 33 udp6 2003:xxx:8584:123 *:* root ntpd 80961 34 udp6 fe80::xxx:8584%re1_vlan77:123 *:* dhcpd dhcpd 29434 7 udp6 *:547 *:* unbound unbound 68368 3 udp6 *:53 *:* unbound unbound 68368 4 tcp6 *:53 *:* rspamd rspamd-2.5 20828 9 udp6 *:* *:* rspamd rspamd-2.5 20828 18 udp6 *:* *:* rspamd rspamd-2.5 20828 21 udp6 *:* *:* rspamd rspamd-2.5 20828 23 udp6 *:* *:* rspamd rspamd-2.5 20828 24 udp6 *:* *:* rspamd rspamd-2.5 20828 25 udp6 *:* *:* rspamd rspamd-2.5 20828 26 udp6 *:* *:* rspamd rspamd-2.5 20828 27 udp6 *:* *:* rspamd rspamd-2.5 20828 28 udp6 *:* *:* rspamd rspamd-2.5 20828 29 udp6 *:* *:* rspamd rspamd-2.5 20828 30 udp6 *:* *:* rspamd rspamd-2.5 20828 31 udp6 *:* *:* rspamd rspamd-2.5 20828 32 udp6 *:* *:* rspamd rspamd-2.5 20828 33 udp6 *:* *:* rspamd rspamd-2.5 20828 34 udp6 *:* *:* rspamd rspamd-2.5 20828 35 udp6 *:* *:* rspamd rspamd-2.5 20828 36 udp6 *:* *:* rspamd rspamd-2.5 20828 37 udp6 *:* *:* rspamd rspamd-2.5 20828 38 udp6 *:* *:* rspamd rspamd-2.5 20828 39 udp6 *:* *:* rspamd rspamd-2.5 20828 40 udp6 *:* *:* rspamd rspamd-2.5 20828 41 udp6 *:* *:* rspamd rspamd-2.5 20828 42 udp6 *:* *:* rspamd rspamd-2.5 20828 43 udp6 *:* *:* rspamd rspamd-2.5 20828 44 udp6 *:* *:* rspamd rspamd-2.5 20828 45 udp6 *:* *:* rspamd rspamd-2.5 20828 46 udp6 *:* *:* rspamd rspamd-2.5 20828 47 udp6 *:* *:* rspamd rspamd-2.5 20828 48 udp6 *:* *:* rspamd rspamd-2.5 20828 49 udp6 *:* *:* rspamd rspamd-2.5 20828 50 udp6 *:* *:* rspamd rspamd-2.5 20828 51 udp6 *:* *:* rspamd rspamd-2.5 85915 9 udp6 *:* *:* rspamd rspamd-2.5 85915 17 udp6 *:* *:* rspamd rspamd-2.5 85915 19 udp6 *:* *:* rspamd rspamd-2.5 85915 21 udp6 *:* *:* rspamd rspamd-2.5 85915 22 udp6 *:* *:* rspamd rspamd-2.5 85915 23 udp6 *:* *:* rspamd rspamd-2.5 85915 24 udp6 *:* *:* rspamd rspamd-2.5 85915 25 udp6 *:* *:* rspamd rspamd-2.5 85915 26 udp6 *:* *:* rspamd rspamd-2.5 85915 27 udp6 *:* *:* rspamd rspamd-2.5 85915 28 udp6 *:* *:* rspamd rspamd-2.5 85915 29 udp6 *:* *:* rspamd rspamd-2.5 85915 30 udp6 *:* *:* rspamd rspamd-2.5 85915 31 udp6 *:* *:* rspamd rspamd-2.5 85915 32 udp6 *:* *:* rspamd rspamd-2.5 85915 33 udp6 *:* *:* rspamd rspamd-2.5 85915 34 udp6 *:* *:* rspamd rspamd-2.5 85915 35 udp6 *:* *:* rspamd rspamd-2.5 85915 36 udp6 *:* *:* rspamd rspamd-2.5 85915 37 udp6 *:* *:* rspamd rspamd-2.5 85915 38 udp6 *:* *:* rspamd rspamd-2.5 85915 39 udp6 *:* *:* rspamd rspamd-2.5 85915 40 udp6 *:* *:* rspamd rspamd-2.5 85915 41 udp6 *:* *:* rspamd rspamd-2.5 85915 42 udp6 *:* *:* rspamd rspamd-2.5 85915 43 udp6 *:* *:* rspamd rspamd-2.5 85915 44 udp6 *:* *:* rspamd rspamd-2.5 85915 45 udp6 *:* *:* rspamd rspamd-2.5 85915 46 udp6 *:* *:* rspamd rspamd-2.5 85915 47 udp6 *:* *:* rspamd rspamd-2.5 85915 48 udp6 *:* *:* rspamd rspamd-2.5 85915 49 udp6 *:* *:* rspamd rspamd-2.5 80404 9 udp6 *:* *:* rspamd rspamd-2.5 80404 14 udp6 *:* *:* rspamd rspamd-2.5 80404 16 udp6 *:* *:* rspamd rspamd-2.5 80404 18 udp6 *:* *:* rspamd rspamd-2.5 80404 19 udp6 *:* *:* rspamd rspamd-2.5 80404 20 udp6 *:* *:* rspamd rspamd-2.5 80404 21 udp6 *:* *:* rspamd rspamd-2.5 80404 22 udp6 *:* *:* rspamd rspamd-2.5 80404 23 udp6 *:* *:* rspamd rspamd-2.5 80404 24 udp6 *:* *:* rspamd rspamd-2.5 80404 25 udp6 *:* *:* rspamd rspamd-2.5 80404 26 udp6 *:* *:* rspamd rspamd-2.5 80404 27 udp6 *:* *:* rspamd rspamd-2.5 80404 28 udp6 *:* *:* rspamd rspamd-2.5 80404 29 udp6 *:* *:* rspamd rspamd-2.5 80404 30 udp6 *:* *:* rspamd rspamd-2.5 80404 31 udp6 *:* *:* rspamd rspamd-2.5 80404 32 udp6 *:* *:* rspamd rspamd-2.5 80404 33 udp6 *:* *:* rspamd rspamd-2.5 80404 34 udp6 *:* *:* rspamd rspamd-2.5 80404 35 udp6 *:* *:* rspamd rspamd-2.5 80404 36 udp6 *:* *:* rspamd rspamd-2.5 80404 37 udp6 *:* *:* rspamd rspamd-2.5 80404 38 udp6 *:* *:* rspamd rspamd-2.5 80404 39 udp6 *:* *:* rspamd rspamd-2.5 80404 40 udp6 *:* *:* rspamd rspamd-2.5 80404 41 udp6 *:* *:* rspamd rspamd-2.5 80404 42 udp6 *:* *:* rspamd rspamd-2.5 80404 43 udp6 *:* *:* rspamd rspamd-2.5 80404 44 udp6 *:* *:* rspamd rspamd-2.5 80404 45 udp6 *:* *:* rspamd rspamd-2.5 80404 46 udp6 *:* *:* rspamd rspamd-2.5 38154 9 udp6 *:* *:* rspamd rspamd-2.5 38154 11 udp6 *:* *:* rspamd rspamd-2.5 38154 13 udp6 *:* *:* rspamd rspamd-2.5 38154 15 udp6 *:* *:* rspamd rspamd-2.5 38154 16 udp6 *:* *:* rspamd rspamd-2.5 38154 17 udp6 *:* *:* rspamd rspamd-2.5 38154 18 udp6 *:* *:* rspamd rspamd-2.5 38154 19 udp6 *:* *:* rspamd rspamd-2.5 38154 20 udp6 *:* *:* rspamd rspamd-2.5 38154 21 udp6 *:* *:* rspamd rspamd-2.5 38154 22 udp6 *:* *:* rspamd rspamd-2.5 38154 23 udp6 *:* *:* rspamd rspamd-2.5 38154 24 udp6 *:* *:* rspamd rspamd-2.5 38154 25 udp6 *:* *:* rspamd rspamd-2.5 38154 26 udp6 *:* *:* rspamd rspamd-2.5 38154 27 udp6 *:* *:* rspamd rspamd-2.5 38154 28 udp6 *:* *:* rspamd rspamd-2.5 38154 29 udp6 *:* *:* rspamd rspamd-2.5 38154 30 udp6 *:* *:* rspamd rspamd-2.5 38154 31 udp6 *:* *:* rspamd rspamd-2.5 38154 32 udp6 *:* *:* rspamd rspamd-2.5 38154 33 udp6 *:* *:* rspamd rspamd-2.5 38154 34 udp6 *:* *:* rspamd rspamd-2.5 38154 35 udp6 *:* *:* rspamd rspamd-2.5 38154 36 udp6 *:* *:* rspamd rspamd-2.5 38154 37 udp6 *:* *:* rspamd rspamd-2.5 38154 38 udp6 *:* *:* rspamd rspamd-2.5 38154 39 udp6 *:* *:* rspamd rspamd-2.5 38154 40 udp6 *:* *:* rspamd rspamd-2.5 38154 41 udp6 *:* *:* rspamd rspamd-2.5 38154 42 udp6 *:* *:* rspamd rspamd-2.5 38154 43 udp6 *:* *:* redis redis-serv 29655 7 tcp6 ::1:6379 *:* root lighttpd 75648 5 tcp6 ::1:43580 *:* root radiusd 95290 12 udp6 *:1812 *:* root radiusd 95290 13 udp6 *:1813 *:* root radiusd 95290 15 udp6 *:62734 *:* root wireguard- 22100 14 udp6 *:56579 *:* root dhcp6c 9921 5 udp6 *:546 *:* root charon 87168 11 udp6 *:500 *:* root charon 87168 12 udp6 *:4500 *:* root sshd 16372 3 tcp6 ::1:56561 *:* ? ? ? ? tcp6 ::1:6379 ::1:42573 ? ? ? ? tcp6 ::1:6379 ::1:19843 ? ? ? ? tcp6 ::1:6379 ::1:65383 ? ? ? ? tcp6 ::1:6379 ::1:7437 ? ? ? ? tcp6 ::1:6379 ::1:17283 ? ? ? ? tcp6 ::1:53097 ::1:6379 ? ? ? ? tcp6 ::1:3405 ::1:6379 ? ? ? ? tcp6 ::1:47959 ::1:6379 ? ? ? ? tcp6 ::1:59771 ::1:6379 ? ? ? ? tcp6 ::1:57391 ::1:6379 ? ? ? ? tcp6 ::1:32496 ::1:6379
Greets

Byte
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: abraxxa on May 26, 2020, 05:42:23 pm: Port 56571 or 56573?
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: Bytechanger on May 26, 2020, 05:57:23 pm: I'm trying both.
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: Maurice on May 26, 2020, 07:12:54 pm: To approach this from a different angle: Can you access any IPv6 service running on OPNsense from the Internet? VPN, SSH, ...?
If you're not sure, you could e. g. allow SSH access from the Internet for testing.
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: Bytechanger on May 27, 2020, 08:15:18 am: OK, you are right.

It seems to be an problem with opnsense/pppoe, not haproxy.

On InternetServer I can reach ipv6
Code: [Select]
[b]dig AAAA +short www.heise.de[/b] 2a02:2e0:3fe:1001:7777:772e:2:85 [b]wget --no-check-certificate https://[2a02:2e0:3fe:1001:7777:772e:2:85][/b] --2020-05-27 05:48:12-- https://[2a02:2e0:3fe:1001:7777:772e:2:85]/ Connecting to [2a02:2e0:3fe:1001:7777:772e:2:85]:443... connected. WARNING: certificate common name ‘www.heise.de’ doesn't match requested host name ‘2a02:2e0:3fe:1001:7777:772e:2:85’. HTTP request sent, awaiting response... 200 OK Length: 76 [text/plain] Saving to: ‘index.html.6’ index.html.6 100%[===============================================>] 76 --.-KB/s in 0s 2020-05-27 05:48:13 (10.8 MB/s) - ‘index.html.6’ saved [76/76] [b]works fine[/b]
SSH to OPNSense over PPPOE works over ipv4 but also not over ipv6:
Code: [Select]
ssh -i /home/blabla/.ssh/homekey -p 56561 -vvv testuser@2003:(WAN address) OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n 7 Dec 2017 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 19: Applying options for * debug2: resolving "2003:(WAN address)" port 56561 debug2: ssh_connect_direct: needpriv 0 debug1: Connecting to 2003:xxx [2003:(WAN address)] port 56561. nothing....
on client, try to connect
Code: [Select]
sudo tcpdump -ni ens192 'tcp port 56561' [sudo] password for blabla: tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes 05:54:19.387975 IP6 2001:(IP Client).54516 > 2003:(WAN Firewall).56561: Flags [S], seq 1051216040, win 64800, options [mss 1440,sackOK,TS val 2362155641 ecr 0,nop,wscale 6], length 0 05:54:20.396577 IP6 2001:(IP Client).54516 > 2003:(WAN Firewall).56561: Flags [S], seq 1051216040, win 64800, options [mss 1440,sackOK,TS val 2362156649 ecr 0,nop,wscale 6], length 0 05:54:22.412581 IP6 2001:(IP Client).54516 > 2003:(WAN Firewall).56561: Flags [S], seq 1051216040, win 64800, options [mss 1440,sackOK,TS val 2362158665 ecr 0,nop,wscale 6], length 0 05:54:26.604603 IP6 2001:(IP Client).54516 > 2003:(WAN Firewall).56561: Flags [S], seq 1051216040, win 64800, options [mss 1440,sackOK,TS val 2362162857 ecr 0,nop,wscale 6], length 0 05:54:34.796572 IP6 2001:(IP Client).54516 > 2003:(WAN Firewall).56561: Flags [S], seq 1051216040, win 64800, options [mss 1440,sackOK,TS val 2362171049 ecr 0,nop,wscale 6], length 0 --------------------------- sudo tcpdump -vv -ni ens192 'tcp port 56561' tcpdump: listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes 06:27:20.701131 IP6 (flowlabel 0xd3867, hlim 64, next-header TCP (6) payload length: 40) 2001:xxx.54520 > 2003:xxx.56561: Flags [S], cksum 0xcbfd (incorrect -> 0xc709), seq 1683830560, win 64800, options [mss 1440,sackOK,TS val 2364136905 ecr 0,nop,wscale 6], length 0 06:27:21.708591 IP6 (flowlabel 0xf0fdf, hlim 64, next-header TCP (6) payload length: 40) 2001:xxx.54520 > 2003:xxx.56561: Flags [S], cksum 0xcbfd (incorrect -> 0xc31a), seq 1683830560, win 64800, options [mss 1440,sackOK,TS val 2364137912 ecr 0,nop,wscale 6], length 0
on OPNSense-Firewall
Code: [Select]
sudo tcpdump -ni pppoe0 'tcp port 56561' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on pppoe0, link-type NULL (BSD loopback), capture size 262144 bytes 07:54:19.396907 IP6 2003:(IP Client).54516 > 2003:(FIRST PART of IP OPNSense WAN) e92:8583.56561: Flags [S], seq 1051216040, win 64800, options [mss 1440,sackO K,TS val 2362155641 ecr 0,nop,wscale 6], length 0 07:54:19.396972 IP6 2003:(IP OPNSense WAN).56561 > 2001:(FIRST PART of IP Client) :803c::1.54516: Flags [S.], seq 1796056796, ack 1051216041, win 65228, option s [mss 1432,nop,wscale 9,sackOK,TS val 1751506195 ecr 2362155641], length 0 07:54:20.405540 IP6 2003:(IP Client).54516 > 2003:(FIRST PART of IP OPNSense WAN) e92:8583.56561: Flags [S], seq 1051216040, win 64800, options [mss 1440,sackO K,TS val 2362156649 ecr 0,nop,wscale 6], length 0 07:54:20.405579 IP6 2003:(IP OPNSense WAN).56561 > 2001:(FIRST PART of IP Client) :803c::1.54516: Flags [S.], seq 1796056796, ack 1051216041, win 65228, option s [mss 1432,nop,wscale 9,sackOK,TS val 1751506195 ecr 2362156649], length 0 07:54:22.421526 IP6 2003:(IP Client).54516 > 2003:(FIRST PART of IP OPNSense WAN) e92:8583.56561: Flags [S], seq 1051216040, win 64800, options [mss 1440,sackO K,TS val 2362158665 ecr 0,nop,wscale 6], length 0 07:54:22.421564 IP6 2003:(IP OPNSense WAN).56561 > 2001:(FIRST PART of IP Client) :803c::1.54516: Flags [S.], seq 1796056796, ack 1051216041, win 65228, option s [mss 1432,nop,wscale 9,sackOK,TS val 1751506195 ecr 2362158665], length 0 07:54:25.427714 IP6 2003:(IP OPNSense WAN).56561 > 2001:(FIRST PART of IP Client) :803c::1.54516: Flags [S.], seq 1796056796, ack 1051216041, win 65228, option s [mss 1432,nop,wscale 9,sackOK,TS val 1751506195 ecr 2362158665], length 0 07:54:26.613695 IP6 2003:(IP Client).54516 > 2003:(FIRST PART of IP OPNSense WAN) e92:8583.56561: Flags [S], seq 1051216040, win 64800, options [mss 1440,sackO K,TS val 2362162857 ecr 0,nop,wscale 6], length 0 07:54:26.613735 IP6 2003:(IP OPNSense WAN).56561 > 2001:(FIRST PART of IP Client) :803c::1.54516: Flags [S.], seq 1796056796, ack 1051216041, win 65228, option s [mss 1432,nop,wscale 9,sackOK,TS val 1751506195 ecr 2362162857], length 0 07:54:29.613738 IP6 2003:(IP OPNSense WAN).56561 > 2001:(FIRST PART of IP Client) :803c::1.54516: Flags [S.], seq 1796056796, ack 1051216041, win 65228, option s [mss 1432,nop,wscale 9,sackOK,TS val 1751506195 ecr 2362162857], length 0 07:54:32.867815 IP6 2003:(IP OPNSense WAN).56561 > 2001:(FIRST PART of IP Client) :803c::1.54516: Flags [S.], seq 1796056796, ack 1051216041, win 65228, option s [mss 1432,nop,wscale 9,sackOK,TS val 1751506195 ecr 2362162857], length 0 07:54:34.805559 IP6 2003:(IP Client).54516 > 2003:(FIRST PART of IP OPNSense WAN) e92:8583.56561: Flags [S], seq 1051216040, win 64800, options [mss 1440,sackO K,TS val 2362171049 ecr 0,nop,wscale 6], length 0 07:54:34.805592 IP6 2003:(IP OPNSense WAN).56561 > 2001:(FIRST PART of IP Client) :803c::1.54516: Flags [S.], seq 1796056796, ack 1051216041, win 65228, option s [mss 1432,nop,wscale 9,sackOK,TS val 1751506195 ecr 2362171049], length 0 07:54:37.805475 IP6 2003:(IP OPNSense WAN).56561 > 2001:(FIRST PART of IP Client) :803c::1.54516: Flags [S.], seq 1796056796, ack 1051216041, win 65228, option s [mss 1432,nop,wscale 9,sackOK,TS val 1751506195 ecr 2362171049], length 0 07:54:41.006098 IP6 2003:(IP OPNSense WAN).56561 > 2001:(FIRST PART of IP Client) :803c::1.54516: Flags [S.], seq 1796056796, ack 1051216041, win 65228, option s [mss 1432,nop,wscale 9,sackOK,TS val 1751506195 ecr 2362171049], length 0 07:54:44.205453 IP6 2003:(IP OPNSense WAN).56561 > 2001:(FIRST PART of IP Client) :803c::1.54516: Flags [S.], seq 1796056796, ack 1051216041, win 65228, option s [mss 1432,nop,wscale 9,sackOK,TS val 1751506195 ecr 2362171049], length 0 ------------ sudo tcpdump -vv -ni pppoe0 'tcp port 56561' tcpdump: listening on pppoe0, link-type NULL (BSD loopback), capture size 262144 bytes 08:27:20.709034 IP6 (flowlabel 0xd3867, hlim 57, next-header TCP (6) payload length: 40) 2001:xxx.54520 > 2003:xxx.56561: Flags [S], cksum 0xc709 (correct), seq 1683830560, win 64800, options [mss 1440,sackOK,TS val 2364136905 ecr 0,nop,wscale 6], length 0 08:27:20.709120 IP6 (flowlabel 0x3245d, hlim 63, next-header TCP (6) payload length: 40) 2003:xxx.56561 > 2001:xxx.54520: Flags [S.], cksum 0xcbfd (incorrect -> 0xf118), seq 3931317341, ack 1683830561, win 65228, options [mss 1432,nop,wscale 9,sackOK,TS val 472230242 ecr 2364136905], length 0 08:27:21.739742 IP6 (flowlabel 0xf0fdf, hlim 57, next-header TCP (6) payload length: 40) 2001:xxx.54520 > 2003:xxx.56561: Flags [S], cksum 0xc31a (correct), seq 1683830560, win 64800, options [mss 1440,sackOK,TS val 2364137912 ecr 0,nop,wscale 6], length 0 08:27:21.739780 IP6 (flowlabel 0x3245d, hlim 63, next-header TCP (6) payload length: 40) 2003:xxx.56561 > 2001:xxx.54520: Flags [S.], cksum 0xcbfd (incorrect -> 0xed29), seq 3931317341, ack 1683830561, win 65228, options [mss 1432,nop,wscale 9,sackOK,TS val 472230242 ecr 2364137912], length 0
So, I don´t know what´s my problem....
It seems, ipv6 doesn´t work from Internet to WAN (over pppoe), but otherwhise from LAN to Internet works fine.

In dumps there is somthing like checksum incorrect?!

Greets

Byte
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: Maurice on May 27, 2020, 03:07:40 pm: Another user reported similar issues with a Deutsche Telekom DSL line on the German forum:
IPv6 auf WAN nicht erreichbar (https://forum.opnsense.org/index.php?topic=16825)
Maybe you can work on this together. I don't have a PPPoE line so I can only guess.
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: Bytechanger on May 27, 2020, 03:56:09 pm: Thanks,

seems it could be the same problem.
I´m also telekom. But robgnu writes on pfsense it works.
So is it possible, it´s a opnsense bug?

Greets

Byte
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: Bytechanger on May 27, 2020, 04:08:00 pm: OK, It seems, there is something special crazy?!

When I disable automatic created Gateway in

Gateway->Single->WAN_DHCP6
WAN_DHCP6 WAN IPv6 254 fe80::f6b5:2fff:fef0:a2eb

(crazyly it has a link-local address)

it works, I can acces ssh from outside!!
So I think, there is something wrong with opnsense ?!?!

Greets

Byte
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: abraxxa on May 27, 2020, 04:31:56 pm: IPv6 default gateways are always link-local addresses (at least if your product follows the RfCs).

Check the destination layer 2 fields of your outgoing IPv6 packets and compare the syn-ack of an incoming connection with a regular packet going through the firewall.
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: Bytechanger on May 28, 2020, 06:10:45 am: Thanks,
but what/how exactly should I do?

Greets

Byte
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: abraxxa on May 28, 2020, 09:20:07 am: Compare the destination mac address of IPv6 packets from your network to the Internet with syn-ack response packets when accessing a service on the firewall.
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: Bytechanger on May 28, 2020, 09:53:01 am: OK, can you tell me the command for this please?

with
sudo tcpdump -eni pppoe0 'tcp port 56561'
on OPNSense, I see only ipv6 no mac?!
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: abraxxa on May 28, 2020, 10:01:15 am: Just the tcpdump you already did before but write it to a file using -w filename.pcap and then copy it to your PC and load it in Wireshark.
Alternatively you can use the -e flag.
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: Bytechanger on May 28, 2020, 12:42:15 pm: I test it, on OPNSense -e doesn´t show MACs, only ipv6 adresses....
On pcap file also??

Code: [Select]
sudo tcpdump -eni pppoe0 'tcp port 56561' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on pppoe0, link-type NULL (BSD loopback), capture size 262144 bytes 12:30:50.888384 AF IPv6 (28), length 84: 2001:(IP of InternetServer).49268 > 2003:(IP of WAN OPNSense).56561: Flags [S], seq 459488888, win 64800, options [mss 1440,sackOK,TS val 3410368836 ecr 0,nop,wscale 6], length 0 12:30:50.888496 AF IPv6 (28), length 84: 2003:(IP of WAN OPNSense).56561 > 2001:(IP of InternetServer).49268: Flags [S.], seq 2539208736, ack 459488889, win 65228, options [mss 1432,nop,wscale 9,sackOK,TS val 2303200564 ecr 3410368836], length 0 12:30:50.906850 AF IPv6 (28), length 76: 2001:(IP of InternetServer).49268 > 2003:(IP of WAN OPNSense).56561: Flags [.], ack 1, win 1013, options [nop,nop,TS val 3410368854 ecr 2303200564], length 0 12:30:50.907048 AF IPv6 (28), length 117: 2001:(IP of InternetServer).49268 > 2003:(IP of WAN OPNSense).56561: Flags [P.], seq 1:42, ack 1, win 1013, options [nop,nop,TS val 3410368855 ecr 2303200564], length 41 12:30:50.907064 AF IPv6 (28), length 76: 2003:(IP of WAN OPNSense).56561 > 2001:(IP of InternetServer).49268: Flags [.], ack 42, win 128, options [nop,nop,TS val 2303200583 ecr 3410368855], length 0 12:30:50.920355 AF IPv6 (28), length 133: 2003:(IP of WAN OPNSense).56561 > 2001:(IP of InternetServer).49268: Flags [P.], seq 1:58, ack 42, win 128, options [nop,nop,TS val 2303200596 ecr 3410368855], length 57 12:30:50.938217 AF IPv6 (28), length 76: 2001:(IP of InternetServer).49268 > 2003:(IP of WAN OPNSense).56561: Flags [.], ack 58, win 1013, options [nop,nop,TS val 3410368886 ecr 2303200596], length 0 12:30:50.938242 AF IPv6 (28), length 1132: 2003:(IP of WAN OPNSense).56561 > 2001:(IP of InternetServer).49268: Flags [P.], seq 58:1114, ack 42, win 128, options [nop,nop,TS val 2303200614 ecr 3410368886], length 1056 12:30:50.938759 AF IPv6 (28), length 1436: 2001:(IP of InternetServer).49268 > 2003:(IP of WAN OPNSense).56561: Flags [P.], seq 42:1402, ack 58, win 1013, options [nop,nop,TS val 3410368886 ecr 2303200596], length 1360 12:30:50.938778 AF IPv6 (28), length 76: 2003:(IP of WAN OPNSense).56561 > 2001:(IP of InternetServer).49268: Flags [.], ack 1402, win 126, options [nop,nop,TS val 2303200614 ecr 3410368886], length 0 12:30:50.959396 AF IPv6 (28), length 124: 2001:(IP of InternetServer).49268 > 2003:(IP of WAN OPNSense).56561: Flags [P.], seq 1402:1450, ack 1114, win 1002, options [nop,nop,TS val 3410368907 ecr 2303200614], length 48 12:30:50.959418 AF IPv6 (28), length 76: 2003:(IP of WAN OPNSense).56561 > 2001:(IP of InternetServer).49268: Flags [.], ack 1450, win 128, options [nop,nop,TS val 2303200635 ecr 3410368907], length 0 12:30:50.967390 AF IPv6 (28), length 584: 2003:(IP of WAN OPNSense).56561 > 2001:(IP of InternetServer).49268: Flags [P.], seq 1114:1622, ack 1450, win 128, options [nop,nop,TS val 2303200643 ecr 3410368907], length 508 12:30:50.989428 AF IPv6 (28), length 92: 2001:(IP of InternetServer).49268 > 2003:(IP of WAN OPNSense).56561: Flags [P.], seq 1450:1466, ack 1622, win 1002, options [nop,nop,TS val 3410368937 ecr 2303200643], length 16 12:30:50.989449 AF IPv6 (28), length 76: 2003:(IP of WAN OPNSense).56561 > 2001:(IP of InternetServer).49268: Flags [.], ack 1466, win 128, options [nop,nop,TS val 2303200665 ecr 3410368937], length 0 12:30:51.007402 AF IPv6 (28), length 120: 2001:(IP of InternetServer).49268 > 2003:(IP of WAN OPNSense).56561: Flags [P.], seq 1466:1510, ack 1622, win 1002, options [nop,nop,TS val 3410368955 ecr 2303200665], length 44 12:30:51.007444 AF IPv6 (28), length 76: 2003:(IP of WAN OPNSense).56561 > 2001:(IP of InternetServer).49268: Flags [.], ack 1510, win 128, options [nop,nop,TS val 2303200683 ecr 3410368955], length 0 12:30:51.007523 AF IPv6 (28), length 120: 2003:(IP of WAN OPNSense).56561 > 2001:(IP of InternetServer).49268: Flags [P.], seq 1622:1666, ack 1510, win 128, options [nop,nop,TS val 2303200683 ecr 3410368955], length 44 12:30:51.025398 AF IPv6 (28), length 144: 2001:(IP of InternetServer).49268 > 2003:(IP of WAN OPNSense).56561: Flags [P.], seq 1510:1578, ack 1666, win 1002, options [nop,nop,TS val 3410368973 ecr 2303200683], length 68 12:30:51.025441 AF IPv6 (28), length 76: 2003:(IP of WAN OPNSense).56561 > 2001:(IP of InternetServer).49268: Flags [.], ack 1578, win 128, options [nop,nop,TS val 2303200701 ecr 3410368973], length 0 12:30:51.030829 AF IPv6 (28), length 120: 2003:(IP of WAN OPNSense).56561 > 2001:(IP of InternetServer).49268: Flags [P.], seq 1666:1710, ack 1578, win 128, options [nop,nop,TS val 2303200707 ecr 3410368973], length 44 12:30:51.048840 AF IPv6 (28), length 216: 2001:(IP of InternetServer).49268 > 2003:(IP of WAN OPNSense).56561: Flags [P.], seq 1578:1718, ack 1710, win 1002, options [nop,nop,TS val 3410368997 ecr 2303200707], length 140 12:30:51.048861 AF IPv6 (28), length 76: 2003:(IP of WAN OPNSense).56561 > 2001:(IP of InternetServer).49268: Flags [.], ack 1718, win 128, options [nop,nop,TS val 2303200725 ecr 3410368997], length 0 12:30:51.049712 AF IPv6 (28), length 176: 2003:(IP of WAN OPNSense).56561 > 2001:(IP of InternetServer).49268: Flags [P.], seq 1710:1810, ack 1718, win 128, options [nop,nop,TS val 2303200725 ecr 3410368997], length 100 12:30:51.069546 AF IPv6 (28), length 304: 2001:(IP of InternetServer).49268 > 2003:(IP of WAN OPNSense).56561: Flags [P.], seq 1718:1946, ack 1810, win 1002, options [nop,nop,TS val 3410369017 ecr 2303200725], length 228 12:30:51.069566 AF IPv6 (28), length 76: 2003:(IP of WAN OPNSense).56561 > 2001:(IP of InternetServer).49268: Flags [.], ack 1946, win 128, options [nop,nop,TS val 2303200745 ecr 3410369017], length 0 12:30:51.075459 AF IPv6 (28), length 104: 2003:(IP of WAN OPNSense).56561 > 2001:(IP of InternetServer).49268: Flags [P.], seq 1810:1838, ack 1946, win 128, options [nop,nop,TS val 2303200751 ecr 3410369017], length 28 12:30:51.093479 AF IPv6 (28), length 188: 2001:(IP of InternetServer).49268 > 2003:(IP of WAN OPNSense).56561: Flags [P.], seq 1946:2058, ack 1838, win 1002, options [nop,nop,TS val 3410369041 ecr 2303200751], length 112 12:30:51.093550 AF IPv6 (28), length 872: 2003:(IP of WAN OPNSense).56561 > 2001:(IP of InternetServer).49268: Flags [P.], seq 1838:2634, ack 2058, win 128, options [nop,nop,TS val 2303200769 ecr 3410369041], length 796 12:30:51.155121 AF IPv6 (28), length 76: 2001:(IP of InternetServer).49268 > 2003:(IP of WAN OPNSense).56561: Flags [.], ack 2634, win 1002, options [nop,nop,TS val 3410369103 ecr 2303200769], length 0 12:30:51.155143 AF IPv6 (28), length 120: 2003:(IP of WAN OPNSense).56561 > 2001:(IP of InternetServer).49268: Flags [P.], seq 2634:2678, ack 2058, win 128, options [nop,nop,TS val 2303200831 ecr 3410369103], length 44 12:30:51.173341 AF IPv6 (28), length 76: 2001:(IP of InternetServer).49268 > 2003:(IP of WAN OPNSense).56561: Flags [.], ack 2678, win 1002, options [nop,nop,TS val 3410369121 ecr 2303200831], length 0 12:30:51.173376 AF IPv6 (28), length 520: 2001:(IP of InternetServer).49268 > 2003:(IP of WAN OPNSense).56561: Flags [P.], seq 2058:2502, ack 2678, win 1002, options [nop,nop,TS val 3410369121 ecr 2303200831], length 444 12:30:51.173390 AF IPv6 (28), length 76: 2003:(IP of WAN OPNSense).56561 > 2001:(IP of InternetServer).49268: Flags [.], ack 2502, win 127, options [nop,nop,TS val 2303200849 ecr 3410369121], length 0 12:30:51.175004 AF IPv6 (28), length 184: 2003:(IP of WAN OPNSense).56561 > 2001:(IP of InternetServer).49268: Flags [P.], seq 2678:2786, ack 2502, win 128, options [nop,nop,TS val 2303200850 ecr 3410369121], length 108 12:30:51.175154 AF IPv6 (28), length 176: 2003:(IP of WAN OPNSense).56561 > 2001:(IP of InternetServer).49268: Flags [P.], seq 2786:2886, ack 2502, win 128, options [nop,nop,TS val 2303200850 ecr 3410369121], length 100 12:30:51.175294 AF IPv6 (28), length 440: 2003:(IP of WAN OPNSense).56561 > 2001:(IP of InternetServer).49268: Flags [P.], seq 2886:3250, ack 2502, win 128, options [nop,nop,TS val 2303200851 ecr 3410369121], length 364 12:30:51.175348 AF IPv6 (28), length 360: 2003:(IP of WAN OPNSense).56561 > 2001:(IP of InternetServer).49268: Flags [P.], seq 3250:3534, ack 2502, win 128, options [nop,nop,TS val 2303200851 ecr 3410369121], length 284 12:30:51.182243 AF IPv6 (28), length 112: 2003:(IP of WAN OPNSense).56561 > 2001:(IP of InternetServer).49268: Flags [P.], seq 3534:3570, ack 2502, win 128, options [nop,nop,TS val 2303200858 ecr 3410369121], length 36 12:30:51.193282 AF IPv6 (28), length 76: 2001:(IP of InternetServer).49268 > 2003:(IP of WAN OPNSense).56561: Flags [.], ack 2886, win 1002, options [nop,nop,TS val 3410369141 ecr 2303200850], length 0 12:30:51.193733 AF IPv6 (28), length 76: 2001:(IP of InternetServer).49268 > 2003:(IP of WAN OPNSense).56561: Flags [.], ack 3534, win 1002, options [nop,nop,TS val 3410369142 ecr 2303200851], length 0 12:30:51.242917 AF IPv6 (28), length 76: 2001:(IP of InternetServer).49268 > 2003:(IP of WAN OPNSense).56561: Flags [.], ack 3570, win 1002, options [nop,nop,TS val 3410369191 ecr 2303200858], length 0 12:31:08.458459 AF IPv6 (28), length 112: 2001:(IP of InternetServer).49268 > 2003:(IP of WAN OPNSense).56561: Flags [P.], seq 2502:2538, ack 3570, win 1002, options [nop,nop,TS val 3410386406 ecr 2303200858], length 36 12:31:08.458506 AF IPv6 (28), length 76: 2003:(IP of WAN OPNSense).56561 > 2001:(IP of InternetServer).49268: Flags [.], ack 2538, win 128, options [nop,nop,TS val 2303218134 ecr 3410386406], length 0 12:31:08.458883 AF IPv6 (28), length 112: 2003:(IP of WAN OPNSense).56561 > 2001:(IP of InternetServer).49268: Flags [P.], seq 3570:3606, ack 2538, win 128, options [nop,nop,TS val 2303218134 ecr 3410386406], length 36 12:31:08.458978 AF IPv6 (28), length 112: 2003:(IP of WAN OPNSense).56561 > 2001:(IP of InternetServer).49268: Flags [P.], seq 3606:3642, ack 2538, win 128, options [nop,nop,TS val 2303218134 ecr 3410386406], length 36 12:31:08.476409 AF IPv6 (28), length 76: 2001:(IP of InternetServer).49268 > 2003:(IP of WAN OPNSense).56561: Flags [.], ack 3606, win 1002, options [nop,nop,TS val 3410386424 ecr 2303218134], length 0 12:31:08.476645 AF IPv6 (28), length 76: 2001:(IP of InternetServer).49268 > 2003:(IP of WAN OPNSense).56561: Flags [.], ack 3642, win 1002, options [nop,nop,TS val 3410386424 ecr 2303218134], length 0
OK, with sudo tcpdump -w test3.pcap
I get a looooot of data and don´t know, what to do...

Greets

Byte
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: robgnu on May 28, 2020, 07:39:25 pm: Hello,

I am the user from the german forum with the same issue:
I have two separate OPNsense boxes with 20.1.7 connected by PPPoE (VDSL, Deutsche Telekom). Both machines do exactly the same:
- Connecting to the ISP works perfect. MTU was not set, but calculated to 1492, which is correct.
- All PCs behind the Firewall get clean IPv6 connections and traffic to the internet.
- One machine BEHIND the firewall is reachable from the internet (SSH; I configured the according rule)
- IPv6 connections from the Internet to the WAN port of the OPNsense are getting a timeout (SSH, HTTPS etc.)
- IPv4 connections from the Internet to the WAN port of the OPNsense working as usual and fast.
- Both OPNsense machines were fresh installed with 20.1.0, then I configured the PPPoE connection and then I upgraded to the latest version.

I figured out, that disabling the Gateway "WAN_DHCP6" solves that issue. On one machine I have now a continuously restarting radvd daemon (every 2 seconds) which is another IPv6 problem, I think.

I logged into these two machines an did a "clog -f /var/log/system.log". There is a massive flooding of messages (both OPNsenses). In parallel I did the same on another pfSense, there aren't those messages:

Code: [Select]
May 28 19:31:12 sense dhcp6c[67533]: script "/var/etc/dhcp6c_wan_script.sh" terminated May 28 19:31:12 sense dhcp6c[67533]: removing an event on pppoe0, state=REQUEST May 28 19:31:12 sense dhcp6c[67533]: removing server (ID: 00:02:00:00:05:83:34:30:3a:37:31:3a:38:33:3a:61:38:3a:63:38:3a:30:30:00:00:00) May 28 19:31:12 sense dhcp6c[67533]: got an expected reply, sleeping. May 28 19:31:12 sense dhcp6c[67533]: Sending Solicit May 28 19:31:12 sense dhcp6c[67533]: a new XID (15754b) is generated May 28 19:31:12 sense dhcp6c[67533]: set client ID (len 14) May 28 19:31:12 sense dhcp6c[67533]: set identity association May 28 19:31:12 sense dhcp6c[67533]: set elapsed time (len 2) May 28 19:31:12 sense dhcp6c[67533]: set option request (len 4) May 28 19:31:12 sense dhcp6c[67533]: send solicit to ff02::1:2%pppoe0 May 28 19:31:12 sense dhcp6c[67533]: reset a timer on pppoe0, state=SOLICIT, timeo=0, retrans=1016 May 28 19:31:12 sense dhcp6c[67533]: receive advertise from fe80::200:ff:fe00:0%pppoe0 on pppoe0 May 28 19:31:12 sense dhcp6c[67533]: get DHCP option client ID, len 14 May 28 19:31:12 sense dhcp6c[67533]: DUID: 00:01:00:01:21:39:d6:87:00:0d:b9:54:2b:6c May 28 19:31:12 sense dhcp6c[67533]: get DHCP option server ID, len 26 May 28 19:31:12 sense dhcp6c[67533]: DUID: 00:02:00:00:05:83:34:30:3a:37:31:3a:38:33:3a:61:38:3a:63:38:3a:30:30:00:00:00 May 28 19:31:12 sense dhcp6c[67533]: get DHCP option identity association, len 59 May 28 19:31:12 sense dhcp6c[67533]: IA_NA: ID=0, T1=0, T2=0 May 28 19:31:12 sense dhcp6c[67533]: get DHCP option status code, len 43 May 28 19:31:12 sense dhcp6c[67533]: status code: no addresses May 28 19:31:12 sense dhcp6c[67533]: get DHCP option DNS, len 32 May 28 19:31:12 sense dhcp6c[67533]: server ID: 00:02:00:00:05:83:34:30:3a:37:31:3a:38:33:3a:61:38:3a:63:38:3a:30:30:00:00:00, pref=-1 May 28 19:31:12 sense dhcp6c[67533]: reset timer for pppoe0 to 0.979450 May 28 19:31:13 sense dhcp6c[67533]: picked a server (ID: 00:02:00:00:05:83:34:30:3a:37:31:3a:38:33:3a:61:38:3a:63:38:3a:30:30:00:00:00) May 28 19:31:13 sense dhcp6c[67533]: Sending Request May 28 19:31:13 sense dhcp6c[67533]: a new XID (ac85ba) is generated May 28 19:31:13 sense dhcp6c[67533]: set client ID (len 14) May 28 19:31:13 sense dhcp6c[67533]: set server ID (len 26) May 28 19:31:13 sense dhcp6c[67533]: set status code May 28 19:31:13 sense dhcp6c[67533]: set identity association May 28 19:31:13 sense dhcp6c[67533]: set elapsed time (len 2) May 28 19:31:13 sense dhcp6c[67533]: set option request (len 4) May 28 19:31:13 sense dhcp6c[67533]: send request to ff02::1:2%pppoe0 May 28 19:31:13 sense dhcp6c[67533]: reset a timer on pppoe0, state=REQUEST, timeo=0, retrans=1059 May 28 19:31:13 sense dhcp6c[67533]: receive reply from fe80::200:ff:fe00:0%pppoe0 on pppoe0 May 28 19:31:13 sense dhcp6c[67533]: get DHCP option client ID, len 14 May 28 19:31:13 sense dhcp6c[67533]: DUID: 00:01:00:01:21:39:d6:87:00:0d:b9:54:2b:6c May 28 19:31:13 sense dhcp6c[67533]: get DHCP option server ID, len 26 May 28 19:31:13 sense dhcp6c[67533]: DUID: 00:02:00:00:05:83:34:30:3a:37:31:3a:38:33:3a:61:38:3a:63:38:3a:30:30:00:00:00 May 28 19:31:13 sense dhcp6c[67533]: get DHCP option identity association, len 59 May 28 19:31:13 sense dhcp6c[67533]: IA_NA: ID=0, T1=0, T2=0 May 28 19:31:13 sense dhcp6c[67533]: get DHCP option status code, len 43 May 28 19:31:13 sense dhcp6c[67533]: status code: no addresses May 28 19:31:13 sense dhcp6c[67533]: get DHCP option DNS, len 32 May 28 19:31:13 sense dhcp6c[67533]: Received REPLY for REQUEST May 28 19:31:13 sense dhcp6c[67533]: nameserver[0] 2003:180:2:7000::53 May 28 19:31:13 sense dhcp6c[67533]: nameserver[1] 2003:180:2:9000::53 May 28 19:31:13 sense dhcp6c[67533]: make an IA: NA-0 May 28 19:31:13 sense dhcp6c[67533]: status code for NA-0: no addresses May 28 19:31:13 sense dhcp6c[67533]: IA NA-0 is invalidated May 28 19:31:13 sense dhcp6c[67533]: remove an IA: NA-0 May 28 19:31:13 sense dhcp6c[67533]: reset a timer on pppoe0, state=INIT, timeo=0, retrans=113 May 28 19:31:13 sense dhcp6c[67533]: executes /var/etc/dhcp6c_wan_script.sh May 28 19:31:13 sense dhcp6c: dhcp6c REQUEST on pppoe0 - running newipv6 May 28 19:31:14 sense opnsense: /usr/local/etc/rc.newwanipv6: IPv6 renewal is starting on 'pppoe0' May 28 19:31:14 sense opnsense: /usr/local/etc/rc.newwanipv6: On (IP address: 2003:a:xxxx:xxxx:20d:b9ff:fe54:2b6c) (interface: WAN[wan]) (real interface: pppoe0).
For now, I'm out of ideas. I switched over from pfSense for some other good reasons, but I never had any problems with IPv6 on pfSense.

Robert.
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: robgnu on May 28, 2020, 09:16:21 pm: Hello,

I did some more investigation this evening. One post in this forum (https://forum.opnsense.org/index.php?topic=17434.msg79254#msg79254) gave the hint, to disable reply-to in the firewall settings. After this tip and re-enableing the IPv6 gateway back, I am able to connect to the WAN interface (TCP/UDP).

I don't understand why - I don't have a Multi-WAN setup, just simple PPPoE. Is there a difference to pfSense?

Robert.
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: robgnu on May 28, 2020, 09:17:14 pm: Quote from: robgnu on May 28, 2020, 07:39:25 pm
Hello,

I am the user from the german forum with the same issue:
I have two separate OPNsense boxes with 20.1.7 connected by PPPoE (VDSL, Deutsche Telekom). Both machines do exactly the same:
- Connecting to the ISP works perfect. MTU was not set, but calculated to 1492, which is correct.
- One machine BEHIND the firewall is reachable from the internet (SSH; I configured the according rule)
- IPv6 connections from the Internet to the WAN port of the OPNsense are getting a timeout (SSH, HTTPS etc.)
- IPv4 connections from the Internet to the WAN port of the OPNsense working as usual and fast.
- Both OPNsense machines were fresh installed with 20.1.0, then I configured the PPPoE connection and then I upgraded to the latest version.

I figured out, that disabling the Gateway "WAN_DHCP6" solves that issue. On one machine I have now a continuously restarting radvd daemon (every 2 seconds) which is another IPv6 problem, I think.

I logged into these two machines an did a "clog -f /var/log/system.log". There is a massive flooding of messages (both OPNsenses). In parallel I did the same on another pfSense, there aren't those messages:

Code: [Select]
May 28 19:31:12 sense dhcp6c[67533]: script "/var/etc/dhcp6c_wan_script.sh" terminated May 28 19:31:12 sense dhcp6c[67533]: removing an event on pppoe0, state=REQUEST May 28 19:31:12 sense dhcp6c[67533]: removing server (ID: 00:02:00:00:05:83:34:30:3a:37:31:3a:38:33:3a:61:38:3a:63:38:3a:30:30:00:00:00) May 28 19:31:12 sense dhcp6c[67533]: got an expected reply, sleeping. May 28 19:31:12 sense dhcp6c[67533]: Sending Solicit May 28 19:31:12 sense dhcp6c[67533]: a new XID (15754b) is generated May 28 19:31:12 sense dhcp6c[67533]: set client ID (len 14) May 28 19:31:12 sense dhcp6c[67533]: set identity association May 28 19:31:12 sense dhcp6c[67533]: set elapsed time (len 2) May 28 19:31:12 sense dhcp6c[67533]: set option request (len 4) May 28 19:31:12 sense dhcp6c[67533]: send solicit to ff02::1:2%pppoe0 May 28 19:31:12 sense dhcp6c[67533]: reset a timer on pppoe0, state=SOLICIT, timeo=0, retrans=1016 May 28 19:31:12 sense dhcp6c[67533]: receive advertise from fe80::200:ff:fe00:0%pppoe0 on pppoe0 May 28 19:31:12 sense dhcp6c[67533]: get DHCP option client ID, len 14 May 28 19:31:12 sense dhcp6c[67533]: DUID: 00:01:00:01:21:39:d6:87:00:0d:b9:54:2b:6c May 28 19:31:12 sense dhcp6c[67533]: get DHCP option server ID, len 26 May 28 19:31:12 sense dhcp6c[67533]: DUID: 00:02:00:00:05:83:34:30:3a:37:31:3a:38:33:3a:61:38:3a:63:38:3a:30:30:00:00:00 May 28 19:31:12 sense dhcp6c[67533]: get DHCP option identity association, len 59 May 28 19:31:12 sense dhcp6c[67533]: IA_NA: ID=0, T1=0, T2=0 May 28 19:31:12 sense dhcp6c[67533]: get DHCP option status code, len 43 May 28 19:31:12 sense dhcp6c[67533]: status code: no addresses May 28 19:31:12 sense dhcp6c[67533]: get DHCP option DNS, len 32 May 28 19:31:12 sense dhcp6c[67533]: server ID: 00:02:00:00:05:83:34:30:3a:37:31:3a:38:33:3a:61:38:3a:63:38:3a:30:30:00:00:00, pref=-1 May 28 19:31:12 sense dhcp6c[67533]: reset timer for pppoe0 to 0.979450 May 28 19:31:13 sense dhcp6c[67533]: picked a server (ID: 00:02:00:00:05:83:34:30:3a:37:31:3a:38:33:3a:61:38:3a:63:38:3a:30:30:00:00:00) May 28 19:31:13 sense dhcp6c[67533]: Sending Request May 28 19:31:13 sense dhcp6c[67533]: a new XID (ac85ba) is generated May 28 19:31:13 sense dhcp6c[67533]: set client ID (len 14) May 28 19:31:13 sense dhcp6c[67533]: set server ID (len 26) May 28 19:31:13 sense dhcp6c[67533]: set status code May 28 19:31:13 sense dhcp6c[67533]: set identity association May 28 19:31:13 sense dhcp6c[67533]: set elapsed time (len 2) May 28 19:31:13 sense dhcp6c[67533]: set option request (len 4) May 28 19:31:13 sense dhcp6c[67533]: send request to ff02::1:2%pppoe0 May 28 19:31:13 sense dhcp6c[67533]: reset a timer on pppoe0, state=REQUEST, timeo=0, retrans=1059 May 28 19:31:13 sense dhcp6c[67533]: receive reply from fe80::200:ff:fe00:0%pppoe0 on pppoe0 May 28 19:31:13 sense dhcp6c[67533]: get DHCP option client ID, len 14 May 28 19:31:13 sense dhcp6c[67533]: DUID: 00:01:00:01:21:39:d6:87:00:0d:b9:54:2b:6c May 28 19:31:13 sense dhcp6c[67533]: get DHCP option server ID, len 26 May 28 19:31:13 sense dhcp6c[67533]: DUID: 00:02:00:00:05:83:34:30:3a:37:31:3a:38:33:3a:61:38:3a:63:38:3a:30:30:00:00:00 May 28 19:31:13 sense dhcp6c[67533]: get DHCP option identity association, len 59 May 28 19:31:13 sense dhcp6c[67533]: IA_NA: ID=0, T1=0, T2=0 May 28 19:31:13 sense dhcp6c[67533]: get DHCP option status code, len 43 May 28 19:31:13 sense dhcp6c[67533]: status code: no addresses May 28 19:31:13 sense dhcp6c[67533]: get DHCP option DNS, len 32 May 28 19:31:13 sense dhcp6c[67533]: Received REPLY for REQUEST May 28 19:31:13 sense dhcp6c[67533]: nameserver[0] 2003:180:2:7000::53 May 28 19:31:13 sense dhcp6c[67533]: nameserver[1] 2003:180:2:9000::53 May 28 19:31:13 sense dhcp6c[67533]: make an IA: NA-0 May 28 19:31:13 sense dhcp6c[67533]: status code for NA-0: no addresses May 28 19:31:13 sense dhcp6c[67533]: IA NA-0 is invalidated May 28 19:31:13 sense dhcp6c[67533]: remove an IA: NA-0 May 28 19:31:13 sense dhcp6c[67533]: reset a timer on pppoe0, state=INIT, timeo=0, retrans=113 May 28 19:31:13 sense dhcp6c[67533]: executes /var/etc/dhcp6c_wan_script.sh May 28 19:31:13 sense dhcp6c: dhcp6c REQUEST on pppoe0 - running newipv6 May 28 19:31:14 sense opnsense: /usr/local/etc/rc.newwanipv6: IPv6 renewal is starting on 'pppoe0' May 28 19:31:14 sense opnsense: /usr/local/etc/rc.newwanipv6: On (IP address: 2003:a:xxxx:xxxx:20d:b9ff:fe54:2b6c) (interface: WAN[wan]) (real interface: pppoe0).
For now, I'm out of ideas. I switched over from pfSense for some other good reasons, but I never had any problems with IPv6 on pfSense.

Robert.
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: penM on May 29, 2020, 06:16:57 pm: I can connect with haproxy from LAN via IPv6, but I could not connect via WAN from IPv6.

I used ipv6 port forwarding to transfer to linux of LAN and confirmed whether it can be accessed by IPv6. This was successful.(I'm running ipv6nat because my network only gets IPv6 address / 64.)

I was able to communicate with haproxy from the WAN by adding the following port forwarding.
The settings currently in operation are as follows.
In my environment em0 is the physical interface ipv6 IPoE. Also, ppoe0 is ipv4 and bridhe0 is LAN.
The LAN (bridge0) has addresses 192.168.1.1 and fd4b: 5bb3: b8d2: 1c9e.(Example)

pfctl -sn
Code: [Select]
rdr on em0 inet6 proto tcp from any to (self) port = https -> fd4b:5bb3:b8d2:1c9e::1 port 443 rdr on pppoe0 inet proto tcp from any to (self) port = https -> 192.168.1.1 port 443 rdr on bridge0 inet proto tcp from any to (self) port = https -> 192.168.1.1 port 443 rdr on bridge0 inet6 proto tcp from any to (self) port = https -> fd4b:5bb3:b8d2:1c9e::1 port 443
Title: Re: haproxy for ipv4 and ipv6 to ipv4; ipv6 doesnt work?
Post by: abraxxa on May 29, 2020, 09:13:05 pm: I don't know the FreeBSD specifics of the reply-to feature but the OPNSense docs description reads like if it remembers the layer 2 address a packet is received from and sends the reply to the same layer 2 address regardless of the layer 3 routing table.
Can someone from the OPNSense team comment on this which more knowledge?