[SOLVED]Site2Site openvpn Tunnel OK, Clients not

[hcso-tm]

Hi,

I did encounter some really strange behavior.
We did Setup a OpenVPN Tunnel between 2 Opensense FWs. The Tunnel seems to be fine. When I Ping/Portprobe "ServerFW" from "ClientFW" it just works as expected.
When I try the same from a PC connected to "ClientFW" I can capture Packages leaving ClientFW, not arriving at ServerFW.


First, i tried nc from a pc connected to ClientFW. You can see that the ClientFW tried to send those Packages to ServerFW. On the Server-Side there is nothing captured.
After that, I used the Interface->Debuging->Port-Probe on the ClientFW. Those Packages are captured at both ends.

See attached Captures.
Capture Setting on both firewalls are set to: Tunnel Interface, promiscuos Mode, TCP Port 443, started at the same Time.

Any ideas why traffic in the Tunnel is behaving so strange?

[zerwes]

can you describe your network setup, the phase1 tunnel and phase2 subnets if the type is policy based, otherwise the routing?

[hcso-tm]

Hi,
we´re currently preparing the first branch-office.

The Main-Site is hosting 2 OPNsenses in HA. Currently running 3 OpenVPN-Servers on different public IP-Adresses.
Server-Mode is P2P ssl/tls
Device Mode tun

Tunnel-Settings:
Tunnel-Network: 172.24.240.0/24 (172.24.240.1 Server in Main-Net, 172.24.240.2 Client in Branch Office)
Redirect Gateway false
Local Network 172.24.0.0/15
Remote Network 172.26.0.0/16
Compression off
Type of Service false
Duplicate Connections true (tested with false too)

Client Settings:
Dynamic IP true
Address Pool true
Topology true
Client Management Port false

No Advanced config


It is Route-Based and those Routes were automatically created in the Routing table.
Our Main-Net is 172.24.0.0/15 (172.24.0.0/16 for Server/Infrastructure and 172.25.0.0/16 for Clients)
Branch-Office will be 172.26.0.0/16 (172.26.0.0/17 for Server/Infrastructure and 172.26.128.0/17 for Clients)


As I mentioned: Routing and the Tunnel itself seems to work. I see the packages in the Firewall logs leaving the FW on the Tunnel-Interface, I see them on the Packet-Capture on the Tunnel, I just miss them arriving on the Tunnel Interface on the Server.

I just tried to connect to a Service in Main net: From the Firewall in the Branch-Office it works, from the Client it doesn´t (same as connecting to the OpenVPN Server itself)

[zerwes]

Sorry, my mistake, I overlooked the OpenVPN mentioning in your first post ... I had my mind on a ipsec topic before ..
do you have some rules defined on the OpenVPN interface?

[hcso-tm]

For testing purposes something like:
Allow Any * * * *
on both Firewalls

I do see the Rule on the Branch-Office Firewall-Log and nothing on Main-Net, but again:
Packet capture shows outgoing traffic that gets lost somewhere in the tunnel while the tunnel itself stays up... That is the strange part.

BTW: the other way round (Ping from Main-Net to Branch) does only work on the Tunnel-IP. Any other IP is unreachable and no packets are captured on the Branch Firewall

[zerwes]

I still have routing in suspicion ...
can you post the output of

Code Select Expand

# netstat -rn
# route get 172.24.240.1
# route get 172.24.240.2
# route get 172.24.0.1
# route get 172.26.0.1from both nodes on the console?

[hcso-tm]

Yeah, that was my first guess, but it seems not to be the case

Routes on Server

Code (Routes on Server) Select Expand

   route to: 172.24.240.1
destination: 172.24.240.1
        fib: 0
  interface: lo0
      flags: <UP,HOST,DONE,STATIC,PINNED>
recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0     16384         1         0
   route to: 172.24.240.2
destination: 172.24.240.2
        fib: 0
  interface: ovpns3
      flags: <UP,HOST,DONE,PINNED>
recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0
   route to: 172.24.0.1
destination: 172.24.0.1
        fib: 0
  interface: lo0
      flags: <UP,HOST,DONE,STATIC,PINNED>
recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0     16384         1         0
   route to: 172.26.0.1
destination: 172.26.0.0
       mask: 255.255.0.0
    gateway: 172.24.240.2
        fib: 0
  interface: ovpns3
      flags: <UP,GATEWAY,DONE,STATIC>
recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0



Code ( Routes on Client) Select Expand

   route to: 172.24.240.1
destination: 172.24.240.1
        fib: 0
  interface: ovpnc1
      flags: <UP,HOST,DONE,PINNED>
recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0
   route to: 172.24.240.2
destination: 172.24.240.2
        fib: 0
  interface: lo0
      flags: <UP,HOST,DONE,STATIC,PINNED>
recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0     16384         1         0
   route to: 172.24.0.1
destination: 172.24.0.0
       mask: 255.254.0.0
    gateway: 172.24.240.1
        fib: 0
  interface: ovpnc1
      flags: <UP,GATEWAY,DONE,STATIC>
recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0
   route to: 172.26.0.1
destination: 172.26.0.1
        fib: 0
  interface: lo0
      flags: <UP,HOST,DONE,STATIC,PINNED>
recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0     16384         1         0


Code (all routes Server) Select Expand

Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            XXX.XXX.XXX.XXX    UGS      vtnet1

...

127.0.0.1          link#5             UH          lo0
172.24.0.0/24      link#16            U      vtnet0_v
172.24.0.0/22      link#16            U      vtnet0_v
172.24.0.1         link#16            UHS         lo0
172.24.0.2         link#16            UHS         lo0
172.24.8.0/24      link#10            U      vtnet0_v
172.24.8.1         link#10            UHS         lo0
172.24.8.2         link#10            UHS         lo0
172.24.12.0/24     link#11            U      vtnet0_v
172.24.12.1        link#11            UHS         lo0
172.24.12.2        link#11            UHS         lo0
172.24.16.0/24     link#15            U      vtnet0_v
172.24.16.1        link#15            UHS         lo0
172.24.16.2        link#15            UHS         lo0
172.24.20.0/24     link#17            U      vtnet0_v
172.24.20.1        link#17            UHS         lo0
172.24.20.2        link#17            UHS         lo0
172.24.24.0/24     link#18            U      vtnet0_v
172.24.24.1        link#18            UHS         lo0
172.24.24.2        link#18            UHS         lo0
172.24.28.0/24     link#14            U      vtnet0_v
172.24.28.1        link#14            UHS         lo0
172.24.28.2        link#14            UHS         lo0
172.24.32.0/24     link#41            U      vtnet0_v
172.24.32.1        link#41            UHS         lo0
172.24.32.2        link#41            UHS         lo0
172.24.240.0/24    172.24.240.2       UGS      ovpns3
172.24.240.1       link#12            UHS         lo0
172.24.240.2       link#12            UH       ovpns3
172.24.248.0/24    link#3             U        vtnet2
172.24.248.2       link#3             UHS         lo0
172.25.0.0/22      link#20            U      vtnet0_v
172.25.0.1         link#20            UHS         lo0
172.25.0.2         link#20            UHS         lo0
172.25.4.0/22      link#21            U      vtnet0_v
172.25.4.1         link#21            UHS         lo0
172.25.4.2         link#21            UHS         lo0
172.25.8.0/22      link#22            U      vtnet0_v
172.25.8.1         link#22            UHS         lo0
172.25.8.2         link#22            UHS         lo0
172.25.12.0/22     link#23            U      vtnet0_v
172.25.12.1        link#23            UHS         lo0
172.25.12.2        link#23            UHS         lo0
172.25.16.0/22     link#24            U      vtnet0_v
172.25.16.1        link#24            UHS         lo0
172.25.16.2        link#24            UHS         lo0
172.25.20.0/22     link#25            U      vtnet0_v
172.25.20.1        link#25            UHS         lo0
172.25.20.2        link#25            UHS         lo0
172.25.24.0/22     link#26            U      vtnet0_v
172.25.24.1        link#26            UHS         lo0
172.25.24.2        link#26            UHS         lo0
172.25.28.0/22     link#27            U      vtnet0_v
172.25.28.1        link#27            UHS         lo0
172.25.28.2        link#27            UHS         lo0
172.25.32.0/22     link#28            U      vtnet0_v
172.25.32.1        link#28            UHS         lo0
172.25.32.2        link#28            UHS         lo0
172.25.36.0/22     link#29            U      vtnet0_v
172.25.36.1        link#29            UHS         lo0
172.25.36.2        link#29            UHS         lo0
172.25.40.0/22     link#30            U      vtnet0_v
172.25.40.1        link#30            UHS         lo0
172.25.40.2        link#30            UHS         lo0
172.25.44.0/22     link#31            U      vtnet0_v
172.25.44.1        link#31            UHS         lo0
172.25.44.2        link#31            UHS         lo0
172.25.48.0/22     link#32            U      vtnet0_v
172.25.48.1        link#32            UHS         lo0
172.25.48.2        link#32            UHS         lo0
172.25.52.0/22     link#42            U      vtnet0_v
172.25.52.1        link#42            UHS         lo0
172.25.52.2        link#42            UHS         lo0
172.25.56.0/22     link#8             U      vtnet0_v
172.25.56.1        link#8             UHS         lo0
172.25.56.2        link#8             UHS         lo0
172.25.128.0/24    link#38            U      vtnet0_v
172.25.128.1       link#38            UHS         lo0
172.25.128.2       link#38            UHS         lo0
172.25.232.0/22    link#33            U      vtnet0_v
172.25.232.1       link#33            UHS         lo0
172.25.232.2       link#33            UHS         lo0
172.25.236.0/22    link#34            U      vtnet0_v
172.25.236.1       link#34            UHS         lo0
172.25.236.2       link#34            UHS         lo0
172.25.240.0/22    link#35            U      vtnet0_v
172.25.240.1       link#35            UHS         lo0
172.25.240.2       link#35            UHS         lo0
172.25.244.0/22    link#36            U      vtnet0_v
172.25.244.1       link#36            UHS         lo0
172.25.244.2       link#36            UHS         lo0
172.25.248.0/22    link#37            U      vtnet0_v
172.25.248.1       link#37            UHS         lo0
172.25.248.2       link#37            UHS         lo0
172.25.252.0/24    172.25.252.2       UGS      ovpns2
172.25.252.1       link#40            UHS         lo0
172.25.252.2       link#40            UH       ovpns2
172.25.253.0/24    172.25.253.2       UGS      ovpns1
172.25.253.1       link#39            UHS         lo0
172.25.253.2       link#39            UH       ovpns1
172.26.0.0/16      172.24.240.2       UGS      ovpns3


Code (all routes client) Select Expand

Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            172.26.124.20      UGS      vtnet1
127.0.0.1          link#5             UH          lo0
172.24.0.0/15      172.24.240.1       UGS      ovpnc1
172.24.240.0/24    172.24.240.1       UGS      ovpnc1
172.24.240.1       link#11            UH       ovpnc1
172.24.240.2       link#11            UHS         lo0
172.26.0.0/22      link#9             U      vtnet0_v
172.26.0.1         link#9             UHS         lo0
172.26.0.2         link#9             UHS         lo0
172.26.8.0/24      link#10            U      vtnet0_v
172.26.8.1         link#10            UHS         lo0
172.26.8.2         link#10            UHS         lo0
172.26.12.0/24     link#12            U      vtnet0_v
172.26.12.1        link#12            UHS         lo0
172.26.12.2        link#12            UHS         lo0
172.26.120.0/24    link#3             U        vtnet2
172.26.120.2       link#3             UHS         lo0
172.26.124.0/24    link#2             U        vtnet1
172.26.124.1       link#2             UHS         lo0
172.26.124.2       link#2             UHS         lo0
192.168.78.0/24    link#4             U        vtnet3
192.168.78.2       link#4             UHS      vtnet3
192.168.78.81      link#4             UHS         lo0

Internet6:
...


[zerwes]

Yeah, that was my first guess, but it seems not to be the case

Indeed, the routing lgtm ...Can you run a tcpdump on the openvpn interfaces listening for icmp packages and try some pings to the remote site, first from the vpn device and then from the network behind, in both directions?  (at least this would be my next step)

[hcso-tm]

From main to branch nothing happens, I see Packets going out on Tunnel@MainFW, nothing arriving at Tunnel@branch (see attachments from this post).
tcp-Dumps from branch to main are in the first post.

It is really strange...

[zerwes]

... weird ... one can simply say the tunnel is not up ...
I would tear it all down and start over from scratch ... something seems wrong ...
In fact there is a quite straight forward howto available at https://docs.opnsense.org/manual/how-tos/sslvpn_s2s.html

[hcso-tm]

Yes, I followed that documentation.

I did just capture the WAN-Interface too and realized, that those missing packets don´t reach the WAN (even though you see them "leaving" in the tunnel-interface dump). Successful pings do correlate to WAN-Packets

[hcso-tm]

I did rebuild the complete Tunnel, still the same behavior  :-\

[hcso-tm]

After some further testing (included HW and IP-Changes) I found those lines in OpenVPN Debug11 Logs:

Code Select Expand

2022-04-29T14:39:44 openvpn[52367] test_pcsm/XXX.XXX.XXX.XXX:49277 MULTI: bad source address from client [192.168.78.76], packet dropped
2022-04-29T14:39:44 openvpn[52367] test_pcsm/XXX.XXX.XXX.XXX:49277 GET INST BY VIRT: 192.168.78.76 [failed]

=> seems like broken Routing

route get 192.168.78.76 tells me the correct route...

Is it possible, that OpenVPN does not use the OS-Routingtable?

[hcso-tm]

Finally solved it:

You need to configure the Local and Remote Network in "Client Specific Overrides" on the Server-Side.
Settings made in the Server-Settings are ignored...

Please Update the Documentation to make that clear! Is there any reason to have those Options in Server-Settings? Is there any Case where they work?