OpnSense FRR BGP and AWS Site-to-Site VPN - hosts not reachable (ping/ssh fails)

[imconfusedallthetime]

Hi,

OpnSense newbie here.

I have configured an IPsec tunnel between a bespoke cloud platform and AWS and the tunnel is up with 1 BGP route advertised.

I can SSH from an EC2 instance in AWS to the VM running OpnSense on the bespoke platform using 192.168.1.1. However, I am unable to SSH from the same EC2 instance to another VM 192.168.1.100 in the same network as OpnSense. SSH reports "ssh: connect to host 192.168.1.100 port 22: Connection timed out".

I can SSH from 192.168.1.1 to 192.168.1.100.

Is this firewall related or routing problem? Any guidance is appreciated.

Code: [Select]

Bespoke cloud platform
VM hosting OpnSense - 192.168.1.1 (ASN 64514, BGP advertised 192.168.1.0/24)
VM web server - 192.168.1.100

AWS Site-to-Site VPN (AWS ASN 64512)
VM - 172.31.86.32

OpnSense > Firewall > Rules > IPsec

Protocol: IPv4
Source: *
Port: *
Destination: LAN net
Port: *
Gateway: *

OpnSense > Firewall > Rules > LAN

Default allow LAN to any IPv4 and IPv6.

OpnSense > Firewall > Rules > WAN

Allow 500 (ISAKMP), 4500 (IPsec NAT-T), 179 (BGP).

Routing > Diagnostics > BGP > IPv$ Routing Table

Valid Best Internal Network Next Hop Metric LocPrf Weight Path Origin
Y N N 172.31.0.0/16 169.254.220.77 200 0 0 64512 IGP
Y Y N 172.31.0.0/16 169.254.135.37 100 0 0 64512 IGP
Y Y N 192.168.1.0/24 0.0.0.0 0 0 32768 Internal IGP



[imconfusedallthetime]

Does anyone know what this means and what I exactly need to do?

Code: [Select]

OpnSense > Routing > BGP > General

Network: 192.168.1.0/24
Select the network to advertise, you have to set a Null route via System -> Routes

I don't think I've done this.

[mimugmail]

In Azure you need to create a Routing table since all vms use Azure gateways. No idea If this is same in AWS.

[imconfusedallthetime]

In AWS, routing is propagated and appears under VPC > Route Tables:

Code: [Select]

Destination, Target, Status, Propagated
0.0.0.0/0, igw-27d7295d, Active, No
172.31.0.0/16, local, Active, No
192.168.1.0/24, vgw-01c10b23ec5e24488, Active, Yes
Do I need to add a Null route as mentioned under Routing (FRR) > BGP > General tab in OpnSense?

[mimugmail]

Sure, thats why its mentioned there

[imconfusedallthetime]

Being a complete newbie, why is the the null route required? Also, what would the null route look like and do I need to specify the AWS gateways as the gateway?

[mimugmail]

Your network with gateway null .. thats ist. Just try it

[imconfusedallthetime]

Thanks. So I've added the null route for 192.168.1.0/24, which is also the advertised route under OpnSense > System > Routes > Configuration.

Network: 192.168.1.0/24
Gateway: Null4 - 127.0.0.1

I am still unable to connect from my EC2 instance (172.31.40.15) to the VM 192.168.1.100 behind OpnSense (192.168.1.1). This is what Wireshark shows:

Code: [Select]

25 20.305718 172.31.40.15 192.168.1.100 TCP 66 [TCP Retransmission] 49418 → 80 [SYN, ECN, CWR] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
25 20.305718 172.31.40.15 192.168.1.100 TCP 66 [TCP Retransmission] 49418 → 80 [SYN, ECN, CWR] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
27 20.557705 172.31.40.15 192.168.1.100 TCP 66 [TCP Retransmission] 52171 → 80 [SYN, ECN, CWR] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
I don't think it's firewall related, as I disabled this to check.

I should also mention that tracert from my EC2 instance (172.31.40.15) yields...

Code: [Select]

Tracing route to ip-192-168-1-100.ec2.internal [192.168.1.100] over a maximum of 30 hops:

  1     8 ms     8 ms     8 ms  169.254.126.28
  2     *        *        *     Request timed out.
  3     *        *        *     Request timed out.
...
169.254.126.28 is what AWS refers to as the inside address for the customer gateway for tunnel 1.

Code: [Select]

Tunnel Number Outside IP Address Inside IPv4 CIDR Inside IPv6 CIDR Status Status Last Changed Details
Tunnel 1 4.279.84.223 169.254.126.26/30 - UP July 12, 2021 at 1:43:21 PM UTC+1 1 BGP ROUTES
Tunnel 2 100.20.252.220 169.254.222.26/30 - UP July 12, 2021 at 1:43:16 PM UTC+1 1 BGP ROUTES
AWS also shows routing is propagated.

Code: [Select]

Destination Target Status Propagated
172.31.0.0/16 local Active No
0.0.0.0/0 igw-67d7675d Active No
192.168.1.0/24 vgw-01b10b25ec5e28844 Active Yes
vgw-01b10b25ec5e28844 is the gateway on the AWS side.

[random1104]

Assuming the correct firewall rules are in place, do you have the correct default gateway (or needed static rules if the VMs have multiple interfaces in different subnets) on both sides?

[imconfusedallthetime]

I'm confident firewall rules are correct as I can ssh to 192.168.1.1 (OpnSense on custom cloud) from an AWS EC2 server. I can also proxy to 192.168.1.100 (web server) via 192.168.1.1 from said server.

To be sure, I disabled firewall on OpnSense and tested with the same outcome - SSH and ping directly to 192.168.1.100 fails from EC2 server.

It feels like routing related and I don't have any static routes defined under System > Routes > Configuration. I'm not quite sure what to select for Network Address (192.168.1.100?) and Gateway (WAN?).

NAT is set to Automatic and I can see the AWS WAN interfaces are listed.

AmazonIKEvpn03139b9b80 networks, AmazonIKEvpn03139b9b81 networks, LAN networks, Loopback networks, 127.0.0.0/8