Messages

to close out this thread, I have just managed to get this to work.


A NAT rule was needed to get things to work as expected:


Thank you for your help and support

It's not really clear what you're trying to accomplish - e.g. is there an actual WAN here, or are you using the WAN interface as an additional LAN? I'm not familiar with Azure, and don't know what a "VNet Peering" entails, but your network design doesn't look right. You have a host on subnet 10.13.1.0/24 and a route supposedly pointing to 10.1.1.250, but that's on a different subnet, so what's in between? You have the same next hop (10.1.1.250) on the other side, but that side of your hub is 10.1.0.250 ...

VNet peering is basically a way to connect two virtual network together, so spoke can talk to hub and vice versa as if they were in the same VNet.

Spoke2, spoke3 and hub are separate VNets.

In my diagram, spoke3 is connected to hub and spoke1 is connected hub. However this doesn't make spoke1 is accessible from spoke1 just because both are connected (via VNet peering) to hub. To make them talk to each other, you can route the traffic from spoke3 to spoke1 to a NVA (network virtual appliance) in our case opnsense is the NVA which in turn should be able to route/allow the traffic to the destined network. In my case, I am using a static route for this matter which points to the LAN interface of my NVA (10.1.1.250).
The WAN interface is going to be used for internet traffic from spoke VNets.

10.13.1.4 is not part of 10.1.1.0/24, so it will take the default route, which is the gateway associated with your WAN interface. You'll need an additional "LAN" interface for "spoke3"

How to add an additional LAN interface? Under Interfaces--Assignments, I can't seem to find any option to add a new interface.
I am totally new to opnsense, please excuse my ignorance.

Your LAN interface can't be both 10.11.1.0/24 and 10.13.1.0/24. You'll need two separate interfaces. You haven't shared any information at all about how OPNsense is configured, so we could only guess...........

My bad... Here how opnsense configured
Lan interface 10.1.1.250
Wan interface 101.0.250

I have a hub-and-spoke model in Azure (as shown below)


opnsense:
LAN interface:10.1.1.250
WAN interface:10.1.0.250

Spoke1 is peered to hub vnet where opnsense lives.
Spoke3 is peered to hub vnet where opnsense lives.

For spoke1 to talk to spoke3, the traffic is routed through opnsese LAN interface in the hub.
Also for spoke3 to talk to spoke1, the traffic is routed through opnsese LAN interface in the hub.

I have added specific firewall rules on opnsense to allow the traffic from spoke1 to spoke3 and vice versa. However when I try to reach from spoke1 to spoke3 but I can't. I don't see any traffic on spoke3 at all from spoke1. Looking at opnsense firewall logs. I see the traffic is hitting my rule, but then the traffic is let out from firewall through WAN interface as shown in the following snippet:



Why this is happening? What do I need to do to get the traffic from spoke1 to reach to spoke3 and vice versa?

any help is appreciated.

I finally managed to get it to work just fine. I resorted to a fresh installation of opnsense with fresh config, and then I was able to get the IPsec tunnel up. Thank you all for your help

Did you install a policy on both sides? (Install policy checked)

yes

Its not a Phase 1 issue if Phase 1 is established and there is a Phase 2.

I expect there are no rules that allow traffic through the tunnel.

Just create a rule in Floating allowing any any on both Firewalls to troubleshoot, if this is not production yet.

Just a floating rule but still the connection is up but there is no traffic. I can't ping/ssh either anything on the other side.

Seems to be a p1 issue.

You have to state the outside public IP as "My identifier" on both sites.

They are there already

Here is some outputs if you can spot anything wrong:

site1:

Code Select Expand

swanctl --list-sas
no files found matching '/usr/local/etc/strongswan.opnsense.d/*.conf'
con1: #1, ESTABLISHED, IKEv2, 233b879dde1990fc_i* 7f1b33d2a0738eca_r
  local  '4.213.xx.xx' @ 10.1.0.250[4500]
  remote '4.188.xx.xx' @ 4.188.xx.xx[4500]
  AES_CBC-256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048
  established 1073s ago, rekeying in 12368s
  con1: #2, reqid 1, INSTALLED, TUNNEL-in-UDP, ESP:AES_GCM_16-256
    installed 1073s ago, rekeying in 2050s, expires in 2527s
    in  cb2eaa17,      0 bytes,     0 packets
    out ca26585e,      0 bytes,     0 packets,   186s ago
    local  10.1.0.0/16
    remote 10.2.0.0/16

site2:

Code Select Expand

root@OPNsense:~ # swanctl --list-sas
no files found matching '/usr/local/etc/strongswan.opnsense.d/*.conf'
con1: #1, ESTABLISHED, IKEv2, 233b879dde1990fc_i 7f1b33d2a0738eca_r*
  local  '4.188.xx.xx' @ 10.2.0.250[4500]
  remote '4.213.xx.xx' @ 4.213.xx.xx[4500]
  AES_CBC-256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048
  established 1067s ago, rekeying in 12126s
  con1: #2, reqid 1, INSTALLED, TUNNEL-in-UDP, ESP:AES_GCM_16-256
    installed 1067s ago, rekeying in 2128s, expires in 2533s
    in  ca26585e,      0 bytes,     0 packets
    out cb2eaa17,      0 bytes,     0 packets,   180s ago
    local  10.2.0.0/16
    remote 10.1.0.0/16

site1: ipsec logs:

Code Select Expand

2024-11-28T09:36:33-06:00 Informational charon 13[IKE] <con1|2> sending keep alive to 4.188.xx.xx[4500]
2024-11-28T09:36:01-06:00 Informational charon 12[NET] <con1|2> sending packet: from 10.1.0.250[4500] to 4.188.xx.xx[4500] (224 bytes)
2024-11-28T09:36:01-06:00 Informational charon 12[ENC] <con1|2> generating IKE_AUTH response 1 [ IDr AUTH N(ESP_TFC_PAD_N) SA TSi TSr ]
2024-11-28T09:36:01-06:00 Informational charon 12[IKE] <con1|2> CHILD_SA con1{2} established with SPIs c18aeef8_i cdd9b5d2_o and TS 10.1.0.0/16 === 10.2.0.0/16
2024-11-28T09:36:01-06:00 Informational charon 12[CFG] <con1|2> selected proposal: ESP:AES_GCM_16_256/NO_EXT_SEQ
2024-11-28T09:36:01-06:00 Informational charon 12[IKE] <con1|2> maximum IKE_SA lifetime 15691s
2024-11-28T09:36:01-06:00 Informational charon 12[IKE] <con1|2> scheduling rekeying in 14251s
2024-11-28T09:36:01-06:00 Informational charon 12[IKE] <con1|2> IKE_SA con1[2] established between 10.1.0.250[4.213.xx.xx]...4.188.xx.xx[4.188.xx.xx]
2024-11-28T09:36:01-06:00 Informational charon 12[IKE] <con1|2> authentication of '4.213.xx.xx' (myself) with pre-shared key
2024-11-28T09:36:01-06:00 Informational charon 12[IKE] <con1|2> received ESP_TFC_PADDING_NOT_SUPPORTED, not using ESPv3 TFC padding
2024-11-28T09:36:01-06:00 Informational charon 12[IKE] <con1|2> authentication of '4.188.xx.xx' with pre-shared key successful
2024-11-28T09:36:01-06:00 Informational charon 12[CFG] <con1|2> selected peer config 'con1'
2024-11-28T09:36:01-06:00 Informational charon 12[CFG] <2> looking for peer configs matching 10.1.0.250[4.213.xx.xx]...4.188.xx.xx[4.188.xx.xx]
2024-11-28T09:36:01-06:00 Informational charon 12[ENC] <2> parsed IKE_AUTH request 1 [ IDi N(INIT_CONTACT) IDr AUTH N(ESP_TFC_PAD_N) SA TSi TSr N(MULT_AUTH) N(EAP_ONLY) N(MSG_ID_SYN_SUP) ]
2024-11-28T09:36:01-06:00 Informational charon 12[NET] <2> received packet: from 4.188.xx.xx[4500] to 10.1.0.250[4500] (256 bytes)
2024-11-28T09:36:01-06:00 Informational charon 12[NET] <2> sending packet: from 10.1.0.250[500] to 4.188.xx.xx[500] (472 bytes)
2024-11-28T09:36:01-06:00 Informational charon 12[ENC] <2> generating IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(CHDLESS_SUP) N(MULT_AUTH) ]
2024-11-28T09:36:01-06:00 Informational charon 12[IKE] <2> remote host is behind NAT
2024-11-28T09:36:01-06:00 Informational charon 12[IKE] <2> local host is behind NAT, sending keep alives
2024-11-28T09:36:01-06:00 Informational charon 12[CFG] <2> selected proposal: IKE:AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048
2024-11-28T09:36:01-06:00 Informational charon 12[IKE] <2> 4.188.xx.xx is initiating an IKE_SA
2024-11-28T09:36:01-06:00 Informational charon 12[ENC] <2> parsed IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
2024-11-28T09:36:01-06:00 Informational charon 12[NET] <2> received packet: from 4.188.xx.xx[500] to 10.1.0.250[500] (464 bytes)
2024-11-28T09:36:00-06:00 Informational charon 12[IKE] <con1|1> received NO_PROPOSAL_CHOSEN notify error
2024-11-28T09:36:00-06:00 Informational charon 12[ENC] <con1|1> parsed IKE_SA_INIT response 0 [ N(NO_PROP) ]
2024-11-28T09:36:00-06:00 Informational charon 12[NET] <con1|1> received packet: from 4.188.xx.xx[500] to 10.1.0.250[500] (36 bytes)
2024-11-28T09:36:00-06:00 Informational charon 13[NET] <con1|1> sending packet: from 10.1.0.250[500] to 4.188.xx.xx[500] (464 bytes)
2024-11-28T09:36:00-06:00 Informational charon 13[ENC] <con1|1> generating IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
2024-11-28T09:36:00-06:00 Informational charon 13[IKE] <con1|1> initiating IKE_SA con1[1] to 4.188.xx.xx
2024-11-28T09:36:00-06:00 Informational charon 13[KNL] creating acquire job for policy 10.1.0.250/32 === 4.188.xx.xx/32 with reqid {1}
2024-11-28T09:35:46-06:00 Informational charon 13[CFG] installing 'con1'
2024-11-28T09:35:46-06:00 Informational charon 13[CFG] added vici connection: con1
2024-11-28T09:35:46-06:00 Informational charon 13[CFG] loaded IKE shared key with id 'ike-p1-0' for: '4.188.xx.xx'
2024-11-28T09:35:46-06:00 Informational charon 00[JOB] spawning 16 worker threads
2024-11-28T09:35:46-06:00 Informational charon 00[LIB] loaded plugins: charon aes des blowfish rc2 sha2 sha1 md4 md5 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs12 pgp dnskey sshkey pem openssl pkcs8 fips-prf curve25519 xcbc cmac hmac kdf gcm drbg curl attr kernel-pfkey kernel-pfroute resolve socket-default stroke vici updown eap-identity eap-md5 eap-mschapv2 eap-radius eap-tls eap-ttls eap-peap xauth-generic xauth-eap xauth-pam whitelist addrblock counters
2024-11-28T09:35:46-06:00 Informational charon 00[CFG] loaded 0 RADIUS server configurations
2024-11-28T09:35:46-06:00 Informational charon 00[CFG] loading secrets from '/usr/local/etc/ipsec.secrets'
2024-11-28T09:35:46-06:00 Informational charon 00[CFG] loading crls from '/usr/local/etc/ipsec.d/crls'
2024-11-28T09:35:46-06:00 Informational charon 00[CFG] loading attribute certificates from '/usr/local/etc/ipsec.d/acerts'
2024-11-28T09:35:46-06:00 Informational charon 00[CFG] loading ocsp signer certificates from '/usr/local/etc/ipsec.d/ocspcerts'
2024-11-28T09:35:46-06:00 Informational charon 00[CFG] loading aa certificates from '/usr/local/etc/ipsec.d/aacerts'
2024-11-28T09:35:46-06:00 Informational charon 00[CFG] loading ca certificates from '/usr/local/etc/ipsec.d/cacerts'
2024-11-28T09:35:46-06:00 Informational charon 00[CFG] using '/sbin/resolvconf' to install DNS servers
2024-11-28T09:35:46-06:00 Informational charon 00[LIB] providers loaded by OpenSSL: default legacy
2024-11-28T09:35:46-06:00 Informational charon 00[DMN] Starting IKE charon daemon (strongSwan 5.9.14, FreeBSD 14.1-RELEASE-p6, amd64)

site2: ipsec logs

Code Select Expand

2024-11-28T09:36:33-06:00 Informational charon 14[IKE] <con1|2> sending keep alive to 4.213.xx.xx[4500]
2024-11-28T09:36:01-06:00 Informational charon 12[IKE] <con1|2> CHILD_SA con1{2} established with SPIs cdd9b5d2_i c18aeef8_o and TS 10.2.0.0/16 === 10.1.0.0/16
2024-11-28T09:36:01-06:00 Informational charon 12[CFG] <con1|2> selected proposal: ESP:AES_GCM_16_256/NO_EXT_SEQ
2024-11-28T09:36:01-06:00 Informational charon 12[IKE] <con1|2> received ESP_TFC_PADDING_NOT_SUPPORTED, not using ESPv3 TFC padding
2024-11-28T09:36:01-06:00 Informational charon 12[IKE] <con1|2> maximum IKE_SA lifetime 15499s
2024-11-28T09:36:01-06:00 Informational charon 12[IKE] <con1|2> scheduling rekeying in 14059s
2024-11-28T09:36:01-06:00 Informational charon 12[IKE] <con1|2> IKE_SA con1[2] established between 10.2.0.250[4.188.xx.xx]...4.213.xx.xx[4.213.xx.xx]
2024-11-28T09:36:01-06:00 Informational charon 12[IKE] <con1|2> authentication of '4.213.xx.xx' with pre-shared key successful
2024-11-28T09:36:01-06:00 Informational charon 12[ENC] <con1|2> parsed IKE_AUTH response 1 [ IDr AUTH N(ESP_TFC_PAD_N) SA TSi TSr ]
2024-11-28T09:36:01-06:00 Informational charon 12[NET] <con1|2> received packet: from 4.213.xx.xx[4500] to 10.2.0.250[4500] (224 bytes)
2024-11-28T09:36:01-06:00 Informational charon 12[NET] <con1|2> sending packet: from 10.2.0.250[4500] to 4.213.xx.xx[4500] (256 bytes)
2024-11-28T09:36:01-06:00 Informational charon 12[ENC] <con1|2> generating IKE_AUTH request 1 [ IDi N(INIT_CONTACT) IDr AUTH N(ESP_TFC_PAD_N) SA TSi TSr N(MULT_AUTH) N(EAP_ONLY) N(MSG_ID_SYN_SUP) ]
2024-11-28T09:36:01-06:00 Informational charon 12[IKE] <con1|2> establishing CHILD_SA con1{2}
2024-11-28T09:36:01-06:00 Informational charon 12[IKE] <con1|2> authentication of '4.188.xx.xx' (myself) with pre-shared key
2024-11-28T09:36:01-06:00 Informational charon 12[IKE] <con1|2> remote host is behind NAT
2024-11-28T09:36:01-06:00 Informational charon 12[IKE] <con1|2> local host is behind NAT, sending keep alives
2024-11-28T09:36:01-06:00 Informational charon 12[CFG] <con1|2> selected proposal: IKE:AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048
2024-11-28T09:36:01-06:00 Informational charon 12[ENC] <con1|2> parsed IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(CHDLESS_SUP) N(MULT_AUTH) ]
2024-11-28T09:36:01-06:00 Informational charon 12[NET] <con1|2> received packet: from 4.213.xx.xx[500] to 10.2.0.250[500] (472 bytes)
2024-11-28T09:36:01-06:00 Informational charon 13[NET] <con1|2> sending packet: from 10.2.0.250[500] to 4.213.xx.xx[500] (464 bytes)
2024-11-28T09:36:01-06:00 Informational charon 13[ENC] <con1|2> generating IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
2024-11-28T09:36:01-06:00 Informational charon 13[IKE] <con1|2> initiating IKE_SA con1[2] to 4.213.xx.xx
2024-11-28T09:36:01-06:00 Informational charon 13[KNL] creating acquire job for policy 10.2.0.250/32 === 4.213.xx.xx/32 with reqid {1}
2024-11-28T09:36:00-06:00 Informational charon 13[CFG] installing 'con1'
2024-11-28T09:36:00-06:00 Informational charon 13[CFG] added vici connection: con1
2024-11-28T09:36:00-06:00 Informational charon 16[CFG] loaded IKE shared key with id 'ike-p1-0' for: '4.213.xx.xx'
2024-11-28T09:36:00-06:00 Informational charon 16[NET] <1> sending packet: from 10.2.0.250[500] to 4.213.xx.xx[500] (36 bytes)
2024-11-28T09:36:00-06:00 Informational charon 16[ENC] <1> generating IKE_SA_INIT response 0 [ N(NO_PROP) ]
2024-11-28T09:36:00-06:00 Informational charon 16[IKE] <1> no IKE config found for 10.2.0.250...4.213.xx.xx, sending NO_PROPOSAL_CHOSEN
2024-11-28T09:36:00-06:00 Informational charon 16[ENC] <1> parsed IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
2024-11-28T09:36:00-06:00 Informational charon 16[NET] <1> received packet: from 4.213.xx.xx[500] to 10.2.0.250[500] (464 bytes)
2024-11-28T09:35:59-06:00 Informational charon 00[JOB] spawning 16 worker threads
2024-11-28T09:35:59-06:00 Informational charon 00[LIB] loaded plugins: charon aes des blowfish rc2 sha2 sha1 md4 md5 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs12 pgp dnskey sshkey pem openssl pkcs8 fips-prf curve25519 xcbc cmac hmac kdf gcm drbg curl attr kernel-pfkey kernel-pfroute resolve socket-default stroke vici updown eap-identity eap-md5 eap-mschapv2 eap-radius eap-tls eap-ttls eap-peap xauth-generic xauth-eap xauth-pam whitelist addrblock counters
2024-11-28T09:35:59-06:00 Informational charon 00[CFG] loaded 0 RADIUS server configurations
2024-11-28T09:35:59-06:00 Informational charon 00[CFG] loading secrets from '/usr/local/etc/ipsec.secrets'
2024-11-28T09:35:59-06:00 Informational charon 00[CFG] loading crls from '/usr/local/etc/ipsec.d/crls'
2024-11-28T09:35:59-06:00 Informational charon 00[CFG] loading attribute certificates from '/usr/local/etc/ipsec.d/acerts'
2024-11-28T09:35:59-06:00 Informational charon 00[CFG] loading ocsp signer certificates from '/usr/local/etc/ipsec.d/ocspcerts'
2024-11-28T09:35:59-06:00 Informational charon 00[CFG] loading aa certificates from '/usr/local/etc/ipsec.d/aacerts'
2024-11-28T09:35:59-06:00 Informational charon 00[CFG] loading ca certificates from '/usr/local/etc/ipsec.d/cacerts'
2024-11-28T09:35:59-06:00 Informational charon 00[CFG] using '/sbin/resolvconf' to install DNS servers
2024-11-28T09:35:59-06:00 Informational charon 00[LIB] providers loaded by OpenSSL: default legacy
2024-11-28T09:35:59-06:00 Informational charon 00[DMN] Starting IKE charon daemon (strongSwan 5.9.14, FreeBSD 14.1-RELEASE-p6, amd64)

Did you even allow remote access in Azure?

Yes - I can ssh into each opnsense machine and I can access the https portal just fine

Did you create firewall rules that allow traffic?

If you mean the IPSec firewall rule, yes. I was following this article https://docs.opnsense.org/manual/how-tos/ipsec-s2s.html
Is there anything else to be allowed?

I am testing opnsese on two VMs on Azure both live in two different virtual network and each has a single public ip address and a since nic as shown the below diagram.

I managed to get the S2S tunnel up but there is no traffic between the two opnsense servers.

I tried to ping a vm (or even the other opnsense) from opnsense1 server, but I get this message:

Code Select Expand

root@OPNsense:~ # ping 10.2.1.4
PING 10.2.1.4 (10.2.1.4): 56 data bytes
ping: sendto: Permission denied
ping: sendto: Permission denied

Same error I get when I try to ssh into the vm on the other side:

Code Select Expand

root@OPNsense:~ # ssh 10.2.1.4
ssh: connect to host 10.2.1.4 port 22: Permission denied

Any help is really appreciated.