Messages

I turned off "far gateway" without any success.
I'm not exactly sure how to set the default gateway for the LAN clients. I did set the router to have routes 0.0.0.0/1 and 128.0.0.0/1 with next hop to 10.103.19.20 (internal opnsense ip)

I did some more tests and instead of tracerouting to google I tried to do it to the external IP of opnsense instead and it doesn't work as it gets stuck at the internal side.

Code Select Expand


sudo traceroute -T 10.103.19.20 (internal ip)
traceroute to 10.103.19.20 (10.103.19.20), 30 hops max, 60 byte packets
1  10.103.19.20 (10.103.19.20)  4.360 ms  6.671 ms  4.454 ms

sudo traceroute -T 10.110.100.20 (external ip)
traceroute to 10.110.100.20 (10.110.100.20), 30 hops max, 60 byte packets
1  _gateway (10.103.10.1)  2.965 ms  2.809 ms  2.695 ms
2  10.103.19.20 (10.103.19.20)  3.575 ms  2.532 ms  3.346 ms


I also tried to curl to google and the internal server using both the internal and external interface on the opnsense server and that worked without any issues.

Hello,
First time setting up an opnsense server and I have a strange issue with my set up. With the setup I have, I am able to connect from the internet to the internal server but I am not able to talk to the internet however the opnsense server is able to communicate with the internet. Both internal and opnsense server are able to curl each other so there doesn't seem to be any connection issues between the two. If I change the routes to another firewall (that we are migrating away from) it all works fine.

OK:
Internet -> opnsense -> internal server
opnsense -> internal server
internal server -> opnsense
opnsense -> internet

Failing:
internal server -> opnsense -> internet


The setup is in an openstack environment if that affects anything.



My networks are:
- ext-net (10.110.100.0/24)
- int-net1 (10.103.10.0/24)
- int-net2 (10.103.10.0/24)

IPs:
opnsense
- ext-net: 10.110.100.15
- int-net2: 10.103.19.10

internal server:
- int-net: 10.103.10.20


Openstack router with int-net1 & int-net2 connected
Have routes 0.0.0.0/1 and 128.0.0.0/1 with next hop to 10.110.100.15

In opnsense I currently have:

System: Gateways: Configuration
WAN_DHCP - Upstream Gateway
LAN_DHCP - Far Gateway

Firewall: Settings: Advanced
All checkboxes for NAT checked

Firewall: NAT: Outbound
Interface: LAN, src/dst any, NAT Address: Interface address, Static port: No
Tried WAN as well



From the internal server when I run traceroute I get:

Code Select Expand


sudo traceroute -T google.com
traceroute to google.com (216.58.207.238), 30 hops max, 60 byte packets
1  _gateway (10.103.10.1)  4.701 ms  3.134 ms  3.112 ms
2  10.103.19.20 (10.103.19.20)  3.763 ms  3.868 ms  3.849 ms
3  * * *
4  * * *



Also I ran package capture (and curl to google) and to my eyes it looks like it goes out and comes back...

Code Select Expand


Interface Timestamp SRC DST output

pflog0 2024-12-11
06:36:15.607765 ttl 62, id 57784, offset 0, flags [DF], proto TCP (6), length 60)
    10.110.100.15.43468 > 216.58.207.238.443: Flags [S], cksum 0xa29a (correct), seq 3051706853, win 64240, options [mss 1460,sackOK,TS val 721005954 ecr 0,nop,wscale 7], length 0
LAN
vtnet0 2024-12-11
06:36:15.607732 fa:16:3e:0a:4d:fa
***
fa:16:3e:8e:c8:89
***
ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 63, id 57784, offset 0, flags [DF], proto TCP (6), length 60)
    10.103.10.20.33782 > 216.58.207.238.443: Flags [S], cksum 0x2273 (correct), seq 3051706853, win 64240, options [mss 1460,sackOK,TS val 721005954 ecr 0,nop,wscale 7], length 0
LAN
vtnet0 2024-12-11
06:36:15.609338 fa:16:3e:8e:c8:89 fa:16:3e:0a:4d:fa ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 59, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    216.58.207.238.443 > 10.103.10.20.33782: Flags [S.], cksum 0xa3c5 (correct), seq 727142400, ack 3051706854, win 65535, options [mss 1412,sackOK,TS val 378202074 ecr 721005954,nop,wscale 8], length 0
LAN
vtnet0 2024-12-11
06:36:15.912366 fa:16:3e:8e:c8:89 fa:16:3e:0a:4d:fa ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 59, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    216.58.207.238.443 > 10.103.10.20.33782: Flags [S.], cksum 0xa296 (correct), seq 727142400, ack 3051706854, win 65535, options [mss 1412,sackOK,TS val 378202377 ecr 721005954,nop,wscale 8], length 0
LAN
vtnet0 2024-12-11
06:36:16.637422 fa:16:3e:0a:4d:fa fa:16:3e:8e:c8:89 ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 63, id 57785, offset 0, flags [DF], proto TCP (6), length 60)
    10.103.10.20.33782 > 216.58.207.238.443: Flags [S], cksum 0x1e6d (correct), seq 3051706853, win 64240, options [mss 1460,sackOK,TS val 721006984 ecr 0,nop,wscale 7], length 0
LAN
vtnet0 2024-12-11
06:36:16.638664 fa:16:3e:8e:c8:89 fa:16:3e:0a:4d:fa ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 59, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    216.58.207.238.443 > 10.103.10.20.33782: Flags [S.], cksum 0x9fc0 (correct), seq 727142400, ack 3051706854, win 65535, options [mss 1412,sackOK,TS val 378203103 ecr 721005954,nop,wscale 8], length 0
LAN
vtnet0 2024-12-11
06:36:18.685239 fa:16:3e:0a:4d:fa fa:16:3e:8e:c8:89 ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 63, id 57786, offset 0, flags [DF], proto TCP (6), length 60)
    10.103.10.20.33782 > 216.58.207.238.443: Flags [S], cksum 0x166d (correct), seq 3051706853, win 64240, options [mss 1460,sackOK,TS val 721009032 ecr 0,nop,wscale 7], length 0
LAN
vtnet0 2024-12-11
06:36:18.686489 fa:16:3e:8e:c8:89 fa:16:3e:0a:4d:fa ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 59, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    216.58.207.238.443 > 10.103.10.20.33782: Flags [S.], cksum 0x97c0 (correct), seq 727142400, ack 3051706854, win 65535, options [mss 1412,sackOK,TS val 378205151 ecr 721005954,nop,wscale 8], length 0
LAN
vtnet0 2024-12-11
06:36:20.744355 fa:16:3e:8e:c8:89 fa:16:3e:0a:4d:fa ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 59, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    216.58.207.238.443 > 10.103.10.20.33782: Flags [S.], cksum 0x8fb6 (correct), seq 727142400, ack 3051706854, win 65535, options [mss 1412,sackOK,TS val 378207209 ecr 721005954,nop,wscale 8], length 0
LAN
vtnet0 2024-12-11
06:36:22.717319 fa:16:3e:0a:4d:fa fa:16:3e:8e:c8:89 ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 63, id 57787, offset 0, flags [DF], proto TCP (6), length 60)
    10.103.10.20.33782 > 216.58.207.238.443: Flags [S], cksum 0x06ad (correct), seq 3051706853, win 64240, options [mss 1460,sackOK,TS val 721013064 ecr 0,nop,wscale 7], length 0
LAN
vtnet0 2024-12-11
06:36:22.718643 fa:16:3e:8e:c8:89 fa:16:3e:0a:4d:fa ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 59, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    216.58.207.238.443 > 10.103.10.20.33782: Flags [S.], cksum 0x8800 (correct), seq 727142400, ack 3051706854, win 65535, options [mss 1412,sackOK,TS val 378209183 ecr 721005954,nop,wscale 8], length 0
WAN
vtnet1 2024-12-11
06:36:15.607771 fa:16:3e:7c:1a:a6 fa:16:3e:05:85:93 ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 62, id 57784, offset 0, flags [DF], proto TCP (6), length 60)
    10.110.100.15.43468 > 216.58.207.238.443: Flags [S], cksum 0xa29a (correct), seq 3051706853, win 64240, options [mss 1460,sackOK,TS val 721005954 ecr 0,nop,wscale 7], length 0
WAN
vtnet1 2024-12-11
06:36:15.609325 fa:16:3e:05:85:93 fa:16:3e:7c:1a:a6 ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 60, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    216.58.207.238.443 > 10.110.100.15.43468: Flags [S.], cksum 0x23ed (correct), seq 727142400, ack 3051706854, win 65535, options [mss 1412,sackOK,TS val 378202074 ecr 721005954,nop,wscale 8], length 0
WAN
vtnet1 2024-12-11
06:36:15.912347 fa:16:3e:05:85:93 fa:16:3e:7c:1a:a6 ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 60, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    216.58.207.238.443 > 10.110.100.15.43468: Flags [S.], cksum 0x22be (correct), seq 727142400, ack 3051706854, win 65535, options [mss 1412,sackOK,TS val 378202377 ecr 721005954,nop,wscale 8], length 0
WAN
vtnet1 2024-12-11
06:36:16.637434 fa:16:3e:7c:1a:a6 fa:16:3e:05:85:93 ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 62, id 57785, offset 0, flags [DF], proto TCP (6), length 60)
    10.110.100.15.43468 > 216.58.207.238.443: Flags [S], cksum 0x9e94 (correct), seq 3051706853, win 64240, options [mss 1460,sackOK,TS val 721006984 ecr 0,nop,wscale 7], length 0
WAN
vtnet1 2024-12-11
06:36:16.638655 fa:16:3e:05:85:93 fa:16:3e:7c:1a:a6 ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 60, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    216.58.207.238.443 > 10.110.100.15.43468: Flags [S.], cksum 0x1fe8 (correct), seq 727142400, ack 3051706854, win 65535, options [mss 1412,sackOK,TS val 378203103 ecr 721005954,nop,wscale 8], length 0
WAN
vtnet1 2024-12-11
06:36:18.685254 fa:16:3e:7c:1a:a6 fa:16:3e:05:85:93 ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 62, id 57786, offset 0, flags [DF], proto TCP (6), length 60)
    10.110.100.15.43468 > 216.58.207.238.443: Flags [S], cksum 0x9694 (correct), seq 3051706853, win 64240, options [mss 1460,sackOK,TS val 721009032 ecr 0,nop,wscale 7], length 0
WAN
vtnet1 2024-12-11
06:36:18.686477 fa:16:3e:05:85:93 fa:16:3e:7c:1a:a6 ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 60, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    216.58.207.238.443 > 10.110.100.15.43468: Flags [S.], cksum 0x17e8 (correct), seq 727142400, ack 3051706854, win 65535, options [mss 1412,sackOK,TS val 378205151 ecr 721005954,nop,wscale 8], length 0


I also looked at the states when doing a curl and saw this

Code Select Expand



Int
Dir
Proto
Source
Nat
Destination
Gateway
State
Rule
Commands
all > tcp 10.103.10.20:38960 216.58.207.238:443 SYN_SENT:ESTABLISHED
all < tcp 10.110.100.15:59758 10.103.10.20:38960 216.58.207.238:443 ESTABLISHED:SYN_SENT let out anything from firewall host itself (force gw)



If there is anything that I missed, please let me know!