OPNsense Forum
Archive => 21.1 Legacy Series => Topic started by: tryllz on March 04, 2021, 11:06:41 pm
-
Hi,
I have the following network setup https://i.ibb.co/wwPLH2H/Network.png (https://i.ibb.co/wwPLH2H/Network.png)
All traffic from 10.0.64.0 / 27 behind FirewallB (firewallsm) reaches 192.168.28.0 / 27 network via the LAN interface of FirewallA (firewallwm), and the same traffic also reaches internet in the same way, as follows:
10.0.64.42 (VM) > FirewallB (LAN) > FirewallA (LAN) > FirewallA (WAN) > Laptop's Wireless NIC > Wifi Router
Strangely FirewallB (firewallsm) can ping Google DNS but the VM 10.0.64.42 for some reason cannot ping Google DNS. I have set all protocols, ports as allowed on FirewallB (firewallsm) to reach FirewallA (firewallwm).
FirewallA (firewallwm)
Gateway - https://i.ibb.co/bRC8P8G/Firewall-A-GW.png (https://i.ibb.co/bRC8P8G/Firewall-A-GW.png)
LAN Interface Rule - https://i.ibb.co/XWSnLRd/Firewall-A-1-dell-Rule.png (https://i.ibb.co/XWSnLRd/Firewall-A-1-dell-Rule.png)
WAN Interface Rule - https://i.ibb.co/zZwcnjJ/Firewall-A-2-WAN-Rule.png (https://i.ibb.co/zZwcnjJ/Firewall-A-2-WAN-Rule.png)
FirewallA (firewallsm) logs show 10.0.64.42 traffic is allowed through its WAN
Log - https://i.ibb.co/9cqPSW7/Firewall-A-Packet-log.png (https://i.ibb.co/9cqPSW7/Firewall-A-Packet-log.png)
Log - https://i.ibb.co/kVTX31B/Firewall-A-Packet-log-2.png (https://i.ibb.co/kVTX31B/Firewall-A-Packet-log-2.png)
FirewallB (firewallsm)
Gateway - https://i.ibb.co/pPQC1p8/Firewall-B-GW.png (https://i.ibb.co/pPQC1p8/Firewall-B-GW.png)
LAN Rule - https://i.ibb.co/VY9vFVL/Firewall-B-1-dell-Rule.png (https://i.ibb.co/VY9vFVL/Firewall-B-1-dell-Rule.png)
tcpdump for 10.0.64.42 VM on FirewallB LAN (em0)
root@firewallsm:~ # tcpdump -i vmx0 host 10.0.64.42 and host 8.8.8.8 and icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmx0, link-type EN10MB (Ethernet), capture size 262144 bytes
06:17:09.324409 IP 10.0.64.42 > dns.google: ICMP echo request, id 1, seq 1, length 40
06:17:13.853917 IP 10.0.64.42 > dns.google: ICMP echo request, id 1, seq 2, length 40
06:17:18.858484 IP 10.0.64.42 > dns.google: ICMP echo request, id 1, seq 3, length 40
tcpdump for 10.0.64.42 VM on FirewallA LAN (em0)
root@firewallwm:~ # tcpdump -i em0 host 8.8.8.8 and icmp -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em0, link-type EN10MB (Ethernet), capture size 262144 bytes
06:17:09.335331 IP 10.0.64.42 > 8.8.8.8: ICMP echo request, id 1, seq 1, length 40
06:17:13.865408 IP 10.0.64.42 > 8.8.8.8: ICMP echo request, id 1, seq 2, length 40
06:17:23.870819 IP 10.0.64.42 > 8.8.8.8: ICMP echo request, id 1, seq 4, length 40
tcpdump for 10.0.64.42 VM on FirewallA WAN (em1)
root@firewallwm:~ # tcpdump -i em1 host 8.8.8.8 and icmp -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em1, link-type EN10MB (Ethernet), capture size 262144 bytes
Ping from FirewallB
root@firewallsm:~ # ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=127 time=13.196 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=127 time=12.625 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=127 time=12.609 ms
^C
--- 8.8.8.8 ping statistics ---
16 packets transmitted, 16 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 11.705/13.026/15.910/1.111 ms
tcpdump of FirewallA LAN (em0)
root@firewallwm:~ # tcpdump -i em0 host 8.8.8.8 and icmp -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em0, link-type EN10MB (Ethernet), capture size 262144 bytes
06:22:21.816908 IP 192.168.10.7 > 8.8.8.8: ICMP echo request, id 23594, seq 0, length 64
06:22:21.827095 IP 8.8.8.8 > 192.168.10.7: ICMP echo reply, id 23594, seq 0, length 64
06:22:22.876598 IP 192.168.10.7 > 8.8.8.8: ICMP echo request, id 23594, seq 1, length 64
06:22:22.886317 IP 8.8.8.8 > 192.168.10.7: ICMP echo reply, id 23594, seq 1, length 64
06:22:23.948947 IP 192.168.10.7 > 8.8.8.8: ICMP echo request, id 23594, seq 2, length 64
06:22:23.957978 IP 8.8.8.8 > 192.168.10.7: ICMP echo reply, id 23594, seq 2, length 64
tcpdump of FirewallA WAN (em1)
root@firewallwm:~ # tcpdump -i em1 host 8.8.8.8 and icmp -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em1, link-type EN10MB (Ethernet), capture size 262144 bytes
06:22:21.817029 IP 192.168.47.132 > 8.8.8.8: ICMP echo request, id 24689, seq 0, length 64
06:22:21.826993 IP 8.8.8.8 > 192.168.47.132: ICMP echo reply, id 24689, seq 0, length 64
06:22:22.876700 IP 192.168.47.132 > 8.8.8.8: ICMP echo request, id 24689, seq 1, length 64
06:22:22.886219 IP 8.8.8.8 > 192.168.47.132: ICMP echo reply, id 24689, seq 1, length 64
06:22:23.949057 IP 192.168.47.132 > 8.8.8.8: ICMP echo request, id 24689, seq 2, length 64
06:22:23.957845 IP 8.8.8.8 > 192.168.47.132: ICMP echo reply, id 24689, seq 2, length 64
I'm kind of lost as to what the issue is, any thoughts, thanks..
-
Is VM behide Firewall 2 Traffic NAT'd when its goes to Firewall 1?
-
Is VM behide Firewall 2 Traffic NAT'd when its goes to Firewall 1?
There is no NATting on FirewallB, may be default NAT is being applied, I have not set any, all NATting occurs on FirewallA..
I'll check this on FirewallB.
-
Does Firewall 1 know to route 10.0.64.0/27 via Firewall2 192.168.28.0 address.
So it knows where to send the return traffic to?
-
Does Firewall 1 know to route 10.0.64.0/27 via Firewall2 192.168.28.0 address.
So it knows where to send the return traffic to?
Sorry I'm getting confused as to which one are you referring to as Firewall 1.
Both firewalls have Static routes to all their respective networks. FirewallA has 2 Gateways, 1 for WAN (upstream) and 1 for 10.64.0 network, FirewallB has only 1 Gateway.
FirewallA's LAN VMs can access both, internet via WAN Gateway and 10.0.64.0 via 192.168.10.7 (LAN interface of FirewallB).
FirewallB's LAN VM can access 192.168.28.0 via 192.168.10.3 (LAN interface of FirewallA) (this part works fine).
Currently both firewalls' networks are able to reach other, and as FirewallB's VMs can only reach internet through FirewallA. There seems to be some issue there even though FirewallA's tcpdump on WAN show 10.0.64.42 was allowed out the WAN of FirewallA.
I'm not sure how to set path for return internet traffic, and whether I have to set it specifically, if yes where ?
Thank You
-
FirewallB has default NAT being applied.
https://i.ibb.co/v1p2wRW/Firewall-B-NAT.png (https://i.ibb.co/v1p2wRW/Firewall-B-NAT.png)
-
I'm not sure why you configured it as a 2nd gateway on FirewallA, you should be using just static routes "System: Routes: Configuration".
What you should have is a static route to 10.0.64.0/27 to via the FirewallB address on the 192.168.28.0 network. If you start to use gateways you'll complicate things and may hit issues with reply-to.
Once the static route on FirewallA is done, it'll know how to return taffic for a 10.0.64.0/27 VM, which it'll route to 192.168.28.0/27 address of FirewallB. So it can then forward the traffic back to teh VM on the otherside.
I hope this makes more sense?
-
I'm not sure why you configured it as a 2nd gateway on FirewallA, you should be using just static routes "System: Routes: Configuration".
What you should have is a static route to 10.0.64.0/27 to via the FirewallB address on the 192.168.28.0 network. If you start to use gateways you'll complicate things and may hit issues with reply-to.
Once the static route on FirewallA is done, it'll know how to return taffic for a 10.0.64.0/27 VM, which it'll route to 192.168.28.0/27 address of FirewallB. So it can then forward the traffic back to teh VM on the otherside.
I hope this makes more sense?
I'm unsure if my explanation is understood.
VMs in both networks 192.168.28.0 and 10.0.64.0 can communicate with each other via Static Routes.
VMs in network 192.168.28.0 can communicate with the internet.
VMs in network 10.0.64.0 cannot communicate with the internet.
The 1st Gateway (192.168.27.2, upstream) allows for traffic from all networks to reach the internet.
The 2nd gateway on FirewallA (192.168.10.7) allows traffic from 192.168.28.0 to reach 10.0.64.0 (behind FirewallB).
If I just set gateway as 192.168.47.2 on FirewallA, traffic from FirewallB cannot reach 192.168.28.0.
If I just set gateway as 192.168.10.7 on FirewallA, traffic from FirewallB cannot reach the internet.
With the current configuration traffic from FirewallB can reach both 192.168.28.0, and the internet (at least FirewallB can reach the internet as of now, only the Vms behind it can't).
-
I'm still confused to why you have a second gateway its not required, for this setup.
Do you have Teamviewer or something I can remote in and take a look, as that would be easier? :)
You can find me on the IRC room.
-
Remoted on to Tryllz setup and he'd had it all setup correctly, between the two firewalls, and the gateway is required for the static routes, something I forgot ::).
The reason to why the internet didn't work was due to Outbound NAT. Since FirewallA doesn't see any external networks of FirewallB, so when you use Automatic NAT, it doesn't create rules for these networks.
So I put his NAT in to manual mode and created 2 NAT rules which will cover 10.0.0.0/8 and 192.168.0.0/16, so when he creates any new external private networks be hide FirewallB, they'll get internet access if the firewall lets them.
I would of created a ALIAS called RFC1918 and then used that as the source on the NAT rule, but since the browser was being glitchy and wouldn't open then new alias model, I had to do it this way.
This issue should now be resolved as Tryllz can now reach the internet on the VM he couldn't do before.
-
Remoted on to Tryllz setup and he'd had it all setup correctly, between the two firewalls, and the gateway is required for the static routes, something I forgot ::).
The reason to why the internet didn't work was due to Outbound NAT. Since FirewallA doesn't see any external networks of FirewallB, so when you use Automatic NAT, it doesn't create rules for these networks.
So I put his NAT in to manual mode and created 2 NAT rules which will cover 10.0.0.0/8 and 192.168.0.0/16, so when he creates any new external private networks be hide FirewallB, they'll get internet access if the firewall lets them.
I would of created a ALIAS called RFC1918 and then used that as the source on the NAT rule, but since the browser was being glitchy and wouldn't open then new alias model, I had to do it this way.
This issue should now be resolved as Tryllz can now reach the internet on the VM he couldn't do before.
Thank you FingerlessGloves for taking the timeout and helping me solve this issue via remote access, its been going on for a month now, extremely grateful.
As for the modal pop-up window glitch, I realized I was using the previous version 20.7 in which I did not face this glitch.
I tested the the modal window on a different browser and it worked fine.