Suggestions to configure OPNsense with Juniper switch

[dantargent]

I'm trying to setup my network, but would like advice on the best way.
Below a simplified network outline; I have an OPNSense box sitting between the internet and my private network (in reality multiple VLANs but for simplicity I'm only showing one VLAN).

The managed Juniper EX2200 switch has an l3 routing interface and also runs a DHCP server.
When I move the DHCP server to the OPNsense box (and thus the OPNsense box becomes the gateway) and remove the l3 routing interface on the switch, everything works fine.
However when I run the DHCP server on the switch the packages from the LAN don't enter the WAN side on the OPNsense box. Any suggestions on how to properly configure this?

Currently I have this configuration
Juniper EX2200 switch:

Code Select Expand


set routing-options static route 0.0.0.0/0 next-hop 192.168.99.10


OPNSense box:

   
• Created a gateway 192.168.99.11 on the re0 interface.
• Interface on re0 configured to have static IP 192.168.99.10/24, upstream gateway set to 192.168.99.11
• Add allow all firewall rule on this interface
• I've also tried adding Outbound NAT as well, but to no avail.

Some debug information

Traceroute from 192.168.99.61 to 8.8.8.8

Code Select Expand


traceroute to 8.8.8.8 (8.8.8., 30 hops max, 60 byte packets
1  192.168.99.11 (192.168.99.11)  4.442 ms  4.561 ms  4.718 ms
2  192.168.99.10 (192.168.99.10)  0.265 ms  0.257 ms  0.249 ms
3  * * *
4  * * *
5  * * *


Ping from 192.168.99.61 to 8.8.8.8

Code Select Expand


PING 8.8.8.8 (8.8.8. 56(84) bytes of data.
From 192.168.99.11: icmp_seq=1 Redirect Host(New nexthop: 192.168.99.10)
--- 8.8.8.8 ping statistics ---
83 packets transmitted, 0 received, 100% packet loss, time 83951ms


Attached a visual overview of the network topology

[lilsense]

How does the EX 2200 port set up to connect to the OPNsense?

Also, looks like that OPNsense is unable to send the returned packets. you may need to add routes on the OPNsense.

[Demusman]

Why would you set a gateway to your switch??
You're sending packets from your switch to OPNsense back to your switch.
Makes no sense.

[dantargent]

Thanks Demusman for phrasing it like that, doesn't make sense indeed! I've removed the gateway on the re0 interface and put gateway back to auto. Also removed the NAT.

@lilsense I have the following relevant routes in the routing table (automatically added)

Code Select Expand


Proto Destination Gateway Flags Use MTU Netif Netif (name)
ipv4 192.168.99.0/24 link#3 U NaN 1500 re0 LAN
ipv4 192.168.99.10 link#3 UHS NaN 16384 lo0 Loopback


When running ping from the switch directly nothing comes back:

Code Select Expand


juniperuser@sw01> ping 8.8.8.8 routing-instance route99
PING 8.8.8.8 (8.8.8.8): 56 data bytes
^C
--- 8.8.8.8 ping statistics ---
10 packets transmitted, 0 packets received, 100% packet loss


Package capture on OPNSense on the interface shows the following when running a ping command from the switch (so there is a reply coming back?)

Code Select Expand

LAN
re0 21:47:17.122304 JUNIPERSWITCH > MACADDRESSOFOPNSENSE, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 9977, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.99.11 > 8.8.8.8: ICMP echo request, id 33291, seq 0, length 64
    8.8.8.8 > 192.168.99.11: ICMP echo reply, id 33291, seq 0, length 64
    192.168.99.11 > 8.8.8.8: ICMP echo request, id 33291, seq 1, length 64
    8.8.8.8 > 192.168.99.11: ICMP echo reply, id 33291, seq 1, length 64
    192.168.99.11 > 8.8.8.8: ICMP echo request, id 33291, seq 2, length 64
    8.8.8.8 > 192.168.99.11: ICMP echo reply, id 33291, seq 2, length 64
    192.168.99.11 > 8.8.8.8: ICMP echo request, id 33291, seq 3, length 64
    8.8.8.8 > 192.168.99.11: ICMP echo reply, id 33291, seq 3, length 64
    192.168.99.11 > 8.8.8.8: ICMP echo request, id 33291, seq 4, length 64
    8.8.8.8 > 192.168.99.11: ICMP echo reply, id 33291, seq 4, length 64
    192.168.99.11 > 8.8.8.8: ICMP echo request, id 33291, seq 5, length 64
    8.8.8.8 > 192.168.99.11: ICMP echo reply, id 33291, seq 5, length 64
    192.168.99.11 > 8.8.8.8: ICMP echo request, id 33291, seq 6, length 64
    8.8.8.8 > 192.168.99.11: ICMP echo reply, id 33291, seq 6, length 64
    192.168.99.11 > 8.8.8.8: ICMP echo request, id 33291, seq 7, length 64
    8.8.8.8 > 192.168.99.11: ICMP echo reply, id 33291, seq 7, length 64
    192.168.99.11 > 8.8.8.8: ICMP echo request, id 33291, seq 8, length 64
    8.8.8.8 > 192.168.99.11: ICMP echo reply, id 33291, seq 8, length 64
    192.168.99.11 > 8.8.8.8: ICMP echo request, id 33291, seq 9, length 64
    8.8.8.8 > 192.168.99.11: ICMP echo reply, id 33291, seq 9, length 64

A ping from the switch to the OPNsense box does work though

Code Select Expand


juniperuser@sw01> ping 192.168.99.11 routing-instance route99
PING 192.168.99.11 (192.168.99.11): 56 data bytes
64 bytes from 192.168.25.11: icmp_seq=0 ttl=64 time=2.517 ms
64 bytes from 192.168.25.11: icmp_seq=1 ttl=64 time=0.704 ms
64 bytes from 192.168.25.11: icmp_seq=2 ttl=64 time=0.324 ms
^C
--- 192.168.25.11 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.324/1.182/2.517/0.957 ms


[Demusman]

I think you need to explain a lot more than you are.
Why is your gateway using .11 for starters?
Yeah, it'll work, but why? There's a reason network people use a convention and you're going against it completely.
Can you go into more detail about your setup?
Show pics of your interface settings, vlans included.

You're really making this harder than it is.
No need to use layer3 on the switch, let the router handle that like it's made for.
You should literally just plug the switch into the router, devices into the switch and it should work.
No gateways to the switch, no NAT, just plug it in.

You say you have many vlans. Then you would trunk the port going to the router and tag all your vlans on it. Then untag vlans on ports they're needed on.

[dantargent]

Thanks Demusman, your reply really got me thinking again. And I realized you are completely right, I've added a lot of needles complexity (gateways, NAT, layer3 routing) in the mix. I took your comment 'plug it in' quite literally and just plugged a spare laptop in the re0 port on OPNsense and you know what, it just worked (when pointing the gateway on the laptop to 192.168.25.10)!

So this actually isolated the issue towards the Juniper switch and not the firewall. After removing more complexity, I realized I had turned on arp-inspection and ip-source-guard  (see: https://www.juniper.net/documentation/us/en/software/junos/security-services/topics/example/port-security-protect-from-spoofing-els.html)

So by running on the Juniper switch

Code Select Expand

delete ethernet-switching-options secure-access-port vlan LAN arp-inspection
delete ethernet-switching-options secure-access-port vlan LAN ip-source-guard


everything works like a charm!

Thanks a lot for your help and getting my thinking straight!