Firewall not accessible after changing lan VLAN

[surfrock66]

I apologize in advance for my limited information because I cannot get to the web interface at this point and I'm not as familiar with the console, but I can provide any information requested if you show me how to provide it.

I recently got a new layer 3 switch so that I could enable some 10 gig fiber for my storage network.  Doing so, I wanted to move most of my networking off of VLAN 1 and onto a new VLAN, VLAN 99.  Before I did this move, I created a new VLAN interface on the open sense firewall.  I added the new 99 VLAN to all the required trunks on all my other switches, changed the IP addresses on all of my other devices on that VLAN, and even added a new management interface on OpnSense which was reachable on the 99 VLAN at 10.99.1.40. 

I thought I was ready to cut over and I did, replacing the old layer 3 switch with the new one.  For the most part everything went well, and everything on the lan appears to work as expected.  That being said, my firewall is now completely unreachable.  Fired up the console and reconfigured the interfaces and IPs, but it appears to be unreachable on the lan or the wan.  My lan interface is an LACP lagg which should be configured on the L3 switch as well, And the ports show as up on the switch but I cannot ping it even from the l3 switch where a gateway is configured on the 99 VLAN.  The wan is a single connection to my Comcast business gateway, And I believe it is properly configured as I am getting a DHCP address on it.  From the shell, I cannot ping 8.8.8.8.  as far as I can see, both the LAN and WAN interfaces are not communicating out and I am kind of stumped.  Here is as much information as I have right now but I can provide any more information as requested.

[surfrock66]

One additional bit of information, if I am able to get into the shell and do a ping with "-S" specifying the IP of the WAN interface I am able to successfully paying 8888.  I think I just have some sort of a routing configuration problem that I do not understand how to fix from the console

[surfrock66]

Ok, for now I took LACP out of the equation and it appears to be working, I have an interface on an unused port directly between my opnsense firewall and my L3 switch and all is well to get to the firewall on the LAN via vlan 99.

That being said, I have one final issue...I'm not passing traffic to WAN, and I think I've been looking at it too long to see the issue.  I have 2 interfaces with 2 gateways; the WAN interface has a gateway from DHCP from comcast and is getting auto-created with weight 254, and my LAN interface gets auto-detected with a gateway on the L3 switch on the 99 VLAN (as would be expected) but with weight 255, and is being tagged as (active) in the interface.

I haven't messed with routing rules on the LAN, but it appears I'm getting into a routing loop:

Code Select Expand

domainname@prefix-thelio:~/.scripts$ ping 10.99.1.40
PING 10.99.1.40 (10.99.1.40) 56(84) bytes of data.
64 bytes from 10.99.1.40: icmp_seq=1 ttl=63 time=0.166 ms
64 bytes from 10.99.1.40: icmp_seq=2 ttl=63 time=0.103 ms
^C
--- 10.99.1.40 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1005ms
rtt min/avg/max/mdev = 0.103/0.134/0.166/0.031 ms
domainname@prefix-thelio:~/.scripts$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
From 10.99.1.40 icmp_seq=1 Time to live exceeded
From 10.99.1.40 icmp_seq=2 Time to live exceeded
^C
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1000ms

domainname@prefix-thelio:~/.scripts$ traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
1  prefix-prosafe-00.subdomain.domainname.com (10.4.1.254)  0.385 ms  1.415 ms  0.664 ms
2  prefix-opnsense-1.subdomain.domainname.com (10.99.1.40)  0.302 ms  0.285 ms  0.268 ms
3  10.99.1.254 (10.99.1.254)  0.754 ms  0.914 ms  1.078 ms
4  prefix-opnsense-1.subdomain.domainname.com (10.99.1.40)  1.250 ms  1.231 ms  1.210 ms
5  * * *
6  prefix-opnsense-1.subdomain.domainname.com (10.99.1.40)  1.126 ms  0.324 ms  0.424 ms
7  * * 10.99.1.254 (10.99.1.254)  0.543 ms
8  prefix-opnsense-1.subdomain.domainname.com (10.99.1.40)  0.433 ms  0.470 ms  0.458 ms
9  10.99.1.254 (10.99.1.254)  0.692 ms  0.831 ms *
10  prefix-opnsense-1.subdomain.domainname.com (10.99.1.40)  0.560 ms  0.596 ms  0.714 ms
11  * * *
12  prefix-opnsense-1.subdomain.domainname.com (10.99.1.40)  0.814 ms  0.645 ms  0.778 ms
13  * * *
14  prefix-opnsense-1.subdomain.domainname.com (10.99.1.40)  0.893 ms  0.879 ms  0.819 ms
15  * * *
16  prefix-opnsense-1.subdomain.domainname.com (10.99.1.40)  1.104 ms  0.966 ms  0.994 ms
17  * 10.99.1.254 (10.99.1.254)  0.995 ms  1.168 ms
18  prefix-opnsense-1.subdomain.domainname.com (10.99.1.40)  0.812 ms  0.846 ms  0.874 ms
19  10.99.1.254 (10.99.1.254)  1.256 ms * *
20  prefix-opnsense-1.subdomain.domainname.com (10.99.1.40)  0.988 ms  0.867 ms  0.918 ms
21  * * *
22  prefix-opnsense-1.subdomain.domainname.com (10.99.1.40)  1.250 ms  1.360 ms  1.320 ms
23  * * *
24  prefix-opnsense-1.subdomain.domainname.com (10.99.1.40)  1.222 ms  1.842 ms  1.292 ms
25  * * *
26  prefix-opnsense-1.subdomain.domainname.com (10.99.1.40)  1.430 ms  1.460 ms  2.181 ms
27  * * *
28  prefix-opnsense-1.subdomain.domainname.com (10.99.1.40)  1.817 ms  1.835 ms  1.717 ms
29  * * *
30  prefix-opnsense-1.subdomain.domainname.com (10.99.1.40)  1.827 ms  1.811 ms  1.862 ms


I've been looking at this for so long I'm not seeing what to do.  In System -> Routes -> Status the default route is my LAN gateway on the 99 network, so it makes sense that the L3 switch sends traffic to the firewall, then that is sending traffic to the L3 switch, and it's a loop. 

In the Firewall rules, my WAN just has port forward rules which used to work, and the LAN just has the rules in the attached photos (and some chromecast internal dns rules which aren't in play)

I'm stuck and I've been looking at this for long enough I'm not seeing clearly, any advice is appreciated.

[surfrock66]

The good news is that I didn't have a NAT rule in; I changed it to hybrid and added an outbound manual rule on the LAN interface (from information in this thread https://forum.opnsense.org/index.php?topic=18889.0 ).  Now I can access the internet, which is great!

Several things are NOT working though.

1) The firewall itself is not accessing the WAN, for example, I cannot check for updates.  I do have an internal DNS and DHCP server which are working and Opnsense is using this DNS server for resolution.
2) My port forward rules aren't working at all, from the outside I appear to be unable to get to anything on the inside.
3) I'm still on the non-lacp interface; I was never able to figure out why vlan traffic wasn't passing through the LACP interface.

I think all three of those are solvable though; any advice is appreciated but it's less urgent as the family has internet at the moment.

[surfrock66]

I've provided some more screenshots to help with the config; I'm not sure if it's helpful or not.  My L3 gateway is the router and it's default route points to the opnsense firewall, which has 1 interface internal and 1 to comcast, and I'm sure I messed something up.

[surfrock66]

2 more pics of configs