Automatically generated rules - is the reason I stopped migrating to OPNSense

[CJ]

Are you still using LAN as the parent of all of the VLANs?

Yes.

However, I come across this question couple of times now. Is that not how we supposed to do it?

The default LAN rule allows all.  I know there is a recommendation not to mix tagged and untagged traffic on the same interface but I forget the exact details for why.  If you change your VLANs to use something other than LAN as the parent, you may see things work as you expect.

That all said, why not try testing with separate interfaces and no VLANs.  That way you eliminate all of the variables of incorrect VLAN setup, etc.  It will help isolate if the problem is truly OPNSense or not.

[newjohn]

Are you still using LAN as the parent of all of the VLANs?

Yes.

However, I come across this question couple of times now. Is that not how we supposed to do it?

The default LAN rule allows all.  I know there is a recommendation not to mix tagged and untagged traffic on the same interface but I forget the exact details for why.  If you change your VLANs to use something other than LAN as the parent, you may see things work as you expect.

That all said, why not try testing with separate interfaces and no VLANs.  That way you eliminate all of the variables of incorrect VLAN setup, etc.  It will help isolate if the problem is truly OPNSense or not.

Ahh now I understand. I will dig out some 4 ports NIC cards. Ok for this i need to swap from 2 port NIC to 4 port NIC so i can use the 3rd port as VLANs parent interface.

So the new setup will look like this:

Port 1 = WAN
Port 2 = LAN
Port 3 = Parent Interface for all VLANS. - This needs to be Trunk/Tagged Ports for all VLANS?

TIA

[netnut]

I know there is a recommendation not to mix tagged and untagged traffic on the same interface but I forget the exact details for why.

Because in many cases you get weird (and hard to debug) behavior depending on the underlying infrastructure. Take the example below, nothing to do with OPNSense itself, but if you plug your firewall into this (trunk) port you get three different results (mind the minor differences). This is "just" configuration, I'm not even opening the can-of-worms of your $5 vlan aware "smart-switch"....

VLAN RED = 100
VLAN BLUE = 200

# Trunk

Code: [Select]

ge-0/0/0 {
    unit 0 {
        family ethernet-switching {
            port-mode trunk;
            vlan members [ RED BLUE ];
        }
    }
}

# Trunk with Tagged Native

Code: [Select]

ge-0/0/0 {
    unit 0 {
        family ethernet-switching {
            port-mode trunk;
            vlan members [ RED BLUE ];
            native-vlan-id 100;
        }
    }
}

# Trunk with Untagged Native

Code: [Select]


ge-0/0/0 {
    unit 0 {
        family ethernet-switching {
            port-mode trunk;
            vlan members [ BLUE ];
            native-vlan-id 100;
        }
    }
}


[newjohn]

Whats is your take on this please?
Please see the attached screenshot.

I explained my take extensivily in previous post:

- Go back to the drawing board
- Simplify your setup
- From here proof your initial statement that "Automatically generated rules" allows traffic that isn't expected to be allowed (ICMP or whatever.)

So a default install, 1 WAN, 1 LAN and 1 OPT interface, no VLAN's, no manual config except for 3 interface IP configs. If in this setup you can proof your statement many people are willing to look into your issue. Should take you less time than posting screenshoits of state reset buttons.

For me, I'm too old to look into virtualised infra's with VLAN trunk ports without a detailed low level design and not knowing if basic network skills are in place.

As they say in the famous movie "Houston we have a problem"

I kept drilling down as I kept getting replies to my pings.

In the end to remove any middleman issues and confirm once and for all this is opnsense issue, I removed the switch, the virtualization and went baremetal.

Testing env:
Two baremetal win11 PCs.
4 port fresly installed Opnsense.

PCs: Both baremetal:
PC1 - connected directly to LAN port on the opnsense firewall = IP add 10.1.1.1
PC2 - connected directly to OPT1 port on the opnsense firewall = IP add 10.2.2.2

Opnsense:
Fresh install, the only config I added was to enable OPT1 and assing IP Address 10.2.2.254. Nothing else changed.

Test result, I still get a response to the ping.
Steps:
Issued the ping command form PC2, at first dont get a response
However, as soon as you issue the ping command on PC1, both PCs can ping each other.

Please see the screenshots.
Note: The system due to the size does not allow me to attach all the screenshots in one go. I will add them one by one.

[newjohn]

Image addition:

[newjohn]

Image addition:

[newjohn]

Image addition:

[newjohn]

Image addition:

[newjohn]

Image addition:

[Monviech (Cedrik)]

I have actually confirmed this behavior in OPNsense 23.7 now since the test methology was specified properly.

I live booted a opnsense VM (latest ISO from the website), put 3 virtual private switches on it in hyper-v, configured lan and opt 1 with the TUI Wizard in the shell.

Booted up two Ubuntu 22.04, one in LAN and the other in OPT1, and yes I could ping between the VMs.

Ping from OPT1 to LAN was possible after the first ping from LAN to OPT1. I have looked into pfctl and there were only allow rules for LAN loaded.

Code: [Select]

VM1 192.168.2.3 <----> OPT1 192.168.2.1/24 OPNSENSE 192.168.1.1/24 LAN <----> 192.168.1.101 VM2
Will post packet captures soon here as edit:

- Both ICMP echo request and ICMP echo reply have the same ID. There is no new ID generated, which must means it all happens in the same stateful session.

PACKET CAPTURE:

Code: [Select]

root@pc07:/home/administrator# ip a
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:15:5d:00:c9:76 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.101/24 metric 100 brd 192.168.1.255 scope global dynamic eth0
       valid_lft 4933sec preferred_lft 4933sec
    inet6 fe80::215:5dff:fe00:c976/64 scope link
       valid_lft forever preferred_lft forever

root@pc07:/home/administrator# ping 192.168.2.3
PING 192.168.2.3 (192.168.2.3) 56(84) bytes of data.
64 bytes from 192.168.2.3: icmp_seq=1 ttl=63 time=0.653 ms
64 bytes from 192.168.2.3: icmp_seq=2 ttl=63 time=0.602 ms
64 bytes from 192.168.2.3: icmp_seq=3 ttl=63 time=0.666 ms
64 bytes from 192.168.2.3: icmp_seq=4 ttl=63 time=1.01 ms
64 bytes from 192.168.2.3: icmp_seq=5 ttl=63 time=0.673 ms
64 bytes from 192.168.2.3: icmp_seq=6 ttl=63 time=1.38 ms
^C
--- 192.168.2.3 ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5085ms
rtt min/avg/max/mdev = 0.602/0.829/1.377/0.278 ms

root@pc07:/home/administrator# tcpdump -i any proto ICMP -n
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
08:32:25.001798 eth0  Out IP 192.168.1.101 > 192.168.2.3: ICMP echo request, id 8, seq 1, length 64
08:32:25.002434 eth0  In  IP 192.168.2.3 > 192.168.1.101: ICMP echo reply, id 8, seq 1, length 64
08:32:26.012412 eth0  In  IP 192.168.2.3 > 192.168.1.101: ICMP echo request, id 8, seq 5, length 64
08:32:26.012453 eth0  Out IP 192.168.1.101 > 192.168.2.3: ICMP echo reply, id 8, seq 5, length 64
08:32:26.022547 eth0  Out IP 192.168.1.101 > 192.168.2.3: ICMP echo request, id 8, seq 2, length 64
08:32:26.023114 eth0  In  IP 192.168.2.3 > 192.168.1.101: ICMP echo reply, id 8, seq 2, length 64
08:32:27.036417 eth0  In  IP 192.168.2.3 > 192.168.1.101: ICMP echo request, id 8, seq 6, length 64
08:32:27.036456 eth0  Out IP 192.168.1.101 > 192.168.2.3: ICMP echo reply, id 8, seq 6, length 64
08:32:27.046642 eth0  Out IP 192.168.1.101 > 192.168.2.3: ICMP echo request, id 8, seq 3, length 64
08:32:27.047275 eth0  In  IP 192.168.2.3 > 192.168.1.101: ICMP echo reply, id 8, seq 3, length 64
08:32:28.060402 eth0  In  IP 192.168.2.3 > 192.168.1.101: ICMP echo request, id 8, seq 7, length 64
08:32:28.060440 eth0  Out IP 192.168.1.101 > 192.168.2.3: ICMP echo reply, id 8, seq 7, length 64
08:32:28.070543 eth0  Out IP 192.168.1.101 > 192.168.2.3: ICMP echo request, id 8, seq 4, length 64
08:32:28.071518 eth0  In  IP 192.168.2.3 > 192.168.1.101: ICMP echo reply, id 8, seq 4, length 64
08:32:29.071726 eth0  Out IP 192.168.1.101 > 192.168.2.3: ICMP echo request, id 8, seq 5, length 64
08:32:29.072363 eth0  In  IP 192.168.2.3 > 192.168.1.101: ICMP echo reply, id 8, seq 5, length 64
08:32:29.084260 eth0  In  IP 192.168.2.3 > 192.168.1.101: ICMP echo request, id 8, seq 8, length 64
08:32:29.084279 eth0  Out IP 192.168.1.101 > 192.168.2.3: ICMP echo reply, id 8, seq 8, length 64
08:32:30.086534 eth0  Out IP 192.168.1.101 > 192.168.2.3: ICMP echo request, id 8, seq 6, length 64
08:32:30.087880 eth0  In  IP 192.168.2.3 > 192.168.1.101: ICMP echo reply, id 8, seq 6, length 64
08:32:30.108482 eth0  In  IP 192.168.2.3 > 192.168.1.101: ICMP echo request, id 8, seq 9, length 64
08:32:30.108501 eth0  Out IP 192.168.1.101 > 192.168.2.3: ICMP echo reply, id 8, seq 9, length 64
08:32:31.132446 eth0  In  IP 192.168.2.3 > 192.168.1.101: ICMP echo request, id 8, seq 10, length 64
08:32:31.132479 eth0  Out IP 192.168.1.101 > 192.168.2.3: ICMP echo reply, id 8, seq 10, length 64


root@pc08:/home/administrator# ip a
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:15:5d:00:c9:77 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.3/24 metric 100 brd 192.168.2.255 scope global dynamic eth0
       valid_lft 5305sec preferred_lft 5305sec
    inet6 fe80::215:5dff:fe00:c977/64 scope link
       valid_lft forever preferred_lft forever

root@pc08:/home/administrator# ping 192.168.1.101
PING 192.168.1.101 (192.168.1.101) 56(84) bytes of data.
64 bytes from 192.168.1.101: icmp_seq=5 ttl=63 time=0.736 ms
64 bytes from 192.168.1.101: icmp_seq=6 ttl=63 time=0.777 ms
64 bytes from 192.168.1.101: icmp_seq=7 ttl=63 time=0.713 ms
64 bytes from 192.168.1.101: icmp_seq=8 ttl=63 time=0.500 ms
64 bytes from 192.168.1.101: icmp_seq=9 ttl=63 time=0.706 ms
64 bytes from 192.168.1.101: icmp_seq=10 ttl=63 time=0.717 ms
^C
--- 192.168.1.101 ping statistics ---
10 packets transmitted, 6 received, 40% packet loss, time 9211ms
rtt min/avg/max/mdev = 0.500/0.691/0.777/0.088 ms

root@pc08:/home/administrator# tcpdump -i any proto ICMP
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
08:32:21.920856 eth0  Out IP 192.168.2.3 > 192.168.1.101: ICMP echo request, id 8, seq 1, length 64
08:32:22.940118 eth0  Out IP 192.168.2.3 > 192.168.1.101: ICMP echo request, id 8, seq 2, length 64
08:32:23.964114 eth0  Out IP 192.168.2.3 > 192.168.1.101: ICMP echo request, id 8, seq 3, length 64
08:32:24.988127 eth0  Out IP 192.168.2.3 > 192.168.1.101: ICMP echo request, id 8, seq 4, length 64
08:32:25.002180 eth0  In  IP 192.168.1.101 > 192.168.2.3: ICMP echo request, id 8, seq 1, length 64
08:32:25.002212 eth0  Out IP 192.168.2.3 > 192.168.1.101: ICMP echo reply, id 8, seq 1, length 64
08:32:26.012097 eth0  Out IP 192.168.2.3 > 192.168.1.101: ICMP echo request, id 8, seq 5, length 64
08:32:26.012802 eth0  In  IP 192.168.1.101 > 192.168.2.3: ICMP echo reply, id 8, seq 5, length 64
08:32:26.022956 eth0  In  IP 192.168.1.101 > 192.168.2.3: ICMP echo request, id 8, seq 2, length 64
08:32:26.022974 eth0  Out IP 192.168.2.3 > 192.168.1.101: ICMP echo reply, id 8, seq 2, length 64
08:32:27.036133 eth0  Out IP 192.168.2.3 > 192.168.1.101: ICMP echo request, id 8, seq 6, length 64
08:32:27.036881 eth0  In  IP 192.168.1.101 > 192.168.2.3: ICMP echo reply, id 8, seq 6, length 64
08:32:27.047092 eth0  In  IP 192.168.1.101 > 192.168.2.3: ICMP echo request, id 8, seq 3, length 64
08:32:27.047108 eth0  Out IP 192.168.2.3 > 192.168.1.101: ICMP echo reply, id 8, seq 3, length 64
08:32:28.060130 eth0  Out IP 192.168.2.3 > 192.168.1.101: ICMP echo request, id 8, seq 7, length 64
08:32:28.060810 eth0  In  IP 192.168.1.101 > 192.168.2.3: ICMP echo reply, id 8, seq 7, length 64
08:32:28.071295 eth0  In  IP 192.168.1.101 > 192.168.2.3: ICMP echo request, id 8, seq 4, length 64
08:32:28.071311 eth0  Out IP 192.168.2.3 > 192.168.1.101: ICMP echo reply, id 8, seq 4, length 64
08:32:29.072085 eth0  In  IP 192.168.1.101 > 192.168.2.3: ICMP echo request, id 8, seq 5, length 64
08:32:29.072121 eth0  Out IP 192.168.2.3 > 192.168.1.101: ICMP echo reply, id 8, seq 5, length 64
08:32:29.084096 eth0  Out IP 192.168.2.3 > 192.168.1.101: ICMP echo request, id 8, seq 8, length 64
08:32:29.084573 eth0  In  IP 192.168.1.101 > 192.168.2.3: ICMP echo reply, id 8, seq 8, length 64

I couldn't replicate that behavior with a TCP session though. If I create a TCP SSH session from LAN to OPT1, I CAN'T SSH back from OPT1 to LAN.

So there is no real security risk here right now.

What I know is that ICMP doesn't have states. It's a stateless protocol. So what we are seeing might be... normal?

[CJ]

As they say in the famous movie "Houston we have a problem"

I kept drilling down as I kept getting replies to my pings.

In the end to remove any middleman issues and confirm once and for all this is opnsense issue, I removed the switch, the virtualization and went baremetal.

Testing env:
Two baremetal win11 PCs.
4 port fresly installed Opnsense.

PCs: Both baremetal:
PC1 - connected directly to LAN port on the opnsense firewall = IP add 10.1.1.1
PC2 - connected directly to OPT1 port on the opnsense firewall = IP add 10.2.2.2

Opnsense:
Fresh install, the only config I added was to enable OPT1 and assing IP Address 10.2.2.254. Nothing else changed.

Test result, I still get a response to the ping.
Steps:
Issued the ping command form PC2, at first dont get a response
However, as soon as you issue the ping command on PC1, both PCs can ping each other.

Please see the screenshots.
Note: The system due to the size does not allow me to attach all the screenshots in one go. I will add them one by one.

Isn't this just ICMP hole punching?  https://en.wikipedia.org/wiki/ICMP_hole_punching

The ping was correctly blocked until you opened up the return by starting from the other way.  It's the whole way a lot of zero config apps work.

[newjohn]

As they say in the famous movie "Houston we have a problem"

I kept drilling down as I kept getting replies to my pings.

In the end to remove any middleman issues and confirm once and for all this is opnsense issue, I removed the switch, the virtualization and went baremetal.

Testing env:
Two baremetal win11 PCs.
4 port fresly installed Opnsense.

PCs: Both baremetal:
PC1 - connected directly to LAN port on the opnsense firewall = IP add 10.1.1.1
PC2 - connected directly to OPT1 port on the opnsense firewall = IP add 10.2.2.2

Opnsense:
Fresh install, the only config I added was to enable OPT1 and assing IP Address 10.2.2.254. Nothing else changed.

Test result, I still get a response to the ping.
Steps:
Issued the ping command form PC2, at first dont get a response
However, as soon as you issue the ping command on PC1, both PCs can ping each other.

Please see the screenshots.
Note: The system due to the size does not allow me to attach all the screenshots in one go. I will add them one by one.

Isn't this just ICMP hole punching?  https://en.wikipedia.org/wiki/ICMP_hole_punching

The ping was correctly blocked until you opened up the return by starting from the other way.  It's the whole way a lot of zero config apps work.

Thank you for the input. Although I understand and agree in a sense on what you saying, although not 100% sure I know the answer to that.

If someone with definite knowledge can confirm its not a security issue I am happy with that although at the beginning I thought it was the auto generated rules was the cause.

Also in the link you shared it shows that an attacker can pretend to be replying to your ping request in return causing the remote PC to respond and we have an open connection ready to be exploited. But i guess thats a conversation for another day.

I guess the question now is is this a security issue or is it expected as you and
Monviech mentioned.

[Monviech (Cedrik)]

I dont think its a security risk because you can simply mitigate it by not allowing the ICMP protocol if you don't need it in your network.

If or of not ICMP (v4) is a security issue is widely discussed.

I would be more worried about some smart home devices that connect directly to a cloud and they can get into your network through the stateful TCP return connection.

[netnut]

I dont think its a security risk because you can simply mitigate it by not allowing the ICMP protocol if you don't need it in your network.

Nein, Nein, Nein!!! Please don't be "security aware" by disabling ICMP, there are many good reasons why this protocol exists next to UDP/TCP. You allow ICMP in general or filtering it down to _at_ least Destination Unreachable, Time Exceeded and Parameter Problem and NDP. There are things like PMTU discovery and IPv6 to name a few.

If or of not ICMP (v4) is a security issue is widely discussed.

If a packet filter allows whatever packet that it _should_ have filter by policy, you have a security bug. You could technically tunnel out traffic with ICMP reply packets which don't have a fixed data length. You can use Suricata to detect these kind of anomalies.  But back to OP...

You should see a "no response found!" message in your packet capture when pinging "back" from the OPT interface to LAN, which you probably don't.

What if you re-run the test but now first go to:

INTERFACES: SETTINGS and check the "Hardware CRC    Disable hardware checksum offload" box, while you there you could possible check also the two other ones AND disable VLAN Hardware Filtering. You might want to reboot, just to be really sure...

[Monviech (Cedrik)]

I dont think its a security risk because you can simply mitigate it by not allowing the ICMP protocol if you don't need it in your network.

Nein, Nein, Nein!!! Please don't be "security aware" by disabling ICMP, there are many good reasons why this protocol exists next to UDP/TCP. You allow ICMP in general or filtering it down to _at_ least Destination Unreachable, Time Exceeded and Parameter Problem and NDP. There are things like PMTU discovery and IPv6 to name a few.

You misinterpreted what I said. Maybe I should have added an "but I don't believe in it." at the end. I generally allow most ICMP and ICMPv6 types because they are quite essential to have the network running well. You're right about that.