Firewall rules on interfaces do not take effect, floating ones do.

[ananke]

I'd love for any hints/suggestions, as I've been trying to debug this issue for a day to no avail.

Long story short: when trying to set up an EC2 instance with four interfaces, I've been unable to make firewall rules work, unless they are created as floating ones. I've been able to repeat this dozens of times, and the results remain the same: per-interface rules result in logs showing as 'pass', tcpdump shows both request and responses from external hosts, but no traffic leaves the opnsense system until that rule is a floating one.

I'd be happy to post full config, if it would make diagnosis easier.

The set up is as follows:

Code: [Select]

*** OPNsense.localdomain: OPNsense 23.7.12 ***

 AccessNetwork (ena1) -> v4/DHCP4: 100.65.134.171/20
 BlueNetwork1 (ena2) -> v4/DHCP4: 10.3.129.150/25
 BlueNetwork2 (ena3) -> v4/DHCP4: 10.3.201.235/25
 WAN (xn0)       -> v4/DHCP4: 10.3.239.98/25


Firewall rule: (opt3 == ena3)

Code: [Select]

   <rule uuid="05a6eb84-d13b-49bb-a9d6-c4e3b1805b69">
      <type>pass</type>
      <interface>opt3</interface>
      <ipprotocol>inet</ipprotocol>
      <statetype>keep state</statetype>
      <descr>allow all traffic from blue network 2</descr>
      <direction>in</direction>
      <quick>1</quick>
      <source>
        <any>1</any>
      </source>
      <destination>
        <any>1</any>
      </destination>
      <log>1</log>
    </rule>

From an external host, attached to same network as ena3, I can send ping but I get no reply. (same goes for any other protocol)

Code: [Select]

:~$ ping 10.3.201.235
PING 10.3.201.235 (10.3.201.235) 56(84) bytes of data.
opnsense firewall log shows:

Code: [Select]

00:00:52.660957 rule 79/0(match): pass in on ena3: (tos 0x0, ttl 64, id 34272, offset 0, flags [DF], proto ICMP (1), length 84)
    10.3.201.249 > 10.3.201.235: ICMP echo request, id 12, seq 1, length 64
tcpdump on opnsense shows both request and reply, both on the same interface:

 

Code: [Select]

root@OPNsense:~ # tcpdump -n -i ena3
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ena3, link-type EN10MB (Ethernet), capture size 262144 bytes
19:10:45.666195 IP 10.3.201.249 > 10.3.201.235: ICMP echo request, id 12, seq 1, length 64
19:10:45.666235 IP 10.3.201.235 > 10.3.201.249: ICMP echo reply, id 12, seq 1, length 64
19:10:46.673291 IP 10.3.201.249 > 10.3.201.235: ICMP echo request, id 12, seq 2, length 64
19:10:46.673315 IP 10.3.201.235 > 10.3.201.249: ICMP echo reply, id 12, seq 2, length 64

---

Coming out of opnsense and sending ICMP to the host above, no issues:

Code: [Select]

root@OPNsense:~ # ping 10.3.201.249
PING 10.3.201.249 (10.3.201.249): 56 data bytes
64 bytes from 10.3.201.249: icmp_seq=0 ttl=64 time=0.308 ms
64 bytes from 10.3.201.249: icmp_seq=1 ttl=64 time=0.231 ms
firewall log shows:

Code: [Select]

00:00:16.274609 rule 70/0(match) [uid 0]: pass out on ena3: (tos 0x0, ttl 64, id 17845, offset 0, flags [none], proto ICMP (1), length 84)
    10.3.201.235 > 10.3.201.249: ICMP echo request, id 37587, seq 0, length 64
---

Now, after adding same rule, but as floating one (this happens to be allowing just ICMP, but same thing happens if I just allow all protocols):


 

Code: [Select]

<filter>
    <rule uuid="040f670b-e260-4723-b4dd-5217c01bcaa7">
      <type>pass</type>
      <ipprotocol>inet</ipprotocol>
      <statetype>keep state</statetype>
      <descr>allow all ICMP traffic</descr>
      <direction>in</direction>
      <floating>yes</floating>
      <quick>1</quick>
      <protocol>icmp</protocol>
      <source>
        <any>1</any>
      </source>
      <destination>
        <any>1</any>
      </destination>
    </rule>
Pinging from the other host shows a proper response:

Code: [Select]

:~$ ping 10.3.201.235
PING 10.3.201.235 (10.3.201.235) 56(84) bytes of data.
64 bytes from 10.3.201.235: icmp_seq=1 ttl=64 time=0.196 ms
64 bytes from 10.3.201.235: icmp_seq=2 ttl=64 time=0.175 ms

opnsense firewall log shows a match/pass on a different rule:

Code: [Select]

00:00:00.000000 rule 77/0(match): pass in on ena3: (tos 0x0, ttl 64, id 8347, offset 0, flags [DF], proto ICMP (1), length 84)
    10.3.201.249 > 10.3.201.235: ICMP echo request, id 15, seq 1, length 64

While tcpdump shows identical traffic pattern, all on ena3 interface:

Code: [Select]

root@OPNsense:~ # tcpdump -n -i ena3
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ena3, link-type EN10MB (Ethernet), capture size 262144 bytes
19:17:27.701908 IP 10.3.201.249 > 10.3.201.235: ICMP echo request, id 16, seq 1, length 64
19:17:27.701941 IP 10.3.201.235 > 10.3.201.249: ICMP echo reply, id 16, seq 1, length 64
19:17:28.721290 IP 10.3.201.249 > 10.3.201.235: ICMP echo request, id 16, seq 2, length 64
19:17:28.721313 IP 10.3.201.235 > 10.3.201.249: ICMP echo reply, id 16, seq 2, length 64
19:17:29.745275 IP 10.3.201.249 > 10.3.201.235: ICMP echo request, id 16, seq 3, length 64
19:17:29.745295 IP 10.3.201.235 > 10.3.201.249: ICMP echo reply, id 16, seq 3, length 64
I've checked, there is no associated traffic on any other interfaces. What am I missing?

Thank you in advance!

[Maurice]

Well, tcpdump shows echo replies being sent to 10.3.201.249. But are they addressed to the correct MAC address?

[ananke]

Excellent suggestion, thank you. Unfortunately, MAC addresses appear to be correct.

Here's a quick rundown, first three pings with the floating rule disabled, then with it being enabled.

remote linux host: 02:9d:2c:38:69:05 - 10.3.201.249
opnsense: 02:a3:86:44:1b:e3 - 10.3.201.235

Code: [Select]

root@OPNsense:/usr/home/student # tcpdump -n -e -i ena3
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ena3, link-type EN10MB (Ethernet), capture size 262144 bytes

No replies, three pings sent

21:19:14.676928 02:9d:2c:38:69:05 > 02:a3:86:44:1b:e3, ethertype IPv4 (0x0800), length 98: 10.3.201.249 > 10.3.201.235: ICMP echo request, id 25, seq 1, length 64
21:19:14.676964 02:a3:86:44:1b:e3 > 02:e0:27:d1:67:a1, ethertype IPv4 (0x0800), length 98: 10.3.201.235 > 10.3.201.249: ICMP echo reply, id 25, seq 1, length 64
21:19:14.967543 02:e0:27:d1:67:a1 > 02:a3:86:44:1b:e3, ethertype IPv4 (0x0800), length 90: 10.3.201.249.33315 > 104.234.70.173.123: NTPv4, Client, length 48
21:19:15.697625 02:9d:2c:38:69:05 > 02:a3:86:44:1b:e3, ethertype IPv4 (0x0800), length 98: 10.3.201.249 > 10.3.201.235: ICMP echo request, id 25, seq 2, length 64
21:19:15.697650 02:a3:86:44:1b:e3 > 02:e0:27:d1:67:a1, ethertype IPv4 (0x0800), length 98: 10.3.201.235 > 10.3.201.249: ICMP echo reply, id 25, seq 2, length 64
21:19:16.721531 02:9d:2c:38:69:05 > 02:a3:86:44:1b:e3, ethertype IPv4 (0x0800), length 98: 10.3.201.249 > 10.3.201.235: ICMP echo request, id 25, seq 3, length 64
21:19:16.721552 02:a3:86:44:1b:e3 > 02:e0:27:d1:67:a1, ethertype IPv4 (0x0800), length 98: 10.3.201.235 > 10.3.201.249: ICMP echo reply, id 25, seq 3, length 64

Replies are being received, three pings sent:

21:19:36.153819 02:9d:2c:38:69:05 > 02:a3:86:44:1b:e3, ethertype IPv4 (0x0800), length 98: 10.3.201.249 > 10.3.201.235: ICMP echo request, id 26, seq 1, length 64
21:19:36.153845 02:a3:86:44:1b:e3 > 02:9d:2c:38:69:05, ethertype IPv4 (0x0800), length 98: 10.3.201.235 > 10.3.201.249: ICMP echo reply, id 26, seq 1, length 64
21:19:36.431063 02:e0:27:d1:67:a1 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.3.201.235 tell 10.3.201.129, length 28
21:19:36.431076 02:a3:86:44:1b:e3 > 02:e0:27:d1:67:a1, ethertype ARP (0x0806), length 42: Reply 10.3.201.235 is-at 02:a3:86:44:1b:e3, length 28
21:19:37.169553 02:9d:2c:38:69:05 > 02:a3:86:44:1b:e3, ethertype IPv4 (0x0800), length 98: 10.3.201.249 > 10.3.201.235: ICMP echo request, id 26, seq 2, length 64
21:19:37.169573 02:a3:86:44:1b:e3 > 02:9d:2c:38:69:05, ethertype IPv4 (0x0800), length 98: 10.3.201.235 > 10.3.201.249: ICMP echo reply, id 26, seq 2, length 64
21:19:38.193540 02:9d:2c:38:69:05 > 02:a3:86:44:1b:e3, ethertype IPv4 (0x0800), length 98: 10.3.201.249 > 10.3.201.235: ICMP echo request, id 26, seq 3, length 64
21:19:38.193557 02:a3:86:44:1b:e3 > 02:9d:2c:38:69:05, ethertype IPv4 (0x0800), length 98: 10.3.201.235 > 10.3.201.249: ICMP echo reply, id 26, seq 3, length 64
21:19:39.468520 02:e0:27:d1:67:a1 > 02:a3:86:44:1b:e3, ethertype IPv4 (0x0800), length 90: 10.3.201.249.45923 > 216.240.36.24.123: NTPv4, Client, length 48
21:19:43.191794 02:e0:27:d1:67:a1 > 02:a3:86:44:1b:e3, ethertype IPv4 (0x0800), length 90: 10.3.201.249.54456 > 91.189.91.157.123: NTPv4, Client, length 48

remote host

Code: [Select]

2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:9d:2c:38:69:05 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    inet 10.3.201.249/25 metric 100 brd 10.3.201.255 scope global dynamic ens5
       valid_lft 3173sec preferred_lft 3173sec
    inet6 fe80::9d:2cff:fe38:6905/64 scope link
       valid_lft forever preferred_lft forever

OPNsense

Code: [Select]

root@OPNsense:~ # ifconfig -u
xn0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: WAN (wan)
    options=20<JUMBO_MTU>
    ether 02:2f:a8:a6:15:13
    inet 10.3.239.98 netmask 0xffffff80 broadcast 10.3.239.127
    media: Ethernet autoselect (Unknown <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ena1: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: AccessNetwork (opt1)
    options=20<JUMBO_MTU>
    ether 02:01:3b:49:f5:bb
    inet 100.65.134.171 netmask 0xfffff000 broadcast 100.65.143.255
    media: Ethernet autoselect (Unknown <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ena2: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: BlueNetwork1 (opt2)
    options=20<JUMBO_MTU>
    ether 02:86:ab:0a:83:29
    inet 10.3.129.150 netmask 0xffffff80 broadcast 10.3.129.255
    media: Ethernet autoselect (Unknown <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ena3: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: BlueNetwork2 (opt3)
    options=20<JUMBO_MTU>
    ether 02:a3:86:44:1b:e3
    inet 10.3.201.235 netmask 0xffffff80 broadcast 10.3.201.255
    media: Ethernet autoselect (Unknown <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

[ananke]

Couple more pieces of information:

• This problem is repeatable with opt2/ena2, same thing happens there
• Not sure if it's related or not, but the firewall rules on WAN (xn0) seem to apply to traffic on en1/opt1. Despite accessing the opnsense system via en1, no rules on en1 seem to apply: I have no rules to allow traffic from there. Yet when I disable traffic allow on xn0, I no longer can reach the system via ena1

I'd like to mention that it's very unlikely that this is related to AWS networking, as the same identical setup using Linux as firewall/router appliance doesn't exhibit any issues. The networks/routes/vms/etc are deployed via the same set of cloudformation templates, the only difference being the firewall/router vm swapped from Linux to OPNsense.

OPNSense AMI is 2.7.12, with virtually all defaults left on, just added/assigned interfaces.

[Maurice]

Without the floating rule, the echo replies are being sent to 02:e0:27:d1:67:a1, which is not where the echo requests came from, but most likely your default gateway. You might have fallen into the reply-to trap. Lookup the non-floating rule in Firewall: Diagnostics: Statistics: rules. Does it have a reply-to option? Then you have to explicitly disable reply-to in the rule's advanced settings.

[ananke]

That's it!

Thank you so much. I've been beating my head against the wall for at least a day, if not two, trying to figure out what's going on. With all of the network interfaces set to DHCP, looks like reply-to will be always enabled automatically.

I've been reading various threads, and forum posts, and it seems the only solution for us is to explicitly set that flag to 'disabled'. for each and every rule. I haven't found anything that would allow us to set it as a default behavior for an interface managed by DHCP.

Thank you again.

[Maurice]

There is an option in Firewall: Settings: Advanced to disable it globally, and there have been heated debates in the past whether this should be the default.
I'm not aware of an interface-level setting.

[Patrick M. Hausen]

A host must not have more than one interface configured by DHCP.

[Maurice]

@Patrick This unfortunately lacks real world compatibility. The specs e.g. also say that a router mustn't accept Router Advertisements itself. That's just not practical.

[ananke]

@Maurice: thank you again, looks like disabling it globally and removing it from individual rules works in our setting, this will simplify a lot. It's a bit overwhelming at first to go through all of the available settings, and I honestly I completely missed the 'Advanced' tab under individual rules at first.

@Patrick: As to the DHCP for one interface only, that's not feasible. Networks have gotten more complex, and deployments are more dynamic. Setting three out of four interfaces to static configuration is simply not attainable at scale.

Cheers!

[jahlives]

> A host must not have more than one interface configured by DHCP.

it think that depends on what exactly "host" means :-) If host is your opnsense (router) then I would agree to only configure one of the hosts interface via DHCP (mostly WAN/upstream) and all the others statically. But if host means any host inside your network then I'd agree to the others as this is not feasible nowadays. I have multiple hosts with 3 or more interfaces. To manually configure them would really be a pita. One just has to take care that not multiple default routes are pushed and that the interfaces do not share the same broadcast network. But beside from that I see no point to not configure all interfaces of a host via dhcp :-)

[Patrick M. Hausen]

I am referring to the OPNsense system itself 

Yes, laptop systems with e.g. at least wired and wireless connections are a different matter. But e.g. Mac OS does come with an IP stack that can reliably prioritise interfaces and their accompanying configuration.

Kind regards,
Patrick

[Maurice]

Multi-WAN routers are really nothing new, even in the smallest of networks. The idea of only one WAN interface being allowed to have a dynamic configuration is not anchored in reality.

[Patrick M. Hausen]

Which name servers will be used? Which default gateway? OPNsense doesn't really have a mechanism to apply these setting per interface. It's all a single stack.

[Maurice]

Not sure whether these are rhetorical questions, Patrick. OPNsense has many features which augment FreeBSD's capabilities (and sometimes work around its limitations).
Typically, DNS servers received via DHCP / RA / IPCP are ignored. The local recursive resolver or statically configured external DNS servers are used instead. When choosing to allow dynamic DNS servers, this can indeed be configured per interface.
The default gateway is determined by priorities, gateway monitoring and default gateway switching.