Messages

1

General Discussion / custom options for /etc/resolv.conf without unbound

« on: February 21, 2024, 04:23:22 pm »

Hello,

We need to inject `ndots` option into /etc/resolv.conf, but can't seem to find a sanctioned way to do it. Using a custom /etc/dhclient-exit-hooks fails, because OPNsense just rewrites /etc/resolv.conf afterwards.

Unbound service appears like the only other potential option, but we're running without it, as it was problematic in out-of-the-box config in a network that has no direct access to the internet (unbound kept trying to reach root servers, and all DNS requests were taking 15-30 seconds). It's not needed in our case anyway, so we'd like to find a proper way to get that option into /etc/resolv.conf

I don't suppose anybody has any ideas on how to accomplish that?

Thanks!

2

23.7 Legacy Series / Re: Firewall rules on interfaces do not take effect, floating ones do.

« on: January 26, 2024, 05:30:06 pm »

@Maurice: thank you again, looks like disabling it globally and removing it from individual rules works in our setting, this will simplify a lot. It's a bit overwhelming at first to go through all of the available settings, and I honestly I completely missed the 'Advanced' tab under individual rules at first.

@Patrick: As to the DHCP for one interface only, that's not feasible. Networks have gotten more complex, and deployments are more dynamic. Setting three out of four interfaces to static configuration is simply not attainable at scale.

Cheers!

3

23.7 Legacy Series / Re: Firewall rules on interfaces do not take effect, floating ones do.

« on: January 26, 2024, 01:50:56 am »

That's it!

Thank you so much. I've been beating my head against the wall for at least a day, if not two, trying to figure out what's going on. With all of the network interfaces set to DHCP, looks like reply-to will be always enabled automatically.

I've been reading various threads, and forum posts, and it seems the only solution for us is to explicitly set that flag to 'disabled'. for each and every rule. I haven't found anything that would allow us to set it as a default behavior for an interface managed by DHCP.

Thank you again.

4

23.7 Legacy Series / Re: Firewall rules on interfaces do not take effect, floating ones do.

« on: January 25, 2024, 10:41:35 pm »

Couple more pieces of information:

• This problem is repeatable with opt2/ena2, same thing happens there
• Not sure if it's related or not, but the firewall rules on WAN (xn0) seem to apply to traffic on en1/opt1. Despite accessing the opnsense system via en1, no rules on en1 seem to apply: I have no rules to allow traffic from there. Yet when I disable traffic allow on xn0, I no longer can reach the system via ena1

I'd like to mention that it's very unlikely that this is related to AWS networking, as the same identical setup using Linux as firewall/router appliance doesn't exhibit any issues. The networks/routes/vms/etc are deployed via the same set of cloudformation templates, the only difference being the firewall/router vm swapped from Linux to OPNsense.

OPNSense AMI is 2.7.12, with virtually all defaults left on, just added/assigned interfaces.

5

23.7 Legacy Series / Re: Firewall rules on interfaces do not take effect, floating ones do.

« on: January 25, 2024, 10:30:33 pm »

Excellent suggestion, thank you. Unfortunately, MAC addresses appear to be correct.

Here's a quick rundown, first three pings with the floating rule disabled, then with it being enabled.

remote linux host: 02:9d:2c:38:69:05 - 10.3.201.249
opnsense: 02:a3:86:44:1b:e3 - 10.3.201.235

Code: [Select]

root@OPNsense:/usr/home/student # tcpdump -n -e -i ena3
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ena3, link-type EN10MB (Ethernet), capture size 262144 bytes

No replies, three pings sent

21:19:14.676928 02:9d:2c:38:69:05 > 02:a3:86:44:1b:e3, ethertype IPv4 (0x0800), length 98: 10.3.201.249 > 10.3.201.235: ICMP echo request, id 25, seq 1, length 64
21:19:14.676964 02:a3:86:44:1b:e3 > 02:e0:27:d1:67:a1, ethertype IPv4 (0x0800), length 98: 10.3.201.235 > 10.3.201.249: ICMP echo reply, id 25, seq 1, length 64
21:19:14.967543 02:e0:27:d1:67:a1 > 02:a3:86:44:1b:e3, ethertype IPv4 (0x0800), length 90: 10.3.201.249.33315 > 104.234.70.173.123: NTPv4, Client, length 48
21:19:15.697625 02:9d:2c:38:69:05 > 02:a3:86:44:1b:e3, ethertype IPv4 (0x0800), length 98: 10.3.201.249 > 10.3.201.235: ICMP echo request, id 25, seq 2, length 64
21:19:15.697650 02:a3:86:44:1b:e3 > 02:e0:27:d1:67:a1, ethertype IPv4 (0x0800), length 98: 10.3.201.235 > 10.3.201.249: ICMP echo reply, id 25, seq 2, length 64
21:19:16.721531 02:9d:2c:38:69:05 > 02:a3:86:44:1b:e3, ethertype IPv4 (0x0800), length 98: 10.3.201.249 > 10.3.201.235: ICMP echo request, id 25, seq 3, length 64
21:19:16.721552 02:a3:86:44:1b:e3 > 02:e0:27:d1:67:a1, ethertype IPv4 (0x0800), length 98: 10.3.201.235 > 10.3.201.249: ICMP echo reply, id 25, seq 3, length 64

Replies are being received, three pings sent:

21:19:36.153819 02:9d:2c:38:69:05 > 02:a3:86:44:1b:e3, ethertype IPv4 (0x0800), length 98: 10.3.201.249 > 10.3.201.235: ICMP echo request, id 26, seq 1, length 64
21:19:36.153845 02:a3:86:44:1b:e3 > 02:9d:2c:38:69:05, ethertype IPv4 (0x0800), length 98: 10.3.201.235 > 10.3.201.249: ICMP echo reply, id 26, seq 1, length 64
21:19:36.431063 02:e0:27:d1:67:a1 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.3.201.235 tell 10.3.201.129, length 28
21:19:36.431076 02:a3:86:44:1b:e3 > 02:e0:27:d1:67:a1, ethertype ARP (0x0806), length 42: Reply 10.3.201.235 is-at 02:a3:86:44:1b:e3, length 28
21:19:37.169553 02:9d:2c:38:69:05 > 02:a3:86:44:1b:e3, ethertype IPv4 (0x0800), length 98: 10.3.201.249 > 10.3.201.235: ICMP echo request, id 26, seq 2, length 64
21:19:37.169573 02:a3:86:44:1b:e3 > 02:9d:2c:38:69:05, ethertype IPv4 (0x0800), length 98: 10.3.201.235 > 10.3.201.249: ICMP echo reply, id 26, seq 2, length 64
21:19:38.193540 02:9d:2c:38:69:05 > 02:a3:86:44:1b:e3, ethertype IPv4 (0x0800), length 98: 10.3.201.249 > 10.3.201.235: ICMP echo request, id 26, seq 3, length 64
21:19:38.193557 02:a3:86:44:1b:e3 > 02:9d:2c:38:69:05, ethertype IPv4 (0x0800), length 98: 10.3.201.235 > 10.3.201.249: ICMP echo reply, id 26, seq 3, length 64
21:19:39.468520 02:e0:27:d1:67:a1 > 02:a3:86:44:1b:e3, ethertype IPv4 (0x0800), length 90: 10.3.201.249.45923 > 216.240.36.24.123: NTPv4, Client, length 48
21:19:43.191794 02:e0:27:d1:67:a1 > 02:a3:86:44:1b:e3, ethertype IPv4 (0x0800), length 90: 10.3.201.249.54456 > 91.189.91.157.123: NTPv4, Client, length 48

remote host

Code: [Select]

2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:9d:2c:38:69:05 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    inet 10.3.201.249/25 metric 100 brd 10.3.201.255 scope global dynamic ens5
       valid_lft 3173sec preferred_lft 3173sec
    inet6 fe80::9d:2cff:fe38:6905/64 scope link
       valid_lft forever preferred_lft forever

OPNsense

Code: [Select]

root@OPNsense:~ # ifconfig -u
xn0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: WAN (wan)
    options=20<JUMBO_MTU>
    ether 02:2f:a8:a6:15:13
    inet 10.3.239.98 netmask 0xffffff80 broadcast 10.3.239.127
    media: Ethernet autoselect (Unknown <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ena1: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: AccessNetwork (opt1)
    options=20<JUMBO_MTU>
    ether 02:01:3b:49:f5:bb
    inet 100.65.134.171 netmask 0xfffff000 broadcast 100.65.143.255
    media: Ethernet autoselect (Unknown <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ena2: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: BlueNetwork1 (opt2)
    options=20<JUMBO_MTU>
    ether 02:86:ab:0a:83:29
    inet 10.3.129.150 netmask 0xffffff80 broadcast 10.3.129.255
    media: Ethernet autoselect (Unknown <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ena3: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: BlueNetwork2 (opt3)
    options=20<JUMBO_MTU>
    ether 02:a3:86:44:1b:e3
    inet 10.3.201.235 netmask 0xffffff80 broadcast 10.3.201.255
    media: Ethernet autoselect (Unknown <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

6

23.7 Legacy Series / Firewall rules on interfaces do not take effect, floating ones do.

« on: January 25, 2024, 09:27:17 pm »

I'd love for any hints/suggestions, as I've been trying to debug this issue for a day to no avail.

Long story short: when trying to set up an EC2 instance with four interfaces, I've been unable to make firewall rules work, unless they are created as floating ones. I've been able to repeat this dozens of times, and the results remain the same: per-interface rules result in logs showing as 'pass', tcpdump shows both request and responses from external hosts, but no traffic leaves the opnsense system until that rule is a floating one.

I'd be happy to post full config, if it would make diagnosis easier.

The set up is as follows:

Code: [Select]

*** OPNsense.localdomain: OPNsense 23.7.12 ***

 AccessNetwork (ena1) -> v4/DHCP4: 100.65.134.171/20
 BlueNetwork1 (ena2) -> v4/DHCP4: 10.3.129.150/25
 BlueNetwork2 (ena3) -> v4/DHCP4: 10.3.201.235/25
 WAN (xn0)       -> v4/DHCP4: 10.3.239.98/25


Firewall rule: (opt3 == ena3)

Code: [Select]

   <rule uuid="05a6eb84-d13b-49bb-a9d6-c4e3b1805b69">
      <type>pass</type>
      <interface>opt3</interface>
      <ipprotocol>inet</ipprotocol>
      <statetype>keep state</statetype>
      <descr>allow all traffic from blue network 2</descr>
      <direction>in</direction>
      <quick>1</quick>
      <source>
        <any>1</any>
      </source>
      <destination>
        <any>1</any>
      </destination>
      <log>1</log>
    </rule>

From an external host, attached to same network as ena3, I can send ping but I get no reply. (same goes for any other protocol)

Code: [Select]

:~$ ping 10.3.201.235
PING 10.3.201.235 (10.3.201.235) 56(84) bytes of data.
opnsense firewall log shows:

Code: [Select]

00:00:52.660957 rule 79/0(match): pass in on ena3: (tos 0x0, ttl 64, id 34272, offset 0, flags [DF], proto ICMP (1), length 84)
    10.3.201.249 > 10.3.201.235: ICMP echo request, id 12, seq 1, length 64
tcpdump on opnsense shows both request and reply, both on the same interface:

 

Code: [Select]

root@OPNsense:~ # tcpdump -n -i ena3
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ena3, link-type EN10MB (Ethernet), capture size 262144 bytes
19:10:45.666195 IP 10.3.201.249 > 10.3.201.235: ICMP echo request, id 12, seq 1, length 64
19:10:45.666235 IP 10.3.201.235 > 10.3.201.249: ICMP echo reply, id 12, seq 1, length 64
19:10:46.673291 IP 10.3.201.249 > 10.3.201.235: ICMP echo request, id 12, seq 2, length 64
19:10:46.673315 IP 10.3.201.235 > 10.3.201.249: ICMP echo reply, id 12, seq 2, length 64

---

Coming out of opnsense and sending ICMP to the host above, no issues:

Code: [Select]

root@OPNsense:~ # ping 10.3.201.249
PING 10.3.201.249 (10.3.201.249): 56 data bytes
64 bytes from 10.3.201.249: icmp_seq=0 ttl=64 time=0.308 ms
64 bytes from 10.3.201.249: icmp_seq=1 ttl=64 time=0.231 ms
firewall log shows:

Code: [Select]

00:00:16.274609 rule 70/0(match) [uid 0]: pass out on ena3: (tos 0x0, ttl 64, id 17845, offset 0, flags [none], proto ICMP (1), length 84)
    10.3.201.235 > 10.3.201.249: ICMP echo request, id 37587, seq 0, length 64
---

Now, after adding same rule, but as floating one (this happens to be allowing just ICMP, but same thing happens if I just allow all protocols):


 

Code: [Select]

<filter>
    <rule uuid="040f670b-e260-4723-b4dd-5217c01bcaa7">
      <type>pass</type>
      <ipprotocol>inet</ipprotocol>
      <statetype>keep state</statetype>
      <descr>allow all ICMP traffic</descr>
      <direction>in</direction>
      <floating>yes</floating>
      <quick>1</quick>
      <protocol>icmp</protocol>
      <source>
        <any>1</any>
      </source>
      <destination>
        <any>1</any>
      </destination>
    </rule>
Pinging from the other host shows a proper response:

Code: [Select]

:~$ ping 10.3.201.235
PING 10.3.201.235 (10.3.201.235) 56(84) bytes of data.
64 bytes from 10.3.201.235: icmp_seq=1 ttl=64 time=0.196 ms
64 bytes from 10.3.201.235: icmp_seq=2 ttl=64 time=0.175 ms

opnsense firewall log shows a match/pass on a different rule:

Code: [Select]

00:00:00.000000 rule 77/0(match): pass in on ena3: (tos 0x0, ttl 64, id 8347, offset 0, flags [DF], proto ICMP (1), length 84)
    10.3.201.249 > 10.3.201.235: ICMP echo request, id 15, seq 1, length 64

While tcpdump shows identical traffic pattern, all on ena3 interface:

Code: [Select]

root@OPNsense:~ # tcpdump -n -i ena3
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ena3, link-type EN10MB (Ethernet), capture size 262144 bytes
19:17:27.701908 IP 10.3.201.249 > 10.3.201.235: ICMP echo request, id 16, seq 1, length 64
19:17:27.701941 IP 10.3.201.235 > 10.3.201.249: ICMP echo reply, id 16, seq 1, length 64
19:17:28.721290 IP 10.3.201.249 > 10.3.201.235: ICMP echo request, id 16, seq 2, length 64
19:17:28.721313 IP 10.3.201.235 > 10.3.201.249: ICMP echo reply, id 16, seq 2, length 64
19:17:29.745275 IP 10.3.201.249 > 10.3.201.235: ICMP echo request, id 16, seq 3, length 64
19:17:29.745295 IP 10.3.201.235 > 10.3.201.249: ICMP echo reply, id 16, seq 3, length 64
I've checked, there is no associated traffic on any other interfaces. What am I missing?

Thank you in advance!