Degraded printer functionality until ICMP enabled on LAN side of firewall

[Lu]

I'm posting this in the hope others benefit from our pain. After a large Toshiba printer/MFC was replaced on our network with a newer model (an e-STUDIO3525AC), it had a great deal of trouble. The previous model had worked fine, and there were no changes to the OPNsense box's config between the two. Despite trying both dynamic and static network configs, IPv4-only, IPv6-only, etc., the new one could not get DNS resolution of any address, could not ping public IP addresses (even directly, like 8.8.8.8), and was generally poor at obtaining and holding onto its network config. It even complained at various points that the network cable wasn't connected. I used OPNsense's Interfaces > Diagnostics > Packet Capture, limited to the printer's MAC, and saw it was fairly chatty. I tested the new printer on a secondary physical network and all was okay, so it was something about the main network.

When I realised I could ping public addresses from my own PC, but not the firewall's, I found this thread about it. I enabled ICMP with this rule on the LAN interface, in order to test ping from the printer again:

Protocol	Source	Port	Destination	Port	Gateway	Schedule
IPv4+6 ICMP	*	*	This Firewall	*	*	*

To my surprise, everything started behaving. I'm not blaming OPNsense; I think the printer was deciding it wouldn't or couldn't do basic communication without the router responding to certain queries, or something. If you're experiencing such issues, they may be being triggered by default firewall policies.

[meyergru]

...which by default allow any to any for any IP protocols on LAN? ;-)

[drosophila]

Would be interesting to know if the printer would also work normally if it had no gateway set at all. IMO a printer has no business outside its local network, anyway, so doesn't need / shouldn't have a gateway.

[meyergru]

Depends on what the machine can do. For example, if it can scan/fax, it needs the time, which is also needed for logging. If that is the case, it could use an NTP server. Of course, it could get the NTP sever IP via DHCP, but as it turns out, there are plenty of devices which use arbitrary cloud NTP servers - like AppleTVs, e.g.

Also, some devices look for firmware updates, can order consumables in advance a.s.o., let alone collect statistics and push that to the manufacturer without anyone knowing.

There are good reasons to put "smart" devices into a separate VLAN.

[drosophila]

Also, some devices look for firmware updates, can order consumables in advance a.s.o., let alone collect statistics and push that to the manufacturer without anyone knowing.

There are good reasons to put "smart" devices into a separate VLAN.

Indeed, that's what I was implicitly hinting at, and rather than ordering consumables I'd rather have it send a mail to the admin instead, and use a local NTP server, but I accept that the default configuration might be catering to the person who plops down the device, pops in some cables and then expects it to work until it's decommissioned. Of course, this affords privacy and security only to those select "elites" who at least know what might be happening and therefore might try to prevent them.

[Lu]

...which by default allow any to any for any IP protocols on LAN? ;-)

Can you elaborate? I don't understand what you're referring to, specifically.

so doesn't need / shouldn't have a gateway.

It's a big office printer with a support contract, we don't own it. It uses WAN for emailing and FTP upload of scanned documents, NTP, and firmware updates. It probably could be made to function well enough without WAN but, regardless, it has difficulty communicating with other machines on the LAN.

I'm having trouble with it again and have had to configure it to use IP addresses instead of names, e.g. for SMTP. Dnsmasq on OPNsense flat-out refuses to answer its DNS queries. I've done (redacted) packet captures on the OPNsense device, to compare requests from my own machine with those of the printer's, and I don't know why they go unanswered. I don't see anything blocking them when I watch the live view of the firewall.

[meyergru]

...which by default allow any to any for any IP protocols on LAN? ;-)

Can you elaborate? I don't understand what you're referring to, specifically.

In the default configuration, OpnSense comes with a preinstalled "allow any -> any" rule for the first LAN interface. Without it, you initially would not be able to access either the internet or the firewall itself. When people create another VLAN, in 99% of the cases, the first question they come up with is: "Why is it that I cannot access the internet from my second VLAN?" It is because they did not create a similar rule on that new VLAN.

[Lu]

Ah, Ta. I'm aware of the auto-generated rules, and I'm not using VLANs. I added it out of frustration, really.

[meyergru]

I was not referring to the auto-generated rules. There is an initial explicit allow-any rule for the first LAN, which you can manually delete, not an auto-generated one. So, it is strange that you had to create such a rule by yourself, unless you deleted the existing one at some point.

[Lu]

I was not referring to the auto-generated rules

My mistake, I meant the automatically provided ones, not the auto-generated ones. I followed the guide for WAN failover, which said to edit those rules and assign to them a gateway. I have since added another rule after those, without a gateway:

statetype

state-policy

sequence

action

quick

interfacenot

interface

direction

ipprotocol

protocol

icmptype

icmp6type

source_net

source_not

source_port

destination_net

destination_not

destination_port

divert-to

gateway

keep

51

pass

1

0

lan

in

inet

any

lan

0

any

0

v4

keep

61

pass

1

0

lan

in

inet6

any

lan

0

any

0

v6

keep

71

pass

1

0

lan

in

inet46

any

lan

0

any

0

Though this has improved the LAN stability, the Printer still misbehaves, and not in a consistent manner. Sometimes it'll work as desired, but usually stops, so I have to leave it configured with IP addresses rather than names, and accept the degradation.

I'm still inclined to believe it is at least partially caused by the router because it has other odd behaviours like near total failure of IPv6 after a random amount of uptime. Restarting the router, without changing any config, will 'fix' that.

[drosophila]

This sounds like there might be some sort of conflict inside the main network, since things are working properly in the second network. Is there any possibility to apply the same OPNSense configuration on the second LAN to see if it's an issue with the rules? The times of duplicate MAC addresses (caused by chinese knockoff NICs) should be long gone but this is the sort of issue I used to experience in cases like these. Though in that case the conflicting device would likely also experience issues. Also seems similar to issues with MTU / MSS, which might be misconfigured in the printer itself by default, especially with VLANs / IPv6.

[Lu]

I think this was all hampered by old entries in ARP cache, as the new printer used the same IP address as the old, but obviously has a different MAC. Clearing it made things work again. The weird part is it that it kept happening after it got an IP address and things would work for a while, which suggests that the ARP entries were corrected, then became corrupt again. Very odd since the old printer was no longer on the premises, so couldn't be replying and poisoning ARP.

It has also happened since with at least two other machines. A static IP was assigned in Dnsmasq, the machine would receive its assignment via DHCP but then had no IPv4 connectivity to the Internet, though communicating with other machines on the LAN was okay. The OPNsense ARP table contained an entry for that IP but with a MAC address of a machine that had that address months ago. Clearing the ARP table did not fix it permanently, it came back. The other machine was on the network, but using a different ethernet port with a different MAC, and there was no record on that other machine that it thought it was still assigned that address; all diagnostics show only its expected addresses. I doubt it was responding to ARP queries for its disconnected port with an IP address it no longer knew about.

Similar thing again on the second machine, IP assigned in Dnsmasq statically, DNS leases and ARP table all showing the correct entries, machine itself showing the right addresses. Rebooted OPNsense box after an update, and the ARP table contained the previous entry again. I think it was OPNsense, not the machine, because the machine was wiped and had a new OS install. The previous ARP responses for that IP and the bad MAC would have come from Windows, but the ARP responses with that IP and the good MAC would have come from Linux. Linux could never have known that the other MAC would have been assigned that address.

[drosophila]

Good detective work! This seems to indicate that there is some cruft stored in the config files, possibly leftovers from old configuration changes, now masked by their wrapper services being disabled. IDK why sense would store ARP things in the config, but it might be old entries for IP assignments on some obscure sub-service page, or maybe even MAC-based rules for the firewall that have a side-effect. Fixed ARP entries are possible at least in the config files (you can manually add ARP entries from the console, so scripts can also do it). The fact that it comes back only after some time hints that it might coincide with provider-induced WAN IP reassignments, resulting in the respective services are being refreshed.
Of course, it is possible to assign arbitrary MACs to interfaces in most OS/drivers, but that would also have been set up manually, just like custom scripts.

It might work to "clean up" the config by exporting it, clearing the current one, then re-importing the previously exported one. That would get rid of everything that is in auto-generated config files but not in the actual config. You could however grep through the generated xml and see if those MACs appear in it, and if so, where.