Messages

Just to report back after user test: today the printer was not visible on the Mac via Airpint and also via avahi-browse I could not see it anymore. But if the printer is rebooted, both works. Not sure if/why the printer sends the mDNS multicast packages only once (or how often it is resend). But I'm willing to assume that this is a quirk of the printer and not of OPNsense.
Thanks again everybody for your support!

A better approach would seem to be using mDNS reflectors.

I'm using a mDNS reflector, as pointed out in my first post. It's the OPNsense os-mdns-repeater plugin. But without the additional firewall rules to allow multicast traffic between the two VLANs it (obviously) does not work.

Thanks for all the support. Adding a multicast specific firewall rule above the allow internet rule seems to work, at least avahi-browse now return some result. I'll let it test by the users next week.
 

Should work. So why not?
I just don't know if you really need to allow mDNS on multiple Interfaces.

Indeed only one interface would be enough. But then the rule is not within the global scope, but only comes later in the interface scope, resulting in not working.

As someone who really hates mDNS forcing devices like that I would like to suggest the following :
- Setup a CUPS Printer Server by using something like a Raspberry Pi 2B/3B or a small Intel Atom NUC running Linux.

While I do understand your rationale and agree with your dislike, I'm still hoping that it should be easier and less maintenance to allow mDNS between two VLANs than administering yet another device on the net.

Use Firewall->Log Files->Live View and watch traffic going to and from the printer.

That might have brought me one step closer. I activated logging for all rules on the firewall. One hit (using port 5353) caught my eyes, as it was passed by a rule I didn't have on my mind: We're using following rule to allow outgoing internet traffic:

• type: global rule
• description: allow internet
• interfaces: <basically all>
• quick: yes
• action: pass
• direction: in
• version: IPv4
• protocol: any
• source: any
• source port: any
• invert target: yes
• target: single host or network -> 10.0.0.0/8
• target port: any
• section: source routing
• gateway: WAN_gateway

Obviously negating 10.0.0.0/8 also catches multicast 224.0.0.251. A side-effect I didn't have on my mind. I'm assume that the wan gateway prevents to get it working. Thus I'm wondering what is best practice?

• Adding another rule covering mDNS/mutlicast and more than one interface (to make it a global rule) and moving it above the allow internet rule
• Altering the allow internet rule to not match mDNS/multicast (not sure how)
• Something completely different

I'm happy about a hint.

[Firewall rules]

Thanks for all your replies!

Firewall rules
positioned as top most interface rules:
1) rule VLAN 65

• • target: 70ClientIntern network
  • target port: single port -> 5353

The target of the mDNS packets is the multicast IP 224.0.0.251. So you have to allow this here.

Makes sense. So I changed it to:

• interface: 65SrvIntern
• target: IP address -> 224.0.0.251
• target port: single port -> 5353

Unfortunately it still doesn't work. tcpdump and avahi-browse show the same result as before. I'm probably overlooking something very obvious here... :-/

I temporarily tried out to broaden the rule to:

• interface: 65SrvIntern
• direction: in
• version: IPv4
• protocol: UDP
• source: any
• source port: any
• target: any
• target port: any

But even then it did not work... So should I look somewhere else?

You will also need firewall rules to allow printing in addition to the MDNS. MDNS just locates the printers address.

Yes, I already have a more generic rule, which I assume should work (just added the other rule as I wanted to rule out other side effects):

• active: yes
• interface: 70ClientsIntern
• action: allow
• direction: in
• version: IPv4
• protocol: any
• source: any
• source port: any
• target: 65SrvIntern network
• target port: any

[mDNS-Repeater]

A user with an Apple Mac (VLAN 70; IP 10.70.0.1) wants to print on a printer (VLAN 65; IP 10.64.65.203) in a different VLAN. The printer only supports Airprint (using mdns) for Mac OS (no specific printer driver, using a generic one does not print images). I installed the os-mdns-repeater and also tried the os-udpbroadcastrelay but fail to get it working. Can someone please support in how to debug. What I tried so far/my current config & information level:
Client:
$ avahi-browse -art | grep "Brother MFC-L" within the VLAN 65 provides output, within the VLAN 70 does not. Ping from client in VLAN 70 to printer in VLAN 65 works.

OPNsense:
mDNS-Repeater
GUI config

• activated
• interfaces: 65SrvIntern, 70ClientIntern

Via ssh

Code Select Expand

# cat /etc/rc.conf.d/mdns_repeater
mdns_repeater_enable="YES"
mdns_repeater_interfaces="ix0_vlan65 vlan0.70

# ps aux | grep -i mdns
root          48892   0.0  0.0   14088    2628  2  S+   17:04       0:00.00 grep -i mdns
root@fw:/home/user # ps aux | grep -i mdns
root          51533   0.0  0.0   14004    2488  -  Ss   17:04       0:00.00 /usr/local/bin/mdns-repeater -p /var/run/mdns-repeater.pid ix0_vlan65 vlan0.70
root          57371   0.0  0.0   14088    2632  2  S+   17:04       0:00.00 grep -i mdns

# sockstat -l | grep 5353
root          mdns-repea 62350  3 udp4    *:5353                *:*                 
root          mdns-repea 62350  4 udp4    10.64.65.254:5353     *:*                 
root          mdns-repea 62350  5 udp4    10.70.255.254:5353    *:*

# ifconfig | grep -E "ix0_vlan65|vlan0.70"
ix0_vlan65: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
vlan0.70: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500

# tcpdump -i ix0_vlan65 -n 'udp port 5353' -v | grep "Brother MFC-L"
tcpdump: listening on ix0_vlan65, link-type EN10MB (Ethernet), snapshot length 262144 bytes
    10.64.65.203.5353 > 224.0.0.251.5353: 0*- [0q] 1/0/1 Brother MFC-L9570CDW series._uscans._tcp.local. (Cache flush) TXT "txtvers=1" "vers=2.62" "adminurl=https://BROTHERMFCL9570CDW.local./net/net/airprint.html" "representation=https://BROTHERMFCL9570CDW.local./icons/device-icons-128.png" "rs=eSCL" "ty=Brother MFC-L9570CDW series" "note=Stiftung" "pdl=application/pdf,image/jpeg" "UUID=e3248000-80ce-11db-8000-b42200665636" "cs=binary,grayscale,color" "is=adf,platen" "duplex=T" (435)
16:50:36.026741 IP6 (hlim 255, next-header UDP (17), payload length 443) fe80::b622:ff:fe66:5636.5353 > ff02::fb.5353: [udp sum ok] 0*- [0q] 1/0/1 Brother MFC-L9570CDW series._uscans._tcp.local. (Cache flush) TXT "txtvers=1" "vers=2.62" "adminurl=https://BROTHERMFCL9570CDW.local./net/net/airprint.html" "representation=https://BROTHERMFCL9570CDW.local./icons/device-icons-128.png" "rs=eSCL" "ty=Brother MFC-L9570CDW series" "note=Stiftung" "pdl=application/pdf,image/jpeg" "UUID=e3248000-80ce-11db-8000-b42200665636" "cs=binary,grayscale,color" "is=adf,platen" "duplex=T" (435)
    10.64.65.203.5353 > 224.0.0.251.5353: 0*- [0q] 2/0/3 _ipp._tcp.local. PTR Brother MFC-L9570CDW series._ipp._tcp.local., _ipps._tcp.local. PTR Brother MFC-L9570CDW series._ipps._tcp.local. (1437)
16:54:01.317889 IP6 (hlim 255, next-header UDP (17), payload length 1445) fe80::b622:ff:fe66:5636.5353 > ff02::fb.5353: [udp sum ok] 0*- [0q] 2/0/3 _ipp._tcp.local. PTR Brother MFC-L9570CDW series._ipp._tcp.local., _ipps._tcp.local. PTR Brother MFC-L9570CDW series._ipps._tcp.local. (1437)
###SHORTEND###
^C58979 packets captured
7589695 packets received by filter
0 packets dropped by kernel

# tcpdump -i vlan0.70 -n 'udp port 5353' -v | grep "Brother MFC-L"
tcpdump: listening on vlan0.70, link-type EN10MB (Ethernet), snapshot length 262144 bytes
^C511099 packets captured
13203846 packets received by filter
0 packets dropped by kernel

Running mdns-repeater manually in the foreground (mdns-repeater -f ix0_vlan65 vlan0.70) did not make a difference. It showed forwarding lots of packages from VLAN 70, but non from VLAN 65.


Firewall rules
positioned as top most interface rules:
1) rule VLAN 65

• active: yes
• interface: 65SrvIntern
• quick: yes
• action: pass
• direction: incoming
• version: any
• protocol: udp
• source: any
• source port: any
• target: 70ClientIntern network
• target port: single port -> 5353
• protocol: yes

2) rule VLAN 70

• active: yes
• interface: 70ClientIntern
• quick: yes
• action: pass
• direction: incoming
• version: any
• protocol: udp
• source: any
• source port: any
• target: 65SrvIntern network
• target port: single port -> 5353
• protocol: yes

info to each rule (live log) shows 0 hits.

*push* Anyone an idea? The issue persists with OPNsense 24.7.11_2-amd64. Thx!

Since the last update to Opnsense 24.7 with squid as transparent http proxy and SSL SNI, some users report about being not able to access some https-websites. I tried to capture the relevant cache.log output excerpt (with "debug_options ALL, 1 11,6 26,6 83,6"):

Code Select Expand

2024/12/16 14:42:23.116 kid1| 83,5| Session.cc(96) NewSessionObject: SSL_new session=0x1c15fae80000
2024/12/16 14:42:23.116 kid1| 83,5| Session.cc(154) CreateSession: link FD 1247 to TLS session=0x1c15fae80000
2024/12/16 14:42:23.117 kid1| 83,5| bio.cc(114) write: FD 1247 wrote 2452 <= 2452
2024/12/16 14:42:23.118 kid1| 83,5| bio.cc(137) read: FD 1247 read -1 <= 5
2024/12/16 14:42:23.118 kid1| 83,5| Io.cc(92) Handshake: -1/35 for TLS connection 0x1c15fae80000 over conn19210 local=216.194.167.35:443 remote=10.63.10.46:60964 FD 1247 flags=33
2024/12/16 14:42:23.119 kid1| 83,5| bio.cc(137) read: FD 1247 read 5 <= 5
2024/12/16 14:42:23.119 kid1| 83,5| bio.cc(137) read: FD 1247 read 19 <= 19
2024/12/16 14:42:23.119 kid1| 83,5| Io.cc(92) Handshake: -1/0 for TLS connection 0x1c15fae80000 over conn19210 local=216.194.167.35:443 remote=10.63.10.46:60964 FD 1247 flags=33
2024/12/16 14:42:23 kid1| ERROR: failure while accepting a TLS connection on conn19210 local=216.194.167.35:443 remote=10.63.10.46:60964 FD 1247 flags=33: SQUID_TLS_ERR_ACCEPT+TLS_LIB_ERR=A000412+TLS_IO_ERR=1
2024/12/16 14:42:23.119 kid1| 83,5| Session.cc(201) SessionSendGoodbye: session=0x1c15fae80000
2024/12/16 14:42:23.119 kid1| 83,5| Session.cc(93) operator(): SSL_free session=0x1c15fae80000
Without squid, I can access the website just fine. The ssl parameters of the website connection seem to be ok: TLS_AES_256_GCM_SHA384. 256-Bit-Key. TLS 1.3), the cert is valid (according to Firefox).

What could be the issue here? How to debug further? Any help appreciated. :)

Sorry for expressing myself not clear enough. And I'm also not sure if I understand your question about "client export vs. simple OpenVPN client side".
I tested a little further and for me it seems to work by:
1. creating a user without adding/assigning any certificate
2. client export within OpenVPN, with the certificate I used to attach to my VPN users.
So there might actually not be a (technical) problem for me right now. And I might just have been confused ("usability problem") by not being able to add the cert to the user anymore and not seeing any users linked anymore to the cert in OpenVPN > Client export. Sry for the noise.

My current situation here is, that I'm assigning one existing cert to multiple users (not matching CN) for VPN access (like, as a MFA next to a password). (Please let's not discuss if this is a sensible setup, I inherited it and need a working setup now to buy time to design and introduce a better concept for my users, e.g., using individual certs per user.)
@franco if I understood you correctly, this should still be doable. I fail so in the Web UI. Can you please elaborate how I can assign an existing non matching cert to a user?

Thanks for your support!
Simon

I have not touched the OPNsense configuration since the upgrade. SSH is disabled on the router so the web interface is the only way to manage it.

Can you try a locale console (if you have physical access and can connect monitor + keyboard)?

No idea about the root cause of the error message. If the web GUI has worked after the update, maybe there's a chance that it is a temporary error. In that case, maybe forcing a reboot (e.g. cutting of power) could help?

Good luck!

Don't worry. I'm interested in finding a good/correct solution myself. :)
I only fear my knowledge might be too limited to help. In case a view at our installation would help, just let me know via PM and we can figure that out.
Simon

Thanks for your confirmation. I opened a ticket.
Simon

Thanks for the replies!

Just an idea, because I saw symptoms like these: Did you try lowering your MTU or do MSS clamping?

If you use VLANs, PPPoE or anything else that limits ord reduces your real MTU, you will experience packet loss with sites that cannot do correct PMTU discovery along the way to some sites. Such sites will then be much slower, because eventually, the dropped packets are being corrected.

Gateways are indeed PPPoE and internally all interfaces use VLANs. The assigned/default MTU is 1492. I once lowered the MTU and MRU in the settings to 1450, which led to a connection breakdown (but reasons for the breakage might be elsewhere as well).
In the meantime we noticed, that suricata was running in IDS and IPS mode, which is apparently neither supported for VLANs nor for PPPoE. One hypothesis was, that suricata might have messed up with the packages as well (leading to MTU issues). We deactivated suricata and will observe it for a couple of weeks to see if the problem persists.

I have had the same problem for some time, and also the same temporary solution. I thought it had something to with cache, memory, logs etc. But i think I solved it by turning on PowerD. All settings are set to HiAdaptive. It was turned off by default.

It has been running for 2 weeks now without any issues, no reboots. Something which wasn't possible before. I hope this helps.

It is turned off over here as well. I would not have thought of it as a reason but by now I will not rule out anything... I'll keep it in the back of my mind. If disabling suricata will not be the solution and another attempt of lowering the MTU will also not help, I'll for sure try it out.

In any case, I'll report back here. Thx! Simon

[both gateways receive the same upstream gateway IP address]

Following set up: two PPPoE gateways (two separate contracts) from the same ISP: one smaller uplink for VoIP and one bigger uplink for all the other traffic (mainly surfing the web). Firewall/NAT rules to choose the correct gateway. IP addresses of the gateway are assigned by DHCP from the ISP. The ISP differentiates them by the PPPoE credentials and assigns us the respective WAN IP address. As both gateways come from the same ISP and are only two different sized contracts, both gateways receive the same upstream gateway IP address by the ISP.

As we do have a mixture of different connection problems and I cannot really find the solution, I was researching on the net and stumbled upon a github issue. If I understand the comments by fichtner correct, this set up might not be supported at all.

Is my understanding correct, that it is not supported to have two gateways with public IP addresses A and B which both have the same gateway with IP address C?
I'm asking, because I do not see any error message in the Web GUI and I would have assumed that I do get an error message in the Web GUI if it is not supported. If someone can confirm my understanding, I'm happy to open a ticket. :)
Simon

*push* anyone having an idea?  :)