Messages

I recently rebuilt my Opnsense firewall and split the network into multiple VLANs to isolate the security cameras/alarm, iot, streaming, management and internal traffic.  Because of the number of IoT devices and streaming devices, it's been a nightmare process, but finally have multicast and firewall rules in a good place.
BUT I keep seeing randomly blocked groups of traffic to open ports; these are mainly from the Ring alarm base unit (outbound, 443/tcp), plex servers (443/tcp) and Apple mobile devices (5223/tcp). These are all showing flags of RA/FPS/FA; they happen between 5 and 25 times at once, then stop again for a random time.
I've read these can be ignored, but i'd like to try and stop them if i can!
I've set the firewall to "conservative" mode, configured adaptive timeouts, increased the state table size, and even changed the individual rules to "sloppy State" and extended the timeout to 24 hours; but it still keeps randomly happening, and frankly driving me nuts!

i've got a single WAN/ISP port; and a single internal interface hosting the VLANs; these are both on an intel x550-T2 card with the WAN running at 2.5GB and the internal connected vias SFP to a unifi switch at 10GB.

Is there anything else I can tweak to avoid thid rtepeeated red traffic in my log!

After some poor network performance, I decided to rebuild my network to try and optimise and secure it better.  I was also sick of having to turn off (R)STP because the Sky Q box would disable ports due to it's network mesh catastrophe!

I have (I think!) quite a robust setup - 6 Unifi Access Points, Unifi Enterprise 24 port multi-gig PoE+ switch, with a trunk port to a Unifi 8 port flex PoE switch.
The Opnsense Firewall is an i3-7100 with 8gb RAM and a 500GB SSD. I have an intel x550-T2 PCIe network card installed, with one port set at 2.5gbps and connected to a VM Superhub 5 @ 1150Mbps, and the other port connected at 10GBps via an SFP+ adapter into the Unifi main switch.

I have split the network into 6 different VLANs (Plus WAN); Internal, Management, Streaming, IoT, Security and Garage.  iPhones, iPads, Laptops, Desktops (Including two Plex/Emby Server) and Macs are all on the internal VLAN, Management VLAN holds the switches, Unifi APs and the server running Homebridge and the Unifi Controller. Streaming VLAN contains the "trusted" devices, SkyQ boxes, nvidia Shield, Amazon Fire TV, Network Audio Players, AV Amps and Sonos Speakers.  The IoT VLAN contains all the other "Smart" devices - Meross smart plugs/extensions/energy meters/radiator valves/smoke alarms, Twinkly(tm) Lights, smart washer/dryer, smart fridge smart kettle etc; plus Apple Homepods to make it easier to control them.  The Garage VLAN is for using diagnostic software and tools that i don't want attached to the internal networks in any way; and the security VLAN houses all my Ring internal and external cameras, alarm, doorbell etc.  This is also isolate from the main VLANs.

Trunk ports and uplink ports are all working great, RSTP is enabled with tweaked switch priorities to make the Sky Q behave, and the ring cameras aren't dropping off the network like they were due to lost packets.  All seems good....except some REALLY annoying glitches!

At the moment, I have set up individual rules for each device group (amazon Echo, Amazon Fire, Plex servers etc) and have been assigning rules between the VLANS as they've popped up blocked and ensured they are a locked down as possible.  I have configure IGMP Proxy, mDNS Repeater for mDNS traffic, and UDP Broadcast Relay to handle all other multicast (SSDP etc).  that brought A LOT of multicast traffic which clearly means things are talking!  Everything seems to works fine, i can even control my Twinkly lights through the app on the internal network which has never worked properly before.  I've had to all ephermeral ports between amazon and streaming devices as expected as well as Specific Smartthings ports and Apple specific APN ports etc to make things behave.

The problem I'm seeing a lot more blocked packets than i was before; Desktop (Plex/Emby) with a lot of SYN flags to port 8883, RST/ACK, FIN/ACK to 443/t; with amazon devices and Ring Alarm base unit seeing lots of PSH/ACK, FIN/ACK and FIN/PSH/ACK to 443/t - all to the internet. They all get stopped by the default "Block All" rule on each VLAN ruletable, completely ignoring the specific rules allowing the traffic before it.  I have even enabled the "options" checkbox under advanced to see if that helps as a lot of traffic is likely multicast around the network; but no dice.

I've also seen a lot of people saying "Oh ignore it, normal traffic" - but i never saw this sort of traffic before.  Can anyone offer any suggestions of pointers - or tell me i'm being stupid and ignore it!?

Thanks!

Sky Q is why I originally put this package together. 😊

I suspect you need to add a firewall entry on your PC. Windows will block the responses from the Q box as it's coming from an address on a different VLAN.

Open windows firewall, Select Advanced Settings

Select Inbound Rules
New Rule

Name: Sky Q Pass
Enabled: Ticked

Protocol and Ports Tab
Proto Type: Any

Ports: Local and Remote: Any

Scope Tab
Local IP address: Any
Remote IP: YOUR Q BOX IP - In my case 10.4.15.91

Advanced Tab:
Specify profiles to which this rule apples
Tick all of them

That should do you.

Sir you are indeed correct - Thank You!  As a chance, i looked at the eset firewall on my pc, and it was blocking SSDP traffic and traffic from the SkyGo executable.  unblocking this immediately fixed the issue with Sky Q connecting on my PC.  I've now added the streaming and IoT Subnets to trusted zones on the basis that if i've allowed the traffic through the firewall, then it's safe!

Of course I'm now left with other issues, but that's another thread..... :)

A Small update; i have managed to get Plex and Emby working via an Amazon Fire Stick over the VLANs, which is great.
for reference, you need to open up 32400/udp from the fire tv to the media server for plex, and 8096/8920 TCP for Emby (7359/U) for server discovery.

But i've just run into a weird problem!  whilst the i-apps work perfectly for Sky Q, when i try to connect to the Sky Q box from the windows app on an internal LAN PC, it hangs, before telling me it needs to be connected to a sky Q box for recordings.  No rejects in the firewall when this happens, lost of 1900 and multiple requests back from each sky Q/mini box - but it fails.

Can't see why they would be different!

I'm been fighting with getting my Sky devices to behave properly on the separate VLAN for a few weeks; finally after reading this thread i've got it working using UDPBR. Have to say great work on the application, it's the only one i've found to successfully work so far!

I've also got a few other devices going into the Streaming VLAN: Sonos speakers, nVidia Shield, Yamaha AV Amp, couple of LG TVs, Amazon Fire TVs and a couple of Apple Pods. This means I'll be running a near full complement of stuff, including Airplay/Bonjour, mDNS, DLNA SSDP etc.  I've also got a separate IoT VLAN which will soon be housing Meross smart plugs, sockets, twinkly lights etc - I've seen nothing so far about what protocols are needed for the Twinkly app to work correctly - that's picky at the best of times!

I've got my firewall rules pretty restrictive with multiple VLANS (Security/Streaming/IoT/Management/Internal LAN); although internal LAN is configured to allow all traffic to other VLANs. To make things even more complex, i'm using a Unifi PoE 2.5GBe switch and multiple Unifi APs on the network. 

I added the default rule to allow anything from the Sky Q boxes to the LAN, and added several relay rules in UDPBR:
239.255.255.20:1900; 239.255.255.250:9131; 239.255.255.250:51200; and 224.0.0.51/224.0.0.251:5353 (source 1.1.1.1).  I had to add the extra options as the Sky Q box was failing on requests (even though it appeared to be working ok on the app). I also found that putting 1.1.1.1 instead of blank resulted in the connection failing - but what i've now got is a weird problem, where the media servers (running Plex and Emby) are staying on the internal network, but need to talk to the shield and the firesticks (and later the TVs and a hifi network streamer). It seems the media servers are getting a bit confused because they are getting bounced requests to the broadcast address of the Streaming device VLAN - the only way i can remove that is to add a specific rule in the Streaming FW rules to allow it.  is this expected behaviour?  As I have a default allow any rule to the streaming VLAN I expected that to also encompass the broadcast IP - but i would not have expected to have to allow the traffic specifically for broadcast IPs?
I also seem to be seeing some multicast traffic from the shield to 255.255.255.250: 51200, 1901 and 9131 which i don't recognise? Has anyone setup like this with Emby/PLEX and come across this issue and a workaround?

On a final point; i've found i've had to add specific floating rules in to allow multicast traffic to come through in order for it to be processed by UDPBR - while i expected it not to handle the outbound client request, I didn't think i needed to add specific rules to allow the traffic into the interface in the first place? 

Any tips on what i need to configure for my IoT and Media Servers would really be appreciated!

Sir you are a legend! Exactly the thing I'd forgotten! Had to enable hybrid NAT rules, but as soon as I create one for TCP and one for ICMP everything worked. 

Knew it would be something stupid I'd forgotten....  ;D

I've seen a couple of Apple devices showing ipv6 addresses as well as ipv4, but any ipv6 is disabled - although saying that I think there's an ipv6 default allow rule that Opnsense has set up...

Both the Ubuntu server and the firewall were rebuilt at the same time as I used new hardware, so no rogue routes etc left to cause issues.

Will look into the dev tools - thanks for the suggestion. The only tuning on dnsmasq is to redirect Remote Desktop services (teamviewer etc) to null and to redirect legacy music streaming services to the Ubuntu box.

With re Garda to IPv6 - the firewall was set to handle ipv6 but dhcpv6 was disabled. If disabled on fw settings as well now; is there anywhere else to look?

I think I'm having one of those weeks where nothing works right!
I am playing around with Docker at the moment to see if there is viable reason to put Plex/Emby into isolated containers. Currently experimenting with networking.

I configured a macvlan network and managed to get it almost working (didn't enable promiscuous mode on firewall) and I've been playing with IPVlan v3 networking now.  No matter what I do, I can't get internet traffic routing back to the subnet and it's driving me nuts - I'm sure it's something simple but I can't figure it out...I'm hoping someone can help me feel silly!

I have a route setup on the firewall to push any traffic for the docker subnet to the host Ubuntu vm (vm it set as a gateway). I have also added a lan rule on the firewall to allow all traffic from the docker subnet to the internet from the lan interface. I can see in the logs that traffic is being allowed since I added this rule, and I can ping the firewall and all other devices on the lan from the container in the v3 ipvlan network. But if I ping a web address (Google for example), it resolves the name, I see the traffic being allowed in the fw logs....but no I no response back.  From the ubuntu host it works fine.

Am I missing something obvious here?

I have always used Opnsense as the firewall with a Ubuntu box acting as the network router and dnsmasq for dhcp/dns, which has worked great.

At last firewall build I installed an Intel x550 lan card so decided to use the fw for routing/dhcp/unbound dns too to make use of the enhanced 2.5gbe bandwidth. Unfortunately since I've done this, I've had issues.  When trying to browse websites on any device, it sometimes takes an age before the site comes up, sometimes it times out and a refresh works.  Other times I'll be using a site then all of a sudden the connection will drop and I'll have to reload (normally when entering info into a page).

I've checked the config and everything seems fine - the only oddity I've setup is that only the firewall is allowed to contact wan dns, all other devices on the network use the fw for dns and are blocked from external services. I had this setup previously, with only the Ubuntu box being allowed external dns and it worked fine.

I know on hardware firewalls in the past we've been advised not to use it as the default router for the network (eg Sonicwall) as they had some unexpected behavior, I was wondering if anyone had similar experiences with Opnsense?

I am hoping someone can help me solve a really frustrating problem!

I am using a Lenovo branded intel x550-t2 card in a custom build pc using an i3-7th gen cpu and ssd.

I have been using a cheapo Realtek based dual 2.5gbps network card and this has been connecting to a 2.5gb switch and the 2.5gb port on my virgin router - both auto negotiate to 2.5gbps and when testing using another windows machine connected to a 2.5gbps nic (intel i219) I get around 1150mbps from my 1gig virgin service.
But when I swap out for the x550, it negotiates to a 10gbps port fine but otherwise only connects at 1gbps. If o force to 2.5, they connect at this speed for the most I can get using the same pc as earlier is 6-700mbps.

I do not have any ids running and only have a basic config on the firewall. I've tried the offloading settings but no change.

Can anyone offer any advice how to get this card to negotiate properly and more to the point provide comparable
Speed to the cheap card?

FYI I have just done a completely clean install to confirm it's not that and it isn't.

Can someone direct me to a guide to configure GoDaddy in the new ddclient?  I've been looking everywhere, but so far found nothing helpful!

I have moved my Let's Encrypt and Dynamic DNS duties to Opnsense; and have both of these working fine.

I have set an automation task up to upload the certificate to my Ubuntu server via SFTP task; this then rebuilds the certificate into a full chain and makes it available via a network share to other machines to access for SSL services.

this used to work, but i've since replaced my Ubuntu server and installed Ubuntu 20.  no matter what i try to do, I cannot get them to talk.  I have set up an sftp user on the ubuntu server and have connected rom the opnsense shell and accepted the encryption key; but when trying to connect through the GUI process, it always fails to connect with an access denied issue due to the encryption key.

I've gone through loads of different guides, all saying something different, but no definitive guide on connecting Opnsense Acme task with ubuntu.

Please can someone point me in the right direction before i tear my last few hairs out..?!

sorry for re-upping an old post, but i thought it would help someone else if they run into these problems!

Turns out the issue ISN'T to do with OpnSense at all, its to do with the Ciphers that are used by onedrive to sign in.

It seems that windows updated removed some older ciphers, that are required for OneDrive to work; this causes the issues with being unable to sign in.  Initially, I thought this was to do with SSLv3 not being enabled, so used IISCrypto to fix and it seemed to work...mostly.  then i read the below article and made the changes suggested.  this fixes the problem for me every time - hopefully it's going to stop someone else tearing their hair out like me!

https://docs.microsoft.com/en-us/sharepoint/troubleshoot/administration/error-0x8004de40-in-onedrive

I have been fighting with Onedrive for business for the last couple of months - previous to this I had used onedrive without any issues on my home network.  Suddenly, i started having problems with it connecting, or once connected having issues communicating.  I first put this down to Onedrive being, well, microsoft - removed all traces of the app from my win10 machine, deleted reg entries, repaired windows, deleted profiles - even created a new user profile.  same issues.  No traffic blocked in the firewall and can see 443 traffic from my laptop going out fine - it simply doesn't connect. 

today, I did some more digging, and did the latest fw update to see if it helped (it didn't).  whilst the firewall was offline, i switched to using my iphone hotspot - onedrive immediately connected and worked seamlessly!  Putting this down to coincidence, i have tried multiple times and can repeat this every time i connect using my phone as internet instead of through my home network.  I have tried via wifi and through wired connection - same issue.  It's currently sitting the same as it has been for the last 15 minutes saying "signing in".

Can anyone offer me any help?  It seems to have coincided with the last update i applied (not sure what it was as i didn't pay attention...clever).  the problem isn't the laptop, the network or the software - and oddly i can log onto the same machine and communicate via the web page to access onedrive while it's still refusing to connect on the app. meanwhile i can use the app on my phone on 4g without any issue. 

i have no traffic management enabled, and the most basic configuration - an allow all rule out from the internal lan and nat translation to a single Virgin media internet connection.  No Intrusion detection or any packet shaping or filtering enabled.