I live in The Netherlands and use an OPNsense router as a replacement for the ISP (KPN) modem/router.
We also have two TVs; these STBs use multicast (IGMP) so I use os-igmp-proxy to get this to work.
According to the traffic graph, each stream uses around 14.4 Mbps of bandwidth. On 24.1, switching TV channels resulted in a short increase in traffic (28.8 Mbps) which would very quickly drop down back to 14.4 Mbps. I believe this is how IGMP works due to a subscriber unsubscribing.
After upgrading to 24.7 this does not happen anymore... the traffic graph keeps showing an increase in traffic after each TV channel switch but also does not come down anymore. Eventually (after 5-10 minutes it seems) it does come down (in 14.4 Mbps steps), but it's no more instant like it was on 24.1 and causes the STB to freeze once the traffic gets too much.
What could be causing this? I have not changed anything configuration-wise. The STBs are on a VLAN and both are connected directly to my OPNsense box through a bridge (this is fine performance wise).
Just noticed these errors in the logs:
igmpproxy MRT_DEL_MFC; Errno(49): Can't assign requested address
igmpproxy select() failure; Errno(4): Interrupted system call
Searching for this error results in some older topics, also after an OPNsense update:
https://forum.opnsense.org/index.php?topic=18347.0
and
https://forum.opnsense.org/index.php?topic=18490.msg84154#msg84154
Hopefully this can be fixed soon, because this is flooding the STB causing it to freeze.
Thanks!
Same issue for me, also with KPN as ISP.
It seems that the streams are beeing dropped after a while (4 mins orso), but if you change channels quickly the bandwith increases very quickly.
IGMP Proxy version is the same as before, and re-installing the proxy doesn't fix the issue.
Quote from: DutchCrownNL on July 26, 2024, 03:06:20 PM
Same issue for me, also with KPN as ISP.
It seems that the streams are beeing dropped after a while (4 mins orso), but if you change channels quickly the bandwith increases very quickly.
IGMP Proxy version is the same as before, and re-installing the proxy doesn't fix the issue.
Yep, it seems to be a kernel issue (that's where they fixed it before if I read correctly). The problem lies with the multicast group unsubscribe failing. The streams are eventually dropped by KPN; I've noticed the streams staying up even after rebooting OPNsense.
I just hope the amazing folks at Deciso can solve it because otherwise I'm afraid I'm going to have to downgrade to 24.1 (the people I live with are complaining haha).
Any idea which patch could be missing? stable/14 may be incomplete or regressed a bit oO
Well I see two larger changes from 2023 done for FreeBSD 14. Maybe this is groundhog day all over? :)
Will look at it closer next week. Tickets on GitHub welcome...easier to follow up.
Cheers,
Franco
And yes there are some fixes on FreeBSD main that have no backport. This is great (stable) stuff.
Can you install this kernel and reboot?
# opnsense-update -zkr 24.7_2
Better or same?
Cheers,
Franco
Thanks for the replies @franco!
I will try the new kernel in about half an hour and will report back.
Quote from: franco on July 26, 2024, 05:44:52 PM
Can you install this kernel and reboot?
# opnsense-update -zkr 24.7_2
Better or same?
Cheers,
Franco
Will try this later tonight aswell and report back.
Quote from: franco on July 26, 2024, 05:44:52 PM
Can you install this kernel and reboot?
# opnsense-update -zkr 24.7_2
Better or same?
Cheers,
Franco
Just tried this one; unfortunately the problem remains the same so I have switched back to the current kernel using `opnsense-update -k`.
Looking back at the posts from 2020, this change that seemed to have fixed it back then (not sure if there's more changes from that thread that are relevant though):
https://forum.opnsense.org/index.php?topic=18490.msg84346#msg84346
Perhaps comparing these files might lead to the solution (I'm not familiar enough with C to understand the code), but of course it could also be something else:
IPv4:https://github.com/opnsense/src/blob/stable/24.1/sys/netinet/in_mcast.c
https://github.com/opnsense/src/blob/stable/24.7/sys/netinet/in_mcast.c
IPv6:https://github.com/opnsense/src/blob/stable/24.1/sys/netinet6/in6_mcast.c
https://github.com/opnsense/src/blob/stable/24.7/sys/netinet6/in6_mcast.c
As for GitHub issues: I'd be more than happy to create an issue there, but in which repo should I create the issue?
I checked the old stuff but the commit in question is still there. Let's try to revert one of the possible breaking candidates instead:
# opnsense-update -zkr 24.7_1
The bug report should go to opnsense/src.
Cheers,
Franco
Quote from: franco on July 26, 2024, 06:39:08 PM
I checked the old stuff but the commit in question is still there. Let's try to revert one of the possible breaking candidates instead:
# opnsense-update -zkr 24.7_1
The bug report should go to opnsense/src.
Cheers,
Franco
Thanks very much for the effort, but unfortunately this one doesn't fix the issue either (is it correct that the version naming is lower compared to the previous kernel test version?).
I have just created the GitHub issue:
https://github.com/opnsense/src/issues/210
Thanks again and have a nice weekend!
Yes I did not push the two backports to GitHub because they were useless so the revert was just one commit after 24.7 tag. Ok, so that's not either. I'm willing to roll back one more commit but if that's not it I'm unsure what it actually is.
Cheers,
Franco
# opnsene-update -zkr 24.7_5
Last one for triage, then we should continue structurally in the ticket.
Thanks,
Franco
Quote from: franco on July 26, 2024, 08:05:55 PM
# opnsene-update -zkr 24.7_5
Last one for triage, then we should continue structurally in the ticket.
Thanks,
Franco
Thanks for this last try this evening on a Friday!
Unfortunately, this one didn't fix the issue either. :(
For now, we'll just have to be very careful not to 'zap' between TV channels too fast and better plan which channels to watch. I hope the people I live with accept that for now. :P Hopefully more luck next week; have a nice weekend!
Thanks so far. At least we have a starting point.
Cheers,
Franco
As i said yesterday i would test the updates, but just like Joris neither of them worked for me. :-\
Thanks, but no worries. These were the obvious things to try, but it just showed us that we need to look elsewhere in the kernel which will take a bit more coordination and code review.
Cheers,
Franco
Thanks Franco. Unfortunately I have just discovered another related issue.
After a day of OPNsense running and the TV being used occasionally, somehow TV channels start stuttering after a few seconds (even if the traffic graph does not show abnormal things). I have checked the logs but didn't see anything related to this.
Restarting the IGMP proxy service seems to have fixed the stuttering... probably related to all the streams that haven't been released?
While I'm happy to assist with the operational things pertaining to software I have no way to test this myself.
Cheers,
Franco
I understand. I think the main issue is that unsubscribe requests aren't handled correctly. That's just a multicast thing. IGMP proxy probably doesn't handle that properly causing the stuttering (which explains why restarting the service fixes the stuttering).
I think this issue will be fixed as well if the streams are correctly released after the unsubscribe request. I will be available to test stuff next week; I'm more than happy to test potential fixes.
Interesting. Im using IGMP to watch tv too.
So I tried to replicate issue (to know how Im affected), but it is not happening to me - cant replicated described behavior.
Can you please help me diagnose, how do you noticed that issue is happening?
Quote from: mnaim on July 28, 2024, 12:22:32 AM
Can you please help me diagnose, how do you noticed that issue is happening?
It's quite simple; basically open the OPNsense dashboard and watch the traffic graph. Turn on your TV and you should see the traffic increase to a certain level. Then switch to the next TV channel and you will see the traffic graph increase again. It should only increase shortly (it did before) but after the 24.7 update, the traffic no longer reduces. Now switch again to the next TV channel and you will see another increase. This keeps happening with each switch to the next TV channel.
Quote from: jorisvervuurt on July 27, 2024, 03:39:35 PM
I think this issue will be fixed as well if the streams are correctly released after the unsubscribe request. I will be available to test stuff next week; I'm more than happy to test potential fixes.
I seem to have the same problems. Restarting IGMP proxy is the workaround I use for now. Happy to help with testing/sharing my config when needed.
Best regards.
Good morning.
New test kernel:
# opnsense-update -zkr 24.7_9
Cheers,
Franco
Quote from: franco on July 29, 2024, 10:42:39 AM
Good morning.
New test kernel:
# opnsense-update -zkr 24.7_9
Cheers,
Franco
Already replied on GitHub, but that one seems to have fixed the problem!
Thanks so much. :)
Just out of curiosity; what was the actual issue? Not sure if it's too complex to explain of course.
Thanks for confirming. 8)
Here is the timeline...
1. Bad fix introduced to FreeBSD main 31.08.23 https://cgit.freebsd.org/src/commit/?id=fa03d37432
2. Bad fix cherry-picked to FreeBSD stable/14 branch07.09.23 https://cgit.freebsd.org/src/commit/?id=70fbe797c0
3. FreeBSD 14.0 released based on broken releng/14.0 (from stable/14) state 20.11.23
3. Bad fix removed from FreeBSD main 14.12.23 https://cgit.freebsd.org/src/commit/?id=c196e43243b8
4. Proper fix introduced to FreeBSD main 14.12.23 https://cgit.freebsd.org/src/commit/?id=c2e340452c1
5. FreeBSD 14.1 released based on broken releng/14.0 (from stable/14) state 04.06.24
I've added these two commits for 24.7.1 (which are in 24.7_9 as tested):
https://github.com/opnsense/src/commit/48570316
https://github.com/opnsense/src/commit/e81800f6a
Cheers,
Franco
Cool! I noticed only the IPv4 file has changes; I haven't checked the IPv6 file, but should that one be changed as well? :)
https://github.com/opnsense/src/blob/stable/24.7/sys/netinet6/in6_mcast.c
No, no changes vs. FreeBSD main.
What I find odd is that the committer had 5 months to bring this to FreeBSD 14.1 after breaking it in FreeBSD 14.0 but it would rather have it remain broken in both versions. This is literally 10 seconds worth of work it would have needed to be fixed in FreeBSD 14.1. ;)
Cheers,
Franco
Oh well, it's fixed now anyway. :P
Thanks again.
Quote from: franco on July 29, 2024, 12:22:44 PM
What I find odd is that the committer had 5 months to bring this to FreeBSD 14.1 after breaking it in FreeBSD 14.0 but it would rather have it remain broken in both versions. This is literally 10 seconds worth of work it would have needed to be fixed in FreeBSD 14.1. ;)
I sincerely hope I will be able to buy you and a certain other individual a beer or two in Dublin and have a chat about upstreaming/MFC policy without anyone pulling the knives out ... 8)
I just made a €50 donation to say thanks for picking this up so fast but also as a way to say thanks for OPNsense in general! Feel free to add me (Joris Vervuurt) to the list if you want to. 😄
Looking forward to the next update.
Thanks, highly appreciated. Most of the timing is related to how good the testers are so I have to say you were a great one. :)
Cheers,
Franco
;)
Quote from: franco on July 29, 2024, 12:22:44 PM
What I find odd is that the committer had 5 months to bring this to FreeBSD 14.1 after breaking it in FreeBSD 14.0 but it would rather have it remain broken in both versions. This is literally 10 seconds worth of work it would have needed to be fixed in FreeBSD 14.1. ;)
Odd? Looing at the committer / maintainer, I'm kinda getting convinced these stunts are done on purpose for the firewall distro that's based on -CURRENT/-HEAD.
I chose "odd" to be friendly. But this isn't the first time -- not sure if you saw some of the other things. ;)
Cheers,
Franco
Unfortunately, yes... the concept of stable seems to get completely lost. Seems like someone wants "stable" well-known bugs.
Good work!
Last question though ... on both LAN- and WAN-side I see that while one stream is about 14Mbps, when I switch channels the consumed bandwidth on the WAN-side doubles for about 3 minutes (to about 28Mbps) , while on the LAN-side it takes about half a minute before everything is back to normal. Is that expected behaviour? Never looked at it in this detail before, so I hope you don't mind me asking ::)
Best regards.
Quote from: i81b4u on July 29, 2024, 05:37:27 PM
Good work!
Last question though ... on both LAN- and WAN-side I see that while one stream is about 14Mbps, when I switch channels the consumed bandwidth on the WAN-side doubles for about 3 minutes (to about 28Mbps) , while on the LAN-side it takes about half a minute before everything is back to normal. Is that expected behaviour? Never looked at it in this detail before, so I hope you don't mind me asking ::)
Best regards.
In my case, traffic is dropping back pretty much instant (well, maybe a second) on both interfaces. I think something else is causing this in your case, not sure what though. Have you tried connecting the STB directly to the router instead of behind (non-IGMP snooping enabled?) switches to see if that changes anything?
Quote from: franco on July 29, 2024, 11:55:23 AM
I've added these two commits for 24.7.1 (which are in 24.7_9 as tested):
So, just to conform the aforementioned sentence --> Has this fix been included in the standard (now released) 24.7_9 or will they only be released in 24.7.1 and are they only in the test kernel for now?
I am waiting with my upgrade for this fix as I am also a KPN fiber user in NLD... :)
24.7.1 kernel will have it, for now on 24.7_x use the test kernel.
Cheers,
Franco
Quote from: franco on July 30, 2024, 04:07:16 PM
24.7.1 kernel will have it, for now on 24.7_x use the test kernel.
Copy all, thanks. I'l just wait then. I can install a testkernel, but when stuff goes south I am too noob to dive in and fix on the commandline. :)
Quote from: jorisvervuurt on July 30, 2024, 09:41:47 AM
In my case, traffic is dropping back pretty much instant (well, maybe a second) on both interfaces. I think something else is causing this in your case, not sure what though. Have you tried connecting the STB directly to the router instead of behind (non-IGMP snooping enabled?) switches to see if that changes anything?
After another reboot I now see that everything works according to specs ???.
Anyway ... thanks for pointing out what to expect. Made me go and question myself and my setup ;).
Best regards.
Quote from: Rene78 on July 30, 2024, 04:40:56 PM
Quote from: franco on July 30, 2024, 04:07:16 PM
24.7.1 kernel will have it, for now on 24.7_x use the test kernel.
Copy all, thanks. I'l just wait then. I can install a testkernel, but when stuff goes south I am too noob to dive in and fix on the commandline. :)
Testing has been concluded and there's a known good kernel that fixes your issue, which you can install with
opnsense-update -zkr 24.7_10 && opnsense-shell rebootReverting to the non-patched kernel - should you consider it for some reason - is as easy as checking for updates again.
There aren't that many updates to be pushed yet, nothing critical requiring an immediate fix that I can see, hence the probable time frame you're looking at is mid next week for 24.7.1
Just another KPN user here too. The TV just plays a few seconds and then freeze. I've tried the 24.7_10 kernel but didn't solve the issue.
24.7 was working fine when I had it virtialized using virtual bridges. Now I've decided to move baremetal on x710 nic, and it's not working (same config). I can't think in any other change I made now, so I assume it's something with my nic.
How can I check if the connections are being released quickly? To avoid family issues, I just set the iptv box to unicast and it's working, so I think I'm having issues with multicast and igmp.
For one moment I thought also maybe it was Zenarmor/netmap, but I set it in passive mode, and nothing has changed. I really don't know what else to try.
Quote from: furfix on August 01, 2024, 11:09:33 AM
Just another KPN user here too. The TV just plays a few seconds and then freeze. I've tried the 24.7_10 kernel but didn't solve the issue.
Haven't tried the _10 kernel, but _9 does fix the issue:
opnsense-update -zkr 24.7_9 && opnsense-shell rebootEasiest way to check if streams are released is to monitor the traffic graph. Each HD stream is around 14.4 Mbps. If you switch to the next TV channel, you should see the traffic graph shortly increase to 28.8Mbps and then pretty much instantly drop down back to 14.4 Mbps. It shouldn't keep increasing (which was the initial issue that has been fixed in the _9 kernel).
Quote from: newsense on July 31, 2024, 04:18:58 AM
Testing has been concluded and there's a known good kernel that fixes your issue, which you can install with
opnsense-update -zkr 24.7_10 && opnsense-shell reboot
Testing was done with _9 though, not _10.
Did you mean to write _9 or is _10 another follow-up?
I only tested with _9 which fixed the issue; haven't tested _10.
I tried the 10. Going to test 9 now. Rebooting.
Just for clarity: since _10 is one more than _9 both have the same fixes for IGMP, but _7 doesn't have the two relevant IGMP fixes.
Cheers,
Franco
Understood. With 9 the issue remains :(
Probably something totally unrelated, but the freeze it's only happening with channels below 100. From 96 and up, looks like it's working without problems. No freeze
Quote from: franco on August 01, 2024, 11:34:40 AM
Just for clarity: since _10 is one more than _9 both have the same fixes for IGMP, but _7 doesn't have the two relevant IGMP fixes.
Cheers,
Franco
Alright, good to know that the increment means it includes everything from the earlier version.
I thought it was just a random number. ;-)
Just to be sure, I will test _10 later today as well.
Quote from: furfix on August 01, 2024, 11:38:16 AM
Understood. With 9 the issue remains :(
Probably something totally unrelated, but the freeze it's only happening with channels below 100. From 96 and up, looks like it's working without problems. No freeze
Can you check the traffic graph please? Traffic should only very shortly increase and then drop back to 14.4Mbps (or ~8 for some lower quality channels). If it does drop back then you have another issue.
Also check the logs and filter on 'igmpproxy'.
Sorry if I missunderstood you, but you mean this?
That is my WAN IPTV gateway and my IPTV LAN. For giving you a little more context, basically when I switch channels there is around 20m pike traffic for a few seconds, and then it's dead...same moment when the tv freeze. In the graph, basically I switch 3 channels.
Quote from: furfix on August 01, 2024, 11:51:40 AM
Sorry if I missunderstood you, but you mean this?
That is my WAN IPTV gateway and my IPTV LAN. For giving you a little more context, basically when I switch channels there is around 20m pike traffic for a few seconds, and then it's dead...same moment when the tv freeze. In the graph, basically I switch 3 channels.
That's not the issue this thread was about; it's probably a configuration issue on your side.
Please send a direct message to me so I can try to help you there (we don't have to continue in this thread since its unrelated).
I will, thanks Franco and you and sorry for bothering you both!
Quote from: franco on August 01, 2024, 11:34:40 AM
Just for clarity: since _10 is one more than _9 both have the same fixes for IGMP, but _7 doesn't have the two relevant IGMP fixes.
To confirm, everything works fine with _10 as well over here. ;-)
Just installed 24.7_10 and no more freeze. Thanks!
for some reason i cant install the kernel?
Not Found ] Failed, no signature found.
24.7_10
do i need to be in the beta channel?
You're too late. Enjoy 24.7.1 where the fix is baked in already.
Cheers,
Franco