OPNsense Forum
English Forums => Development and Code Review => Topic started by: marjohn56 on December 28, 2017, 10:56:34 am
-
I am about to look at this as it's a requirement for Orange France users.
Now, there are three things, well probably more but at the moment only three come to mind. ;)
1. The addition of the priority setting to filter.lib.inc at line 294 this 'set-prio' => '2'
and I'm using the value 2 as an example (when implemented this will be selectable in the GUI) it does show in rules.debug, but it has no effect, this leads us to 2.
1. I believe the lack of "net.link.vlan.mtag_pcp" => "1", from system.inc is the reason, I'm waiting on Kev to check this for me as he has a test unit with orange france settings. There are several other sysctl values that appear in p****** that are not in opnsense, perhaps this needs to be looked at. In this instance though, I am only looking at the vlan priority stuff.
3. Where to put this option - On the darkside I added it to the dhcp6c settings section, but I am not sure this is the correct location, perhaps the system->advanced.network.
Thoughts?
-
Hmm, different question: do you want to prioritise DHCP6-only or the whole VLAN? Because there is a VLAN-priority setting as well.
-
DHCP6 only.
I'll bring Kev in here, he can describe the requirements much better than I can.
-
3. Where to put this option - On the darkside I added it to the dhcp6c settings section, but I am not sure this is the correct location, perhaps the system->advanced.network.
It's the right place as we could have different priorities per outgoing interface.
-
Hi Franco
Orange France supply a FTTP service
The requirement is for the dhcp and dhcp6c request to have prio 6 but all other traffic priory 0.
Tagging the whole vlan means dhcp and dhcp6 request get an IP but flow is reduced massively
As dhclient uses raw sockets that has been solved via the modified client so we are done there
Dhcp6 however passes through the firewall and it should be possible to tag the packet but even when rules.debug has the correct value the priority is not honoured
We had this working on the darkside but I know they did quiet some work around vlans as things changed from freeBSD 10 to 11
-
1. I believe the lack of "net.link.vlan.mtag_pcp" => "1", from system.inc is the reason, I'm waiting on Kev to check this for me as he has a test unit with orange france settings. There are several other sysctl values that appear in p****** that are not in opnsense, perhaps this needs to be looked at. In this instance though, I am only looking at the vlan priority stuff.
Ok so the "set prio 2" is written in the pf.conf indeed and:
# sysctl -a | grep mtag_pcp
net.link.vlan.mtag_pcp: 0
I don't know the impact of that sysctl, but flipping it for testing should be easy. :)
Edit: Code says this... but somehow we should try to figure out and set the sysctl without user interaction.
159 /*
160 * For now, make preserving PCP via an mbuf tag optional, as it increases
161 * per-packet memory allocations and frees. In the future, it would be
162 * preferable to reuse ether_vtag for this, or similar.
163 */
-
I hand modified sytems.inc to add "net.link.vlan.mtag_pcp" => "1"
I dont have the GITHIB skills :-)
But a wireshark trace is still showing the dhcp6 request with 0 as the priority
I can see in rules.debug that the changes Martin and I made to both systems.inc and filter.lib.inc seem to be creating the correct rule modification heres the modified rule.
pass out log quick on igb0_vlan832 proto udp from {any} port {546} to {any} port {547} set prio 6 label "allow dhcpv6 client in WAN"
-
As dhclient uses raw sockets that has been solved via the modified client so we are done there
Dhcp6 however passes through the firewall and it should be possible to tag the packet but even when rules.debug has the correct value the priority is not honoured
Sorry Kev, it did not occur to before, but could we use the same VLAN tagging we are applying to IPv6 to the dhcpv4 packets, thus making the use of a modified dhclient mute?
-
I dont think so but I may be wrong
But I believe 'raw sockets' don't pass through the firewall rules in the same way
-
no... its me having an idiot moment... :-X
-
Well I've spent a couple of hours on this tonight, it appears it should work but it does not. In fact I cannot get anything to go over the VLAN!
I'll have another look in the morning if I get time
-
Hmm, both the VLAN and the firewall priority settings were added in early 17.1.x by @djGrrr and this would mean it was written and tested on FreeBSD 11.0.
I'm fairly sure nothing changed since then, but just in case there is also a 11.1 test around.
https://forum.opnsense.org/index.php?topic=6257.0
-
I've got the VLAN's working now, it was some oddity with my APU device. I've switched now to a VM and i'm getting the VLANs OK, still cannot see why the set priority is not working on the dhcp6 packets, rules.debug shows its there, and here's the pfinfo. Can you see anything wrong there?
@49 pass out log quick on em0_vlan832 proto udp from any port = dhcpv6-client to any port = dhcpv6-server set ( prio 6 ) keep state label "allow dhcpv6 client in WAN"
[ Evaluations: 6171 Packets: 17 Bytes: 1972 States: 1 ]
[ Inserted: uid 0 pid 68116 State Creations: 11 ]
-
Looks ok, no priority at all? Or is the priority tag being overwritten by the VLAN priority setup value?
-
Priority is being sent out as 0, I hadn't though that maybe the VLAN priority is overiding it.
-
This is pretty nasty if true.
https://github.com/opnsense/core/blob/master/src/etc/inc/interfaces.lib.inc#L160
You can temporarily defang the VLAN setup by deleting the "vlanpcp" stuff there.
-
Confirmed, if I set the overall VLAN priority it all follows that, the dhcp6 packets get changed to Pri 6.
-
I just set the VLAN priority to 3 in the GUI
My entry in rules.debug shows
pass out log quick on igb0_vlan832 proto udp from {any} port {546} to {any} port {547} set prio 6 label "allow dhcpv6 client in WAN"
BUT wireshark trace show the VLAN still tagged as 3 so I conclude that the VLAN priority is overriding it or the rule is not modifying it
This is at 18.1
-
Ok, that means we need an empty "keep default" default in the VLAN device setup. This will still blow up with user setups, I'll file a FreeBSD bug report next year.
For now you can edit out the vlanpcp setup instruction in interfaces.lib.inc and that should start working.
-
it's also at 17.11, it's an unusual situation though, we set a pri on an VLAN then overide a specific port.
-
it's also at 17.11, it's an unusual situation though, we set a pri on an VLAN then overide a specific port.
That certainly stops the pri being set on the VLAN, sadly it has no effect on the dhcp6c packet, which is stubbornly still at 0
-
Ok, that means we need an empty "keep default" default in the VLAN device setup. This will still blow up with user setups, I'll file a FreeBSD bug report next year.
For now you can edit out the vlanpcp setup instruction in interfaces.lib.inc and that should start working.
It seems that's not the problem either, even with that edited out, we're still not getting the pri set on the dhcp6c packet.
-
Just tested at 18.1 as well
Same result with the vlanpcp edited out the priority no longer gets set via the GUI BUT its staying at 0
-
Wow, creating any VLAN already seems to get it stuck at vlanpcp 0 according to ifconfig output, which makes escalating this to FreeBSD more urgent... But for now, time for some kernel code reading. :/
-
Got it.
It is the net.link.vlan.mtag_pcp setting. Although I have set it to 1 in system.inc, that is not being honoured. I did a sysctl -a from the shell and it was showing 0, set it to 1 in the shell and voila, dhcp6c packets are now showing pri 6.
Ok, so why is the setting not being honoured in system.inc...
-
It is the net.link.vlan.mtag_pcp setting.
Ok, that is what the code said. A bit counter-intuitive, but now all makes sense.
Let me take a look at that sysctl.
-
You are a shining star. :)
-
I'll get on and do the GUI setting for it, seems like we have it nailed now.
-
Loving the proactive support over here in the light
The darkside was so different :)
-
Well, tunables GUI worked here even with reboot. All in all this teaches us it's something we should take care of automatically so changed the ticket accordingly and reverted the VLAN PCP configure code.
But... I'm not even convinced this will be a performance impact as we pass the packets through pf anyway and that setting only affects VLAN driver during transmission... ?
https://github.com/opnsense/core/issues/2032
-
Ok here is the rub-in:
The vlan_mtag_pcp sysctl is only used twice.
https://github.com/opnsense/src/blob/master/sys/net/if_vlan.c#L1105
This is the first usage spot that checks wether a PCP was stored, also via pf. It does not allocate.
https://github.com/opnsense/src/blob/master/sys/net/if_vlan.c#L1211
And this second spot tries to preserve the PCP from a packet where the actual allocations take place.
Circling back to the description, it mentions the allocations:
https://github.com/opnsense/src/blob/master/sys/net/if_vlan.c#L160
So the value protects agains performance degradation for allocations and frees, but in the end that only pertains to the second spot which causes these allocations. Even more so, if pf made allocations, the free happens anyway, so the performance is already lost.
My preference would be to kill the first conditional so that pf modifications work always, which won't make 18.1-RC1, but there is still time for 18.1-RC2 or 18.1. That adds a list traversal overhead, but I don't see how this patch wouldn't be accepted by FreeBSD unless they want to go the other route and protect "set prio" in pf with that same sysctl (I can't see how).
This would eliminate the need for fiddling with the sysctl at all.
What do you think?
Cheers,
Franco
-
I would go with whatever you think is best.
As I am adding the GUI for this option, maybe the solution is to set it to one ONLY IF this type of situation occurs, there will be a flag that will be set in the config, that could be read and the value set to 1 then.
-
The problematic bit is that we're generating a side-effect while unclogging "set prio" with the sysctl:
"While uncommon, it is possible that we will find a 802.1q packet encapsulated inside another packet that also had an 802.1q header. For example, ethernet tunneled over IPSEC arriving over ethernet. In that case, we replace the existing 802.1q PCP m_tag value."
So options are:
o removing the sysctl check in the reading spot, or
o adding a different sysctl for the reading spot, or
o making the current sysctl aware of "read-only" + "read-write" PCP mode.
Unfortunately, none of these can be fully worked around in user space. :)
Will try to get more feedback from FreeBSD committers and then commit something appropriate next week.
Cheers,
Franco
-
I'm sure Orange France knew of all the problems they would create when they decided on this daft idea!
-
You can probably hear them laughing right now. :)
Still, fun times chasing this down.
-
There will be a lot of Orange France users happy when this is finally sorted. Still need the dhclient on FreeBSD sorted and dhcp6c too.
Perhaps we could create a package for them to replace the existing clients with the modified versions just for them.
-
Great work tracking this down
-
So the sysctl usage is made obsolete here:
https://github.com/opnsense/src/commit/dabc3cf4ef
Then I added...
https://github.com/opnsense/src/commit/5f9b4916
And now I'm working on simplifying the option 77 patch:
https://github.com/opnsense/src/commit/c24697cb
I'm not sure if the latter compiles at the moment, but except for the BPF magic that needs review it should work if it builds. :)
-
Small build issue, otherwise ready for testing...
https://github.com/opnsense/src/commit/f6b3e14
It's the dhclient_77 branch:
# opnsense-code src
# git checkout dhclient_77
# cd /usr/src/sbin/dhclient
# make
# make install
Cheers,
Franco
-
Excellent thanks Franco
Will be tomorrow before I can test (family party tonight)
-
Is the plan to compile a dhcp6c as well or will that need to go upstream
-
# opnsense-code src
# git checkout dhclient_77
# cd /usr/src/sbin/dhclient
# make
# make install
Cheers,
Franco
Should the be:
# cd /usr
# opnsense-code src
#cd /usr/src
# git checkout dhclient_77
# cd /usr/src/sbin/dhclient
# make
# make install
The build issue I assume is with Kyuafile, not important?
-
Im getting
fatal: Not a git repository (or any of the parent directories) .git
sure its me but I don't know what Im doing wrong
-
Oh, sorry, /usr/src exists in stock install. Use this the first time:
# opnsense-code -f src
-
Our next issue is here...
https://github.com/opnsense/src/commit/c24697cb44ee5ac9963ba1a7e767ae590e1ea97f#diff-4832d06a0637c8fc27150af6b8f05b19R100
The change now requests the packet for *every* packet reaching the filter instead of checking because it's returning in this first step.
-
Ok, this looks reasonable now, but only reverse engineered...
https://github.com/opnsense/src/commit/c0056914
https://github.com/opnsense/src/commit/f841d1d3
Now we have 5 individual commits on top of the dhclient_77 branch which will help pin down what works and what not. :)
Happy testing,
Franco
-
OK, now having recovered from NYE, I am back on the case. :)
dhcp6c is looking good, all working on the VLAN and the prio is setting correctly and doing its thing.
Now, the problem is dhclient, although the VLAN is correct, the prio will not set, it's sticking at 0.
I have cheated a little and just copied the changes made for dhcp6c and changed the ports as needed, pasted it directly beneath the dhcp6 set prioty rule in filter.lib.inc - like this:
$dhcpv4_opts = array(
'label' => 'allow dhcpv client in ' . $intfinfo['descr'],
'direction' => 'out',
'interface' => $intf,
'protocol' => 'udp',
'from_port' => 68,
'to_port' => 67,
);
if (isset($intfinfo['dhcp6vlanprio'])) {
$dhcpv4_opts['set-prio'] = $intfinfo['dhcp6vlanprio'];
}
$fw->registerFilterRule(1, $dhcpv4_opts, $defaults['pass']);
rules.debug shows:
pass out log quick on em0_vlan832 proto udp from {any} port {546} to {any} port {547} set prio 6 label "allow dhcpv6 client in WAN"
pass out log quick on em0_vlan832 proto udp from {any} port {68} to {any} port {67} set prio 6 label "allow dhcpv client in WAN"
but wireshark shows priority 0.
-
Forgot to add, I see this in the firewall logs
WAN Dec 31 18:44:15 0.0.0.0:68 255.255.255.255:67 udp block bogon IPv4 networks from WAN
-
I confirm my tests give the same results as marjohn
The pro flag is not being set on the dhcp request despite the rule being present in rules.debug
-
Well I've been running 18.1.2_2 now for 3 days on the Orange FTTP service
Seeing max throughput of 500/250 no issues
I used the modified dhclient, dhcp6c (i shared these binaries previously) and the vlan flag in tuneables
Also used a marjohn modified interface.inc pending his pull request #2090 being merged
Getting a score of 19/20 on ipv6-test.com
So we know the theory I tested with wireshark work in the wild.
-
Next step is 2090 then?
What's still on the TODO list? I recall:
* dhclient VLAN fix
* dhcp6c raw options
Which VLAN flag are you referring to? mtag_pcp? It should not be necessary on 18.1 anymore.
Cheers,
Franco
-
2090 - I have to wait that long, feels like I'm back on the darkside ;D ;D
I assume you mean 20/20 result. Until Orange set up a reverse DSN entry for my IPv6 there is not much hope of that :-)
Yes setting the mtag_pcp to 1, I may get chance to try without that flag set later (heading back to UK soon)
You are correct on the to do list though
-
Confirmed removed net.link.vlan.mtag_pcp from tunables
All still working
Does that mean the I get a full solution in 2018 now :-)
-
Hopefully. :)
I'll work on 2090 this week... it's bigger than I liked so it had to be sidestepped for 18.1.
-
Next step is 2090 then?
What's still on the TODO list? I recall:
* dhclient VLAN fix
* dhcp6c raw options
I'll work on 2090 this week... it's bigger than I liked so it had to be sidestepped for 18.1.
It's not THAT big.. :)
if you look at it, it just breaks down a rather cumbersome call into more logical ones. For example rather than having the existing call writing all the config files AND starting rtsold and dhcp6c, the config file creation is broken out and is a separate call(s), the starting and stopping of the clients is also separate calls.
-
It keeps eluding... meh... sorry!
-
If it provides any motivation I'm back in France on the 18th March and could test :-)
-
I've had Martin explain everything to me... again... that he is still motivated shows great character!
All help welcome, I think we'll be ready when you are back in France.
Cheers,
Franco
-
@franco
I see you guys have completed dhcp6c now
Any ETA for the modified dhclient :-)
-
Hello :)
Not yet, but at least things keep moving into the right direction. RAW isn't final yet, we decided to run our own dhcp6c client for better review and modification so that comes in rather sooner than later (18.1.8 or 18.1.9 ?)
The patch for dhclient was reviewed and superficially ready, but I'm not sure I broke it during fixing the BPF filter which had a big XXX in the original patch.
Small discussion on this condensed commit: https://github.com/opnsense/src/commit/b179b4628b
Cheers,
Franco