Messages

Hey everyone,

I've playing with x86 server hardware for a good two decades now.

IN most of that time, the rule of thumb has always been if you are doing something that requires reliability, always, always, always go with the Intel NIC, and no matter what, avoid anything Realtek.

But in recent years things seem to ahve changed a little bit.   Intel has had some issues with their multigig NIC chips, and many people are - GASP - even recommending Realtek chips as alternatives.

Apparently pigs are flying, cats are sleeping with dogs, and snowballs are surviving in hell.

So, I guess my question is this.   How reliable are Realtek chips in something like OPNSense these days?

I am planning on building an embedded system, and locating it remotely where it is inconvenient (but not impossible) to get at should it go unresponsive.

Meanwhile I am seeing some really attractive embedded industrial boards with dual Realtek 8111H ports on them.

Are the old paradigms still true?   Would it be a bad idea to use one of these?   Or is it fine these days?

I'd appreciate any input.

So, I noticed that I used "00" instead of "0" in my minutes field. 

I wonder if that did it, so I went in and changed all of the '00"'s for "0"'s.

the 21:00 cronjob today worked.

I wonder if that was it, or maybe I just forgot to press the "Apply" button last time  ::)

Hi,

Can't get it to work. I'm using a vlan tag (added interface) to pluck the internet traffic out of my fiber connection. Are there additional steps I must take? Would you consider a complete step by step write-up (verbose) please?

Kind regard.

Hi,

Just logged in for the first time in a while and saw your message.

Honestly, It has been about 7 months since I did this now, and my memory is a little hazy.

I'll see if I can re-assemble what I did from my post, and put something together, but not tonight.

Bridges work.

They may not perform quite as well as a switch, which has a lot of hardware acceleration specifically for the switching task, but from a functional perspective they are not a problem.

You may just see:

1.) Slightly higher latency than you would with a hardware switch
2.) Slightly lower throughput than you would with a hardware switch
3.) Slightly higher CPU use on the machine running OPNSense

If you can live with the above downsides, there really is not a problem.

Also worth noting, some of this is historical (particularly #2) as with modern hardware a bridge can probably saturate gigabit just fine, though admittedly I haven't tested in a long time.

To be fair, it is still not best practice, and you will still find a lot of people online telling you not to, but a lot of the concerns stem from back in the day with weaker hardware.   If you have a modern system running OPNSense chances are you won't even notice, but YMMV.

I'd test it with your hardware, see what performance you get, see how much CPU load you get, compare your latencies (ping) across the bridge and then decide if this is something you can live with, or if it warrants spending the extra money on a switch.

That said, as long as you don't need a managed switch with VLANs and other managed capabilities, you can get a very basic switch dirt cheap switch, and they don't use a notable amount of power either.

I still prefer actual switches for everything myself, but that is mostly from a historical perspective and a possibly misguided sense of it being "right".

The only place I use bridges is on my KVM/LXC host as I need the individual guests to have their own IP's.   

(My NIC's technically support SR-IOV which could allow me to achieve the same goals but with hardware acceleration rather than bridges  and gain a little performance and reduce a little CPU overhead, but I haven't gotten around to setting it up.  I've been thinking about doing it for going on a decade now, but I never quite seem to get around to it...   ;D )

[EDIT:]

Hi everyone,

So, I have a sneaky teenager in the house who would - if he could - play video games literally all night instead of sleeping.

Initially I solved this by putting his access on a separate interface using VLANs, and controlled access by just firewalling off all access to the WAN on a schedule using the firewall->Settings->schedule and assigning that schedule to a firewall rule in his VLAN.

However, after a couple of months he figured out that while this did not allow any new connections, it didn't terminate existing connections.    At first he would just start his game before the scheduled end time of internet access, but then he figured out he could use a free proxy/VPN and as long as he started it before the scheduled downtime went into effect, he had complete internet access.

So, to combat this, I decided to instead bring down the entire interface for his VLAN on a schedule using cron.

I created a couple of mini "scripts" in /usr/local/bin/ as follows:

wup.sh

Code Select Expand


#!/bin/sh
ifconfig ix0_vlan5 up
exit 0
root@shut

wdown.sh

Code Select Expand


#!/bin/sh
ifconfig ix0_vlan5 down
exit 0

Testing these scripts directly from the command line via SSH, they work as intended bringing the interface down and up again.

Then I decided to add these scripts to cron jobs using the example in this ancient thread.

I created the following conf file in /usr/local/opnsense/service/conf/actions.d/actions_vlan5updown.conf

Code Select Expand


[wup]
command:/usr/local/bin/wup.sh
parameters:
type:script
message:Bring VLAN5 up
description: VLAN5 up

[wdown]
command:/usr/local/bin/wdown.sh
parameters:
type:script
message:Bring VLAN5 down
description: VLAN5 down


Then I restarted the config.d service:

Code Select Expand

# service configd restart

And proceeded to test the actions:

Code Select Expand


# configctl vlan5updown wup

and

Code Select Expand


# configctl vlan5updown wdown


Both worked exactly as expected, bringing down the VLAN5 interface and bringing it back up again.

So, I know my "scripts" work, and I know my .conf file works pointing at those wup.sh and wdown.sh scripts successfully.

The next part - however - does not work as well.   Namely, the actual cron part.



This schedule was supposed to bring VLAN5 up at 9 in the morning this morning (Sunday), but it did nothing.  I had to go in manually via SSH and execute /usr/local/bin/wup.sh to get it going.

Can anyone tell me what I might have done wrong?    I have checked and double checked everything and I can't seem to figure out why it isn't working.

I am no stranger to using cron (though I usually do it under Linux using crontab, not with a web interface).

Is there something here that I've missed?

I'd appreciate any input.

Thanks,
Matt

EDIT:

My apologies,

looks like I can't seem to properly embed the image to show the whole thing.   Clicking on the filename in the attachments seems to work though.

EDIT2:

If I go to the command line and use crontab like I am used to, I find that it works and already has a few jobs in it:

Code Select Expand


# crontab -l
# or /usr/local/etc/cron.d and follow the same format as
# /etc/crontab, see the crontab(5) manual page.
SHELL=/bin/sh
PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
REQUESTS_CA_BUNDLE=/etc/ssl/cert.pem
#minute hour mday month wday command
1 * * * * (/usr/local/sbin/configctl -d syslog archive) > /dev/null
2 * * * * (/usr/local/sbin/expiretable -v -t 3600 sshlockout) > /dev/null
3 * * * * (/usr/local/sbin/expiretable -v -t 3600 virusprot) > /dev/null
4 * * * * (/usr/local/etc/rc.expireaccounts) > /dev/null
*/4 * * * * (/usr/local/sbin/ping_hosts.sh) > /dev/null
0 22 * * * (/usr/local/sbin/configctl -d firmware changelog cron) > /dev/null
* * * * * (test -e /var/db/rrd/updaterrd.sh && /usr/local/bin/flock -n -E 1 /var/db/rrd/updaterrd.sh /var/db/rrd/updaterrd.sh) > /dev/null
1,16,31,46 * * * * (/usr/bin/logger "reload filter for configured schedules" ; /usr/local/etc/rc.filter_configure) > /dev/null
1 3 1 * * (/usr/local/sbin/configctl -d filter schedule bogons) > /dev/null
* * * * * (/usr/local/bin/flock -n -E 0 -o /tmp/filter_update_tables.lock /usr/local/opnsense/scripts/filter/update_tables.py) > /dev/null

Notably, my cron jobs from the web interface are absent.

Maybe it executes them as a different user?   Maybe this is a permissions issue, and I need to change the owner/permissions of my scripts?    They are currently owned by root:wheel like everything else in /usr/local/bin, and I set permissions to 775:

Code Select Expand


# ls -l /usr/local/bin/w*
-rwxrwxr-x  1 root wheel 41 Sep  7 18:58 /usr/local/bin/wdown.sh
-rwxrwxr-x  1 root wheel 39 Sep  7 19:00 /usr/local/bin/wup.sh


Are there different users/groups I should be using for these?

Appreciate any input.

--Matt

So, what functionality are you looking to replicate?

I used Unifi AP's only for years.   I started back before they sold switches, routers and the like, when the Unifi controller was just there to control AP's.

Initially I used pfSense for my router/firewall, but since then I have switched to OPNSense for the router/firewall.

For switches I use Mikrotik in SwOS mode.   Mikrotiks RouterOS is very powerful, but it can be an absolute nightmare to get working, and most of their hardware has nowhere near the capability to actually do some of the routing RouterOS can pull off, so I just stick with SwOS for that.  All I really need is Layer 2 switching (with some managed features like VLAN's) anyway.

Last year I decided to move away from Unifi.  It was a long time coming, but lets just say that when I started using them they were the little upstart, and overtime they became really big and tried to become an ecosystem and take over everything, and I hated that.

I hated logging into my Unifi controller and seeing red X:es for missing Unifi switches and routers that I didn't have, and didn't want to have.   When they decided to discontinue the local server for Unifi Video and force everyone to the cloud at the same time as news broke about their massive cloud breach in 2021 was when I had had enough, and decided I would be moving away from Unifi. 

But it still took me a couple of years to actually do it, as I had other projects that were ahead of it in the queue.

Last year I pulled the trigger on a couple of Mikrotik cAP ax accesspoints.   I figured since i liked their switches, the access points would be great too.    I tried.  I really did.  I'm no stranger to configuring networks, but I just couldn't get the Mikrotik AP's to work properly with VLAN's. In part this was complicated as they were RouterOS only.  I gave myself a deadline.   I would try really hard to make it work, but if I didn't have it working by the time my Amazon return policy was close to being up, they would be going back.

Now I am no stranger to configuring VLAN's, and it has always just worked for me with other vendors, and I put a lot of time and torubleshooting effort into it, but in the end I just could never get it to work.   I felt like RouterOS was poorly documented, had some bugs, and in general felt like using a beta software.  Whereas on other products (including Mikrotik's own SwOS) you make sure you have all the right VLAN's assigned to the uplink port both on the switch and on the AP, and then just select which SSID uses which VLAN, and it just works.   But with the cAP ax under RouterOS it was a nightmare.  There were certain things that needed to be configured on multiple screens, and you had to enable VLAN aware in some places but not in other places, and they had to be in the right order etc. etc. etc. or it just wouldn't work.

I was able to get my primary VLAN to work, but the rest never saw traffic.  So when the Amazon return deadline approached after a month, they went back.

Instead I bought two Ruckus R650 access points and flashed them with the Ruckus Unleashed firmware.    This means that no local server is necessary.  The management interface runs on the Ruckus Unleashed firmware itself.

These access points were a little bit pricy, but I am very very impressed with their performance and range.    I got two, because with Unifi I needed two to cover the entire house, but with the Ruckus R650 units, I probably could have used just one.

They are a little bit pricy, but they are well worth it.



As for the access points

So, I have done some more tinkering and reading.

One guide suggested that checking the box for obtaining IPV6 over IPV4 (I forget the exact name of the ckeckbox right now, but it is the one in the interface settings)

I tried that, but my WAN interface still did not pull an IPV6 address other than the fe80: hardware/linklocal/whatever (I don't know the ipv6 terminology yet) address.

Should I be expecting the WAN interface to pull an IPV66 address if it is working?  Or, since it is no longer NAT with IPV6 maybe it doesn't even need one?

In the IPV4 side, Verizon is notorious for being tricky with their DHCP, often requiring a manual release and renew of the lease by calling their support if you are doing something like changing a router.

Maybe I need to request a manual release renew to be able to start using IPV6?

Is there any way I can even confirm whether or not it is enabled upstream?   Can I somehow query Verizon's DHCP and get an output as to what is available, so I know whether or not I am wasting my time?

I'd appreciate any input anyone may have.  Despite IPV6 being around for a long time now, I still feel like I am fumbling in the dark here.

UNDER INTERFACES > WAN:
IPv4 Configuration Type: keep using DHCP
IPv6 Configuration Type: DHCPv6
Request only an IPv6 Prefix: ✓ (enabled)
Prefix delegation size: 56
Send IPv6 prefix hint: ✓ (enabled)

Well, I'm not ready to allow any of my local networks to use IPV6 yet (as mentioned above, I want to learn how to secure it first) I decided to see if I could at least get DHCPv6 to communicate with upstream by making the above changes.

My WAN interface now has an IPV6 address but sadly it starts with fe80: which I understands means it is a non-routable local hardware address only.

I guess this means it failed to grab an IPV6 address range from upstream?

Maybe it is not yet enabled here?

Appreciate any thoughts.

[Anyway, moving along to my issue:]

Ugh,

I have been dreading having to learn how to configure IPV6 for over a decade now.   

I have my complicated network set up the way I like it, with many local VLAN's each on a separate IPV4 subnet inside the 10.0.0.0/8 private range.

Ever since I first learned of IPV6 in the 90's I thought it was a terrible implementation.   I don't even use DNS or hostnames for my local hosts because I have all of their IPV4 addresses memorized and haven't had to.   I know IP address exhaustion is a real problem, and has been for a very long time, but creating these insanely long and human non-readable IP addresses has been a huge pet peeve of mine when we could have just added another octet to IPV4 and called it a day.   We don't need a unique IP address for every atom on the planet.

I'm also not a fan of IPV6 creating hardware addresses based on mac address and using it to communicate on the local network.  I feel like it removes a level of control I previously had.

And if I'm honest, while I know lots of people hate it, I really like NAT.   I like having a single exit point for everything from my network and keeping control over everything that is internal.   I'm not a fan of "everything on the internet being 1:1" as it breaks my concept of my own little private network that just happens to be connected to the internet, and turns it into an "everything is the internet" model.    It's just like nails on chalkboard to me.  I've never though of my home LAN as "part of the internet".   I've thought of it as my home network.  And occasionally things transit my gateway to the public internet. (but most traffic is local) and I like that NAT:ed single point of entry (as long as I control the gateway)

I've also been using the fact that Android doesn't support DHCPv6 as an excuse to keep my entire network all IPV4 until such time as everything on my local network supports DHCPv6, as the concept of SLAAC really makes me uncomfortable and makes me feel like I am no longer in control

Thus I've just kept IPV6 disabled on the WAN and LAN interfaces of the OPNSense box just because I haven't had the time at any point in 10 years to sit down and re-learn how to properly firewall off ipv6 so I don't have unintentional leaks. I also disable IPV6 on all local machines and devices on my network and have every network firewall configured to block all IPV6 traffic. 

My theory here is that if I just block and disable IPV6 everywhere, my network remains firewalled the way I want it, until I have the time to relearn everything.  I know this means I don't necessarily have IPV6 tunneling blocked, but I try.  But to be true, I can't stay on top of blocking all IPV4 tunneling providers either.

I figured I'd get around to learning it at some point, but I just haven't had the time for over a decade now, and I have no idea when I will.  I'm in my mid 40's now.  I may very well go to my grave without ever enabling IPV6  :p

Everything about this is about control to me.  I am the dictator of my LAN.   I don't want a single bit moving on my network or through my WAN without me intentionally telling it to.  I hate all forms of automatic scanning/discovery  of networks.  With the exception of DHCP I don't like my clients or servers to even behave as if the LAN exists until I tell it which IP address to connect to, and click OK (or hit enter)

So in other words, I use local network like its the 90's, and that's the way I like it.   I consider it to be distinct and separate from the internet, and that's the way I like it.

It may very well be that I just don't understand IPV6 well enough yet, but I feel like with IPV6 I am giving up a lot of control I previously had, and it makes me very uncomfortable.

Anyway, moving along to my issue:

I recently had a cloud backup service offer me a significant discount if I am able to connect only via ipv6, so they don't have to burn one of their precious remaining ipv4 addresses, so this ipv6 thing is finally getting real  :'(

Any way I can easily allow only a single ipv6 address through to the source system of the cloud backups and keep it blocked for anything and everything everything else until I have time to properly read up on how to configure IPV6 correctly to get the level of control I want? 

I'm sure you'll take an image backup beforehand anyway.

Absolutely.

I always image drives before major changes.

Hey everyone,

So I have read this page in the docs, and have been considered trying to enable hardware offloading to see if it helps with some of my high CPU usage.

The goal here is to reduce some of the high CPU usage I have seen during high bandwidth (~gigabit) use over WireGuard.   I am not sure how much of that load is encryption vs. packet handling / interrupts and how much potential there is to help things by enabling NIC hardware offload.



Any hints, tips and tricks before I do?

The docs seem to warn of misbehaving hardware.  Is there a list anywhere of which hardware people have used with hardware offload successfully?

I use an Intel i210 for WAN, and an Intel x520 for LAN.    I understand older enterprise grade Intel NIC's are some of the most well behaved in Unix and Linux, but should these not be a good fit I have a drawer full of disused NIC's I could experiment with.

Just want to make sure there isn't any helpful info I am missing before I start spending a ton of time experimenting.

It looks like the biggest candidates for useful improvement are hardware CRC and Hardware TSO.

Hardware LRO maybe be helpful to CPU load, but the impacts to latency due to buffer bloat may be undesirable, so I wasnt planning on trying that one.

I do use VLAN's so Hardware VLAN filtering may help (if it works and is reliable)

ARP handling is more ambiguous to me.   Not sure what there is to gain here and what the implications are.

Anyway, I'd appreciate any tips, tricks, suggestions, references for further reading, or any other input anyone might provide.

Thanks,
Matt

Depends. There is no universal documented approach that I know of.
Restore config brings everything that has been configured via the GUI including plugins.
Anything else added or changed via console won't be recorded there so needs to be re-done.

Thank you.

In my case I don't think I have set anything from the console.  At least not that I can remember, so I should be good with this.

Also remember the interfaces will be different so you need to account for that. Many do a search and replace before importing the config file, i.e. save config, search and replace the interface names in it, import.

That is a good point, but again, I think I will be fine here.

I passed through the PCIe NIC's to the VM, so they should show up as the same hardware.

I have one i210 and one x520 NIC in the machine.  The i210 is the WAN interface, and the x520 is the LAN interface.

Of course, these are both dual port, and I only passed through one of the ports of each, but I think I passed through the first one (igb0 and ix0) so I think they will wind up being correct without any changes.   I will just have two extra ports (igb1 and ix1) that will be unused.

Of course, if the order changes, I'll have to swap the cables from one port to the other, but that won't be the end of the world.

You can say that again.  It is generally not recommended if you need switching between those ports.  I tend to think of traffic on multiple ports on OPNSense (and other PC based hardware) as best used for routing only.   If you need any switching you are better off using VLANS on a single port (or multiple link aggregated ports if you need the redundancy or capacity) and a switch in between.

Fundamentally correct, but I also understand users with a small home network who (rightfully, IMHO) expect the same as from any consumer DSL router - 1 uplink, 3 or 4 LAN ports.

And OPNsense can do that, the FreeBSD bridge has been greatly improved and can easily switch 1 Gbit/s and more. Foundation sponsored project.

If you have one fixed PC at your desk, one printer, one WiFi access point - why an extra switch?

The fun starts - and lots of endless forum threads and disappointments and debugging ... when people want to get fancy with VLANs and the whole kitchen sink and still don't want to simply buy a switch.

I have a Mikrotik CRS326-24G-2S+IN - 24 Gbit ports, 2 SFP+ at < 200€/$. No-brainer.

Kind regards,
Patrick

Yeah, Mikrotik switches offer a fantastic bargain for managed Layer2 switching.   I have several of them myself.



Their SwOS is fantastic, but their RouterOS is IMHO a bit much.   It requires you to do everyhting manually, in some pretty terrible ways, and documentation is bad, and often is a nightmare to get working.   It feels a bit like using a beta product to me.

And while they support layer3, apart from a few dedicated router product, the CPU's in their devices lack the horsepower to really do any kind of real routing.

So, Mikrotik is awesome, as long as you use SwOS and stick to layer2 :p

Yes, this should ideally be moved, but we can start here until it is.

If you have traffic between the two sites, and it just isn't exiting to the outside network on the side you want, I wonder if the only part that is missing is changing the gateway address.

Lets assume site1 is the side you want traffic to exit to the outside world from, and site 2 is the remote site.

You can test if this is the case by using a machine at site 2, going into manual IP settings and set the gateway to the ip address of the gateway (usually the same as the router) the machines at site 1 use.   I think this ought to then direct the internet traffic over the VPN to site 1.

If that works, you can then return the settings to DHCP, and then change the DHCP server settings at site 2 to use the site 1 as the default gateway.  You might also want to change the address for the default DNS server to that of site 1 as well, or you may find that your DNS requests exit in the wrong place.

If this doesn't work, then you may need someone who is better at this stuff than I am to help.

For them to do that, they will likely need more information though.   A network diagram and a document list of settings currently in use will probably be necessary.

OPNsense can act as a switch, but a slow one coz instead of dedicated chip it will use CPU. It is not recomended, but it is possible.

You can say that again.  It is generally not recommended if you need switching between those ports.  I tend to think of traffic on multiple ports on OPNSense (and other PC based hardware) as best used for routing only.   If you need any switching you are better off using VLANS on a single port (or multiple link aggregated ports if you need the redundancy or capacity) and a switch in between.