Messages

1

24.7 Production Series / Re: Set DNS server to use with Kea DHCP service

« on: October 12, 2024, 01:09:26 am »

And... it's official.  I am stupid.  That worked!  Somehow it never crossed my mind that the checkbox would work that way.  Thank you!

2

24.7 Production Series / Set DNS server to use with Kea DHCP service

« on: October 11, 2024, 09:54:22 pm »

I switch to Kea from ISC for DHCP services.  In ISC it has a very clear field where you can set the DNS server to use.  In my case, I want to use a pihole for one of my networks.  How can I set this?  I looked in the GUI and searched around and either the function doesn't exist in the WebGUI, or I'm stupid and/or blind.

Thanks!

3

Virtual private networks / Re: No zerotier interfaces

« on: September 04, 2023, 09:34:22 am »

Okay, so I fixed it. I believe the system was all messed up because I had told the interfaces to "prevent interface removal".  Here's what I did:

1.  Uninstalled Zerotier plugin.
2.  Went to the interfaces and unchecked the "prevent interface removal" for each of the zerotier interfaces I had.
3.  Removed the interfaces from the system.
4.  Reinstalled Zerotier plugin.
5.  Reconfigured everything as necessary in the WebGUI.  This included adding the interfaces again, etc.
6.  Went into zerotier's website and added the "new" device to the appropriate network and removed the "old" device from the appropriate network.

I can't remember why I checked 'prevent interface removal' in the first place.  I think there was some odd stuff with zerotier making a new interface on every reboot or every update or something, and instead of having to add my "new" device to the network every time, I simply prevent its removal.

4

Virtual private networks / No zerotier interfaces

« on: September 01, 2023, 10:17:49 pm »

Hello everyone.  So I decided to replace my boot device in opnsense with a mirrored set with ZFS.  I was on the last version of 23.1, and since 23.7 had just come out, I made the choice to go ahead and install 23.7 and then import my config file.  I figured it would be smooth sailing.

Not so much.

Zerotier has been broken since that day.  I haven't had a chance to troubleshoot since it was a low priority, but nwo I need to get it working again.

Looking in the WebGUI of opnsense under the Zerotier section, everything looks normal.

However the zerotier interfaces are not being created.  ifconfig doesn't show them.  I then went to Interfaces -> Assignments and opt7 and opt8 (my two zerotier interfaces) show as "missing" (which I already knew) but I can't get them to be recreated.

I tried uninstalling and reinstall the os-zerotier plugin along with reboots.  I've even installed several of the opnsense updates (along with appropriate reboots) and the interfaces are still not being created.

I also tried creating a new network to see if it would create a new interface.  Still no.  I'm at a loss to know how to either recreate the interfaces that I had or have it create new ones.

I will say that if I log into the Zerotier website to look at device last login, my opnsense box is over 30 days since last login, corresponding with when I did the reinstall.

Any ideas?

Thanks!

5

Hardware and Performance / Re: 10Gbit performance problems with Chelsio T520-SO-CR

« on: June 22, 2023, 04:28:05 am »

@JamesFrisch

Can you provide specs on your opnsense system?  Is it virtualized?

6

23.1 Legacy Series / Re: 23.1.4:1 Update to 23.1.5 not working --- fixed

« on: April 05, 2023, 12:21:01 pm »

After posting this I noticed the last entry for the OP wasn't a signature but was what he did to fix it.  I tried both US repos, and no change.  I then changed to the DE repo the OP used, and its updating right now.

Sounds like some kind of problem with the repos or something.

7

23.1 Legacy Series / Re: 23.1.4:1 Update to 23.1.5 not working --- fixed

« on: April 05, 2023, 12:16:30 pm »

I'm having similar issues at both friends' Opnsense systems I manage.  I manage to get mine to update, but it took like an hour.

One friend updated and rebooted as though everything was normal; typical 10-15 minutes for installation and reboot.

The other friend I cannot update despite trying multiple times from the WebGUI and the CLI.  Here's the CLI output from tonight.

Code: [Select]

  0) Logout                              7) Ping host
  1) Assign interfaces                   8) Shell
  2) Set interface IP address            9) pfTop
  3) Reset the root password            10) Firewall log
  4) Reset to factory defaults          11) Reload all services
  5) Power off system                   12) Update from console
  6) Reboot system                      13) Restore a backup

Enter an option: 12

Fetching change log information, please wait... done

This will automatically fetch all available updates and apply them.

load: 0.28  cmd: sh 70400 [wait] 358.96r 0.00u 0.00s 0% 2980k
mi_switch+0xc2 sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1f2 kern_wait6+0x527 sys_wait4+0x7d amd64_syscall+0x10c fast_syscall_common+0xf8
load: 0.26  cmd: sh 70400 [wait] 361.24r 0.00u 0.00s 0% 2980k
mi_switch+0xc2 sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1f2 kern_wait6+0x527 sys_wait4+0x7d amd64_syscall+0x10c fast_syscall_common+0xf8
load: 0.36  cmd: sh 70400 [wait] 1318.61r 0.00u 0.00s 0% 2980k
mi_switch+0xc2 sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1f2 kern_wait6+0x527 sys_wait4+0x7d amd64_syscall+0x10c fast_syscall_common+0xf8
This update requires a reboot.

Proceed with this action? [y/N]: y


Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
Updating OPNsense repository catalogue...
pkg-static: https://pkg.opnsense.org/FreeBSD:13:amd64/23.1/latest/packagesite.txz: No route to host
Unable to update repository OPNsense
Error updating repositories!
Starting web GUI...done.
Generating RRD graphs...done.

Note that the part where you see CTRL+T output ultimately took over an hour before I got the "proceed with this action" query.

Clearly, *something* is going on, but I have no idea what.

At this particular location, ipv6 is disabled on the WAN side as there is no ipv6 support. I just tried to do the upgrade from the CLI again, but got a new message:

Code: [Select]

Enter an option: 12

Fetching change log information, please wait... fetch: transfer timed out

This will automatically fetch all available updates and apply them.

This has been frozen with no additional output for probably 30 minutes or so.  I'm hoping this is just a problem with a cdn or something and its an easy fix.

8

23.1 Legacy Series / Re: Update didn't reboot.. did it finish?

« on: February 07, 2023, 10:21:42 am »

So about 2 minutes after I posted this the system rebooted.  It then took over an hour to do the reboots and install the OS.  Upgrade on first reboot was slow.  Bootups were slow.  Package installs and updates were slow.  I waited it out (very patiently) and it did eventually come up and "just work".

Digging deeper, once opnsense was booted back up and fully functional, my SSD write speeds were around 1MB/sec maximum (nope, that's not a typo).  After some more thinking and analyzing, ZFS property autotrim was set to off.  As I was never running the "zpool trim <poolname>" command from the command line, I assume it needed a trim.  After a zpool trim and waiting a few minutes, I could write at over 100MB/sec.

I've since changed the autotrim property to on with the command "zpool autotrim=on zroot".  Probably would be a good idea for the default to have autotrim set to on.  As a ZFS guru, I will admit that there are situations where autotrim=off may have it's place for some situations and some workloads, I suspect the greater masses would benefit from autotrim being set to on when a zpool is created.

Edit: For anyone that is curious, the total upgrade time from when I first clicked "update" in the WebGUI to the system booted up and functional was almost 2 hours.  LOL.

9

23.1 Legacy Series / Update didn't reboot.. did it finish?

« on: February 07, 2023, 08:53:50 am »

So I just tried to update from 22.7.11_1 to 23.1(.x???) via the WebGUI.  From the WebGUI everything looked pretty normal, except the reboot never happened.

From the WebGUI:

***GOT REQUEST TO UPGRADE***
Currently running OPNsense 22.7.11_1 (amd64/OpenSSL) at Tue Feb  7 00:17:51 MST 2023
Fetching packages-23.1-OpenSSL-amd64.tar: ......................................................... done
Fetching base-23.1-amd64.txz: ............. done
Fetching kernel-23.1-amd64.txz: ...... done
!!!!!!!!!!!! ATTENTION !!!!!!!!!!!!!!!
! A critical upgrade is in progress. !
! Please do not turn off the system. !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Extracting packages-23.1-OpenSSL-amd64.tar... done
Extracting base-23.1-amd64.txz... done
Installing kernel-23.1-amd64.txz... done
Please reboot.
***REBOOT***
***DONE***

Okay, so it's been more than 20 minutes and the system hasn't rebooted.  Uptime is 12+ days.

IIRC when I upgraded from 22.1 to 22.7 I did a reboot when i thought something was wrong with the upgrade, and the system was rendered unbootable.  Now I'm wondering what I should do next, and if I should gather some logs or anything before going further.  At the time I blamed myself for being too impatient, but this time I'm much more patient and trying to avoid having to do a reinstall and upload the config file again.

Note that during the entire 22.7 lifetime, if an update required a reboot, it always went off without a hitch.

So what should I check or do as next steps?  I feel like the next step is to reboot, but if I'm going to reboot and have an unusable system again, I'd really like to know what is going on.  Any log files that might be useful for troubleshooting?

10

22.7 Legacy Series / Re: Revoked certificate after upgrading to 22.7.4

« on: October 01, 2022, 12:23:19 am »

@franco



Enter an option: 12

Fetching change log information, please wait... The file was signed with revoked certificate pkg.opnsense.org.20210903

This will automatically fetch all available updates and apply them.

Proceed with this action? [y/N]: ^C
*** opnsense4.site.business: OPNsense 22.7.4 (amd64/OpenSSL) ***


root@opnsense4:~ # opnsense-update -M
https://pkg.opnsense.org/FreeBSD:13:amd64/22.7


Just got this error myself on my opnsense install.  I just upgraded from 22.1 to 22.7.4

11

Virtual private networks / Re: Zerotier and odd performance issues...

« on: September 18, 2022, 09:33:37 am »

I won't disagree with you as I've read plenty of people having that experience.  However, my experience has been pretty good, except for this issue.  I did a lot of homework and tried to incorporate as many "lessons learned" from other people as I could before we started using it at all.  Let alone deciding to rely on it for "production".

But I have no direction to go with this particular issue.  I did try emailing the package maintainer per the plugin details, but that was more than a month ago with no response at all.

I'm a little disappointed because I have no idea where to go with this issue.  What if I had found some serious significant security issue?  I honestly have no idea where I'd go as I've already used the first avenue to correct the issue.

At this point, since we're paying for the Zerotier service, I'm going to try reaching out to them next week to see what they think of this.

Overall, 5MB/sec between sites is not going to work for us, so we'll ultimately have to find another solution if we cannot fix this issue.

Edit:  When I originally started using Zerotier, I was planning to put together a guide (and a friend who's a Youtuber wanted to do a video guide) on this.  I thought that could help solve a lot of people's problems with Zerotier by having some kind of well-documented guide or video.  But with the significance of the performance issue I don't know that I could really recommend this to anyone.

12

Virtual private networks / Re: Zerotier and odd performance issues...

« on: August 31, 2022, 11:22:27 pm »

Wow.  I'm surprised nobody had anything to try or ideas.  Did I break the hive mind?  LOL!

I'm gonna do more testing later this week on this, so I may have more information as time progresses.

13

Virtual private networks / Re: VPN Licenses

« on: August 27, 2022, 02:21:15 am »

Well, your question is a bit confusing.

If you are going to be a VPN server yourself, you don't need any kind of license or anything.  All of the software is free and you just set it up.

If you're wanting to connect to a paid VPN service like NordVPN, ExpressVPN, etc. then you have to have an account with them.  As for whether you need 1 or more depends on the service and what your account at that VPN provider offers.  Some paid accounts are 1 device, some might be 5 devices, and then they may have some premium account for 20 devices.  It just depends.

Hope this helps!

14

Virtual private networks / Zerotier and odd performance issues...

« on: August 26, 2022, 09:33:44 pm »

I've always been relatively disappointed in the zerotier VPN performance despite us using it for almost a year.  I've always thought ZT was "just slow" until I did some googling and more testing.  A few websites compared ZT to be nearly identical to IPSEC and others.  When I saw some people doing 400Mb/sec, I started questioning if my performance is expected or if something is actually wrong on my part.  So I started digging deeper...

I have 2 different ZT networks and different sites.  All are using Opnsense and all are on the latest versions.  I've chosen 2 sites close to each other by latency (and geographically) that I can do testing on that are not using HA (High Availability adds another layer of complexity, so I'm trying to forgo dealing with that until I can get non-HA sites working better).

Let's pretend I have 2 sites.  Site one has router-A and desktop-A (it's business internet account, but at a home) with internet speeds of 500Mb down and 35Mb up.  Site two has router-B and Server-B with internet speeds of 1Gb down and 1Gb up (this is also a home, but has fiber to the home).

My sites have a config file that is like this:

Code: [Select]

{
   "physical": {
      "192.168.0.0/16": { "blacklist": true },
      "10.0.0.0/16": { "blacklist": true }
   },
   "settings": {
      "primaryPort": 9993,
      "portMappingEnabled": false,
      "allowSecondaryPort": false,
      "allowTcpFallbackRelay": false
   }
}
I then have the appropriate firewall rules to allow traffic from 9993.

If I do iperf3 from router-A to router-B, I get 36Mbit/sec.  That saturates the uplink at site A so I'm okay with that.

If I do iperf3 from router-B to router-A, I get 312Mb/sec.  I'm okay with that speed since it is 14ms between the two routers.  More would be great, but I'm gonna take this issue as a bunch of baby steps instead of being upset that it doesn't saturate 1Gb.  (I'm expecting to need some very large buffers if I really want to do 1Gb with 14ms of latency.  I'm no expert at adjusting buffers on opnsense, and I haven't found a link that would teach me enough to figure it out on my own, so I'm gonna work on the issues I do understand first.)

For "the lolz" I did do iperf3 from router-B to router-A using the external IPs to compare against the VPN.  I got 443Mb/sec.

To summarize thus far:
1.  I can saturate the upstream at site A.
2,  I get speeds of 443Mb without the VPN, and 312Mb with the VPN going from router-B to Router-A over the VPN.

If I then connect to an SMB share on Server-B (TrueNAS) and start downloading a file to Desktop-A (Windows 10) and I get 9-12MB/sec fluctuating (I'll just call it a 100Mb link for simplicity).  That's about 1/3 of what I got with iperf3 from router-B to router-A.  So I did a bunch of things here because I wanted to see exactly what was going wrong:

1.  I did a packet capture on Desktop-A using wireshark.  Things go smoothly for the smb transfer, but every 1 second or so I get inundated with a crapload of TCP DUP ACKs.  I'll get 100 to 120 DUP ACKs in less than 2ms.  At the same time I'll get a dozen or so TCP out of orders and a fast retransmit.  Then another second of what looks like normal SMB traffic, followed by another huge group of dup acks, out of order packets, and fast retransmits.
2.  I did an iperf3 test from Server-B to Desktop-A.  I get 9-12MB/sec with 5-50 retransmits almost every second per iperf3.  SMB and iperf3's performance seems to be inline with each other, so that means I probably don't have an SMB problem.  I also did check with Wireshark and I have the same dup acks and such as I mentioned in #1 above.
3.  I did iperf3 from Desktop-A to Router-A and vice versa and I got 931Mb/sec (basically a 1Gb connection).
4.  I did an iperf3 from Server-B to Router-B and vice versa and I got 930Mb/sec (basically a 1Gb connection).
5.  So I did iperf3 from Router-B to Desktop-A and I get about 100Mb/sec with 5-20 retransmits every second from iperf3 output.
6.  I did iperf3 from Server-B to Router-A and I got about 100Mb/sec with 5-20 retransmits every second from iperf3 output.

So it seems if I go from router to router directly, all is good.  I get good speeds and no retransmits.  But as soon as I want to go from a desktop or server on one site to anything on the other (including the router on the other side), performance and reliability take a nasty nosedive.

To put it another way, I can go from router to router just fine, but if I want to actually use the VPN in any useful way using other servers and desktops, it's unreliable and slow.

Any ideas where to even start to investigate this issue? 

I did try emailing the zerotier plugin maintainer to start a conversation more than a week ago, but he didn't respond.  I'd really rather not bother him with additional emails unless I can really prove this is a zerotier issue.

As this is affecting business functionality, the company is open to the idea of paying for someone to troubleshoot and identify the issue.  But I don't know where to start.  Is it an Opnsense problem and I need an Opnsense expert?  Is it a Zerotier problem?  If so, is it the plugin itself or is it the zerotier code.  Is it just a configuration problem on my part.  I know that Zerotier documentation at https://docs.zerotier.com/devices/opnsense makes it pretty clear that Zerotier, Inc doesn't maintain the opnsense implementation.  From some of the "official" posts in the Zerotier forums I get the impression Zerotier, Inc. has the attitude of "we don't do opnsense, so if it works, great, and if it doesn't, don't talk to us about it either".

Thanks to whoever read this all the way to the end.  I realize this is a lot to swallow.

15

High availability / Re: When should a "failover" automatically occur?

« on: July 21, 2022, 12:01:31 am »

Just wanted to provide an update.  For reasons outside my control, the #2 box had to be power cycled.  #1 picked up and took over all workloads.  When #2 came up, it sat in a BACKUP state.

I guess this issue will be left unsolved as I can't investigate it more as the issue is gone.

Thanks to everyone that helped.