Topics

[In short:]

I don't know if this is an OPNSense bug or not - so I thought I would post here and get comments before reporting as a bug (if appropriate) on Github.


In short: NTP is not working for devices on my Wifi Mesh network that are assigned dynamic addresses, however it does work if I assign them a static address.

Question: is this an OPNSense bug or just the way things should work?


Additional background and detail:

I am running OPNSense (current community version) with various interfaces, including:
   LAN
   LAN_IOT
   Master_Clock

I have two separate physical Wi-Fi networks.  One is attached to LAN the other to LAN_IOT.
Both the Wi-Fi LAN and LAN_IOT are running in Access Point mode.
The Wi-Fi LAN device is a TP-Link Deco mesh system (with three TP-Link Deco connected in the Mesh). 
The Wi-Fi LAN_IOT is a legacy TP-Link Access Point with also connects to a legacy TP-Line Extender.

OPNSense is running the Network Time Service.

The Network Time Service runs on the Master_Clock interface.

The Network Time Service gets its time from an ESP32 - GPS based Stratum 1 Time Server I developed and released in 2023.

The Network Time Service delivers NTP date/time data to all other interfaces.

This setup has been working very well for me since 2023, however almost all my devices connected on all interfaces have been set up, via OPNSense, with static IP addresses.

Recently I have been programming another set of ESP32 projects and noticed that while there could connect to either the LAN Wi-Fi network or the LAN_IOT Wi-Fi network they would not retrieve the NTP date/time data.

I spent several days trying to resolve this; but most of this time was chasing down the rabbit hole that my ESP32 code was bad / bad board configurations / etc (non of which seemed to be the issue in the end).

Also, in debugging this, at one point I stopped the OPNSense Time Service, and with it stopped my other device could get NTP date/time data but my ESP32 projects could not.  So that was not it.

However, late yesterday I found that if in OPNSense I set the ESP32 project device up with a static IP address it was able to get the NTP date/time date just fine. However when it ran as with a dynamic IP address the problem returned.

In conclusion:

In any case, I am reporting it here. 

I don't know if its an issue with OPNsense, TP-Link, both. or simply how things work.

If it is an as of yet unknown/unreported issue with OPNSense, the above offers a work around.

Also, if the comments here so indicate, I will open it up as a bug on the OPNSense Github page.

With thanks.

On my OPNSense system I use Unbound and Caddy.

With the help of Caddy, I access the OPNSense WebGUI from a PC on my LAN interface via https.

I was trying to get something else working today, and disabled Unbound.  Immediately upon doing that I was locked out of the OPNSense WebGUI.

I had to connect a keyboard and monitor to my OPNSense box and do a restore from a backup earlier in the day (when Unbound was enabled) to get WebGUI access back again.

Is this a known problem?

I've been using the OPNSense plugin for Caddy for a little while now, still working out some of the kinks on my system but today I ran into a new one.

I had a device referenced by https://pikvm.example.com which had been working well for at least a week now, maybe two.

However, this morning I could no longer access it via https://pikvm.example.com as I could as recently as yesterday.  Regardless, I could access via http://xxx.xxx.xxx.xxx (its IP v4 address).  Also, I could ping it just fine at its IP v4 address.

After some digging, I realized I did not have the device assigned to a static IP v4 address in OPNSense on the page Services: ISC DHCPv4: [LAN]

I assigned a static IP address to it, and vola it was working again.

Just thought I would share.

In OPNSense, on my LAN interface I have:

    IPv4 Configuration Type set to Static IPv4
    IPv4 Configuration Type set to None

However, from Windows, if I ping one of my devices within the LAN from another device in the LAN using the command as follows:

ping venus.local

I get an IP v6 address.

However, if I ping the same device with

ping -4 vensu.local

I get the expected IP v4 address.

I found this out the hard way, after something broke and it took me a couple hours to find the issue was this behaviour (i.e. venus.local reporting an IPv6 address rather than an IPv4 address).

I'm not sure, but I suspect the IP v6 address is being generated from a switch within the LAN interface. 

Is there anything that can be done in OPNSense to prevent this?

(I suspect no, but thought I would ask all the same).

Yesterday, with help from this forum, I got Caddy to provide ssl access to my OPNSense router - which was for me a great first step.

Today, for the second step, I had hoped to get it working for some local machines - for example one running Home Assistant and another PiKVM.

At first I tried the subdomain route - but couldn't get it to work.  Also, I found it didn't let me use specific ports - so I attempted to follow the advice of the documentation here: https://docs.opnsense.org/manual/how-tos/caddy.html#wildcard-domain-with-subdomains

If in doubt, do not use subdomains. If there should be foo.example.com, bar.example.com and example.com, just create them as three base domains. This way, there is the most flexibility, and the most features are supported.

but did not succeed.

For this this I removed the wildcard entries I had previously used, and replaced them with three domains:

router.example.com
ha.example.com
pikvm.example.com

router.example.com is working fine.

ha.example.com and pikvm.example.com are not.

When I try to browse to either ha.example.com or pikvm.example.com I get a CloudFlare page served up that says Connection timed out - 502.

There are no error, warning or even informational entries in my Caddy log to show what might be going on.

My registry domain dns records are set up as follows:

Name                     Type        TTL          RDATA 
@                           A            14400      xxx.xxx.xxx.xxx
@                           NS          86400      xxxxx.ns.cloudflare.com
@                           NS          86400      xxxx.ns.cloudflare.com
ha.example.com      A            14440      xxx.xxx.xxx.xxx
pikvm.example.com A            14440      xxx.xxx.xxx.xxx
router.example.com A            14440      xxx.xxx.xxx.xxx
www                        CNAME      14440      example.com

My Cloudflare records are as follows:
Type     Name            Content               Proxy status    TTL      
A          ha                 xxx.xxx.xxx.xxx   Proxied          Auto
A          pikvm            xxx.xxx.xxx.xxx   Proxied          Auto
A          example.com  xxx.xxx.xxx.xxx   Proxied          Auto
A          router            xxx.xxx.xxx.xxx   Proxied          Auto
CNAME  www              example.com       Proxied          Auto

For the IP addresses of the two machine, even though I think the right think to do is use the external addresses, I have also tired using my internal IP address (non proxied) such as 192.168.1.123 ; but when I used the internal address they didn't work either.  Also, CloudFlare throws an error if I try to use the internal ones and Proxy them.  My gut feel I should be using external proxied addresses so Caddy can set them to what it needs.

In any case, here too is my Caddy file:

Code Select Expand



# DO NOT EDIT THIS FILE -- OPNsense auto-generated file


# caddy_user=root

# Global Options
{
log {
output net unixgram//var/run/caddy/log.sock {
}
format json {
time_format rfc3339
}
}

dynamic_dns {
provider cloudflare xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
domains {
ha.example.com @
pikvm.example.com @
router.example.com @
}
versions ipv4
}

email me@example.com
grace_period 10s
import /usr/local/etc/caddy/caddy.d/*.global
}
# DO NOT EDIT THIS FILE -- OPNsense auto-generated file


# caddy_user=root

# Global Options
{
log {
output net unixgram//var/run/caddy/log.sock {
}
format json {
time_format rfc3339
}
}

dynamic_dns {
provider cloudflare xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
domains {
ha.example.com @
pikvm.example.com @
router.example.com @
}
versions ipv4
}

email me@example.com
grace_period 10s
import /usr/local/etc/caddy/caddy.d/*.global
}

# Reverse Proxy Configuration


# Reverse Proxy Domain: "bf960db4-a3f6-432c-8be4-be6ed247d7b2"
ha.example.com {
tls {
issuer acme {
dns cloudflare xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
resolvers 1.1.1.1
}
}

@b91182d4-6b27-4ef1-a8a3-8d45e1578a76 {
client_ip 192.168.0.0/16 172.16.0.0/12 10.0.0.0/8
}
handle @b91182d4-6b27-4ef1-a8a3-8d45e1578a76 {
handle {
reverse_proxy 192.168.1.173:8123 {
transport http {
tls_insecure_skip_verify
}
}
}
}
}
# Reverse Proxy Domain: "9a3a66c5-aeff-4401-a697-b34de6525a10"
pikvm.example.com {
tls {
issuer acme {
dns cloudflare xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
resolvers 1.1.1.1
}
}

@b91182d4-6b27-4ef1-a8a3-8d45e1578a76 {
client_ip 192.168.0.0/16 172.16.0.0/12 10.0.0.0/8
}
handle @b91182d4-6b27-4ef1-a8a3-8d45e1578a76 {
handle {
reverse_proxy 192.168.1.166:443 {
transport http {
tls_insecure_skip_verify
}
}
}
}
}
# Reverse Proxy Domain: "17af584e-7fcf-4ca0-b5f9-fbd9712f95e4"
router.example.com {
tls {
issuer acme {
dns cloudflare xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
resolvers 1.1.1.1
}
}

@b91182d4-6b27-4ef1-a8a3-8d45e1578a76 {
client_ip 192.168.0.0/16 172.16.0.0/12 10.0.0.0/8
}
handle @b91182d4-6b27-4ef1-a8a3-8d45e1578a76 {
}
}

import /usr/local/etc/caddy/caddy.d/*.conf





I have also tried both handlers without tls_insecure_skip_verify - but it still didn't work.


Also, and as and aside although I don't think it matters much, when I deleted the wild card entry from before, and when I created and then deleted some other Services: Caddy Web Server: Reverse Proxy - Domains it appears their certificates are still hanging around (as I see them in the Dashboard under the Caddy Certificates widget) rather being cleaned up when their associated domains were deleted.  I suppose it is possible these unused certificates may be getting in the way some how - but I don't know how to otherwise get rid of them.

Any help would be much appreciated.

Using the OPNSense Caddy plug in - I can't seem to the automatic certificates to work.

I've followed and reviewed the instructions at https://docs.opnsense.org/manual/how-tos/caddy.html (several times) and watched the video at https://www.youtube.com/watch?v=6ip8Sx4zcDA many times, however, I can't seem to get automatic certificates to work.

screenshot of what I am getting on the dashboard: https://ibb.co/5TLqKzM
greyed out is my hostname . domain name - but it matches the hostname and domain name found in OPNSense - System - Settings - General.  For example:  router.example.com

Having that said, I do seem to have a valid certificate for the hostname . domain in system trust certificates - due to expire in December.  Not sure how it got there - I've tried many things, and prior to this I had been using the acme client (now uninstalled - so maybe it was created from there).   I am however hesitant to delete it from trust so that I could take a fresh run at all this; Regardless, I tried and it said it was being used by webgui and AcmeClient - validation for OPNSense router {AcmeClient.certificates.certificate.xxxx-xxxxx-xxx...-xxx}


My domain dns records look like this:

Name                        Type          TTL   RDATA 
@                             A           14400   xxx.xxx.xxx.xxx
@                             NS           86400   xxxx.ns.cloudflare.com
@                             NS           86400   xxxx.ns.cloudflare.com
www                         CNAME   14440   example.com
router.example.com   CNAME   14440   example.com
*.example.com          CNAME   14440   example.com

These are the errors I see in the log is:

Code Select Expand

2024-09-18T11:11:08-04:00 Error caddy "error","ts":"2024-09-18T15:11:08Z","logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"router.example.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:dns - DNS problem: NXDOMAIN looking up A for router.example.com - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for router.example.com - check that a DNS record exists for this domain"}
2024-09-18T11:11:08-04:00 Error caddy "error","ts":"2024-09-18T15:11:08Z","logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"router.example.com","problem":{"type":"urn:ietf:params:acme:error:dns","title":"","detail":"DNS problem: NXDOMAIN looking up A for router.example.com - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for router.example.com - check that a DNS record exists for this domain","instance":"","subproblems":[]},"order":"https://acme-v02.api.letsencrypt.org/acme/order/1952357436/306227172126","attempt":2,"max_attempts":3}
2024-09-18T11:11:08-04:00 Error caddy "error","ts":"2024-09-18T15:11:08Z","logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"router.example.com","challenge_type":"tls-alpn-01","problem":{"type":"urn:ietf:params:acme:error:dns","title":"","detail":"DNS problem: NXDOMAIN looking up A for router.example.com - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for router.example.com - check that a DNS record exists for this domain","instance":"","subproblems":[]}}
2024-09-18T11:11:06-04:00 Error caddy "error","ts":"2024-09-18T15:11:06Z","logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"router.example.com","problem":{"type":"urn:ietf:params:acme:error:dns","title":"","detail":"DNS problem: NXDOMAIN looking up A for router.example.com - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for router.example.com - check that a DNS record exists for this domain","instance":"","subproblems":[]},"order":"https://acme-v02.api.letsencrypt.org/acme/order/1952357436/306227166306","attempt":1,"max_attempts":3}
2024-09-18T11:11:06-04:00 Error caddy "error","ts":"2024-09-18T15:11:06Z","logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"router.example.com","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:dns","title":"","detail":"DNS problem: NXDOMAIN looking up A for router.example.com - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for router.example.com - check that a DNS record exists for this domain","instance":"","subproblems":[]}}
2024-09-18T11:11:04-04:00 Error caddy "error","ts":"2024-09-18T15:11:04Z","logger":"tls","msg":"job failed","error":"router.example.com: obtaining certificate: [router.example.com] Obtain: [router.example.com] solving challenges: [router.example.com] context canceled (order=https://acme.zerossl.com/v2/DV90/order/jmMN4DSlnzSi3PaZlGl9aQ) (ca=https://acme.zerossl.com/v2/DV90)"}
2024-09-18T11:11:04-04:00 Error caddy "error","ts":"2024-09-18T15:11:04Z","logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"router.example.com","issuer":"acme.zerossl.com-v2-DV90","error":"[router.example.com] solving challenges: [router.example.com] context canceled (order=https://acme.zerossl.com/v2/DV90/order/jmMN4DSlnzSi3PaZlGl9aQ) (ca=https://acme.zerossl.com/v2/DV90)"}
2024-09-18T11:11:04-04:00 Error caddy "error","ts":"2024-09-18T15:11:04Z","logger":"tls.issuance.acme.acme_client","msg":"deactivating authorization","identifier":"router.example.com","authz":"https://acme.zerossl.com/v2/DV90/authz/9L4C-rZlm-gfP4mrONWqDw","error":"attempt 1: https://acme.zerossl.com/v2/DV90/authz/9L4C-rZlm-gfP4mrONWqDw: context canceled"}
2024-09-18T11:11:04-04:00 Error caddy "warn","ts":"2024-09-18T15:11:04Z","logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://acme.zerossl.com/v2/DV90/authz/9L4C-rZlm-gfP4mrONWqDw","error":"performing request: Post \"https://acme.zerossl.com/v2/DV90/authz/9L4C-rZlm-gfP4mrONWqDw\": context canceled"}

Note: all references to example.com and router.example.com have been included above to safeguard privacy, the actual domain name is used on my system in real life.

When I make changes, I wait an hour - which is making this a long process, but have yet to have success so I though at this point it would be best to reach out for help.

Any help would be appreciated.

Looks great!

I've been working on this for hours, can you please help me figure out what am I missing?

Why is this ping not being blocked?

Here are screen shots of my ping and the rules.
https://ibb.co/P6PGcxS
https://ibb.co/qgyJcLK

I also tried disabling all the floating rules, but they had no impact on my test results.

Should the ping not be returning that it can't reach the destination?

It looks like that via the OPNsene API you can enable/disable rules that defined on the Firewall - Filter - Automation window, but you not those that are defined under a Firewall - interface window.  For example if you have rules defined under Firewall - LAN you cannot enable/disable them using the OPNsense API.

I am using the Community edition, so maybe this is possible in the business edition - but the help documentation doesn't seem to say that it is.  Is this the case? 

I just want to be sure before opening up a feature request for it.

Have been having some fun lately building an open source Home Internet Kill Switch using the OPNsense API

If your interested in taking a look here's the video, the OPNSense solution starts around 4 mins 8 seconds.

Also, here's the project page: 
https://rlatour.com/esp32networkblocker/
(its Project 2)

I have three interfaces: WAN, LAN and LAN_IOT.

What is the best way to block all internet traffic to LAN and LAN_IOT while still allowing traffic between and within LAN and LAN_IOT?

.

Following the documentation here:
https://docs.opnsense.org/development/api/plugins/firewall.html

via the api I can retrieve the enabled status of a firewall rule, and I can toggle it from enabled to disabled (or the other way), and I can apply the change.

However, I would rather simply either a) enable it, or in other cases b) disabled it, without needing to first retrieve the current status.  Afterwards I can apply the change.

The documentation above, does reference a setRule feature of the API, but I  can't find it otherwise documented, and the parms for it don't seem to include an way to indicate if the rule should be enabled or disabled.

Is there a way to enable or disable a rule with a direct api post (i.e. not toggle it, rather just either enable or disable it directly)?

I have a program (which I wrote) that basically casts sound/video files to chromecast devices.

I also have aliases and firewall rules set up in OPNSense as follows:

https://ibb.co/tsMxV2w

Thing is, if I enable the rule with the aliases and disable the rule right below it, the casting does not work. 

However, if I enable the rule below the one with the aliases the casting does work.

Am I missing something, or should I open this up as a bug with OPNSense?

I just took another look at the new unbound reporting (which has been a really nice add) but this time I noticed no activity for several hours (since late last night):

Odd thing is that the OPNsense dashboard was reporting the Unbound service as green.

I restarted the service and reporting resumed as expected.

Here is the reporting view:

https://ibb.co/qMkZ0z8

I looked at the Unbound log, and found this entry

2023-03-02T19:17:19-
05:00
Error   unbound   Unable to open pipe. This is likely because Unbound isn't running.   

Anyone else have this problem?

Is there a way to get an alert if the service stops working?

I have a bunch of tp-link kasa devices (switchs, plugs, etc.) on my home network.   Normally their local ip addresses remain constant, which is great because I need them to be fixed for use with Home Assistant.

However, if I enable or disable Intrusion Detection ( Services: Intrusion Detection: Administration ) their IP addresses suddenly change.

Is there a workaround for this?

FYI, there is a broken link on your main web page https://opnsense.org/

Under where it says "INTRUSION DETECTION & PREVENTION" there is a line that reads:

"Get rid of the Trojans & CNC bots with state of the art inline intrusion prevention utilizing Suricata and Proofpoint's Emerging Threats Open rules integrated."

The word 'Suricata' is a hyperlink, and that link is broken.

I had a failed log-on attempt to Home Assistant reported to me last night:

https://ibb.co/xXHLrBt

However, I have this alias and set of rules in place which I thought would prevent it:

https://ibb.co/7KjWgTt

https://ibb.co/6FvXLkk

The rules are associated with the WAN interface.

What am I doing wrong?

Of note: the Home Assistant device works within my LAN and is assigned an IP address within that (LAN) interface - but I figured access to it needs to go thru the WAN - so I associated the rules to the WAN interface - is this the problem?  Should I have associated them within the LAN interface?

I am currently using an alias to block specific external ip v4 addresses; which is great.

However, I would like to be able to all external IP v4 addresses from a specific range of IP addresses.

For example x.y.*.*  (where x and y are specific numbers that I enter, and the * can be anything from 0 to 255)

What is the best approach for this?

After upgrading to 23.1 I had/have an issue with accessing ntopng

In the previous version of OPNsense (22.7) I had set up the ntopng service to use a certificate related to duckdns.org

When I sign on to opnsense I use the address:
https://xxxx.duckdns.org/index.php
where xxxx is my duckdns name
this worked fine in the prior version of OPNsense and continues to work fine in 23.1

However, signing on to ntopng using
https://xxxx.duckdns.org:3000
no longer works.

The work-around for now is to use
http://xxxx.duckdns.org:3000
(no 's' after 'http')
and I can connect, but with an unsecured connection.

I am posting this here, but I am not sure if it is an opnsense or an ntopng issue.

Any thoughts?