Messages

I noticed this odd behavior with the traffic graphs on the Lobby:Dashboard page.  I was able to observe it in multiple OPNSense firewalls of differing devices.  The version I am running (everywhere) is 25.1.10-amd64.

When the tab/window isn't in view, the stats drop down to 0.  Then when you return to the window it recovers back to it's normal rate.

I recently got my hands on a Mudi V2 (GL-E750V2) Portable 4G LTE Router.  The device has Wiregaurd capabilities (as well as OpenVPN).  Use Guide: https://docs.gl-inet.com/router/en/4/user_guide/gl-e750/

I'm still new to WG, but I have a good grasp on how it gets configured.  I would like to setup a "site-site" connection between this mobile router and an OPNSense device.  I am having trouble 'translating' the configuration concepts to the mobile router.  The OPNsense device has a static public WAN address.  The mobile router's IP will always be dynamic and may not always be public.  It could be sitting behind a NAT, it could be directly on the public Internet.  It depends where I am.

Has anyone had any experience with this device?  If so, could you share an example configuration?

I've embraced Wireguard and yes, it's good.  Not perfect, but it works good enough for the needs.  It's much closer to the PPTP style VPN protocol that was flawless and is a "Just Works" type of setup.  Unlike OpenVPN (which by the way, current documentation appears to be inaccurate).

There is one thing missing that would turn this into a "great" option.  With PPTP was that you had access to IPSec Tunnels "for free", meaning you didn't need to do anything extra so long as your firewall rules were in place.  Based on forum searches, the solution appears to be that I need to add manual SPD entries to my IPsec configuration.  However I do not see any documentation that describes the process.  Could someone kindly provide a step by step guide?  I have multiple IPSec tunnels and I can't risk breaking something and getting locked out from remote access.

Assume the following:

- 4 OPNSense devices.  One per site.  Some have multiple subnets: 192.168.0/24, 10.8.8/24 (A), 192.168.1/24, 10.8.7/24 (B), 192.168.7/24, 10.8.9/24 (C), 192.168.8/24 (D).
- Each site has a static public IP on the WAN side, and all 4 sites are interconnected using (legacy config) IPSec tunnels.
- Firewall rules are in place that allow all sites to communicate will all hosts in the subnets
- Site A is running a Wireguard server configured using OPNsense documentation for a road warrior setup.
- The Wiregaurd tunnel network is 192.168.20/24
- Site D IPSec is configured slightly different.  It's using a VTI for route a based tunnel to just site B.  This was an experiment that didn't produce any useful results.  If needed I can switch this to the regular policy based type setup.

Other things to note:

Site A also has a mobile IPsec VPN configured as per official docs.  It to lacks the ability to access remote subnets.

Worked as documented for me. You can compare the strongswan configuration files before and after.

Thanks for the conformation.  These are all remote sites that I (obviously) can't be present at simultaneously.  The risk are of course very high if something were to go wrong. 

...slightly off topic, but maybe having some "out of the box" support for cellular modems to be configured for emergency remote access to the OPNsense instance itself only (rather than as a WAN gateway) could be a neat feature.

As per the page at https://docs.opnsense.org/manual/vpnet.html#migrating-from-tunnels-to-connections

1. Does it work exactly the same or will I see any loss/difference in functionality?
2. Do I need to change anything on the remote end?
3. If the IPSec tunnel is between two OPNSense gateways, what must I do to ensure I do not lose connectivity?

Tried it myself, and yes, it works.  But why does this NAT port forward method work? Isn't the TCP/IP protocol expecting a reply back from the assumed "real ip" address of the DNS server?

Will this same trick work with SMTP?

Also, what's the reason for the Firewall Filter rule?  That seems unnecessary (and I confirmed it works without it).  Yet, I am not redirecting DNS to the firewall, but to a different host on the LAN.

After apply the upgrade to the 24.7 series from 24.1, the DHCP service is refusing to assign an IP address to some clients.  I'm not sure how to troubleshoot this.  There are no entries in the logs.

UPDATE:

Looks like it's more than one client.  Some get a response, other's do not.

Do you have IPv6 there at all ?

these suggest a hardware problem. OPN versions would not normally make a difference.
Suggest you start with the usual mem test, etc. it could be bad memory, bad power or something else.

I don't have IPv6.  Going to try a memtest and see what comes back.  At least this thing is still in warranty (less than a year old).

[What I have and currently works]

I just realized that I have been posting my question in the wrong forum, wondering why I'm not getting an answer.  :P.

What I have and currently works

I have a functional route based IPSec VPN tunnel using the instructions at https://docs.opnsense.org/manual/how-tos/ipsec-s2s-route.html.

Site A: LAN=192.168.1.0/24 WAN=1.2.3.4
Site B: LAN=192.168.8.0/24 WAN=4.3.2.1

Phase 2 Local address at Site A: 10.192.168.1
Phase 2 Local address at Site B: 10.192.168.8
Phase 2 Remote address at Site A: 10.192.168.8
Phase 2 Remote address at Site B: 10.192.168.1

Gateways added on each site, and static routes exists for making sure the 192.168.1.0/24 can talk to 192.168.8.0/24 and vice versa.

Good so far.

Context

For the purpose of simplification, there is a website http://showthisip.com that displays my public IP address.  That host name resolves to 63.247.147.167.

At Site A, when I visit that website from a client on the 192.168.1.0/24 subnet, I am shown the IP address 1.2.3.4, which is expected.  That's the public IP of the WAN interface of OPNsense at Site A.

At Site B, when I visit that website from a client on the 192.168.8.0/24 subnet, I am shown the IP address 4.3.2.1, which is expected.  That's the public IP of the WAN interface of OPNsense at Site B

What I want to do

I want, when I visit that website using a client on the 192.168.1.0/24 subnet at Site A, to show the IP address: 4.3.2.1.  The public IP of the WAN interface at Site B.  In other words, I would like to route traffic leaving Site A  for 63.247.147.167 through the VPN tunnel I setup previously so that http://showthisip.com thinks I am at Site B. 

I add a firewall rule on the IPsec interfaces (both sites) to allow all traffic in.  Then I proceed to add a static route at Site A:

Network Address: 63.247.147.167
Gateway: 10.192.168.8

I open my browser at Site A, type in the website address... and it times out.

I trace route and see:

Code Select Expand

traceroute -n 63.247.147.167
traceroute to 63.247.147.167 (63.247.147.167), 64 hops max, 40 byte packets
1  192.168.1.1  0.347 ms  0.272 ms  0.158 ms
2  10.192.168.8  151.799 ms  151.946 ms  151.913 ms
3  * * *
4  * * *
5  * * *


Am I missing another firewall rule?

Maybe it's not NAT related sine it appears OPNsense is properly auto-generating those:



I added a firewall rule for the IPsec interface at Site B to allow anything through (for testing purposes).  That got me one step closer (it seems).

Code Select Expand

# traceroute -n 162.55.60.2
traceroute to 162.55.60.2 (162.55.60.2), 64 hops max, 40 byte packets
1  192.168.1.1  0.347 ms  0.272 ms  0.158 ms
2  10.192.168.8  151.799 ms  151.946 ms  151.913 ms
3  * * *
4  * * *
5  * * *
6  * *^C


I've run through the instructions and I can get hosts on each of the local networks to connect with each other.  I will admit, this is an interesting way of getting IPSec tunnels setup.

Problem is if I add a route to a public IP on Site A that points to the VTI on Site B, the packets seem to go nowhere.

Code Select Expand

traceroute -n 162.55.60.2
traceroute to 162.55.60.2 (162.55.60.2), 64 hops max, 40 byte packets
1  192.168.1.1  0.234 ms  0.168 ms  0.196 ms
2  * * *
3  * * *
4  * * *
5  * * *
6  * * *
7  * * *
8  * * *
9  * * *
10  * * *
...


Probably has something to do with NAT, but I'm not sure how to fix it.

It seems to be getting worse.  There have been 3 kernel panics so far today and it's only 12PM.

I have been noticing my Internet dropping a lot.  At first I thought this was a problem with my ISP.  Turns out it's not them, it's me.  Specifically, OPNSense:

Code Select Expand


2024-06-07T14:31:39 Notice kernel KDB: enter: panic
2024-06-07T14:31:39 Notice kernel panic() at panic+0x43/frame 0xfffffe008565cac0
2024-06-07T14:31:39 Notice kernel vpanic() at vpanic+0x151/frame 0xfffffe008565ca60
2024-06-07T14:31:39 Notice kernel panic: page fault
2024-06-04T08:11:46 Notice kernel KDB: enter: panic
2024-06-04T08:11:46 Notice kernel panic() at panic+0x43/frame 0xfffffe00107aea50
2024-06-04T08:11:46 Notice kernel vpanic() at vpanic+0x151/frame 0xfffffe00107ae9f0
2024-06-04T08:11:46 Notice kernel panic: page fault
2024-04-22T15:28:49 Notice kernel KDB: enter: panic
2024-04-22T15:28:49 Notice kernel panic() at panic+0x43/frame 0xfffffe00107aea60
2024-04-22T15:28:49 Notice kernel vpanic() at vpanic+0x151/frame 0xfffffe00107aea00
2024-04-22T15:28:49 Notice kernel panic: page fault
2024-04-22T03:53:41 Notice kernel KDB: enter: panic
2024-04-22T03:53:41 Notice kernel panic() at panic+0x43/frame 0xfffffe00107aea60
2024-04-22T03:53:41 Notice kernel vpanic() at vpanic+0x151/frame 0xfffffe00107aea00
2024-04-22T03:53:41 Notice kernel panic: page fault
2024-03-01T12:53:33 Notice kernel KDB: enter: panic
2024-03-01T12:53:33 Notice kernel panic() at panic+0x43/frame 0xfffffe00107aea60
2024-03-01T12:53:33 Notice kernel vpanic() at vpanic+0x151/frame 0xfffffe00107aea00
2024-03-01T12:53:33 Notice kernel panic: page fault
2024-02-18T20:55:36 Notice kernel KDB: enter: panic
2024-02-18T20:55:36 Notice kernel panic() at panic+0x43/frame 0xfffffe008565cac0
2024-02-18T20:55:36 Notice kernel vpanic() at vpanic+0x151/frame 0xfffffe008565ca60
2024-02-18T20:55:36 Notice kernel panic: page fault
2024-02-18T04:22:37 Notice kernel KDB: enter: panic
2024-02-18T04:22:37 Notice kernel panic() at panic+0x43/frame 0xfffffe00107aebd0
2024-02-18T04:22:37 Notice kernel vpanic() at vpanic+0x151/frame 0xfffffe00107aeb70
2024-02-18T04:22:37 Notice kernel panic: page fault
2024-01-16T12:38:37 Notice kernel KDB: enter: panic
2024-01-16T12:38:37 Notice kernel panic() at panic+0x43/frame 0xfffffe00107aebd0
2024-01-16T12:38:37 Notice kernel vpanic() at vpanic+0x151/frame 0xfffffe00107aeb70
2024-01-16T12:38:37 Notice kernel panic: page fault


That's apparently been going on since I first installed the device (Protectli Vault).  Here's the full log of most recent panic:

Code Select Expand

2024-06-07T14:31:39 Notice kernel FreeBSD clang version 14.0.5 (https://github.com/llvm/llvm-project.git llvmorg-14.0.5-0-gc12386ae247c)
2024-06-07T14:31:39 Notice kernel FreeBSD 13.2-RELEASE-p7 stable/23.7-n254871-d5ec322cffc SMP amd64
2024-06-07T14:31:39 Notice kernel FreeBSD is a registered trademark of The FreeBSD Foundation.
2024-06-07T14:31:39 Notice kernel The Regents of the University of California. All rights reserved.
2024-06-07T14:31:39 Notice kernel Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
2024-06-07T14:31:39 Notice kernel Copyright (c) 1992-2021 The FreeBSD Project.
2024-06-07T14:31:39 Notice kernel ---<<BOOT>>---
2024-06-07T14:31:39 Notice kernel KDB: enter: panic
2024-06-07T14:31:39 Notice kernel --- trap 0x5424a868, rip = 0x89133a23244e489d, rsp = 0xd149a69345269a6c, rbp = 0xbb3576f2ede5dbc ---
2024-06-07T14:31:39 Notice kernel fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe008565cf30
2024-06-07T14:31:39 Notice kernel fork_exit() at fork_exit+0x7e/frame 0xfffffe008565cf30
2024-06-07T14:31:39 Notice kernel ithread_loop() at ithread_loop+0x25a/frame 0xfffffe008565cef0
2024-06-07T14:31:39 Notice kernel softclock() at softclock+0x79/frame 0xfffffe008565ce60
2024-06-07T14:31:39 Notice kernel softclock_call_cc() at softclock_call_cc+0x133/frame 0xfffffe008565ce40
2024-06-07T14:31:39 Notice kernel nd6_llinfo_timer() at nd6_llinfo_timer+0x478/frame 0xfffffe008565cd90
2024-06-07T14:31:39 Notice kernel __rw_wlock_hard() at __rw_wlock_hard+0x158/frame 0xfffffe008565ccf0
2024-06-07T14:31:39 Notice kernel --- trap 0xc, rip = 0xffffffff80c84288, rsp = 0xfffffe008565cc50, rbp = 0xfffffe008565ccf0 ---
2024-06-07T14:31:39 Notice kernel calltrap() at calltrap+0x8/frame 0xfffffe008565cb80
2024-06-07T14:31:39 Notice kernel trap_pfault() at trap_pfault+0x4f/frame 0xfffffe008565cb80
2024-06-07T14:31:39 Notice kernel trap_fatal() at trap_fatal+0x387/frame 0xfffffe008565cb20
2024-06-07T14:31:39 Notice kernel panic() at panic+0x43/frame 0xfffffe008565cac0
2024-06-07T14:31:39 Notice kernel vpanic() at vpanic+0x151/frame 0xfffffe008565ca60
2024-06-07T14:31:39 Notice kernel db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe008565ca10
2024-06-07T14:31:39 Notice kernel KDB: stack backtrace:
2024-06-07T14:31:39 Notice kernel time = 1717770612
2024-06-07T14:31:39 Notice kernel cpuid = 2
2024-06-07T14:31:39 Notice kernel panic: page fault
2024-06-07T14:31:39 Notice kernel trap number = 12
2024-06-07T14:31:39 Notice kernel current process = 12 (swi4: clock (0))
2024-06-07T14:31:39 Notice kernel processor eflags = interrupt enabled, resume, IOPL = 0
2024-06-07T14:31:39 Notice kernel = DPL 0, pres 1, long 1, def32 0, gran 1
2024-06-07T14:31:39 Notice kernel code segment = base 0x0, limit 0xfffff, type 0x1b
2024-06-07T14:31:39 Notice kernel frame pointer         = 0x28:0xfffffe008565ccf0
2024-06-07T14:31:39 Notice kernel stack pointer         = 0x28:0xfffffe008565cc50
2024-06-07T14:31:39 Notice kernel instruction pointer = 0x20:0xffffffff80c84288
2024-06-07T14:31:39 Notice kernel fault code = supervisor read data, page not present
2024-06-07T14:31:39 Notice kernel fault virtual address = 0x440
2024-06-07T14:31:39 Notice kernel cpuid = 2; apic id = 04
2024-06-07T14:31:39 Notice kernel Fatal trap 12: page fault while in kernel mode
2024-06-07T14:31:39 Notice syslog-ng syslog-ng starting up; version='4.4.0'


I'm not sure what to make of this, but it brings back bad memories from a few years ago when I was having the same exact problem with a different device.

I can try upgrading to 'the latest version', but will that actually help? (it didn't a few years ago).

I know this is probably a question that has been asked an answered multiple times.  Problem is that my searches on this forum are not producing any results.  I think it's because I am probably not working it correctly.

I have 2 OPNSense boxes:

- Site A: IP of WAN: 1.2.3.4
- Site B: IP of WAN: 9.8.7.6

There is a site-to-site tunnel created between the two.  LAN clients can communicate with each other.  Good so far.

I would like to have all LAN clients in Site A that connect to a specific public IP address (or host name if possible), appear as if they are in Site B.

In other words, a LAN client in Site A opens the page http://showthisip.com and see's 9.8.7.6, but only for that page.  If I go to http://seethisip.com (assume for the moment for the purpose of this example that the two sites resolve to different IP's), I will see 1.2.3.4

Am I correct that if I follow the instructions on one of these paged, I will achieve the above?
https://docs.opnsense.org/manual/how-tos/ipsec-s2s-route.html
https://docs.opnsense.org/manual/how-tos/ipsec-s2s-conn-route.html

Which one should I follow?

[Setup Overview]

Setup Overview

As seen in the diagram below I have two offices.  Office 1 has a single LAN (192.168.0.0/24) and Office 2 has a single LAN (192.168.8.0/24).  The two offices are connected to each other using a site-site IPSEC VPN tunnel using OPNSense A and C.



Office 1 has two ISP's.  OPNSense B is connected to the second ISP and there are some hosts in Office 1 that use OPNSense B as the default gateway.

Problem

I would like to make it so 192.168.0.50 can communicate with 192.168.8.2 and 192.168.8.2 can communicate with 192.168.0.50 without having to configure a static route on 192.168.0.50.

Attempts So Far

I created a static route entry on OPNSense B to send anything for 192.168.8.0/24 to 192.168.0.100



This does not work as expected.  The hosts can ping each other, but no network traffic other than that makes it through.  For example, HTTP does not work between them.

Firewall rules for LAN on all OPNSense's:



For the IPSec interface on OPNSense A and C: