Topics

1

Tutorials and FAQs / Use os-ndproxy to proxy ISP provided /64 Prefix from WAN to LAN

« on: November 17, 2024, 09:15:09 pm »

This is a draft how it worked for me in a test environment.

os-ndproxy is part of OPNsense Community Edition 24.7.9 as plugin.

The goal is to get a single IPv6 Prefix mapped from WAN to LAN on an OPNsense connected to a Provider that only provides a single /64 prefix.

https://man.freebsd.org/cgi/man.cgi?query=ndproxy

<-- INTERNET - OPNsense PE - (igc1) Peering <---> WAN (igc1) - OPNsense CPE - (igc0) LAN <---> Clients

OPNsense PE (Provider Edge):

This simulates the Provider Router. On its own WAN, it has a Prefix by the upstream provider, e.g. /48. We simulate being our own provider, so we create a network that delegates a single /64 network downstream.

In a non simulated setup, this would be a router in the network of your provider and can be skipped during setup.

OPNsense CPE (Customer Premises Equipment):

A second OPNsense attached to OPNsense PE, it will receive the delegated /64 Prefix on its WAN interface. The goal is to use it on the LAN interface by proxying NDP with ndproxy. Without it, only the router itself could use this network. Devices on the LAN side of the CPE router could not use it. With ndproxy, NDP (Neighbor Discovery Procotol) will be proxied from LAN to WAN so all LAN IPv6 Devices can be discovered by the OPNsense PE and traffic will be sent back to these devices.

We assume the network we receive is 2001:db8:aaaa:bbbb::/64

OPNsense PE configuration:

Interfaces: [Peering]
Link local IPv6 address: fe80::1/64 (will be an auto generated one, this is needed for ndproxy. It is the default IPv6 Gateway of OPNsense CPE.)
IPv6 Configuration Type: Static IPv6
IPv6 address: 2001:db8:aaaa:bbbb::1/64

Services: ISC DHCPv6: [Peering]
Subnet: 2001:db8:aaaa:bbbb::
Subnet mask: 64
Range: Leave empty
Prefix Delegation Range:
from: 2001:db8:aaaa:bbbb::   to: 2001:db8:aaaa:bbbb::
Prefix Delegation Size: 64

Services: Router Advertisements:[Peering]
Router Advertisements: Router Only
Advertise Default Gateway: X

With these settings in place we now configure the CPE Router.
If you do not simulate the setup, but have a real provider, you only need the OPNsense CPE configuration, since that is your router that you control.

OPNsense CPE configuration:

Interfaces: [WAN]
IPv6 Configuration Type: DHCPv6
Promiscous Mode: X (important since otherwise not all NDP Multicast will be captured)
Prefix delegation size: 64
Request prefix only: X  (important since the WAN should not get a GUA, otherwise the IPv6 routing will not work. If one is needed, make sure it has a /128 prefix length.)

Interfaces: [LAN]
MAC: aa:bb:cc:dd:ee:ff (needed for ndproxy, do not set it here, use the actual interface MAC address)
IPv6 Configuration Type: Track Interface
Parent Interface: WAN
Assign prefix ID: 0

Here it is best not to use "Allow manual adjustement of DHCPv6 and Router Advertisements". It will "Just Work".
If you know what you are doing in manual control, you have to manually configure DHCPv6 and set Router Advertisements to Assisted, or set them to Stateless and use SLAAC without DHCPv6.

Now for the os-ndproxy configuration on the CPE Router:

Enable: X
Uplink Interface: WAN
Downlink MAC Address: aa:bb:cc:dd:ee:ff    <- The LAN MAC address of OPNsense CPE
Uplink IPv6 Addresses: fe80::1     <- The link local address of OPNsense PE peering interface
Exception IPv6 Addresses: leave empty

Apply the configuration.

Ensure the firewall rules allow IPv6 traffic.

Connect the Windows Client (or any other client) to the LAN side of OPNsense CPE, it should automatically receive an IPv6 address like: 2001:db8:aaaa:bbbb::2000/64

Try to "ping -6 ipv6.google.com" and it will work. When you disable ndproxy the ping will stop working after a while. When you enable it the ping will work again.

If it does not work right away, you might have to reboot your OPNsense once.

The benefit of this setup is that it should dynamically continue to work even if the prefix changes due to track interface on LAN. Also, IPv6 addresses with the privacy extension should work as well. We do not need any NAT66 or other tricks.

The downside is, it feels a bit wonky. Sometimes it doesn't work for a few minutes. Deleting the neighbor discovery protocol table on the router "ndp -c" and then restarting ndproxy seems to fix it. So far, when it runs, it seems to work just fine but it can be a little weird. It also works after OPNsense reboots, but only if the WAN interface has been put manually into promiscous mode.
 

2

24.7 Production Series / (SOLVED) I need an os-caddy user with an Exchange Server who can test something

« on: July 03, 2024, 08:49:57 pm »

Hey,

I am going to roll out a change to the os-caddy plugin that needs some testing.

I need somebody who runs os-caddy and an Exchange Server 2016 or higher and is currently reverse proxying it with os-caddy, and also uses an external Outlook 2016 or higher client. (Maybe I'm lucky xD)

There's this feature called NTLM transport in Caddy that is compiled in. In these PRs, I have implemented HTTP version and HTTP keepalive in order to see if they can replace NTLM in the future.

The NTLM module says, it is essentially doing HTTP "versions 1.1" and "keepalive off".
https://github.com/caddyserver/ntlm-transport

So, if anybody could test if these options will still make an Outlook 2016 or higher Client connect successfully to a reverse proxied Exchange Server, would be highly appreciated.

Instead of "NTLM", HTTP Version "HTTP1.1" and HTTP Keepalive "0" have to be chosen.

It works when Outlook has no authentication popup that won't go away.

I want to be able to decide if I can phase the NTLM module out or not.

PRs:
https://github.com/opnsense/plugins/pull/4072
https://github.com/opnsense/plugins/pull/4071

Thank you~

3

Tutorials and FAQs / Tutorial: Caddy (Reverse Proxy) + Let's Encrypt Certificates + Dynamic DNS

« on: February 09, 2024, 01:31:44 pm »

Like this plugin? Consider donating to me.



This plugin is simple to use and very easy to configure. Yet, it also offers plenty of advanced options for more complicated usecases at the same time.

- For Reverse Proxy + automatic Let’s Encrypt Certificates follow these steps:

1. Installation
2. Prepare OPNsense for Caddy after installation
3. Creating a simple reverse proxy
(Please note that the docs have been updated for 24.7, so there might be different terminology at a few steps.)

- For Dynamic DNS follow this additional step:

4. Dynamic DNS

Layer 4 module:

https://docs.opnsense.org/manual/how-tos/caddy.html#caddy-layer4-proxy


If you have questions or find an issue, please ask here or post on Github, I will answer them and fix problems as soon as possible.

4

Development and Code Review / os-caddy plugin

« on: November 04, 2023, 09:41:43 am »

Plugin was merged into OPNsense Plugins, I won't post in this thread anymore.
https://github.com/opnsense/plugins/pull/3840

Latest plugin version is:

Code: [Select]

v1.5.1_1(built on 06.02.2024)
Changelog

- 1.5.1_1

Little warning: A lot of things changed, if you have problems, remove the plugin and reinstall it. Also you might need to restart your firewall. Maybe.

- More DNS Providers added: netlify, namesilo, njalla, vercel, googleclouddns, alidns, powerdns, tencentcloud, dinahosting, metaname, hexonet, ddnss, linode, mailinabox, ovh, namecheap, azure, openstack-designate.
- More input fields and better documentation added for the DNS Provider API Keys.
- Changed rc.d script to standard freebsd poudriere one packaged with the caddy-custom binary, included setup.sh script to rc.conf.d/caddy.
- Updated dependancy to caddy-custom instead of caddy.
- Removed +POST_DEINSTALL.post and +POST_INSTALL.post.
- Turned syslog-ng configuration from template to static file.
- A few typos in the general.volt and reverse_proxy.volt corrected.
- The RealInterfaceField custom Fieldtype was removed and replaced with an OPNsense integrated template function to read the interface name.
- Enable $internalModelUseSafeDelete in ReverseProxyController.php - Items can only be deleted when they are not referenced by other items, making deleting in the GUI safer since there can't be any orphaned configuration left behind.
- Migration script M1_1_3 from "Description" to "description" added. Lower case description is needed to be in line with some OPNsense integrated functions.

Big thank you for all the reviewers on github: @franco, @kulikov-a, @mimugmail

- 1.5.0 Omit vultr from DNS-Providers by @Monviech in #103, General view cleanup by @Monviech in #106, Add ACME-DNS Provider for custom ACME Server support by @Monviech in #107, Hint pressing apply by @Monviech in #108, Create ACL by @Monviech in #109, Code consistency by @Monviech in #110, Built os-caddy-1.5.0.pkg by @Monviech in #111

- 1.4.5 New validate api action + Validation model fix by @Monviech, Add configuration option to log HTTP access to plain JSON files by @pmhausen, Add backend path prepend feature to handler configuration by @pmhausen

- 1.4.4 Route53 DNS Provider added + Dark Mode GUI fix + New caddy binary built that includes more DNS Provider modules as preparation + os-caddy was built with 24.1.1 dev system and tested on new OPNsense release.

- 1.4.2 Added Basic Auth as additional access restriction, made views cleaner, fixed template for new DNS Providers (desec) and added Porkbun for GUI configuration, cleaned up some code and fixed some typos.

- 1.4.0 DynDNS (Dynamic DNS) Feature added, Logging refactored to Syslog-ng to integrate completely into the OPNsense, HTTP Access Logs can be enabled.
Supported DNS Providers:
cloudflare, duckdns, digitalocean, dnspod, hetzner, godaddy, gandi, vultr, ionos, desec, porkbun

- 1.3.4 Added support for "tls_server_name" and "abort" (Reject Unmatched Connections). Fixed a bug in the template with DNS Challenge, DNS-01 checkbox didn't work.

- 1.3.3 Small template bug fixed. Wildcard handles are always placed after their subdomain handles.

- 1.3.2 Small template improvement. Empty Handles are always placed last automatically.

- 1.3.1 Access list support to restrict which IP address can connect to a domain. This is useful for restricting access to local IPs only, or when a CDN and trusted proxy is used.

Latest caddy version is:
Current Built
Caddy Releases

1. Link: How-To Install
2. Link: How-To Use Tutorials

What is Caddy? Caddy is an easy to use powerful Web Server written in Go. It includes a production ready Reverse proxy that is easy to configure. It really does all the complex configurations and Let's Encrypt certificate management and just works automagically. If you have trouble configuring HA Proxy or NGINX, look at how easy you could have it with Caddy, it's literally just a few clicks. You can have a Reverse Proxy in under a minute

5

General Discussion / Testing os-OPNWAF first time - Website looks strange

« on: November 03, 2023, 04:13:13 pm »

Hello,

I'm trying out opnwaf cause it interests me.

My test setup is really simple. I'm reverse proxying the website of my pihole.

WAN ---> IPv4 pihole.external.example.com ---> DEC740 Port 443 ---> OPNWAF + ACME ---> raspi03.internal.example.com with self signed HTTPS cert

I have configured everything like the documentation.

The result looks very strange, I thought it was the SSL termination at first, that's why I implemented a self signed HTTPS cert so the apache proxy can proxy on 443 instead of 80, but the Website of the Pihole looks kinda mangled up.

Then I also tried out with and without mod security enabled, but that didn't change anything.

Since I only used nginx as reverse proxy before, the result surprised me. Is that to be expected, or is my test scenario just a bad choice? I've wanted to use it as reverse proxy choice for internal services like checkmk, otobo or nextcloud in the end.

Thanks for any input. 

Code: [Select]

root@opn02:~ # cat /usr/local/etc/apache24/Includes/acme.conf
# Let's encrypt renewal
MDCertificateAgreement accepted
MDNotifyCmd /usr/local/sbin/configctl apache reload
MDContactEmail admin@example.com


Code: [Select]

/usr/local/etc/apache24/Includes/gateway_vhosts.conf
root@opn02:~ # cat /usr/local/etc/apache24/Includes/gateway_vhosts.conf

Listen 443

<Proxy balancer://ff9fefd4-3a8c-489d-bfb4-3dafb1485380>
    BalancerMember https://raspi03.internal.example.com/admin
</Proxy>

MDomain pihole.external.example.com

<VirtualHost *:443>
    ServerName pihole.external.example.com
    Options -FollowSymLinks
    Options -Indexes
    Options -ExecCGI
    LogLevel warn
    ProxyRequests Off
    SSLProxyEngine On
    SSLProxyCheckPeerName On

    SSLEngine on
    Protocols h2 http/1.1 acme-tls/1

    # https://wiki.mozilla.org/Security/Server_Side_TLS
    # TLS modern configuration
    SSLProtocol             all -SSLv3 -TLSv1 -TLSv1.1 -TLSv1.2
    SSLHonorCipherOrder     off
    SSLSessionTickets       off
    SSLOptions              +StrictRequire

    SSLUseStapling          On


    <Location "/">
        ProxyPreserveHost Off
        ProxyPass "balancer://ff9fefd4-3a8c-489d-bfb4-3dafb1485380/"
        ProxyPassReverse "balancer://ff9fefd4-3a8c-489d-bfb4-3dafb1485380/"
    </Location>

    Header always merge Strict-Transport-Security "max-age=63072000; includeSubdomains; preload"
    # Add security and privacy related headers
    Header set Content-Security-Policy "default-src 'self'; upgrade-insecure-requests;"
    Header always edit Set-Cookie (.*) "$1; HttpOnly; Secure"
    Header set X-Content-Type-Options "nosniff"
    Header set X-XSS-Protection "1; mode=block"
    Header set Referrer-Policy "strict-origin"
    Header set X-Frame-Options: "deny"
    SetEnv modHeadersAvailable true


</VirtualHost>


6

Virtual private networks / [Scrapped for now - Explanation added] Filter/NAT IPsec with IPFW instead of PF

« on: October 16, 2023, 11:36:31 am »

1. DISCLAIMER:
SCRAPPED - I CANT GET IT TO WONT WORK (without help)

DONT USE THIS ANYWHERE ELSE THAN IN TEST ENVIRONMENTS AT THE MOMENT.
IF YOU USE THIS IN PRODUCTION ITS AT YOUR OWN PERSONAL RISK.
This is a proof of concept to see if there's a way to improve current ipsec handling in the OPNsense in some way to have Filtering and NAT at the same time with ipsec and enc interfaces. Don't use this in production. It's here for discussion.
When I talk about NAT, I always mean SNAT here. NATD only supports SNAT.
===========================================================================================================================

2. What's the POC about?

When using IPsec with PF, there are limitations with FILTER and NAT when there are POLICY BASED and ROUTED (VTI) IPsec Tunnels on the same system.

I wanted to find out if there is a way forward that improves and resolves this behavior once and for all.

Since then I found out there is PF, IPFW and NATD available on the OPNsense at the same time, I wondered if I could just:
- Remove IPsec NAT and FILTER from PF
- Add IPsec NAT and FILTER to IPFW and NATD

It seems like OPNsense already uses this concept to some degree to share forwarding between PF and captive portal and traffic shaping IPFW:

Code: [Select]

Firewall: Settings: Advanced

Code: [Select]

Shared forwarding - Use shared forwarding between packet filter, traffic shaper and captive portal
Using policy routing in the packet filter rules causes packets to skip processing for the traffic shaper and captive portal tasks. Using this option enables the sharing of such forwarding decisions between all components to accommodate complex setups.
The following POC demonstrates that it looks like it is indeed possible to do that. Yet the side effects are UNCLEAR.

===========================================================================================================================

3. POC Test Setup explained:

Code: [Select]

      +-------------------+                 +------------------+
      |  OPNsense Site A  |   IPsec VTI      |  OPNsense Site B |
      |-------------------|                  |------------------|
      | LAN  192.168.1.1  |                  | LAN  192.168.2.1 |
      | OPT1 192.168.101.1|                  |                  |
      | WAN  172.16.0.189 |==================| WAN  172.16.0.190|
      | ipsec2 10.20.30.2 |                  | ipsec2 10.20.30.3|
      +-------------------+                  +------------------+
             ||                                       
             || Policy-based Tunnel                   
             ||                                       
      +-------------------+                           
      |   OPNsense Site C |                           
      |-------------------|                           
      | LAN  192.168.3.1  |                           
      | WAN  172.16.0.191 |                           
      +-------------------+                           


All of the below modifications have been made on OPNsense Site A. OPNsense Site B simulates an IPsec VTI implementation I have no control over, and OPNsense Site C simulates an IPSec Policy Based implementation I have no control over. OPNsense B and C are stock out of the box with no hacks.

My goal is to SNAT all requests from OPT1 to OPNsense A LAN IP address, since I assume that its the only accepted IP on Site B and Site C. They both don't know about my OPT1 net.

I have set the tunables to disable all filtering on enc and ipsec interfaces:

Code: [Select]

net.enc.in.ipsec_filter_mask IPsec input firewall filter mask runtime 0
net.enc.out.ipsec_filter_mask IPsec output firewall filter mask runtime 0
net.inet.ipsec.filtertunnel If set, filter packets from an IPsec tunnel. runtime 0
net.inet6.ipsec6.filtertunnel If set, filter packets from an IPsec tunnel. runtime 0

This will pass any traffic past the PF firewall and PF NAT, but it will still hit the IPFW firewall as well as the NATD daemon.

Afterwards I created:
- an IPsec VTI tunnel between OPNsense Site A and OPNsense Site B, but without any gateway or routes set on OPNsense Site A. The IPsec kernel routes (SPD) now take precedence over the routing decisions that PF would impose. This means we have to use IPFW to policy route policy based tunnel traffic to the right destination. And we also have to manage routes manually to the routed VPN tunnels.

Code: [Select]

route add -net 192.168.2.0/24 10.20.30.3
- a Policy Based IPsec Tunnel between OPNsense Site A and OPNsense Site C, with the traffic selectors being the LAN Site A and LAN Site C.


Now I load the IPFW kernel module in the opnsense:

Code: [Select]

kldload ipfw

Afterwards I restart ipfw:

Code: [Select]

service ipfw onerestart

Careful with further ipfw restarts, the rules we add are only in memory and will be lost after restarts.

Now you can see that there seem to be default rules that have been implemented by the already existing "/usr/local/etc/ipfw.rules"

Code: [Select]

ipfw list


Code: [Select]

00100 allow pfsync from any to any
00110 allow carp from any to any
00120 allow layer2 mac-type 0x0806,0x8035
00130 allow layer2 mac-type 0x888e,0x88c7
00140 allow layer2 mac-type 0x8863,0x8864
00150 deny layer2 not mac-type 0x0800,0x86dd
00200 skipto 60000 ip6 from ::1 to any
00201 skipto 60000 ip4 from 127.0.0.0/8 to any
00202 skipto 60000 ip6 from any to ::1
00203 skipto 60000 ip4 from any to 127.0.0.0/8
06000 skipto 60000 tcp from any to any out
06199 skipto 60000 ip from any to any
60000 return
65533 allow ip from any to any
65534 deny ip from any to any
65535 allow ip from any to any

Because there is "65533 allow ip from any to any" and ipfw comes after pf in the packet flow, all packets go through.

Now we can implement our divert rule so that we NAT traffic, and additionally our routes so that the traffic is routed through the right IPsec tunnels:

Code: [Select]

ipfw add 00005 divert 8668 ip from 192.168.101.0/24 to 192.168.2.0/24
ipfw add 00006 divert 8668 ip from 192.168.101.0/24 to 192.168.3.0/24


Code: [Select]

route add -net 192.168.2.0/24 10.20.30.3
route add -net 192.168.3.0/24 172.16.0.191


If you want the rules ipfw restart persistent you have to add them to the /usr/local/etc/ipfw.rules

Afterwards we have to configure "natd" to listen on 8668 and SNAT the packets for us on 192.168.1.1 if they match the above divert rule:

Code: [Select]

/etc/rc.conf


Code: [Select]

keymap="de.noacc.kbd"
natd_enable="YES"
natd_flags="-a 192.168.1.1 -redirect_address 192.168.101.1 192.168.1.1"

Here NATD SNATs source IP address 192.168.101.1 to source IP address 192.168.1.1 when it is diverted by IPFW to the socket 8668 (The default divert socket.) You can also create additional divert sockets on different ports and have multiple different NATs this way when adjusting the IPFW divert rules to them.

Now we just have to restart natd to take this in effect

Code: [Select]

service natd restart

===========================================================================================================================

4. Proof that it works:

Now I can prove with packet captures that the NATing works, on the Routed IPsec tunnel, as well as the Policy Based tunnel at the same time, without any tunables set and without pf in the way:

OPNsense SITE A TO OPNsense SITE B

Code: [Select]

root@OPNsenseA:~ # ping -S 192.168.101.1 192.168.2.1
PING 192.168.2.1 (192.168.2.1) from 192.168.101.1: 56 data bytes
64 bytes from 192.168.2.1: icmp_seq=0 ttl=64 time=0.755 ms
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=0.732 ms
64 bytes from 192.168.2.1: icmp_seq=2 ttl=64 time=0.760 ms


Code: [Select]

root@OPNsenseB:~ # tcpdump -i enc0 -n host 192.168.2.1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enc0, link-type ENC (OpenBSD encapsulated IP), capture size 262144 bytes
08:48:41.013508 (authentic,confidential): SPI 0xc9014910: IP 192.168.1.1 > 192.168.2.1: ICMP echo request, id 61125, seq 0, length 64
08:48:41.013609 (authentic,confidential): SPI 0xc453dc2a: IP 192.168.2.1 > 192.168.1.1: ICMP echo reply, id 61125, seq 0, length 64
08:48:42.064250 (authentic,confidential): SPI 0xc9014910: IP 192.168.1.1 > 192.168.2.1: ICMP echo request, id 61125, seq 1, length 64
08:48:42.064342 (authentic,confidential): SPI 0xc453dc2a: IP 192.168.2.1 > 192.168.1.1: ICMP echo reply, id 61125, seq 1, length 64
08:48:43.136055 (authentic,confidential): SPI 0xc9014910: IP 192.168.1.1 > 192.168.2.1: ICMP echo request, id 61125, seq 2, length 64
08:48:43.136145 (authentic,confidential): SPI 0xc453dc2a: IP 192.168.2.1 > 192.168.1.1: ICMP echo reply, id 61125, seq 2, length 64

I can prove that filtering works too, by denying ICMP before the divert rule at 00050:

Code: [Select]

ipfw add 00040 deny icmp from any to any

Opnsense A cant ping Opnsense B anymore.

Code: [Select]

root@OPNsenseA:~ # ping 192.168.2.1
PING 192.168.2.1 (192.168.2.1): 56 data bytes
ping: sendto: Permission denied
ping: sendto: Permission denied

Opnsense B cant ping Opnsense A anymore.

Code: [Select]

root@OPNsenseB:~ # ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1): 56 data bytes
^C
--- 192.168.1.1 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss

OPNsense SITE A TO OPNsense SITE C

Code: [Select]

root@OPNsenseA:~ # ping -S 192.168.101.1 192.168.3.1
PING 192.168.3.1 (192.168.3.1) from 192.168.101.1: 56 data bytes
64 bytes from 192.168.3.1: icmp_seq=0 ttl=64 time=1.231 ms
64 bytes from 192.168.3.1: icmp_seq=1 ttl=64 time=0.754 ms
64 bytes from 192.168.3.1: icmp_seq=2 ttl=64 time=0.920 ms
^C
--- 192.168.3.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.754/0.969/1.231/0.198 ms


Code: [Select]

root@OPNsenseA:~ # tcpdump -i enc0 -n not net 10.20.30.0/24
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enc0, link-type ENC (OpenBSD encapsulated IP), capture size 262144 bytes
10:55:37.713685 (authentic,confidential): SPI 0xc96a6d76: IP 192.168.1.1 > 192.168.3.1: ICMP echo request, id 53976, seq 0, length 64
10:55:37.714290 (authentic,confidential): SPI 0xc843e965: IP 192.168.3.1 > 192.168.1.1: ICMP echo reply, id 53976, seq 0, length 64
10:55:38.783108 (authentic,confidential): SPI 0xc96a6d76: IP 192.168.1.1 > 192.168.3.1: ICMP echo request, id 53976, seq 1, length 64
10:55:38.783573 (authentic,confidential): SPI 0xc843e965: IP 192.168.3.1 > 192.168.1.1: ICMP echo reply, id 53976, seq 1, length 64
10:55:39.854198 (authentic,confidential): SPI 0xc96a6d76: IP 192.168.1.1 > 192.168.3.1: ICMP echo request, id 53976, seq 2, length 64
10:55:39.854896 (authentic,confidential): SPI 0xc843e965: IP 192.168.3.1 > 192.168.1.1: ICMP echo reply, id 53976, seq 2, length 64


Code: [Select]

root@OPNsense:~ # tcpdump -i enc0 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enc0, link-type ENC (OpenBSD encapsulated IP), capture size 262144 bytes
10:55:37.713917 (authentic,confidential): SPI 0xc96a6d76: IP 192.168.1.1 > 192.168.3.1: ICMP echo request, id 53976, seq 0, length 64
10:55:37.714029 (authentic,confidential): SPI 0xc843e965: IP 192.168.3.1 > 192.168.1.1: ICMP echo reply, id 53976, seq 0, length 64
10:55:38.783298 (authentic,confidential): SPI 0xc96a6d76: IP 192.168.1.1 > 192.168.3.1: ICMP echo request, id 53976, seq 1, length 64
10:55:38.783381 (authentic,confidential): SPI 0xc843e965: IP 192.168.3.1 > 192.168.1.1: ICMP echo reply, id 53976, seq 1, length 64
10:55:39.854391 (authentic,confidential): SPI 0xc96a6d76: IP 192.168.1.1 > 192.168.3.1: ICMP echo request, id 53976, seq 2, length 64
10:55:39.854513 (authentic,confidential): SPI 0xc843e965: IP 192.168.3.1 > 192.168.1.1: ICMP echo reply, id 53976, seq 2, length 64

As reference, here's all established IPsec SAs of all firewalls:

Code: [Select]

root@OPNsenseA:~ # swanctl --list-sas
no files found matching '/usr/local/etc/strongswan.opnsense.d/*.conf'
99e0e1c1-285e-45cd-b4ef-61dcc914c879: #10, ESTABLISHED, IKEv2, 64e822915de2c77c_i* f9322b2b8d08c3f9_r
  local  '172.16.0.189' @ 172.16.0.189[4500]
  remote '172.16.0.190' @ 172.16.0.190[4500]
  AES_CBC-128/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/ECP_256
  established 4904s ago, rekeying in 8431s
  c24cc267-6c7e-4993-8bca-d55a1a36fc92: #6, reqid 2, INSTALLED, TUNNEL, ESP:AES_CBC-128/HMAC_SHA2_256_128
    installed 1419s ago, rekeying in 1863s, expires in 2541s
    in  cc6a1148,  32241 bytes,   445 packets
    out ca9c2e5c,  94340 bytes,   647 packets
    local  172.16.0.189/32
    remote 172.16.0.190/32
2cb5596b-8866-4675-bfe3-6743851fa95e: #29, ESTABLISHED, IKEv2, 3c5f14c4aff43214_i 9fb73ebfee8e3639_r*
  local  '172.16.0.189' @ 172.16.0.189[4500]
  remote '172.16.0.191' @ 172.16.0.191[4500]
  AES_CBC-128/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/ECP_256
  established 716s ago, rekeying in 12374s
  46f91ed0-3152-4fce-9ec5-2644e60ad11d: #7, reqid 1, INSTALLED, TUNNEL, ESP:AES_CBC-128/HMAC_SHA2_256_128
    installed 716s ago, rekeying in 2561s, expires in 3244s
    in  c843e965,    504 bytes,     6 packets,   679s ago
    out c96a6d76,    936 bytes,     6 packets,   679s ago
    local  192.168.1.0/24
    remote 192.168.3.0/24

root@OPNsenseB:~ # swanctl --list-sas
no files found matching '/usr/local/etc/strongswan.opnsense.d/*.conf'
bc96997e-6b9e-4b81-805d-48a1135d4560: #17, ESTABLISHED, IKEv2, 64e822915de2c77c_i f9322b2b8d08c3f9_r*
  local  '172.16.0.190' @ 172.16.0.190[4500]
  remote '172.16.0.189' @ 172.16.0.189[4500]
  AES_CBC-128/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/ECP_256
  established 4903s ago, rekeying in 8853s
  bad2434e-38cd-4d07-9908-8a73e60e1e21: #35, reqid 2, INSTALLED, TUNNEL, ESP:AES_CBC-128/HMAC_SHA2_256_128
    installed 1418s ago, rekeying in 1862s, expires in 2542s
    in  ca9c2e5c,  47253 bytes,   645 packets
    out cc6a1148,  64272 bytes,   444 packets
    local  172.16.0.190/32
    remote 172.16.0.189/32

root@OPNsenseC:~ # swanctl --list-sas
no files found matching '/usr/local/etc/strongswan.opnsense.d/*.conf'
29a73eff-323d-458d-9203-62f64dff2275: #15, ESTABLISHED, IKEv2, 3c5f14c4aff43214_i* 9fb73ebfee8e3639_r
  local  '172.16.0.191' @ 172.16.0.191[4500]
  remote '172.16.0.189' @ 172.16.0.189[4500]
  AES_CBC-128/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/ECP_256
  established 717s ago, rekeying in 12424s
  2f5a11bf-9f31-444a-9986-e5f3b694624d: #5, reqid 1, INSTALLED, TUNNEL, ESP:AES_CBC-128/HMAC_SHA2_256_128
    installed 717s ago, rekeying in 2566s, expires in 3243s
    in  c96a6d76,    504 bytes,     6 packets,   680s ago
    out c843e965,    936 bytes,     6 packets,   680s ago
    local  192.168.3.0/24
    remote 192.168.1.0/24

===========================================================================================================================

5. Things to check:
5.1 Can PF use the NATD socket with a divert rule like IPFW?
RESULT: PF + NATD doesn't work as expected. The packets get diverted and NATed but they're not returned back to the host. With this out of the way, it looks like IPFW with NATD is the only choice for Source NATing both policy and vti tunnels.
https://man.openbsd.org/divert.4

It seems like PF can also use "divert-packet" to send pakets to a NATD socket in userspace.

Testing this soon. If it works there could be a solution by implementing NATD into the opnsense GUI and expanding the PF Gui to allow setting a "divert-packet" parameter that selects the NATD socket.

It seems like the divert-packet options is exclusive to OpenBSDs implementation of PF. But there seems to be a "divert-to" parameter in FreeBSDs PF.
https://man.freebsd.org/cgi/man.cgi?pf.conf

Code: [Select]

divert-to <host> port <port>
     Used to redirect packets to a local  socket  bound to  host  and
     port.  The packets will not be modified, so getsockname(2) on the
     socket  will  return  the original  destination  address of the
     packet.

Code: [Select]

pass in log quick on hn2 from 192.168.101.1 to {any}
pass out log quick on ipsec2 from 192.168.101.1 to 192.168.2.1 divert-to localhost port 8668

===========================================================================================================================

6. Conclusions:

(Careful, so far these conclusions are only verified for firewall to firewall traffic, using ping -S)

PF + PF_NAT = Not working in this setup (either if_ipsec or if_enc)
PF + NATD = Not working in this setup (at all)
IPFW + IPFW_NAT = Not working in this setup (either if_ipsec or if_enc)
IPFW + NATD = Working for if_enc (policy based ipsec) and if_ipsec (route based ipsec) with FILTER and SNAT, but only from OPNsense to OPNsense directly. Hosts behind it don't work with NAT.

Current known drawbacks (to me):
- There is no pfsync equivalent for ipfw, that means if you use this hack you shouldn't use it on HA if you want seamless failover.
- You can't use the GUI to configure any of this. You have to use the shell.
- Traffic only works between OPNsense with NAT, I can't get it to work on hosts...

-So far, all traffic between the firewalls works great (which is a step up), but as soon as an outside host is introduced, things seem to fall apart and I don't know why. Following this alone with nobody to talk to won't result in anything. If anybody is interested to help me to follow this concept through, please PM.

It seems like this is related to this thread:
https://forums.freebsd.org/threads/share-ipsec-vpn-between-gateway-and-clients-using-ipfw-and-nat.87178/

7. Explanation why it doesn't work:

https://man.freebsd.org/cgi/man.cgi?query=ipsec&sektion=4&format=html

enc(4) is a virtual IPsec interface that is used to pass raw IP packets before encapsulation, and after decapsulation of ESP. That means, that the packets can be filtered and nated on it, before they're processed by IPsec. That is also the reason, why you need to set manual SPD (Security Policy Database) entries, when you NAT. Because you NAT before IPsec encapsulates the ESP packet. And if there's no security policy for the source IP address, IPsec will drop the packet before its encapsulated.

ipsec(4) is a virtual tunnel interface, that is used to pass already encapsulated packets. That means, you can't filter and NAT them, because the original IP header was already encapsulated into an ESP packet.

Now, that should mean, that you should be able to filter and nat all packets that come in and out of enc(4), because they all have their original IP header and are raw packets. In practice, this somehow does not work if "ipsec" is also introduced. With VTI, the packets first go into enc(4), and after they have been encapsulated they go into ipsec(4). Something about this process makes filter and NAT impossible, due to an implementation bug.

With the tunable "net.inet.ipsec.filtertunnel" you can change the filter behavior. Per default, you can perform packet filtering ant NAT before and after ESP on the enc(4) interface. With the tunable set, you can filter "inside" the ESP packet, but because enc(4) doesn't have any encapsulated packets, the filter won't work on it anymore. It will then work on the ipsec(4) interface where the encapsulated ESP packets are. The filter is set on the IP header inside the ESP packets.

Because Policy Based tunnel traffic only passes through the enc(4) interface, and not also through an ipsec(4) interface like VTI tunnels, you can either filter on the RAW IP packets, or on the ESP packets, but not on both (at least in FreeBSDs ipsec(4) and enc(4) implementation)

===========================================================================================================================

Let's start a discussion. 

7

Zenarmor (Sensei) / Zenarmor packet flow

« on: October 10, 2023, 02:39:25 pm »

Hello Zenarmor team,

I would like to know where to add your product into a packet flow diagram (non official, community), that shows OPNsense packet handling:

https://forum.opnsense.org/index.php?topic=36326.0

Thank you

8

23.7 Legacy Series / "IPsec internal host to host" rule, where to disable?

« on: October 09, 2023, 01:14:04 pm »

I'm looking for a way to disable the "IPsec internal host to host" rule.

I have enabled "VPN: IPsec: Advanced Settings" - "Disable all auto-added VPN rules."

When looking into pfctl -s "rules" it shows this rule:

Code: [Select]

pfctl -s rules | grep -i enc0
pass out log on enc0 all flags S/SA keep state label "c1eff64cbafdd6b80448f92cd4aff7e5"

So for now I have just set my own rule before this one to block it:

Code: [Select]

block drop out quick on enc0 inet all label "9d362a93e2c802daca5dcc00a0ad8df8"

Having the IPsec internal host to host rule probably makes a lot of sense because you don't have to create your own "direction out" rules, though still having it after disabling all auto added rules might be misleading? But I'm not judging it, just want to know if it's the way it's supposed to be.

9

Tutorials and FAQs / Packet Flow Diagram

« on: October 07, 2023, 06:57:19 pm »

Attention:
This is a community effort. This is not an official packet flow diagram. Please leave a comment if something is wrong.

Ingress Traffic:
1. Ingress Interface
   |
2. Next Generation Firewall (Ingress)
   |----> 2.1 Suricata (IPS mode) (depends on selected Interfaces)
   |      |----> If Block Rule Matches, Drop Packet
   |      |----> Else, Continue
   |----> 2.2 Zenarmor (depends on selected Interfaces)
   |      |----> If Block Rule Matches, Drop Packet
   |      |----> Else, Continue
   |
3. Scrub (normalize, reassemble fragments, etc.) ~PF~
   |
4. 1:1 NAT (Bi-directional NAT) ~PF~
   |----> 4.1 Match Rules (Static NAT - BINAT - 1:1 NAT)
   |
5. Destination NAT (Port Forward or Redirection) ~PF~
   |----> 5.1 Match Rules (DNAT - Port Forward)
   |
6. Source NAT (Outbound NAT) ~PF~
   |----> 6.1 Match Rules (SNAT - Outbound)
   |
7. Is Packet First in Flow? ~PF~
   |----> Yes:
   |      |----> 7.1 Filter Rules
   |      |       |----> 7.1.1 Block/Pass (Quick) in order of rules until
   |      |                          first match, then terminates further
   |      |                          evaluation
   |      |       |----> 7.1.2 Block/Pass (without Quick) until best
   |      |                          match, if no prior quick rule matched
   |      |----> 7.2 Create State Entry (if rule has state tracking)
   |----> No:
   |      |----> 7.3 Use Existing State Entry
   |
8. Routing Decision (determine egress interface)
   |
9. Traffic Shaping ~IPFW with dummynet~
   |
10. Next Generation Firewall (Egress)
   |----> 10.1 Suricata (IPS mode) (depends on selected Interfaces)
   |      |----> If Block Rule Matches, Drop Packet
   |      |----> Else, Continue
   |----> 10.2 Zenarmor (depends on selected Interfaces)
   |      |----> If Block Rule Matches, Drop Packet
   |      |----> Else, Continue
   |
11. Egress Interface

- Added Block and Pass filter rule differentiation with "Quick" enabled, feeback from @CJ
- Added Suricata and Zenarmor to the right spots with feedback from @mb: https://forum.opnsense.org/index.php?topic=36369.0

10

Virtual private networks / SNAT and Virtual Tunnel Interfaces - An in depth analysis

« on: October 07, 2023, 11:10:37 am »

I have posted this in the freebsd forum in hopes of getting some more insight: FreeBSD Forum

Additional tests with IPFW and NATD, and IPFW and IPFW_NAT: https://forum.opnsense.org/index.php?topic=36456.0

Since you can't do SNAT on VTI ipsecXX interfaces without some tunables set (which break enc0 firewalling and natting) I thought to be clever and go on a quest to find a working routing solution with SNAT and firewalling over policy based tunnels.

My eyes fell on GRE:

- I created a GRE tunnel inside an IPsec Policy Based VPN in Transport Mode.
- Routing into the GRE interfaces worked, traffic flowed as expected (there even was an automatic GRE Gateway created for the static routes, neat)
- SNAT into the GRE interface worked, but once the reply packages traffic flowed back from site B to site A, they got stuck in the GRE interface on site A and never returned to the initiator of the request.

So all in all, with those tunnel interfaces the echo request and echo reply worked with SNAT, but on the final step where it would have to get out of the tunnel interface back to the original initiator, the flow just stops. As if pf forgot there's a NAT Table...?

=================================================================================================================

Code: [Select]

       +-------------------+                 +------------------+
       |  OPNsense Site A  |                 |  OPNsense Site B |
       |-------------------|                 |------------------|
       | LAN  192.168.1.1  |                 | LAN  192.168.2.1 |
       | OPT1 192.168.101.1| IPsec Transport |                  |
       | WAN  172.16.11.2  | =============== | WAN  172.16.11.3 |
       | gre0 10.21.1.2    | --------------- | gre0 10.21.1.3   |
       +-------|-----------+   GRE Tunnel    +-------|----------+
               |                                     |   
               |                                     |
               |                                     |
               |                                     |
               | 
         LAN 192.168.1.100
              --OR--
         OPT1 192.168.101.100                 LAN 192.168.2.3
           [Host Site A]                       [Host Site B]
  (Only one interface is connected)
-----------------------------------------------------------------------------------------------------------------

The setup is like this:

OPNsense Site A:
Interfaces:

Code: [Select]


LAN (hn0)       -> v4: 192.168.1.1/24
OPT1 (hn2)      -> v4: 192.168.101.1/24
WAN (hn1)       -> v4: 172.16.11.2/24
gre10 (gre0)    -> v4: 10.21.1.2/24 - MTU 1398
Routes

Code: [Select]

Internet:
Destination        Gateway            Flags     Netif Expire
10.19.1.2          link#9             UH          lo1
10.21.1.3          link#8             UH         gre0
127.0.0.1          link#1             UH          lo0
172.16.11.0/24     link#6             U           hn1
172.16.11.2        link#6             UHS         lo0
192.168.1.0/24     link#5             U           hn0
192.168.1.1        link#5             UHS         lo0
192.168.2.0/24     10.21.1.3          UGS        gre0
192.168.101.0/24   link#7             U           hn2
192.168.101.1      link#7             UHS         lo0

PF filter rules

Code: [Select]

pass out log all flags S/SA keep state allow-opts label "fae559338f65e11c53669fc3642c93c2"
pass out log on enc0 all flags S/SA keep state label "c1eff64cbafdd6b80448f92cd4aff7e5"
pass out log route-to (gre0 10.21.1.3) inet from (gre0) to ! (gre0:network) flags S/SA keep state allow-opts label "f7f077b5334caa29bc835d174f88b548"
pass in quick inet all flags S/SA keep state label "523ba68a597fc0e535b425d2ef260b6b"
pass in quick inet6 all flags S/SA keep state label "523ba68a597fc0e535b425d2ef260b6b"

PF nat rules

Code: [Select]

nat on gre0 inet from (hn2:network) to 192.168.2.0/24 -> (hn0:0) port 1024:65535

Swanctl SAs
(Ignore the 0 bytes, it works but I didn't pass traffic when I made this --list-sas)

Code: [Select]

21924229-8cb2-496e-b1d7-26cc4dc35f7d: #4, ESTABLISHED, IKEv2, e51ca149a80e6d9a_i* 4800a403859d825a_r
  local  '172.16.11.2' @ 172.16.11.2[4500]
  remote '172.16.11.3' @ 172.16.11.3[4500]
  AES_CBC-128/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/ECP_256
  established 14s ago, rekeying in 13406s
  e200d362-fc18-4844-aa62-0529148faa58: #3, reqid 1, INSTALLED, TRANSPORT, ESP:AES_CBC-128/HMAC_SHA2_256_128
    installed 14s ago, rekeying in 3247s, expires in 3946s
    in  cbbd2ddd,      0 bytes,     0 packets
    out ca6ec24e,      0 bytes,     0 packets
    local  172.16.11.2/32
    remote 172.16.11.3/32

Host Site A (Ubuntu):

Code: [Select]

LAN 192.168.1.100
              --OR--
OPT1 192.168.101.100

-----------------------------------------------------------------------------------------------------------------
 
OPNsense Site B:

Code: [Select]

LAN (hn0)       -> v4: 192.168.2.1/24
WAN (hn1)       -> v4: 172.16.11.3/24
gre10 (gre0)    -> v4: 10.21.1.3/24 - MTU 1398

Routes

Code: [Select]

Internet:
Destination        Gateway            Flags     Netif Expire
10.19.1.3          link#8             UH          lo1
10.21.1.2          link#7             UH         gre0
127.0.0.1          link#1             UH          lo0
172.16.11.0/24     link#6             U           hn1
172.16.11.3        link#6             UHS         lo0
192.168.1.0/24     10.21.1.2          UGS        gre0
192.168.2.0/24     link#5             U           hn0
192.168.2.1        link#5             UHS         lo0

PF filter rules

Code: [Select]

pass out log all flags S/SA keep state allow-opts label "fae559338f65e11c53669fc3642c93c2"
pass out log on enc0 all flags S/SA keep state label "c1eff64cbafdd6b80448f92cd4aff7e5"
pass out log route-to (gre0 10.21.1.2) inet from (gre0) to ! (gre0:network) flags S/SA keep state allow-opts label "64abd34393f7bf3840c44e806a347bf6"
pass in quick inet all flags S/SA keep state label "357faa0befdb804e3fe5f8345c9b76c7"
pass in quick inet6 all flags S/SA keep state label "357faa0befdb804e3fe5f8345c9b76c7"

PF nat rules

Code: [Select]

NONE

Swanctl SAs
(Ignore the 0 bytes, it works but I didn't pass traffic when I made this --list-sas)

Code: [Select]

fa5ea186-0bb5-43fe-b570-15dd7f1b728e: #3, ESTABLISHED, IKEv2, e51ca149a80e6d9a_i 4800a403859d825a_r*
  local  '172.16.11.3' @ 172.16.11.3[4500]
  remote '172.16.11.2' @ 172.16.11.2[4500]
  AES_CBC-128/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/ECP_256
  established 68s ago, rekeying in 13094s
  a78735eb-f23b-4284-a4e3-30ddc2e474a2: #3, reqid 1, INSTALLED, TRANSPORT, ESP:AES_CBC-128/HMAC_SHA2_256_128
    installed 68s ago, rekeying in 3483s, expires in 3892s
    in  ca6ec24e,      0 bytes,     0 packets
    out cbbd2ddd,      0 bytes,     0 packets
    local  172.16.11.3/32
    remote 172.16.11.2/32

Host Site B:

Code: [Select]

LAN  192.168.2.3

=================================================================================================================

Now that the setup is described, here come the packet captures:

First a Sanity Check that the routing itself works as expected.

This package capture shows:
- Host Site A (192.168.1.100) initiating an ICMP echo request to Host Site B (192.168.2.3)

Packet is received on LAN interface of OPNsense A from Host A

Code: [Select]

OPNsense A:~ # tcpdump -i hn0 proto ICMP -n
08:47:05.654822 IP 192.168.1.100 > 192.168.2.3: ICMP echo request, id 45, seq 1, length 64
08:47:05.656662 IP 192.168.2.3 > 192.168.1.100: ICMP echo reply, id 45, seq 1, length 64
Packet is forwarded into GRE tunnel on OPNsense A

Code: [Select]

OPNsense A:~ # tcpdump -i gre0 proto ICMP -n
08:47:05.654909 IP 192.168.1.100 > 192.168.2.3: ICMP echo request, id 45, seq 1, length 64
08:47:05.656581 IP 192.168.2.3 > 192.168.1.100: ICMP echo reply, id 45, seq 1, length 64
Packet is received by GRE tunnel on OPNsense B

Code: [Select]

OPNsense B:~ # tcpdump -i gre0 proto ICMP -n
08:47:05.655683 IP 192.168.1.100 > 192.168.2.3: ICMP echo request, id 45, seq 1, length 64
08:47:05.656101 IP 192.168.2.3 > 192.168.1.100: ICMP echo reply, id 45, seq 1, length 64
Packet is forwarded to LAN on OPNsense B and sent to Host B

Code: [Select]

OPNsense B:~ # tcpdump -i hn0 proto ICMP -n
08:47:05.655790 IP 192.168.1.100 > 192.168.2.3: ICMP echo request, id 45, seq 1, length 64
08:47:05.656069 IP 192.168.2.3 > 192.168.1.100: ICMP echo reply, id 45, seq 1, length 64
Echo Request and Echo Reply route as expected between Host A and Host B

-----------------------------------------------------------------------------------------------------------------

Now with SNAT, which DOESN't WORK as expected:

This package captures shows:
- Host Site A (192.168.101.100) initiating an ICMP echo request to Host Site B (192.168.2.3)
- and getting SNATed onto the Interface IP of OPNsense Site A LAN address (192.168.1.1)

Heres the states of the firewalls:

Code: [Select]

OPNsense A:~ # pfctl -ss | grep -i icmp
all icmp 192.168.2.3:49 <- 192.168.101.100:49       0:0
all icmp 192.168.1.1:45468 (192.168.101.100:49) -> 192.168.2.3:45468       0:0

Code: [Select]

OPNsense B:~ # pfctl -ss | grep -i icmp
all icmp 192.168.1.1:45468 -> 192.168.2.3:45468       0:0
all icmp 192.168.2.3:45468 -> 192.168.1.1:45468       0:0

Packet is received on OPT1 interface of OPNsense A from Host A

Code: [Select]

OPNsense A:~ # tcpdump -i hn2 proto ICMP -n
08:55:12.675487 IP 192.168.101.100 > 192.168.2.3: ICMP echo request, id 47, seq 1, length 64
Packet is forwarded into GRE tunnel on OPNsense A and SNATed to the LAN interface IP 192.168.1.1

Code: [Select]

OPNsense A:~ # tcpdump -i gre0 proto ICMP -n
08:55:12.675565 IP 192.168.1.1 > 192.168.2.3: ICMP echo request, id 41970, seq 1, length 64
08:55:12.676751 IP 192.168.2.3 > 192.168.1.1: ICMP echo reply, id 41970, seq 1, length 64
Packet is received by GRE tunnel on OPNsense B

Code: [Select]

OPNsense B:~ # tcpdump -i gre0 proto ICMP -n
08:55:12.675977 IP 192.168.1.1 > 192.168.2.3: ICMP echo request, id 41970, seq 1, length 64
08:55:12.676516 IP 192.168.2.3 > 192.168.1.1: ICMP echo reply, id 41970, seq 1, length 64
Packet is forwarded to LAN on OPNsense B and sent to Host B

Code: [Select]

OPNsense B:~ # tcpdump -i hn0 proto ICMP -n
08:55:12.676171 IP 192.168.1.1 > 192.168.2.3: ICMP echo request, id 41970, seq 1, length 64
08:55:12.676489 IP 192.168.2.3 > 192.168.1.1: ICMP echo reply, id 41970, seq 1, length 64
The Echo Reply comes all the way back to OPNsense A gre0 - But there it won't be translated back by the NAT table and forwarded back to the hn2 OPT interface

Also I tried it with a vxlan interface too, same behavior.

=================================================================================================================

If you got this far, thank you for reading this post. It took some time to write, so it took some time to read too

I hope there's somebody here with some in depth knowledge who will confirm that SNATing into GRE tunnel interfaces won't work because of some PF limitations. (pf is a firewall implementation in FreeBSD that both pfsense and opnsense use.)

11

Hardware and Performance / Question about DEC740 SFP+ (home use)

« on: October 02, 2023, 06:33:31 pm »

I am not that firm in hardware, so I have a question to somebody more knowledgable about it than me, before investing money.

I'm thinking about purchasing a DEC740 because it looks like a great little box for home use. (I know its a bit more expensive than buying one of those ebay or aliexpress boxes, but I just want something reliable at home where I dont want to spend lots of time troubleshooting.)

My switch at home is a "MS510TXPP 8-Port Multi-Gigabit Smart Managed Pro Switch with PoE+ and two 10G Ports"

I wonder how to connect the 10G SFP+ Port to my MS510TXPP. I have seen that theres this " [DAC] NETGEAR AXC761 10G SFP+ PASSIVE COPPER CABLE 1M (MOST RECENT VERSION)" Cable which seems to be supported. And since it's Netgear it should also work with my Netgear Switch?

I've read a few topics and I've seen that the general opinion seems to be that DAC gets hot and is unreliable, is that true in a home setup like this where the port density is low?

Thank you all for your valuable input

PS:
If anybody is interested why I want to do that, my VM host is still Hyper-V and I want to pull the OPNsense VM out onto reliable hardware before Microsoft pulls the plug on the Windows support in the future. Before I change anything hypervisor related I need to have the network basics reliable.

12

Virtual private networks / [SOLVED] VXLAN between OPNsense and Ubuntu - can't get it to work.

« on: September 28, 2023, 10:15:50 pm »

I really need insight of somebody who has managed to get VXLAN on the OPNsense working.

As baseline, I did a basic VXLAN setup between two Ubuntu 22.04 LTS Servers on the same Network as sanity check.

Code: [Select]

VM1 (vxlan0 id 42 10.30.40.3/24) eth0 172.16.0.186/24 <-----> 172.16.0.187/24 eth0 (10.30.40.4/24 vxlan0 id 42) VM2
VM1:

Code: [Select]

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.0.186  netmask 255.255.255.0  broadcast 172.16.0.255

vxlan0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1355
        inet 10.20.40.3  netmask 255.255.255.0  broadcast 10.20.40.255
        vxlan id 42 remote 172.16.0.187 local 172.16.0.186 dev eth0 srcport 0 0 dstport 4789

VM2:

Code: [Select]

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.0.187  netmask 255.255.255.0  broadcast 172.16.0.255

vxlan0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1355
        inet 10.20.40.4  netmask 255.255.255.0  broadcast 10.20.40.255
        vxlan id 42 remote 172.16.0.186 local 172.16.0.187 dev eth0 srcport 0 0 dstport 4789

This works, sanity check complete.
- eth0 vm2: 172.16.0.187 can ping eth0 vm1: 172.16.0.186 and the other way around.
- VXLAN vm2: 10.20.40.4 can ping VXLAN vm1: 10.20.40.3 and the other way around.

-----------------------------------------------------------------------------------------------------------------------------------

Now I thought, I could do it between OPNsense 23.7.4 (FreeBSD) and Ubuntu 22.04.3 LTS. I was wrong it seems. But I don't understand why. Please help me

Code: [Select]

VM1 (vxlan0 id 42 10.30.40.3/24) eth0 172.16.0.186/24 <-----> 172.16.0.254/24 eth0 (10.30.40.1/24 vxlan0 id 42) OPNsense
VM1:

Code: [Select]

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.0.186  netmask 255.255.255.0  broadcast 172.16.0.255

vxlan0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1355
        inet 10.20.40.3  netmask 255.255.255.0  broadcast 10.20.40.3
        vxlan id 42 remote 172.16.0.254 local 172.16.0.186 dev eth0 srcport 0 0 dstport 4789

OPNsense:
(Configured through GUI)

Code: [Select]

hn6: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: hn6_LAN (lan)
options=180018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE,NETMAP>
ether 00:15:5d:00:c9:52
inet 172.16.0.254 netmask 0xffffff00 broadcast 172.16.0.255
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

vxlan2: flags=8803<UP,BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1355
description: vxlan2 (opt6)
options=80020<JUMBO_MTU,LINKSTATE>
ether 58:9c:fc:00:62:52
inet6 fe80::5a9c:fcff:fe00:6252%vxlan2 prefixlen 64 tentative scopeid 0x13
inet 10.20.40.1 netmask 0xffffff00 broadcast 10.20.40.255
groups: vxlan
vxlan vni 42 local 172.16.0.254:4789 remote 172.16.0.186:4789
media: Ethernet autoselect (autoselect <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

OPNsense Firewall rules:

Code: [Select]

root@opn01:~ # pfctl -s rules | grep -i vxlan
scrub on vxlan2 all fragment reassemble
block drop in log on vxlan2 inet6 from fe80::5a9c:fcff:fe00:6252 to any
block drop in log on ! vxlan2 inet from 10.20.40.0/24 to any
pass in log quick on hn6 inet proto udp from any to any port = vxlan keep state label "6937e3837a1e6539757d540ce21cbb7f"
pass in quick on vxlan2 inet all flags S/SA keep state label "33f3a4eb4db2eb24493d44990290c399"

This doesn't work, sanity gone:
- hn6 opnsense: 172.16.0.254 can ping eth0 vm1: 172.16.0.186 and the other way around.
- VXLAN opnsense: 10.20.40.1 CAN'T ping VXLAN vm1: 10.20.40.3 and the other way around.

Packet Capture:
- Ping from 10.20.40.3 (VM1) to 10.20.40.1 (OPNsense) doesn't work.

VM1:

Code: [Select]

root@vm1: tcpdump -i any proto ICMP -n
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
19:53:51.654332 lo    In  IP 10.20.40.3 > 10.20.40.3: ICMP host 10.20.40.1 unreachable, length 92

OPNsense:

Code: [Select]

root@opn01:~ # tcpdump -i hn6 port 4789 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on hn6, link-type EN10MB (Ethernet), capture size 262144 bytes
21:51:58.795397 IP 172.16.0.186.53062 > 172.16.0.254.4789: VXLAN, flags [I] (0x08), vni 42
ARP, Request who-has 10.20.40.1 tell 10.20.40.3, length 28


Code: [Select]

root@opn01:~ # tcpdump -i vxlan2
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vxlan2, link-type EN10MB (Ethernet), capture size 262144 bytes

... NOTHING

Packet Capture:
- Ping from 10.20.40.1 (OPNsense) to 10.20.40.3 (VM1) doesn't work

Code: [Select]

root@opn01:~ # ping -S 10.20.40.1 10.20.40.3
PING 10.20.40.3 (10.20.40.3) from 10.20.40.1: 56 data bytes
ping: sendto: No route to host

-----------------------------------------------------------------------------------------------------------------

Since there was a "No route to host", I have verified the routing tables on OPNsense. They seem to look fine.

Code: [Select]

root@opn01:~ # netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
10.20.40.0/24      link#19            U        vxlan2
10.20.40.1         link#19            UHS         lo0

-------------------------------------------------------------------------------------------------------------------------

I don't know what else to do or where else to look. Either it's broken and really doesn't work, or I don't understand it (more likely). I appreciate any hints.

13

Tutorials and FAQs / [How-To] WireGuard Site-to-Site Setup between two OPNsense

« on: September 27, 2023, 05:02:10 pm »

Now part of the Opnsense Docs: https://docs.opnsense.org/manual/how-tos/wireguard-s2s.html

https://github.com/opnsense/docs/pull/498

Posting this here to prevent link rot when I need to reference it.

14

German - Deutsch / IPv6 Frage - NDP Cache, MaxRtrAdvInterval etc... wenn CARP im Spiel ist.

« on: September 21, 2023, 03:27:32 pm »

Ich hatte vor ein paar Wochen die CARP Dokumentation um IPv6 erweitert: https://docs.opnsense.org/manual/how-tos/carp.html#configuring-carp-with-ipv6

Ich würde jetzt noch gerne das Verhalten besser erklären können, wenn ein Failover mit IPv6 passiert, und benötige dafür mehr Wissen.

Ich interpretiere es so, dass wenn der Failover passiert, die virtuelle MAC der CARP Link Local IPv6 Adresse gleich bleibt. Die Master Rolle geht auf die zweite Firewall über. Das sendende physikalische Interface hat jetzt aber eine andere MAC Adresse, deswegen funktioniert IPv6 nicht mehr, bis die neue MAC Adresse bei den Clients angekommen ist (z.B. durch Router Advertisements). Jetzt kann es also passieren, dass IPv6 für mehrere Minuten nicht richtig funktioniert, bis alle Clients ihren NDP Cache gelöscht haben und neue Router Advertisements erhalten haben (und auch akzeptiert haben).

Zusätzlich gibt es noch die "Unsolicited Neighbor Advertisements" (RFC 2461 - 7.2.6.), würden die in diesem Failover Fall gesendet werden um das aktualisieren des NDP Cache auf den Clients zu beschleunigen?

Oder ist alles vom NDP Cache Timeouts der Clients abhängig, und vom Sendeintervall der "Unsolicited Router Advertisements" (RFC 2461 6.2.4)

Hier würde IPv6 theoretisch so lange nicht funktionieren wenn MaxRtrAdvInterval auf Standard ist:

Code: [Select]

      MaxRtrAdvInterval
                     The maximum time allowed between sending
                     unsolicited multicast Router Advertisements from
                     the interface, in seconds.  MUST be no less than 4
                     seconds and no greater than 1800 seconds.

                     Default: 600 seconds
 

Code: [Select]

     AdvDefaultLifetime
                     The value to be placed in the Router Lifetime field
                     of Router Advertisements sent from the interface,
                     in seconds.  MUST be either zero or between
                     MaxRtrAdvInterval and 9000 seconds.  A value of
                     zero indicates that the router is not to be used as
                     a default router.

                     Default: 3 * MaxRtrAdvInterval
3 * 600 = 1800 Sekunden = 30 Minuten

Würde der Failover von IPv6 geringer ausfallen, wenn man MaxRtrAdvInterval auf 4 Sekunden stellt (Minimaler Wert erlaubt nach RFC)?

Dann wäre es theoretisch 12 Sekunden bis alle Clients wieder funktionieren.

Ich bin für Hinweise dankbar.


Ich hab mal mit GPT4 darüber geredet und das sind die Ergebnisse:

Code: [Select]

In einer CARP-Umgebung (Common Address Redundancy Protocol) gibt es im Kontext von IPv6 einige wichtige Punkte, die du erwähnst, und die sich auf den Failover und das Verhalten der Clients auswirken können. Ich versuche, deine Fragen so präzise und technisch wie möglich zu beantworten:

NDP Cache und Failover: Du hast richtig verstanden, dass bei einem Failover die virtuelle MAC-Adresse konstant bleibt, aber die physikalische MAC-Adresse des sendenden Interfaces sich ändert. Das erfordert ein Update im NDP (Neighbor Discovery Protocol) Cache der Clients, welches durch Router Advertisements oder Unsolicited Neighbor Advertisements beschleunigt werden kann.

Unsolicited Neighbor Advertisements: Diese Pakete werden in der Regel gesendet, um den NDP Cache der Clients schneller zu aktualisieren. In einem Failover-Szenario wären sie äußerst nützlich, um die Zeit bis zur Wiederherstellung der Konnektivität zu minimieren.

MaxRtrAdvInterval und NDP Cache Timeouts: Du hast Recht, dass diese Parameter entscheidend für die Zeit sind, die benötigt wird, um alle Clients wieder zu einem funktionalen Zustand zu bringen. Wenn MaxRtrAdvInterval auf den Standardwert von 600 Sekunden gesetzt ist, dann könnte es theoretisch bis zu 30 Minuten dauern, bis die Konnektivität vollständig wiederhergestellt ist, da das Standard AdvDefaultLifetime 3 * MaxRtrAdvInterval beträgt.

Optimierung durch Anpassung von MaxRtrAdvInterval: Ja, die Reduzierung des MaxRtrAdvInterval auf den minimalen Wert von 4 Sekunden würde den Failover erheblich beschleunigen. Das AdvDefaultLifetime würde dann theoretisch auf 12 Sekunden fallen, was bedeutet, dass die Clients schneller aktualisiert würden.

Beachte jedoch, dass eine zu aggressive Konfiguration von MaxRtrAdvInterval auch Nachteile haben kann, wie eine erhöhte Last auf dem Netzwerk und den beteiligten Geräten.

Insgesamt hängt die Effektivität des Failovers von einer Reihe von Faktoren ab, einschließlich, aber nicht beschränkt auf, die Konfiguration der Router Advertisements, die NDP-Cache-Politik auf den Clients und wie schnell Unsolicited Neighbor Advertisements verschickt werden können. Es könnte sinnvoll sein, diese Parameter in einer Testumgebung zu optimieren und dann die Ergebnisse zu überwachen.

Code: [Select]

User
Würde eine Firewall überhaupt ein Unsolicited Neighbor Advertisement senden? Bei einer CARP Umgebung würde bei einem Hardware Ausfall ja kein "hot swap" der Netzwerkkarte geschehen, sodass auf der gleichen Firewall die MAC Adresse geändert wird. Sondern es gibt einen Failover auf eine andere Hardware. Warum sollte die zweite Firewall das Advertisement senden, an ihr hat sich ja nichts geändert.
ChatGPT
Du bringst einen wichtigen Punkt auf. In einer typischen CARP-Umgebung, bei der die zweite (Backup-) Firewall die Master-Rolle übernimmt, ändert sich die physikalische MAC-Adresse für die zugehörige IPv6-Adresse. Die Backup-Firewall hat keinen Grund, ein Unsolicited Neighbor Advertisement für die vorherige (nun ausgefallene) Master-Firewall zu senden, da sie selbst keine Änderungen erfahren hat.

In der Regel senden Systeme Unsolicited Neighbor Advertisements, um den NDP Cache auf den Clients zu aktualisieren, wenn sich ihre eigenen Link-Layer-Adressinformationen ändern. In einem CARP-Failover-Szenario ändern sich jedoch die Link-Layer-Adressinformationen der virtuellen IP nicht; es ändert sich nur die physikalische Firewall, die die virtuelle IP bedient.

Aus diesem Grund würden Unsolicited Neighbor Advertisements in diesem Kontext nicht automatisch gesendet. Das System, das die Master-Rolle übernimmt, sieht sich selbst nicht als "geändert", auch wenn es die Master-Rolle für die virtuelle IP von einer anderen Firewall übernimmt.

Daher wäre es in einem solchen Szenario wichtig, das Verhalten der NDP Caches der Clients durch andere Mechanismen zu steuern, z. B. durch Anpassung des MaxRtrAdvInterval wie du bereits erwähnt hast. Einige fortgeschrittene CARP-Implementierungen könnten auch Skripte oder spezielle Konfigurationsoptionen bieten, um nach einem Failover gezielt Unsolicited Neighbor Advertisements auszusenden, aber das wäre eine spezielle und keine standardmäßige Funktion.

Code: [Select]

User
Warum senden Clients nicht an die virtuelle MAC Addresse sondern an die physikalische Adresse wenn NDP verwendet wird?
ChatGPT
In einem CARP-Setup (Common Address Redundancy Protocol) wird die virtuelle MAC-Adresse in der Regel nicht für den tatsächlichen Datenverkehr zwischen Client und Server (oder Firewall) verwendet. Stattdessen dient sie als gemeinsame "Anlaufstelle" für ARP- oder NDP-Anfragen, sodass die physikalische Hardware transparent wechseln kann, ohne die IP-Adresse zu ändern.

Im Kontext von IPv6 und dem Neighbor Discovery Protocol (NDP) arbeitet es wie folgt:

Der Client erhält Router Advertisements, die von der Master-Firewall gesendet werden, und fügt die empfangene Information in seinen NDP-Cache ein. Die Information enthält die Link-Layer-Adresse (MAC-Adresse) des Routers (der Master-Firewall) für eine bestimmte IPv6-Adresse.

Wenn der Client Daten an eine externe IPv6-Adresse senden möchte, prüft er seinen NDP-Cache, um die entsprechende Link-Layer-Adresse für die IPv6-Adresse des Routers zu finden.

Der Client sendet dann das Paket an die physikalische MAC-Adresse der Master-Firewall, da diese Information im NDP-Cache hinterlegt ist.

Im Falle eines Failovers:

Die Backup-Firewall übernimmt die Master-Rolle und beginnt, Router Advertisements mit ihrer eigenen physikalischen MAC-Adresse zu senden.

Die Clients aktualisieren schließlich ihren NDP-Cache mit der neuen physikalischen MAC-Adresse, sobald sie die neuen Router Advertisements empfangen.

Die virtuelle MAC-Adresse dient also hauptsächlich dazu, den Failover-Prozess zu erleichtern und die Auswahl der Master-Firewall zu steuern. Der eigentliche Datenverkehr erfolgt jedoch über die physikalische MAC-Adresse der jeweiligen aktiven (Master-) Firewall. Das ist der Grund, warum bei einem Failover eine Aktualisierung des NDP-Caches erforderlich ist, um die neue physikalische MAC-Adresse aufzunehmen.

15

23.7 Legacy Series / IPsec and State Synchronization - unexpected behavior

« on: September 15, 2023, 09:08:25 am »

I had this weird behavior between two OPNsense in HA while using IPsec (between DEC hardware and VM with pcie passthrough, all interface names are the same and theres a lagg)

Quite often, I connected an ikev2 ipsec tunnel, phase 1 and phase 2 were up, but there was no traffic from the opnsense back to the remote peer. It always worked the first time, but the second time it didn't. This behavior mostly affected roadwarrior connections with lots of reconnecting and lesser the site2site tunnels.

My troubleshooting led me to State Synchronization. In Sessions I could also see established TCP sessions even though the tunnel was down.

When I deleted the IP Addresses of the traffic selector (e.g. 192.168.0.0/24) from the state table on both firewalls and restarted the ipsec tunnel, it worked again with Tx and Rx.

To mitigate this behavior:
I created extra firewall rules in Firewall: Rules: IPsec which timed out TCP faster (after 600 seconds).
Then I disabled state syncronisation by setting the "State Type / NO pfsync" parameter for all rules in Firewall: Rules: IPsec.

I didn't come to a conclusion, I just know that my mitigations work and all roadwarriors can connect every time now. It would be interesting to know if that's an expected problem between hardware and vm, or if it could theoretically happen between two hardwares too.