Messages

could you post your openvpn configuration?
And don't worry about the response time on this forum it's pretty slow since it's a small project after all where pfsense is way bigger.
But it's not related anyway to the fact it was in a VM or if it's a dedicated device. Nowadays, if your using the latest versions of linux hypervisor or even bhyve you should have any problems.
So is this device or VM exposed directly to internet or do you have something in between?

so no one has ever tried NAT reflection through vpns ? or noone wants to answer me ?

so for now, my tcpdump looks like that when I try to reach from the vm to the same VM with a curl command but going through nat reflection of opnsense

Can someone tell me what's wrong?

Code Select Expand

20:35:00.103266 IP (tos 0x0, ttl 64, id 36664, offset 0, flags [DF], proto TCP (6), length 60)
   random.io.33302 > galene.random.io.http: Flags [S], cksum 0xb243 (incorrect -> 0x53ff), seq 1243147436, win 64240, options [mss 1460,sackOK,TS val 2966215938 ecr 0,nop,wscale 7], length 0
20:35:01.105994 IP (tos 0x0, ttl 64, id 36665, offset 0, flags [DF], proto TCP (6), length 60)
     random.io.33302 > galene.random.io.http: Flags [S], cksum 0xb243 (incorrect -> 0x5014), seq 1243147436, win 64240, options [mss 1460,sackOK,TS val 2966216941 ecr 0,nop,wscale 7], length 0
20:35:03.122008 IP (tos 0x0, ttl 64, id 36666, offset 0, flags [DF], proto TCP (6), length 60)
     random.io.33302 > galene.random.io.http: Flags [S], cksum 0xb243 (incorrect -> 0x4834), seq 1243147436, win 64240, options [mss 1460,sackOK,TS val 2966218957 ecr 0,nop,wscale 7], length 0
20:35:07.346054 IP (tos 0x0, ttl 64, id 36667, offset 0, flags [DF], proto TCP (6), length 60)
     random.io.33302 > galene.random.io.http: Flags [S], cksum 0xb243 (incorrect -> 0x37b4), seq 1243147436, win 64240, options [mss 1460,sackOK,TS val 2966223181 ecr 0,nop,wscale 7], length 0

well my first vpn was from a big provider.
my second one was more for public routing.

So I don't know what was your first vpn provider, but if you already didn't got any connectivity from opensense by pinging from the first interface from the first vpn then you have already a problem with your first vpn link.
Because at each step you should be able to ping from opensense to the outside world. If it's not the case then the problem is in your setup of your vpn tunnel.

I would suggest you that you start from the start but step by step and well staging your test step by step, not after everything is setup. Especially since you really don't need to use RAM for another vm to do that. Even if I can agree that it can be a more secure schematics since qubes is doing it like that.
also you didn't specify what were you pinging.

this is not a problem at all.
you specify in the second vpn the vpn interface in which you want to nest it.
You take of course a new interface for the DHCP and other routing stuffs to link up your hosts and opnsense.
And in the firewall rules from this last interface you of course put rules to redirect packets to your vpn interface.

You may want to deactivate monitoring because I think it's clearly buggy because it makes it change the routing table and so you have loss of connection etc if there is the slightliest slow down.

And then  that's pretty much it.

You need to test of course if you have connectivity.
Don't hesitate to test it from opnsense with ping and select the good interface. Verify always the command line that is selected by the webui because there might be some bugs, I know that I have from time to time. For example if there is any problems with the interface you want to try, the webui will automatically use the general gateway which we don't want.

If you don't have connectivity inside your clients, then you need to verify your gateway orders, and you need to verify if the ciphers is good. Unfortunately you won't have accurate messages about that in the journal of your vpn because depending of the vpn server configuration it might give you an error or not. You need to verify it from opnsense itself which is the primary client. If opnsense doesn't have any conenctivity from this interface, then it is probably because of your cipher being bad. Ping 1.1.1.1 but also the first gateway inline from the vpn tunnel.

see my post of my setup about it:
https://forum.opnsense.org/index.php?topic=22465.0

I've nested tunnels links like you for differents reasons.

Forgot a little thing:
Don't forget to correctly setup your outbound nat on your first vpn itnerface with all the good rules which includes the one for 127 loopback, the one for all destination and the one for port 500

or at least can someone from the team explain how they implemented the NAT  reflection and what's its limitaitons?

hi,

So I have a specific problem which involves NAT Reflection for a vpn link.
This vpn link is not equiped with a private ip but a public routable ip totally open in terms of port.

So an outbound Nat has been activated on the interface associated to this link.
To link my self-hosted VMs, it's going through internal qemu networks linked to the opnsense VM
So there is an interface between the guests and the vpn link interface.

client at 172.20.0.33 -> interface dhcp 172.20.0.31 -> interface openvpn link with the public ip.

I didn't notice at first the NAT reflexion parameters so it was not activated when I did the prot forwarding but even so it has apparently little importance.
The port forwarding is working great.

The problem is the NAT Reflexion. I can't do a curl from inside the client to the public ip (which means reaching the client itself) and so I can't make certbot from let's encrypt work.

I'm sure it's the NAT Reflexion since we did some testing and by adding a line to /etc/hosts.conf in this debian VM to the actual domain, I have been able to create a certificate. So the problem is not between the Let'sEncrypt servers and the client but well with the client being able to communicate with itself.

Apparently the parameters when activated, don't do a thing.
I don't see any new rules, I tried to recreate one with the port 80 and 443 but it did the same as before.
I know that some will advise to do splitDNS but before you advise that I suggest you do read about yunohost project.

If you read it carefully you will I think clearly understand why create a dozen of zones files with only one to 3 records in it it's more of a hassle. Plus we should be able to rely on NAT Reflexion too even if some of you don't like it.

So what am I missing here? Which rules or parameters should I activated? Is the problem that here are 2 interfaces here ? or is it something else?

Thanks in advance

okey I've succeeded to find the problem to it.

2 things actually,

• a virtualization problem where there was a problem in applying rules of forwarding packets apparently. There is some differences from what you choose, if it's a bridge or a macvlan interface... Depending of your qemu version and how you configured your different internal network and main interface you could encounter some problems. Some examples of configuraitons :https://blog.scottlowe.org/2016/02/09/using-kvm-libvirt-macvtap-interfaces/
• The other thing was the gateways and their priorities. The fact that normally you should be behind the firewall and not in front, organizes the interfaces in specific ways. It's being told in many threads about it. But so it means also that it organizes the priorities of the gateways not in the best way possible for this specific setup.
  In this case, it had kept a LANvm interface which was the secondary interface as the main gateway and active. While the WAN interface was kept at a lower place in term of priority.
  Deleting the LANvm gateway and putting the WAN interface as the first place and deleting other useless potential gateways even deactivated, was teh way to go.

A side note for this second problem, the netflow tool or even the tcpdump was registering packets going through the good vpn interface. I had to ask people on the other end of the link to do a tcpdump to see if packets were coming through and it was not.
So it means that even if tcpdump does show you something, the routing aspect can be much lower in that specific case in the kernel scale and so you won't be able to diagnose this the usual way.


I can also confirm that the vpn link can go through the wireguard tunnel.
That pfsense topology option in the client configuration is actually a server side option as shown in the openvpn documentation https://community.openvpn.net/openvpn/wiki/Topology?__cf_chl_jschl_tk__=6a3f3249ad65cb388ebae5943a61adc70d8c3f72-1618305770-0-AXaURhXgeaZUm_Y8t-UsspsvMRFjMVxNhb2q9pXmpMX8mgSU9GfOTLiZcyTD5Aciv5k7u6KCb6fAqghPymykGiKC-PZoiK71t06XhGvEpblQo1aMLQ3B6uueozRVfvozf5h97_Htn6_fbl_S3q5S-vtdsnScG20W2bl9JxlaGfBswSYn5xRTWCd0sktXhvRmD9UytfNlop6TpQTGvYr7g-cJIkhPsXX602cc4lQVKUJCfyJi7gfQ2R8QtEvPcm05P-86u8q79yTZ4udokKK6a4z9rgJv-hNHp3nJXPnRtNa497gM5QkSq2JXMGxbtpdR1ol2ILlycSqhnM1-8pONPLIwmbW-ARunWuxb1kEqHxYVUcXBdxFDtLGRYjNblHvAezs8YDuaIAtCpFIO4zlgKUM

So apparently it's not necessarily related to opnsense or the firewall settings since it's working from a virtualbox VM ubuntu which is linked to DHCP so through bridge somewhere on the outside LAN. So it must be a windows specific problem which is not using samba/cifs protocol for some reason.

[part one]

hi,

this setup is a bit maybe complex but for me it's quite simple in the idea.

I've setup a virtual opnsense setup on a isolated server on one of my VLANs which needs to act as a gateway for other VMs on the same server.

Among some gateways there are the WAN, an openvpn to a vpn provider, a wireguard connection to a vpn provider, a openvpn to a foundation that provides me a routable public address.

I have problems with that last one.
This openvpn link I want to make it happen through the wireguard link because I don't want them to record my public ip address among other things.
I "think" I succeeded to do that. At least the connection is happening and I'm receiving the public ipv4 and ipv6 that I was supposed to receive.

--------------part one--------------

But my first question is: How can I be sure that it is indeed well executed through the wireguard tunnel already in place?

second question : When I setup the connection I've configured the field of the interface with the wireguard interface. Is that enough ?

The wireguard tunnel has been executed through the excellent script and tutorial here: https://github.com/FingerlessGlov3s/OPNsensePIAWireguard
As you can see, the wireguard tunnel is done through 2 interfaces( the interface that you will assign the new link and the interface of the subnet(s) you want to link to), or 3 if you count the wireguard service.
As you can see also there is an outbound NAT rule for the 500 port for the IKE protocol and for normal usage for all subnets configured in the firewall, which copies both the rules that are automatically generated for WAN.
third question:
I'm not an expert for that the vpn links in that kind of setup so it surprises me that there is a need for a specific port 500 for that since my openvpn setup of the vpn provider didn't need that to make it work. So if someone has documentation or an explanation to give me about that that would be nice  :D

--------------part two--------------

I don't have communication apparently through the tunnel.
I have deactivated from the original script the pull option which might be the reason because then I guess all routes are not in there but I need your insight for that. And I did that because I was using the pull method then it would screwed my wireguard connection, pull it down, and then go back up because of the script put in place and the monitoring but not the openvpn connection of course.

so first question is : is it because of the absence of the pull option and if yes, how to resolve that?

I can't even ping from the firewall itself using the -S switch of the ping command.

I've tried to reproduce the rules fr the outbound NAT as described here: https://forum.opnsense.org/index.php?topic=4979.0

in place of alias I've tried with any and also with the pre-configured alias of the subnet that I was linking to it. You know the ones that are created when you create an interface, you have always the the interface address entry and the one with the name interface net

I've put a rule also in the firewall to let the traffic goes through it.

So what should I provide as logs? as screenshot maybe ? How can I help to make it determine what's the issue here?


We could say that the part of the openvpn link would ressemble to that schematic.
Where the subgateways are there for handling dhcp, routing to the good interface, portforwarding with outside LAN and so also some NAT and can be linked to the appropriate gateways I want for each subnet.
Maybe it's not the way it should be but that's the more appropriate way I could find to compartimentalize.

I then know that the problem does not reside into the subgateways since it is working with all the other gateways. So I wouldn't understand why not for this one unless you explain it to me.
So I must miss something on the ovpn gateway.
I'm pretty sure this is possible since I'm used to qubes and even if it's a bit of a difference since it's handled by VM's for each connection, it means nevertheless that it is possible.

Thanks for all the help you can give me.

by the way a iscsi redirection port is working for example
so it is really seems specific to nfs and cifs and I can't get any ideas why it would be like that

maybe is there any logs that I should copy paste here? I'm not quite sure how to collect them

Are you using the correct protocol for each port? Ports 137 and 138 are UDP iirc

yeah for what I can tell yes.
And if I understand correctly 137-139 is only for netbios correct? So if I specify the address it shouldn't even rely on it correct?
I am going to ask certainly a stupid question here, but Is cifs compatible to go through a NAT? I wouldn't see why theorically but mayeb I missed something about the content of this protocol?

Hi,

So for this specific setup,
opnsense is a VM which act as a Island firewall to isolate this specific lab of VMs, all hosted on the same server.

I'm trying to let through some samba/cifs shares that I've set up on truenas.
The VM are communicating through virtual network to each other and opnsense is the one with the bridging access to the "outside world".

According to the tcpdump of truenas, there is no packets being received from the outside world on port tcp 445. It is being received if I test it through VMs with smbclient.
I can contact the truenas or other VM with my port forwarding rules I created throuhg SSH.
The rules I created for 445,139,138,137 does look exactly the same besides the fact that I've specified the same port as destination and forwarding.

From what I could read of the live log of the firewall the 445 port was contacted and greenlighted.
So I'm a bit out of leads here.
As I said the rules does seem exactly the same than the SSH ones.
So I would like to rule out any misconfiguration of my behalf or maybe any little specifics that I wouldn't know about opnsense.

For example:
- is there any hard rule that would prevent me to forward cifs ports (MS DS)  maybe ?
- or is it something special to do about it

Thanks in advance for all the leads that you could bring me.

just to specify I did  go with a higher level of raid through mdadm on the bare metal server which host then the qcow2 file from opnsense vm.