I am using the latest version of OPNsense, installed as a VM on Proxmox.
My idea is:
• VM1 – OPNsense1: 172.16.0.1/23, LAN1 interface, WAN1 running PPPoE.
• VM2 – OPNsense2: 172.16.0.2/23, LAN2 interface, WAN2 running PPPoE.
• VM3 – OPNsense3: 172.16.0.3/23, LAN3 interface, WAN3 with gateways 172.16.0.1 and 172.16.0.2 to aggregate the networks.
However, I'm running into a problem: when I run a speedtest, the download speed is fine, but the upload speed is stuck at only 0.10 Mbps. If I change the WAN interface to DHCP so that it automatically gets an IP from the DHCP server on OPNsense1, then the upload speed is normal and full bandwidth is available. That workaround works, but the problem is that I have two OPNsense instances running PPPoE and I can't have both acting as DHCP servers.
Do you have any ideas on how to solve this?
You did not show how this is set up topology-wise, but I should probably fear the result... e.g.: are those PPPoE connections to different ISPs (if not, they may have the same gateway, which will break things).
Besides, what you try to do looks like a terrible idea to me. OpnSense can only do its firewall job if it has both a LAN and a WAN interface (Read this first (https://forum.opnsense.org/index.php?topic=42985.0), that is literally point #1 on the list).
Thus, all of the filtering must be done by VM1 and VM2, VM3 cannot do much with it being in the same LAN as VM1, VM2 and all of the clients. So, in order to keep things straight, you need to synchronize the settings of VM1 and VM2 somehow.
OpnSense is well able to handle two WAN connections by itself, so you do not need such a flaky construct with 3 VMs.
Also, using OpnSense on Proxmox has its own problems, see this (https://forum.opnsense.org/index.php?topic=44159.0).
Thank you for replying to my post. I understand the issues you mentioned. Okay, now I will explain in more detail and more specifically.
First of all...
Purpose: I want my clients to be able to change their gateway manually and choose the internet provider they prefer. Therefore, using only one OPNsense instance is not possible, since it only has one interface within the same subnet — in this case, 172.16.0.1/23.
Solution: I have set up a Proxmox server and created 3 virtual machines with OPNsense installed:
VM1 - OPNsense1: WAN1 connects via PPPoE to ISP1, LAN interface is 172.16.0.1/23
VM2 - OPNsense2: WAN2 connects via PPPoE to ISP2, LAN interface is 172.16.0.2/23
VM3 - OPNsense3: LAN interface is 172.16.0.3/23, and it has two additional interfaces on the same vmbr0 bridge as the LAN, used to access the gateways from VM1 and VM2
I added gateways 172.16.0.1 and 172.16.0.2 into OPNsense3 to use the Gateway Group feature (to combine both networks and increase bandwidth).
With that setup, I created a grouped gateway at Tier 1, with interface 172.16.0.3 routing through the grouped gateway.
When a client wants higher download speed, they can point their gateway to 172.16.0.3.
If they want to use ISP1, they can manually set their gateway to 172.16.0.1.
Similarly, if they prefer ISP2, they can change their gateway to 172.16.0.2.
Do you get what I mean?
I get what you want to achieve, but I believe that it does not work that way:
I never tested such a thing, but I do not think that it is feasible: How do you expect that a single client could use more bandwidth? Any TCP stream is limited to one route. You cannot push one packet over one line and the next one over the other.
The best thing you could achieve is that a single stream goes over one route, so the client could then make use of multiple streams. That in turn does not even work where most people think it does, namely with LAGGs, because the algorithm to distribute the packets usually does not look at the port numbers, so a single client takes only one route, regardless of how many streams it uses.
I have not looked into the gateway groups at all, however. If there is a clever trick that I do not know about, then good for you.
Essentially, I successfully combined two ISPs, and it provided double the bandwidth compared to a 1Gb port.
That test is multithreaded (unless you switch to singlethreaded mode, when you will see that only one ISP is being used), so when you download, you will open up multiple TCP streams. Obviously, VM3 distributes those over the two gateways (I assume you point to the internet gateway IPs from VM3).
IDK how the test works with uploads, however your low speed result suggests that the counterpart blocks that test when multiple sender IPs are being used.
Also: Did you set up the Proxmox network hardware correctly? What results do you get when you use the speedtest plugins from the individual VM1/2?
BTW: This all will probably come to a grinding halt when IPv6 is involved, unless you use a proxy or NATv6.
Sorry, I tested and practiced successfully. Thank you for listening to my thoughts while I was reasoning through it. I managed to get it working, and that image shows the result I achieved