WAN/LAN Configuration Issue

[Badante]

Full disclosure. I am not a network guy.

We recently got had a 1gb fiber connection installed. When I work straight from the ONT to my laptop I get 900 down so I know the feed coming in is good. When I connect it to the WAN port on my Protectli Vault, running OPNsense as a VM via proxmox, I begin to have issues. I have the LAN going into a Cisco 1gb unmanaged switch and the OPT1 port goes into a google nest wifi that has been blocked from accessing the LAN.


Here is the issue: The LAN connection is awful.


Here is a traceroute from an ethernet connection -


traceroute to 8.8.8.8 (8.8.8., 64 hops max, 52 byte packets
1  * * opnsense.localopnsense (192.168.1.1)  0.734 ms
2  ro1-dsl-74-215-218-1.fuse.net (74.215.218.1)  1.006 ms  1.029 ms  1.120 ms
3  * * *
4  ip-216-68-14-104.static.fuse.net (216.68.14.104)  1.585 ms  1.843 ms  1.731 ms
5  ip-216-68-14-163.static.fuse.net (216.68.14.163)  13.070 ms  13.320 ms  13.625 ms
6  142.250.164.154 (142.250.164.154)  13.713 ms *  13.614 ms
7  * * *
8  * * *
9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * dns.google (8.8.8.  2349.714 ms

and a ping -

PING 8.8.8.8 (8.8.8.: 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
64 bytes from 8.8.8.8: icmp_seq=1 ttl=117 time=1245.318 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=117 time=243.998 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=117 time=13.524 ms
Request timeout for icmp_seq 5
Request timeout for icmp_seq 6
64 bytes from 8.8.8.8: icmp_seq=7 ttl=117 time=21.152 ms
Request timeout for icmp_seq 8
Request timeout for icmp_seq 9
Request timeout for icmp_seq 10
64 bytes from 8.8.8.8: icmp_seq=11 ttl=117 time=13.653 ms
Request timeout for icmp_seq 12
Request timeout for icmp_seq 13
Request timeout for icmp_seq 14
Request timeout for icmp_seq 15
Request timeout for icmp_seq 16
Request timeout for icmp_seq 17
Request timeout for icmp_seq 18
Request timeout for icmp_seq 19
Request timeout for icmp_seq 20
64 bytes from 8.8.8.8: icmp_seq=21 ttl=117 time=13.627 ms
Request timeout for icmp_seq 22
Request timeout for icmp_seq 23
Request timeout for icmp_seq 24
Request timeout for icmp_seq 25
64 bytes from 8.8.8.8: icmp_seq=26 ttl=117 time=13.432 ms
64 bytes from 8.8.8.8: icmp_seq=25 ttl=117 time=2007.462 ms
64 bytes from 8.8.8.8: icmp_seq=27 ttl=117 time=1000.753 ms
Request timeout for icmp_seq 29
Request timeout for icmp_seq 30
Request timeout for icmp_seq 31
Request timeout for icmp_seq 32
Request timeout for icmp_seq 33
Request timeout for icmp_seq 34
Request timeout for icmp_seq 35
Request timeout for icmp_seq 36
Request timeout for icmp_seq 37
Request timeout for icmp_seq 38
64 bytes from 8.8.8.8: icmp_seq=39 ttl=117 time=13.531 ms
Request timeout for icmp_seq 40
Request timeout for icmp_seq 41
Request timeout for icmp_seq 42
Request timeout for icmp_seq 43
Request timeout for icmp_seq 44
Request timeout for icmp_seq 45

64 bytes from 8.8.8.8: icmp_seq=46 ttl=117 time=13.647 ms

Here are the same tests from the WiFi running off OPT1:

traceroute to 8.8.8.8 (8.8.8., 64 hops max, 52 byte packets
1  192.168.86.1 (192.168.86.1)  19.430 ms  10.017 ms  12.583 ms
2  192.168.3.1 (192.168.3.1)  14.380 ms  11.206 ms  14.076 ms
3  ro1-dsl-74-215-218-1.fuse.net (74.215.218.1)  11.925 ms  14.247 ms  13.704 ms
4  * * *
5  ip-216-68-14-104.static.fuse.net (216.68.14.104)  22.634 ms  20.329 ms  15.133 ms
6  ip-216-68-14-163.static.fuse.net (216.68.14.163)  34.238 ms  24.325 ms  25.669 ms
7  142.250.164.154 (142.250.164.154)  20.089 ms  20.809 ms  21.202 ms
8  * * *

9  dns.google (8.8.8.  30.245 ms  28.115 ms  27.046 ms

PING 8.8.8.8 (8.8.8.: 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=116 time=27.262 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=116 time=27.641 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=116 time=22.321 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=116 time=27.553 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=116 time=28.829 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=116 time=27.248 ms
64 bytes from 8.8.8.8: icmp_seq=6 ttl=116 time=26.121 ms
64 bytes from 8.8.8.8: icmp_seq=7 ttl=116 time=20.781 ms
64 bytes from 8.8.8.8: icmp_seq=8 ttl=116 time=24.250 ms
64 bytes from 8.8.8.8: icmp_seq=9 ttl=116 time=26.296 ms
64 bytes from 8.8.8.8: icmp_seq=10 ttl=116 time=26.558 ms
64 bytes from 8.8.8.8: icmp_seq=11 ttl=116 time=26.071 ms
64 bytes from 8.8.8.8: icmp_seq=12 ttl=116 time=22.371 ms
64 bytes from 8.8.8.8: icmp_seq=13 ttl=116 time=25.409 ms
64 bytes from 8.8.8.8: icmp_seq=14 ttl=116 time=26.521 ms
64 bytes from 8.8.8.8: icmp_seq=15 ttl=116 time=26.536 ms
64 bytes from 8.8.8.8: icmp_seq=16 ttl=116 time=21.931 ms
64 bytes from 8.8.8.8: icmp_seq=17 ttl=116 time=26.919 ms
64 bytes from 8.8.8.8: icmp_seq=18 ttl=116 time=25.642 ms
64 bytes from 8.8.8.8: icmp_seq=19 ttl=116 time=26.383 ms
64 bytes from 8.8.8.8: icmp_seq=20 ttl=116 time=27.860 ms
64 bytes from 8.8.8.8: icmp_seq=21 ttl=116 time=20.671 ms
64 bytes from 8.8.8.8: icmp_seq=22 ttl=116 time=25.843 ms
64 bytes from 8.8.8.8: icmp_seq=23 ttl=116 time=26.701 ms
64 bytes from 8.8.8.8: icmp_seq=24 ttl=116 time=27.554 ms
64 bytes from 8.8.8.8: icmp_seq=25 ttl=116 time=26.999 ms
64 bytes from 8.8.8.8: icmp_seq=26 ttl=116 time=25.575 ms
64 bytes from 8.8.8.8: icmp_seq=27 ttl=116 time=19.702 ms
64 bytes from 8.8.8.8: icmp_seq=28 ttl=116 time=26.850 ms
64 bytes from 8.8.8.8: icmp_seq=29 ttl=116 time=26.483 ms
64 bytes from 8.8.8.8: icmp_seq=30 ttl=116 time=26.355 ms
64 bytes from 8.8.8.8: icmp_seq=31 ttl=116 time=25.175 ms
64 bytes from 8.8.8.8: icmp_seq=32 ttl=116 time=20.195 ms
64 bytes from 8.8.8.8: icmp_seq=33 ttl=116 time=26.760 ms

64 bytes from 8.8.8.8: icmp_seq=34 ttl=116 time=26.468 ms

I am guessing there is some sort of issue with the setup on the LAN port. I don't know what information someone would need to help me diagnose this. Please let me know what settings or whatnot that you would need. Thank you in advance for your assistance!

[knebb]

Without knowing your exact network structure I would guess there is a duplicate IP somewhere.... or some static arp?

[Badante]

How is best to lay out the network structure to you? My apologies as this is not my bailiwick. What information would you need?

[cookiemonster]

the first hop is opnsense so that looks fine. Seems maybe a problem in settings on either proxmox or between proxmox and your VM. Not much we can tell here regarding the right settings for proxmox for your setup.
In my proxmox setup I am passing through the NICs and have no problems. I am using the qemu guest agent.
I suggest narrow down the problem. What the "issues"?
p.s. these long traceroutes are outside your home. Maybe ISP problems?

[Badante]

Thank you for the reply. I will try to answer as best I can:


these long traceroutes are outside your home. Maybe ISP problems? - I thought the same thing so I took my laptop and connected directly to the ONT and the speeds were blistering. Not an ISP issue, pretty certain.

I am using the qemu guest agent. - I will be honest. Not sure what this is or means. I will see what I can find out.

I attached a network layout (rudimentary).

The issue is that anything connected through the LAN continually drops the connection back to the Vault. I have no problems with OPT4 and reaching proxmox. I cannot reach 192.168.1.1 with any consistency. Here is a ping for 192.168.1.1:

[/size]PING 192.168.1.1 (192.168.1.1): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=3027.576 ms
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=2024.192 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=1019.175 ms
Request timeout for icmp_seq 6
64 bytes from 192.168.1.1: icmp_seq=4 ttl=64 time=3524.296 ms
64 bytes from 192.168.1.1: icmp_seq=5 ttl=64 time=2520.588 ms
64 bytes from 192.168.1.1: icmp_seq=6 ttl=64 time=1515.512 ms
64 bytes from 192.168.1.1: icmp_seq=7 ttl=64 time=507.277 ms
64 bytes from 192.168.1.1: icmp_seq=8 ttl=64 time=0.622 ms
Request timeout for icmp_seq 12
64 bytes from 192.168.1.1: icmp_seq=13 ttl=64 time=0.586 ms
Request timeout for icmp_seq 14
Request timeout for icmp_seq 15
Request timeout for icmp_seq 16
64 bytes from 192.168.1.1: icmp_seq=17 ttl=64 time=0.307 ms
[/size]
[/size]I have also attached my network settings on my Mac, connected to the lan.[/size][/size]What settings from my OPNsense (I am guessing the issues is in the Interface setup?

[knebb]

Ok, good pictures. But where is the OPNSense?

In your first post you wrote it is a VM on proxmox. But I do not see it in your graph. What ist the Protectli running? Why do you need a second router/firewall?

Still do not get it...

[Badante]

The proxmox is installed on the Protectli Vault then OPNsense is a VM in the proxmox. I have disabled the proxmox firewall and will use OPNsense as the router, firewall and the Wireguard VPN. I am sorry that I am not doing a better job explaining things. I do appreciate your help.

[knebb]

Ok, so your Vault is the hardware where Proxmox is installed as hypervisor and OPNSense runs as a virtual machine, right? There is no firewalling/ routing done one the Vault except the OPNSense?

In this case I am pretty sure the issue is on the Proxmox which I do not know well enough to troubleshoot.

Make sure you have seperate networks and switches created on Proxmox which are assigned to the different OPNSense LAN ports. MAke sure the do not interact with each other. Once properly separated I expect it will be much better.

/KNEBB