[RESOLVED] opnsense blocking sshv2 and http from inside guest to outside host

[blainer]

I have searched the forum already and not found any clues as to what maybe causing this problem.  I have also searched the pfsense forums and found nothing.  I am going to try and explain this and provide as much documentation as possible, but if I forget something please let me know.

Here is a brief over view of the network topology that I am working with here.


LAN [ em0 (10.254.0.0) / opt1 on em0 VLAN 99 - (192.168.200.2) ] --> HP SWITCH (Layer 2) Trunk Port [VLANS 1 (UT), 99-104,108,112,116(T)] <-- Access Port VLAN 99 (UT) --> CISCO 892 --> MPLS (10.1.0.0/16) --> HP SWITCH DATA CENTER (Layer 2) Trunk Port --> [VLAN 1 (UT), 100-104,108, 112,116 (T) .

At the Data Center I have several devices attached to the HP Switch and some I can access others I cannot.

The switch itself is 10.1.0.2 / SSHv 1.99 good
Observium server on Ubuntu (14.0.4.3) IP-10.1.0.14 SSHv2 unable to connect HTTP 80 unable to connect
Observium server IMPI 10.1.0.4 SSHv 1 good, HTTP and HTTPS 80/443 good
PBX 10.1.3.3 HTTPS: 8089 good.

The problem is the observium server, here is where it gets interesting.

I am able to PING 10.1.0.14, and tracert 10.1.0.14 from inside the OPNSense FW (10.254.0.0/16) but I cannot SSH or access HTTP on 80.   OPSense blocks it as shown in the syslogs.

I can however ssh to OPNSense then SSH to 10.1.0.14 on port 22 fine.

I can also as stated SSH from inside the OPNSense FW to my HP switch running SSHv1.99 fine.

If I go to another site in the MPLS I can ssh to the 10.1.0.14 fine and view the webserver on port 80 as well.

This all happens with the default permit any FW rule for the LAN network.

My vendor has checked my office access on the MPLS and cannot replicate the problem. My Data Center Support team says that there is nothing wrong with their configuration.  I believe that the problem is something with the way pf is working that is mangling the SSH communication.

I have banged my head against the wall for 3 days trying to figure this out.  It just does not make sense why with the permit any rule that traffic to this host is being denied through the default deny rule.

If I create a rule for this host 10.1.0.14 only it is still being blocked.

Attached is my OPNSense Config and SSH Logs.

Please help my sanity is running low.

[AdSchellevis]

Hi Blaine,

Can you try to create some packet captures from the traffic coming from and to that specific host? (diag_packet_capture.php)

Looking at your ssh log, it looks like it's connecting to something and then receives an invalid response from the other side.
If I'm understanding your issue correctly, then all traffic to the IPMI interface on the same machine and network is functioning correctly and the Ubuntu host on the machine doesn't function from your end like it should.

If possible, can you try both http and ssh and capture the lan and the opt traffic (you need to perform the actions twice to do so).

It could be that the mtu size to the MPLS is your issue, in that case you could try to use MSS clamping on the opt interface to lower your mtu.

Regards,

Ad

[blainer]

Thank you for your prompt reply.  Attached is the information you asked for plus putty logs.

[blainer]

Here is some additional info.

[AdSchellevis]

Hi Blaine,

Inspecting the pcap dump, it looks like your host (192.168.100.1) is sending packets with the DF (don't fragment) bit set which then probably don't fit the tunnel to your Ubuntu machine.

Code: [Select]

19:37:59.116826 b0:aa:77:58:cb:98 > 0c:c4:7a:04:cb:25, ethertype IPv4 (0x0800), length 70: (tos 0xc0, ttl 62, id 18653, offset 0, flags [none], proto ICMP (1), length 56)
    192.168.100.1 > 10.1.0.14: ICMP 10.254.8.57 unreachable - need to frag (mtu 1496), length 36
        (tos 0x0, ttl 63, id 50125, offset 0, flags [DF], proto TCP (6), length 1500)
    10.1.0.14.22 > 10.254.8.57.56099: [|tcp]

You could try to enable "IP Do-Not-Fragment" in system_advanced_firewall.php, that should clear the df bit on those packets.

Regards,

Ad

[blainer]

I am happy to report that the problem has been resolved using the suggestions you provided.  I used a tool on my laptop to test the MTU on the given path to the host 10.1.0.14 and it gave an MSS of 1465, so I entered that on the interface connected to the MPLS under MSS Clamping, and enabled the IP Do Not Fragment option on the firewall and it seems to have resolved the issue.

Thank you for all the help Ad, now onto content filtering and VPN... yay!

Sincerely,

Blaine