Problem opening some websites over webproxy

[scream]

Hi together

I currently have some strange issue and atm I don't have a clue why that happens.

I installed opnsense (17.1.18) on two vm's on a esx host. They are a cluster with carp.
Everything look to work fine so far. I can disable CARP on the master and the backup switches to master and internet with proxy is working fine so far.

The only thing that I figured out today is, that I can't open some sites. As example: http://wikipedia.org

It always stuck on loading and times out. In /var/log/squid/access.log appears no entry like the other one when I hit  enter in my browser. (Tried on Windows 10 with Chrome, Firefox and Edge and on my iPhone with Chrome & Safari).
I tried with Proxy.pac Config also as with static proxy config direct in settings.

After it times out... in the log appears this line:

Code: [Select]

1497203664.529  59957 192.168.1.196 TAG_NONE/503 0 CONNECT www.wikipedia.org:443 - HIER_NONE/- -
Yes, there is a blacklist on (adv, tracker, spyware, porn...) but not sites like wikipedia.

Issue exists even if backup firewall is master so it look like both installs having the same issue.
If I reactivate the proxy on my old pfSense installation all work fine again. Also wikipedia. Same ESX, same network devies, browsers, os etc... also same Blacklist & Config.

Does everyone have a idea how to solve this issue?

If you need some more information to track the issue please contact me so I can try to provide more logs or something like this.

Regards
scream

[scream]

Okay... after a long troubleshooting sesstion today it is working now.. but I don't really understand why this happens. (Maybe there is a bug somewhere?)

First some tests with de.wikipedia.org domain:

Code: [Select]

user@fw1:~ % nslookup de.wikipedia.org
Server:         127.0.0.1
Address:        127.0.0.1#53

Non-authoritative answer:
Name:   de.wikipedia.org
Address: 91.198.174.192
Name:   de.wikipedia.org
Address: 2620:0:862:ed1a::1

So, nslookup looks OK for this case.

So I tried a ping, that fails:

Code: [Select]

user@fw1:~ % ping -t 4 de.wikipedia.org
PING de.wikipedia.org (91.198.174.192): 56 data bytes

--- de.wikipedia.org ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
After this I have tried to get the routing entry for this destination:

Code: [Select]

user@fw1:~ %  route get 91.198.174.192
   route to: text-lb.esams.wikimedia.org
destination: 10.0.8.0
       mask: 10.0.8.2
[b]    gateway: 10.0.8.2[/b]
        fib: 0
  interface: ovpns2
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1375         1         0
What looks strange, because 10.0.8.0/30 is the ovpns2 transitnet!

If i try the same for google.com:

Code: [Select]

user@fw1:~ % nslookup www.google.com
Server:         127.0.0.1
Address:        127.0.0.1#53

Non-authoritative answer:
Name:   www.google.com
Address: 216.58.198.68
Name:   www.google.com
Address: 2a00:1450:400b:802::2004

Google also has a ipv4 and a ipv6 adress, but it is working fine.

Code: [Select]

user@fw1:~ % ping -t 4 www.google.com
PING www.google.com (216.58.198.68): 56 data bytes
64 bytes from 216.58.198.68: icmp_seq=0 ttl=53 time=171.396 ms
64 bytes from 216.58.198.68: icmp_seq=1 ttl=53 time=188.146 ms
64 bytes from 216.58.198.68: icmp_seq=2 ttl=53 time=170.902 ms
64 bytes from 216.58.198.68: icmp_seq=3 ttl=53 time=186.810 ms

--- www.google.com ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 170.902/179.313/188.146/8.180 ms
Also the route shows OK, so it goes to the wan gateway (192.168.0.0/24 is transitnet for wan-link).

Code: [Select]

user@fw1:~ % route get 216.58.198.68
   route to: dub08s02-in-f4.1e100.net
destination: default
       mask: default
    gateway: 192.168.0.1
        fib: 0
  interface: em4
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0
So... I first check the routing table and this looks strange.

Code: [Select]

user@fw1:~ % netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.0.1        UGS         em4
[b]10.0.8.0&0xa000802 10.0.8.2           UGS      ovpns2[/b]
10.0.8.1           link#12            UHS         lo0
10.0.8.2           link#12            UH       ovpns2
10.10.0.0/24       link#2             U           em1
10.10.0.2          link#2             UHS         lo0
10.10.0.4          link#2             UHS         lo0
10.20.0.0/24       link#6             U           em5
10.20.0.2          link#6             UHS         lo0
10.20.0.4          link#6             UHS         lo0
10.99.0.0/24       link#1             U           em0
10.99.0.1          link#1             UHS         lo0
127.0.0.1          link#8             UH          lo0
192.168.0.0/24     link#5             U           em4
192.168.0.11       link#5             UHS         lo0
192.168.0.13       link#5             UHS         lo0
192.168.1.0/24     link#4             U           em3
192.168.1.2        link#4             UHS         lo0
192.168.1.4        link#4             UHS         lo0
192.168.20.0/28    192.168.20.2       UGS      ovpns1
192.168.20.1       link#11            UHS         lo0
192.168.20.2       link#11            UH       ovpns1
192.168.40.0/24    link#3             U           em2
192.168.40.2       link#3             UHS         lo0
192.168.40.4       link#3             UHS         lo0

After assign of interface ovpns2, enable it and configure static ipv4 address from the openvpn (10.0.8.0/30) subnet, the routing is now correct:

Code: [Select]

user@fw1:~ % route get 91.198.174.192
   route to: text-lb.esams.wikimedia.org
destination: default
       mask: default
    gateway: 192.168.0.1
        fib: 0
  interface: em4
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0

Code: [Select]

user@fw1:~ % netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.0.1        UGS         em4
10.10.0.0/24       link#2             U           em1
10.10.0.2          link#2             UHS         lo0
10.10.0.4          link#2             UHS         lo0
10.20.0.0/24       link#6             U           em5
10.20.0.2          link#6             UHS         lo0
10.20.0.4          link#6             UHS         lo0
10.99.0.0/24       link#1             U           em0
10.99.0.1          link#1             UHS         lo0
127.0.0.1          link#8             UH          lo0
192.168.0.0/24     link#5             U           em4
192.168.0.11       link#5             UHS         lo0
192.168.0.13       link#5             UHS         lo0
192.168.1.0/24     link#4             U           em3
192.168.1.2        link#4             UHS         lo0
192.168.1.4        link#4             UHS         lo0
192.168.20.0/28    192.168.20.2       UGS      ovpns1
192.168.20.1       link#11            UHS         lo0
192.168.20.2       link#11            UH       ovpns1
192.168.40.0/24    link#3             U           em2
192.168.40.2       link#3             UHS         lo0
192.168.40.4       link#3             UHS         lo0

I don't really have a idea why this happens ... maybe someone have a idea what's wrong with my config... so there is no config for routing something other than three C-Class private IPs in 192.168.x.x range trought that vpn tunnel (remote networks in openvpn config).

Regards
scream