Web Proxy is disabled but I can't reach some websites because they think its on

[sc0ttjm]

I'm using OPNsense 21.4.1-amd64, FreeBSD 12.1-RELEASE-p16-HBSD, OpenSSL 1.1.1k 25 Mar 2021

The Web Proxy service is Disabled, but on some websites I try and access, I get this error message:

Access Denied
You don't have permission to access "http://website.com/" on this server.
Reference #18.6c35068.1625333775.25f6e6e9

I can only replicate this behaviour when using a proxy on another computer, which is why I think it has something to do with the proxy.

Also googling the error suggests that the fix is to disable any proxy service.

To further back this up, I started a premium Proxy trial and setup the proxy on 2 different servers and both could not access these sites either, showing the same error.

I tried enabling the proxy and disabling again on OPNsense, but it makes no difference.

There are quite a few sites we've identified now that are used on a day to day basis for the business but I've been using this one to test as it displays identical behaviour:  https://tui.co.uk

Can anybody help with this please?

[Fright]

hi
https://community.akamai.com/customers/s/article/Why-is-Akamai-blocking-me?language=en_US

[fabian]

Akamai blocks AFAIK IP sources with a bad reputation. Your network could be infected with malicious software that is targeting websites behind Akamai.

[sc0ttjm]

I saw this too which prompted me to try using a proxy to get around it.
I took a dedicated proxy trial so I got my own IP addresses and I tried 10 different IP's from different regions and I got the same result every time.

Whilst using the proxy I could verify that the IP address being presented was different by going to whatsmyip.com but even with the different IP addresses, I kept getting the same result.

I can't see that all of these addresses would be listed with Akamai but their reputation checker link doesn't work when you use any of these addresses, I tried contacting them and they told me they can't do anything as I'm not their customer.  The owner of the websites I'm trying to reach is their customer.

[sc0ttjm]

I've just tested using the Tor Browser on my laptop and I get the same result, so the websites must recognise that your are using a proxy server.

Not knowing how they detect a proxy is being used, I don't know why they think that my servers behind the OPNsense firewall are using a proxy too and therefore denying me access.

If I could work out why they think a proxy is being used, we can hopefully change something on the firewall so they no longer think we're using a proxy and can then access all of these sites again?

I've also tried about 30 different free and paid external proxy services now and they all give the same result so I can't imagine that Akamai is blacklisting every single one of these.

This is turning into a real mystery now.

[chemlud]

Proxy of your provider? But how should that be detected via TOR?

[Fright]

looks like some kind of geo-blocking
I was able to access this site via Netherlands vpn

[sc0ttjm]

We're in an OVHcloud Data Centre and using a purchased IP in their range, so we are not behind a Proxy.

I just tried using a Netherlands free proxy and I still get Access Denied.

VPN might work but I can't really use a VPN on all the RDP Servers in the Data Centre that are behind the OPNsense firewall that can't access these sites, that's why I thought a Proxy would work, just for web traffic.

[chemlud]

UK blocking EU? Brexit thing? Where is your IP located, if you do a IP lookup?

[sc0ttjm]

Hi @chemlud, Thanks for your suggestion, I just browsed to whatsmyip.com directly on the server and it does actually show that the IP location is France. 
I know OVHcloud is French, but we ordered a UK IP and are on a server in their UK Data Centre.
When I try a premium UK Dedicated Proxy and go to whatsmyip.com, it shows the UK as the location but I still get the access denied message so I'm not sure if it is Geo IP now?

[sc0ttjm]

Further update:

It seems to depend on which site you use as to whether you get Paris France or London UK as a location for this IP address.

I just tried this site: https://www.iplocation.net/ which tells me UK London and also that no Proxy is present:

[chemlud]

Hmmm, proxy is blocked, because it's proxy. Native IP is blocked, because its France? How about requesting a native GB IP from the provider and try again.

There are some regular users in the UK (e.g. Marjohn56, Team Rebellion) that have no problems of that kind, I have a hard time imagning reasons for opnsense being involved in the problem. Have you tried a different router?

[sc0ttjm]

Hi @chemlud the IP is definitely UK Based, it seems some sites like whatsmyip.com are reporting the location wrong on the initial page.

When you visit whatsmyip.com and click on "Your IP Address" at the top of the page, it shows the RIPE database lookup and that clearly shows GB and London England as the location so its a mistake on their first page.

I checked again with OVH and it actually says "UK" next to the IP address in the assignment list.

So I still think its being blocked because the site thinks my servers are behind a proxy.

I can replicate the issue anywhere by using a proxy, even on my laptop at home, which is why I think that something on the OPNsense firewall is making the website think it is using a proxy.

I can only replicate this by using a proxy or using my servers that are behind the OPNsense firewall.
If I use no proxy, the website is accessible from anywhere (except OVH behind the OPNsense firewall).

So I need to find out why the sites think a proxy is in use.

[Fright]

I can only replicate this by using a proxy or using my servers that are behind the OPNsense firewall

not necessary. in my case, this site is blocking access anyway (with and without a proxy). the presence of a proxy or the presence of ip-address in the bad-rep list is not the only reason why access can be blocked. I think it is worth contacting the support of the site and use the reference ID to clarify the situation

[sc0ttjm]

UPDATE:

I paid a different Proxy Provider for a "Social Media UK Only Proxy" and as soon as I entered the details in the windows proxy settings, I was able to access the sites for the first time from every RDP server behind the firewall.

I really can't explain why after trying over 30 different proxies but I'm just relieved that it is now working!

Thanks for all your help so far!