Firewall>Aliases>URL Tables fetching not working reliably / Any known limits?

[jbattermann]

Good evening,

I am trying to handle and route a bunch of networks differently through my OPNSense firewall and I was glad to see that it supports, in theory, url tables but my problem is that they are not fetched and imho incorrectly are logged as returning a 500 error.

First things first, I tried one URL table(s) alias with one URL and with multiple URLs, it happens either way. By it I mean, URLs that work and whose server answer the requests coming from my opnsense router with the correct content and HTTP 200, OPNSense logs as errorneous like this:

Code: [Select]

2021-10-25T20:06:23 /update_tables.py[33526] error fetching alias url https://workers.mycloudflaredomain.com/?asn=24424&ipv4=yes&details=yes
2021-10-25T20:06:23 /update_tables.py[33526] error fetching alias url https://workers.mycloudflaredomain.com/?asn=24424&ipv4=yes&details=yes [http_code:500]
I am/was watching this particular request come in and the server returning a 200 OK response, however in the OPNSense log it says 500 and it does not import the values.

I am not a 100% certain what's going on here and I am not 'blaming' OPNsense here, I must be missing something.

Has anyone seen something similar before and knows what to do? I can gladly provide the full URL tables list for debug purposes, if needed.

Thanks!
-Joerg

[Fright]

Hi

workers.mycloudflaredomain.com

cloudflare with load balancing?

[benyamin]

^ +1

Can you curl the url from a console?

[jbattermann]

@benyamin: Yep curl requests on the OPNSense itself work just fine
and @Fright: yep.. but also tried several http://asn.blawk.net/ based ones and then went ahead and even started a local IIS instance on my desktop serving the responses / files

[Fright]

@jbattermann
any clue in IIS logs?

yep..

cloudflare may have some bot-protection. need special library for this imho

tried several http://asn.blawk.net/

tried http://asn.blawk.net/24424 - worked fine for me