GEOip

[neggard]

Now when the new GEOip is here and you create a new license key
should the key be used with  geopipupdate 3.1.1 older or newer?

[fluss]

Neither,
when asked "Will this key be used for GeoIP Update?" select No.

Works fine for me anyways.

/fluss

[JohnnyBeee]

Well I am sorry, but I think there may be some bug here.

For "Will this key be used for GeoIP Update?" I selected "No".
I have then set the URL as explained in the help and nothing happens - the last updated timestamp remains empty and I still get the same reminder ("In order to use GeoIP, you need to configure a source in the GeoIP settings tab").
When I enter the URL in a browser I get the zip file alright.
Even when I put the file on my own web server and enter a URL to my own server (in the form of http://192.168.0.10/geolite2.zip) it does not work.

I have version 19.7.10

Please advise.

PS. What is the expected file format? What should be in the zip? What should the format of the CSV file(s) (name, content) be? Can we use other providers? If yes, how?

[marjohn56]

Check to see what's happening. look at message #62 here:


https://forum.opnsense.org/index.php?topic=15409.msg70705#msg70705


Just tried it here on a new instance of OPNsense and it worked first time. The content of the zip file is a folder named 'GeoLite2-Country-CSV_**DATE**


In that folder are a bunch of csv files.


Just to double check, the format of the url should be:


https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-Country-CSV&license_key=YOUR_KEY&suffix=zip

[JohnnyBeee]

Check to see what's happening. look at message #62 here:
>>> Why would I need to try some Python command if any old browser downloads the zip fine with the URL I provide to OPNsense?
Even when I put the file on my own web server and enter a URL to my own server (in the form of http://192.168.0.10/geolite2.zip) it does not work.
If there were any problem with the URL, shouldn't that show in some log?

Just tried it here on a new instance of OPNsense and it worked first time.
>>> Good. However shouldn't it work for existing installations?

The content of the zip file is a folder named 'GeoLite2-Country-CSV_**DATE**
>>> Yep, the zip downloaded using my web browser (or postman for that matter) contains what you say (folder GeoLite2-Country-CSV_20200128 with a bunch of CSV files). But again, it should work with OPNsense or, if any problems, show some meaningful error.



In that folder are a bunch of csv files.

[marjohn56]

>>> Why would I need to try some Python command if any old browser downloads the zip fine with the URL I provide to OPNsense?

Strangely enough so we can debug what the issue is?

>>> Good. However shouldn't it work for existing installations?

It does, on both my live and test systems. The new instance was a VM install as an added test.

>>> Yep, the zip downloaded using my web browser (or postman for that matter) contains what you say (folder GeoLite2-Country-CSV_20200128 with a bunch of CSV files). But again, it should work with OPNsense or, if any problems, show some meaningful error.

Sod the postman he's useless. I'm going to go and generate a new key selecting 'No' as the key option and test again.

[marjohn56]

Here's another test for you, and I just generated a new key and  that also works. Do the following, it will give more info is there is an error in the link. You can go and have a look at the test.zip too.


From the shell:

Code: [Select]

curl "https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-Country-CSV&license_key=YOUR_KEY&suffix=zip" --output /tmp/test.zip

[JohnnyBeee]

I am not obtuse to testing. Only if two other pieces of other software manage fine where OPNsense does not, I think I have quite a good idea of where the problem is.

I tried your curl test and first it complained about the certificate
curl "https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-Country-CSV&license_key=myKey&suffix=zip" --output d:\temp\test.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (60) SSL certificate problem: unable to get local issuer certificate

When I used option --insecure it downloaded the zip fine.

In my humble opinion we have established that my URL is good.

So why does my URL not work in OPNsense?

[marjohn56]

Can you run this and tell me what you get.


openssl s_client -servername download.maxmind.com -connect download.maxmind.com:443

[JohnnyBeee]

I ran this on my OPNsense box. Btw I ran the curl command from my Windows PC, not knowing how to to run it on the OPNsense box

root@OPNsense:/usr/bin # openssl s_client -servername download.maxmind.com -connect
download.maxmind.com:443
CONNECTED(00000003)
depth=3 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTru
st External CA Root
verify return:1
depth=2 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN
= COMODO RSA Certification Authority
verify return:1
depth=1 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN
= COMODO RSA Organization Validation Secure Server CA
verify return:1
depth=0 C = US, postalCode = 02451, ST = MA, L = Waltham, street = "14 Spring St
reet, 3rd Floor", O = MaxMind Inc., OU = PremiumSSL Wildcard, CN = *.maxmind.com
verify return:1
---
Certificate chain
 0 s:/C=US/postalCode=02451/ST=MA/L=Waltham/street=14 Spring Street, 3rd Floor/O
=MaxMind Inc./OU=PremiumSSL Wildcard/CN=*.maxmind.com
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Org
anization Validation Secure Server CA
 1 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Org
anization Validation Secure Server CA
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Cer
tification Authority
 2 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Cer
tification Authority
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External C
A Root
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIHOjCCBiKgAwIBAgIRAJ5jJD2GDb3mI4b4qgp1HYUwDQYJKoZIhvcNAQELBQAw
gZYxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAO
BgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1pdGVkMTwwOgYD
VQQDEzNDT01PRE8gUlNBIE9yZ2FuaXphdGlvbiBWYWxpZGF0aW9uIFNlY3VyZSBT
ZXJ2ZXIgQ0EwHhcNMTgxMDE1MDAwMDAwWhcNMjAxMTA2MjM1OTU5WjCBrzELMAkG
A1UEBhMCVVMxDjAMBgNVBBETBTAyNDUxMQswCQYDVQQIEwJNQTEQMA4GA1UEBxMH
V2FsdGhhbTEkMCIGA1UECRMbMTQgU3ByaW5nIFN0cmVldCwgM3JkIEZsb29yMRUw
EwYDVQQKEwxNYXhNaW5kIEluYy4xHDAaBgNVBAsTE1ByZW1pdW1TU0wgV2lsZGNh
cmQxFjAUBgNVBAMMDSoubWF4bWluZC5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IB
DwAwggEKAoIBAQCqPNNcIK5N5kOHqtcUH/yh431Ldk+t/XiOoXlmqwot2TcfnSLl
jjk4h2CZfmpfkuWnJugaUd8cB4CFtG0UImgBpq7HHqx9lL+KarKKCGogXuXXvFmv
16abf/lW5CSBzN3A3kzLyWpn8GXnsXryn238tbOK7OU6c91DABWwrGk7hrmF7ZpJ
onPfA+1fm9fNN8GjBHPpS2keEJzwOQ+nU0DTOWzVdpwlaGwUooSe/fD92xFmmP6N
FilwT+w56WNk0SdiZLZ2uGRBs0Q4aDtqEU0YlmnHEj8GxA1fM7yfdc7Wc5bv8CYM
CEESyjjwThFUeS+HkiNPUK0QMWpUrjkzh1V/AgMBAAGjggNmMIIDYjAfBgNVHSME
GDAWgBSa8yvaz61Pti+7KkhIKhK3G0LBJDAdBgNVHQ4EFgQU7VEtw4oNtgdhk+uS
KBQjot3MtJcwDgYDVR0PAQH/BAQDAgWgMAwGA1UdEwEB/wQCMAAwHQYDVR0lBBYw
FAYIKwYBBQUHAwEGCCsGAQUFBwMCMFAGA1UdIARJMEcwOwYMKwYBBAGyMQECAQME
MCswKQYIKwYBBQUHAgEWHWh0dHBzOi8vc2VjdXJlLmNvbW9kby5jb20vQ1BTMAgG
BmeBDAECAjBaBgNVHR8EUzBRME+gTaBLhklodHRwOi8vY3JsLmNvbW9kb2NhLmNv
bS9DT01PRE9SU0FPcmdhbml6YXRpb25WYWxpZGF0aW9uU2VjdXJlU2VydmVyQ0Eu
Y3JsMIGLBggrBgEFBQcBAQR/MH0wVQYIKwYBBQUHMAKGSWh0dHA6Ly9jcnQuY29t
b2RvY2EuY29tL0NPTU9ET1JTQU9yZ2FuaXphdGlvblZhbGlkYXRpb25TZWN1cmVT
ZXJ2ZXJDQS5jcnQwJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLmNvbW9kb2NhLmNv
bTAlBgNVHREEHjAcgg0qLm1heG1pbmQuY29tggttYXhtaW5kLmNvbTCCAX4GCisG
AQQB1nkCBAIEggFuBIIBagFoAHYA7ku9t3XOYLrhQmkfq+GeZqMPfl+wctiDAMR7
iXqo/csAAAFmegz00wAABAMARzBFAiEA4u4V9EYdaY6BSjD5FZ2GOnGGTR57/ip3
HvxNpUVF9QcCIBP7dcXVJlCZwYnb8qinI9Tq2nDIzOMEyoT8tmSkikeAAHYAXqdz
+d9WwOe1Nkh90EngMnqRmgyEoRIShBh1loFxRVgAAAFmegz1IAAABAMARzBFAiEA
1z/e7SEm182ePhrsAJaKcdexDIibFXRRGxHos2PsrcgCIH6Jch9v0Muo88HeTbrE
ieoLvgdPP/ZG/3/3iQEiP8jXAHYAVYHUwhaQNgFK6gubVzxT8MDkOHhwJQgXL6Oq
HQcT0wwAAAFmegz0+AAABAMARzBFAiATZQNqHQet2/xIWnwIdCh4jyNrafJG0NqJ
CJJ7qH+qFgIhAIW/YF2O2yqxTRTz1zGsoCTX4PFgrwvpltdOzepPWH0vMA0GCSqG
SIb3DQEBCwUAA4IBAQAw+ZMSf/U/cGvj7G47WZgarJhIcmpu+3A0RUc/XWOUCsGr
qS5UdrdQZvi23Gh7Wv8tx+dn+p2MpV5SHs6Bay0Vm9WERJEaad0zQwpCUQyq/2FA
+3sobhXZg/U6hjdND25M0RLVp9lizrdDn6STB0VOGLew72Osx0LupeoushKDINSd
s1Yc6vm3Cnz6jMOdZ3dOqSwiZVkWMtELn7f6ldQjTdl/TQhfP1IHQSN2K8EwZ7N3
+9NDhMkW6HrZ0seyiyvp6t7C4Ir1ZGD+InfiSdNBADAKekRpcR1bioT461VWmYq+
gO8hTaIimX+P6Pujx9+5IaKV1wbtmxtBojvR/NJ6
-----END CERTIFICATE-----
subject=/C=US/postalCode=02451/ST=MA/L=Waltham/street=14 Spring Street, 3rd Floo
r/O=MaxMind Inc./OU=PremiumSSL Wildcard/CN=*.maxmind.com
issuer=/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA O
rganization Validation Secure Server CA
---
No client certificate CA names sent
Peer signing digest: SHA256
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 5486 bytes and written 462 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: A78C099CC1D54D091EFEA4ED4998AE8E45DEF5F5521A195A46B21A8B048A338E
    Session-ID-ctx:
    Master-Key: 676EFEE532159A728376DD782D0A1AD0655C52F6DC3EF53D633F75980A3F8769
7619BACE1FF0CD5F7D864B5B3D9665D5
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 64800 (seconds)
    TLS session ticket:
    0000 - a3 62 c2 c9 02 30 30 16-73 b1 ca 15 4a 77 bf 8a   .b...00.s...Jw..
    0010 - 07 1f 2f 9c cf a8 5d 8a-ec 30 c4 ee 20 b3 0b 08   ../...]..0.. ...
    0020 - c0 41 a7 c3 73 34 d0 1d-1c 6c b0 d5 af 64 6d 13   .A..s4...l...dm.
    0030 - 9b 10 35 44 d7 69 e3 df-d7 78 cf cf e3 cc ed 44   ..5D.i...x.....D
    0040 - e9 c8 85 5e e5 d8 8b 25-6f f6 bf 69 69 f3 83 32   ...^...%o..ii..2
    0050 - d4 3f 44 88 2c 0c 05 d6-fb 2c 4a d3 1c 43 6b 90   .?D.,....,J..Ck.
    0060 - 58 fa 04 af 55 d6 01 a9-d3 a8 4b 43 bf a5 73 d8   X...U.....KC..s.
    0070 - 58 2b e6 61 b1 e7 64 ce-96 6e 3b 4a 52 db bf 8b   X+.a..d..n;JR...
    0080 - 51 2b a7 46 03 81 9e 55-09 84 3e 49 92 40 42 64   Q+.F...U..>I.@Bd
    0090 - 0e a6 de b2 23 6a 86 2f-a4 03 98 51 71 52 b3 7e   ....#j./...QqR.~
    00a0 - 3a ea 59 fc dc fe 83 23-15 af b1 aa da 6f d0 09   :.Y....#.....o..

    Start Time: 1580314528
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
closed

[marjohn56]

OK, an issue with /usr/local/etc/ssl/cert.pem may be the cause ( I'm told ).


Run curl again from the shell, we know it's OK when there's no security so don't add --insecure but use the -v flag, it'll give more info.

[JohnnyBeee]

ok, ran curl. Was easier than I thought:

root@OPNsense:/usr/bin # curl "https://download.maxmind.com/app/geoip_download?editi
on_id=GeoLite2-Country-CSV&license_key=MyKey&suffix=zip" --output /tm
p/test.zip -v
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*
  Trying 104.16.38.47:443...
* TCP_NODELAY set
* Connected to download.maxmind.com (104.16.38.47) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /usr/local/etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*
TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [100 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [4824 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; postalCode=02451; ST=MA; L=Waltham; street=14 Spring Street, 3
rd Floor; O=MaxMind Inc.; OU=PremiumSSL Wildcard; CN=*.maxmind.com
*  start date: Oct 15 00:00:00 2018 GMT
*  expire date: Nov  6 23:59:59 2020 GMT
*  subjectAltName: host "download.maxmind.com" matched cert's "*.maxmind.com"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=COMODO CA Limited; CN=COMOD
O RSA Organization Validation Secure Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x46728ea5800)
} [5 bytes data]
> GET /app/geoip_download?edition_id=GeoLite2-Country-CSV&license_key=MyKey&suffix=zip HTTP/2
> Host: download.maxmind.com
> user-agent: curl/7.68.0
> accept: */*
>
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
} [5 bytes data]
< HTTP/2 200
< date: Wed, 29 Jan 2020 18:17:40 GMT
< content-type: application/zip
< content-length: 2031709
< set-cookie: __cfduid=de3cc5371e9ecbec8bca95d4438c4a6a81580321860; expires=Fri,
 28-Feb-20 18:17:40 GMT; path=/; domain=.maxmind.com; HttpOnly; SameSite=Lax
< accept-ranges: bytes
< content-disposition: attachment; filename=GeoLite2-Country-CSV_20200128.zip
< last-modified: Tue, 28 Jan 2020 16:39:16 GMT
< cf-cache-status: DYNAMIC
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-c
gi/beacon/expect-ct"
< server: cloudflare
< cf-ray: 55cd344a4d6fcdcb-CDG
<
{ [1005 bytes data]
100 1984k  100 1984k    0     0  5636k      0 --:--:-- --:--:-- --:--:-- 5620k
* Connection #0 to host download.maxmind.com left intact

So on the OPNsense box curl was happy, not moaning, just downloading.

So could it be someting in/with the zip that OPNsense does not like and therefore does not load the GeoIP data?

Is the downloaded zip different between  "Will this key be used for GeoIP Update?" Yes and No?
Note that the key I used was created with "for GeoIP Update? No"

[marjohn56]

OK, is it still giving you a warning in the GUI?


One last thing, can you run the Python stuff I asked you to try, that will give the final answer. I know you think you should not need to, but it's for a reason.

[JohnnyBeee]

Hi.
As you asked if I were still getting the warning I checked ... and the GeoIP settings were updated !!!
So this does not happen right after we enter the URL and click Apply? Not even during the following hours?
Is OPNsense purposefully waiting until I give up and no longer look to download the zip? 
Is there a scheduler running this?

Anyway, thanks for your help and patience.

[w4rcl0ud]

("In order to use GeoIP, you need to configure a source in the GeoIP settings tab").

I recently reinstalled OPNsense and for some reason this "GeoIP settings tab" is missing? On my previous install I was using the recommended Max Mind process for GeoIP and it was working fine, but it doesn't appear to be an option on the latest build. Can anyone confirm?

Also, I configured the Alias for USA only and configured my firewall rules to block all IP's except for USA, and it appears to be working as I am seeing a bunch blocked connections for that rule (although there are a few USA IP's that are being blocked)

Edit:

Here is a snapshot of the Alias page:


Edit2:

Apparently I didn't run updates - I'm good to go now.