OPNsense Forum
Archive => 19.7 Legacy Series => Topic started by: moellerheide on January 02, 2020, 06:47:24 am
-
Hi there,
GEOIP Service stopt working on Dec 30 / 2019. Obviously maxmind is no more providing the geolite database via
geolite.maxmind.com/download/geoip/database/*
Any hints / solutions to get this fuction working again ?
Best regards
Michael
-
Dev version already has a patch
-
With 19.7.9 you will have to create a maxmind account and Insert your personal link in OPN
-
I use in opnsense 19.7.8 an Alias named GeoIP. Apparently it's populated, according to pftables.
I don't see any option to enter credentials in the Alias definition or do I get this whole thing completely wrong?
-
It still works but you cant download updates to the Geo files.
The input for the link will come in 19.7.9 ;)
-
Ok, I see, many thanks!
Is it a free account (free as in "paid with your telemetry data") or will it cost money? :-)
-
As far as I understood the GitHub ticket, they will have to be capable of notifying you in care someone does not want his IP to be in the database and the record must be deleted. The next update will not contain that data anyway.
-
..so as a hacker I get my IP removed from the list and I'm done? Funny!
Is there no way to get this list compiled from public data available from the countries registrars? Just asking...
-
RIPE lookups get their data from maxmind ;)
-
Have a read of the Maxmind blog
https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/ (https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/)
-
... I applied for an account, but did not receive the email for confirmation. Wait' see...
-
[deleted]
-
...took some time but finally I got an email to my public account and could generate a "Lincense key". Waiting for 19.7.9 to enter it in the GUI.
What bothers me a little is that they want details for invoicing on the login page...
-
Yeah, I noticed that, I put your details in. :P
-
So I assume this explains why I have US IP's being blocked when my rule states to block any country that is not US. So I guess I need to update the firewall and get the Maxmind account rollin'
Oh my 19.7.9 is not out yet. Better temporary disable my GEOIP rule.
-
I didn't see any invoicing. Already got my key, waiting for the OPNsense update now
-
You don't need to enter those details as you're using the Geolite database which is free.
-
As of about 9am this morning my GEOIP tables are no longer populated. Looks like this was the last attempted update from MaxMind. Looking forward to 19.7.9 quickly to fix this.
--Brian
-
Per AD's response on this thread - https://forum.opnsense.org/index.php?topic=15410.0 I installed the patch and setup my Database and plugged in the data to resolve my Geo Issues.
-
Hi,
we're just switching our first devices from pfsense to opnsense.
When will the next version be out, we need GeoIP Aliases...
-
Just wait for the next intermediate release.
-
that means by date?
-
Today or tomorrow
-
Well, by date this means last year:
commit 0229cd54c576ccd0ff8e1b576b0c8d5d34abff56
Author: Ad Schellevis <ad@opnsense.org>
Date: Sun Dec 29 19:29:28 2019 +0100
Where opnsense-patch could be used for the people who do care and want to provide feedback and such and yadda yadda. ;)
Cheers,
Franco
-
Thanks ;D 8)
-
Can I use the GEOIP for more than one opensense or do I need accounts for each box? :-)
-
I just upgraded to 19.7.9, and rebooted. I got my licence key and after figuring out the URL needs to be spelt "license_key" with "s" not a "c" as it gave me an error "Invalid license key", I entered it into the GUI and ....... nothing.
It doesn't update the count and the main Aliases tab just keeps telling me I need to configure for GeoIP.
???
-
I only have a User ID and a License Kex (Maxmind), which URL do You use, I found nothing in the documentation...
https://github.com/opnsense/docs/commit/93f3e9dd41e9c7dfc28c8acae123baa08dcb747c
-
I only have a User ID and a License Kex (Maxmind), which URL do You use, I found nothing in the documentation...
I used "https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-Country-CSV&license_key=<your key here without the angled-brackets>&suffix=zip"
-
If I try to browse
https://download.maxmind.com/app/geoip_download
I get
"Edition ID required"
-
If I try to browse
https://download.maxmind.com/app/geoip_download (https://download.maxmind.com/app/geoip_download)
I get
"Edition ID required"
You must use the whole URL not just the first part, it's all here: https://github.com/opnsense/docs/blob/master/source/manual/how-tos/maxmind_geo_ip.rst
-
If I browse the whole URL including my Licence Key, I get
"Invalid licence key"
Hoooraaayyyy!
-
If I browse the whole URL including my Licence Key, I get
"Invalid licence key"
Hoooraaayyyy!
Did you copy paste it from the document I linked to or mine, because using "licence_key" produces that message, it has to be spelt "license_key" i.e. spelt the American way.
-
Try this:
Login to: https://www.maxmind.com/en/account/login (https://www.maxmind.com/en/account/login)
Next click on the left menu GeoIP2 - Download Files, Next in Download Databases find file which you want and copy link to GZIP, paste this link in OPNsense and this is working :)
-
Try this:
Login to: https://www.maxmind.com/en/account/login (https://www.maxmind.com/en/account/login)
Next click on the left menu GeoIP2 - Download Files, Next in Download Databases find file which you want and copy link to GZIP, paste this link in OPNsense and this is working :)
Those links are date-stamped, so you will never get an updated file using them.You have to use the licence key version of the URL so it gets you the latest file.
-
I generated a new key and now it works in the browser. Yaaaeeyyy!
...but not in the OPNsense, get the window that I have to configure Alias...
-
I generated a new key and now it works in the browser. Yaaaeeyyy!
...but not in the OPNsense, get the window that I have to configure Alias...
Yes, that's where I am stuck as well :(
-
Maxmind updated the updater. Maybe opnsense is outdated? Nice trick...
-
Maxmind updated the updater. Maybe opnsense is outdated? Nice trick...
The latest 19.7.9 was meant to be the update for this.
-
I also just get the message pop-up. Using the URL in a browser works
Reboot also did not work. We need a patch
-
I let it sit there for some time and now:
Last updated
2020-01-06T23:45:56
Total number of ranges
433499
..on the GeoIP page for "Aliases"
***scratchhead***
-
I let it sit there for some time and now:
Last updated
2020-01-06T23:45:56
Total number of ranges
433499
..on the GeoIP page for "Aliases"
***scratchhead***
I left work and just got home and mine has done the same - so it's not quick then, and could need some better feedback for the next release. But at least it works now.
-
So how long did it take. Mine has been sitting for about 45 minutes
-
... checked the first time after 90 min or so.
-
The documentation here is wrong:
https://github.com/opnsense/docs/blob/master/source/manual/how-tos/maxmind_geo_ip.rst
The URL is not licence_key=My_Licence_key&suffix=zip
it shouold be
license_key=My_Licence_key&suffix=zip
-
yeppp, as pointed out some hours ago in this thread ;-)
-
but still doesn't work for me.
Manuel download works, on OPNsense i still get the popup and nothing happens.
-
is it possible to force / debug the download somehow?
-
had the same problem try the other only id (X)
https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-Country-CSV&license_key=XXXXXX&suffix=zip
greeting k0ns0l3 ;)
-
I had the word license misspelled due to a c/p error...note there are a few records written to the log when it fails:
sshd[41635]: error: kex_exchange_identification: Connection closed by remote host
Corrected it and now I'm still waiting (90 mins) for it to try again. A schedule would be handy, or even knowledge of what the default is...perhaps that's documented but I haven't seen it. Also...I'm assuming this data is centralized for ntopng to use as well?
Thanks to the devs for getting this put in! :)
EDIT: Not sure the sshd message is from the GeoIP changes as it occurs every two hours.
EDIT 2: sshd messages are from ntopng network discovery. Also, the GeoIP files are not installed for ntopng...so that's still a manual process after each update, as far as I can tell.
-
I'm waiting for a second install to update the Alias, maybe it get's updated only once a day? Who knows... maybe the documentation...
-
Mine finally updated after 2 hours. That somehow doesn't seem right. At least the devs should put that info in the notice that comes up.
-
Sorry to burst this bubble, but you are all aware that you're being rather unreasonable?
https://twitter.com/opnsense/status/1215380392406069248
-
I have asked for this and the other irrelevant responses to be removed.
-
Frankly, I don't see your point.
If we agree that this is as much your software as everybody else's the trend of "my problem, but not enough of my problem to do something about it but complain a bit" is not helpful.
Meanwhile people update docs, write patches, test and give it enough time to work out.
-
Maybe I wasn't clear so I will reiterate:
We make a conscious choice to not complain about the upstream handling, providing patches and appreciate the help we've gotten to be able to ship this feature relatively quickly.
In turn, it would be nice to have users not complain about technical difficulties directed at us that are out of patch scope or can always be fixed later.
It's just a feature that should be preserved, not the end of the world if this can't be achieved in a day.
Cheers,
Franco
-
It is a brand new feature and it didn't work when it was used and we reported it. My suggestion of putting a note in the message was so others wouldn't also think it was an issue. The message looks like something went wrong.
-
The GeoIP function does not seem to be working. There are no log entries and I use to see a ton of them before the change. Everything looks ok in the GeoIP Settings. Yes, the rule is set to log hits
Is there something I am missing or something else I need to do?
I have the rule set to block all countries except US with an invert. Same rule as before the change. Never get a hit on it and that's just not possible.
Attached is what my floating rule looks like. Please advise if this rule is wrong
-
I can't get it working and unfortunately I don't know what to provide to help figure out the cause. I used the link in my browser and the file downloaded immediately. I've been trying in the GeopIP settings tab for about 14 hours (waiting overnight since I knew some had delays) and still nothing.
-
Did you try to delete the Alias and create a new one? I didn't need to do this, but apparently my Geoblock works. However I don't use invert, but floating (actually 2, one with GeoIP as source and one with destination)...
-
Definitely working here. Rules updated from Maxmind a couple of hours after I corrected my spelling mistake, yes it was me who did the doc, and the rules are working. Just turned on the logging of the inverted, same as dcol, and sure enough log entries started to appear. I normally have it turned off.
Just to confirm, the string should look like this, this is for the non-commercial users:
https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-Country-CSV&license_key= (https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-Country-CSV&license_key=)YOUR_LICENCE_KEY&suffix=zip
It took an hour or so after I changed it for it to update, not sure exactly how it works out the time to poll but its once a day,
You should see the last updated date, which is the file date and the number of entries, at least on mine is 433499 if it has successfully connected and downloaded the data.
-
chemlud, creating a new alias did the trick. Thank you!
-
Here's a way of forcing a download and seeing what is happening.
Go into the shell. Bold chars are what you must enter
root@gateway:~ # cd /usr/local/opnsense/scripts/filter/lib
root@gateway:/usr/local/opnsense/scripts/filter/lib # python3
You will now be seeing the Python interpreter.
>>> from geoip import download_geolite
>>> download_geolite()
Wait a few seconds and if you have got the correct url and licence you should see something like this:
{'address_count': 433499, 'file_count': 499, 'timestamp': '2020-01-06T23:45:56', 'locations_filename': 'GeoLite2-Country-Locations-en.csv', 'address_sources': {'IPv4': 'GeoLite2-Country-Blocks-IPv4.csv', 'IPv6': 'GeoLite2-Country-Blocks-IPv6.csv'}}
Hit Ctrl-d to exit the Python interpreter.
This will download the data, and extract the lists to the /usr/local/share/GeoIP/alias folder and prove that your url is correct. If you run this at anytime it will update the files and you can prove this by looking at the changed date/time on the files in that folder. Note that the free Geolite files are only updated weekly, and according to the Maxmind website this happens on a Tuesday.
-
When I try to delete the old alias it says cannot delete... in use by filter.rule.67/source. I've gone through all my rules in the gui thoroughly and can't find anywhere I missed changing to the new alias. I tried resetting states and reloading pf. It let me disable it but won't let me delete it. Can someone tell me how to figure out what filter.rule.67 is so I can fix this?
-
Here's a way of forcing a download and seeing what is happening.
Thanks very much! I never did get it working yesterday, applied the hotfix just now and manually verified it's correct. Cheers.
-
Here's a way of forcing a download and seeing what is happening.
Thank-you very much for this, works a treat. Checked before and after applying the hotfix and looks fine.
-
Tried recreating rule and alias, still no log entries for GeoIP.
Tried the shell command above and everything is correct. Just not seeing any blocks in the logs. Not sure if it is working or not.
Is there any other way to test this?
-
Enter into your browser
yandex.ru
or
opnsense.org
What happenz?
-
@chemlud It works. Also tried China sites using WebSitePulse and they all work.
So that means the GeoIP is not functioning for me.
-
I would delete the Alias, reboot and establish a fresh Alias. Still not blocking? Did you upgrade to 19.7.9_1? (dunno what got fixed by the latest update though...)
-
Started working after I changed the floating rule to block both directions. Seems most of the blocking was done by IDS already
-
I have two floating rules on all interfaces, one with GeoIP as SOURCE, one with DESTINATION. But I checked now, only the one with SOURCE does fire, if I try to access yandex.ru in the browser. My expectation was that even the traffic from the LAN client (GeoIP as DESTINATION) would be blocked...
-
I found a real disadvantage in using the invert GeoIP floating rule.
For example. I have a GeoIP rule which blocks every country except US and Canada for my Email ports (except 25). Doing this, any local IP's will not be in the Maxmind list so it will be blocked as well.
I am using a floating rule because I have multiple email servers and wanted the same GeoIP blocking for all of them.
So I either have to make a rule which allows all the local ports to pass before the GeoIP rule, or not use invert and have an enormous list in GeoIP.
Floating rules can be tricky to use due to the multiple interfaces and dual direction capabilities.
Any recommendations for the best approach here?
-
One question about Geoip: do I have to subscribe also if I don’t use aliases? Thank you!
-
If you are not using GeoIP rules in the firewall then the answer is no.
-
I have two floating rules on all interfaces, one with GeoIP as SOURCE, one with DESTINATION. But I checked now, only the one with SOURCE does fire, if I try to access yandex.ru in the browser. My expectation was that even the traffic from the LAN client (GeoIP as DESTINATION) would be blocked...
Can someone of the network nerds please comment on this? Where is my mistake in this line of thought? :-)
PS: I created additional block rules on LAN with GeoIP Alias as DESTINATION, but these also do not fire when I try to contact hosts in the blocked regions. But the browser times out while attempting to reach hosts.
-
Transparent Proxy?
-
Nope, plain vanilla:
cabel modem (bridged) - DHCPv4 as WAN on OPNsense (latest x64) - LAN (10.10.10.0/25)
Only thing runnning is Suricata with some rules, but I don't see anything in the Suricata logs...
PS: I did a pcap on the LAN interface and see two outgoing SYN packages to 77.88.55.55 (yandex.ru) when trying to browse yandex.ru and 4 retransmissions. Browser times out, but nothing in the life view for the FW logs (filter for 77.88.55.55).
-
Would be nice if someone could post some GeoIP example rules.
I did seem to get it working, but I think my rules could be cleaner. Since floating rules don't include the local IP's, I have to have a rules for all the local nets before the GeoIP rules. Would be nice to include my own IP list into the Maxmind one.
I have to do this because my GeoIP rule is set to block all counties except US using an invert. See attached.
Is there a better way to handle this?
-
Fun fact: I can't reproduce the non-logging issue for the GeoIP rule on a second machine with same OPNsense version, but the geoip Alias FRESH defined AFTER activating the account.
Would have to delete all geoip rules, delete the geoip Alias, create a fresh Alias and rules and see how logging works then. But too busy currently...
-
I have aswell problems with geoip.
I have done a fresh install of my opnsense router. So I am at OPNsense 19.7.9_1-amd64
I have this url and when I paste it in a browser I get a ZIP file.
https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-Country-CSV&license_key=MYKEY&suffix=zip
I have deleted old alias and created a new geopip alias. When hitting apply I get "In order to use GeoIP, you need to configure a source in the GeoIP settings tab"
The MaxmInd option I choose when generating key are
Will this key be used for GeoIP Update? : YES
[CHECK] Generate a license key and config file for use with geopipupdate version 3.1.1 or newer.
-
You should be able to paste that whole URL into a browser and it should download the zip file.. does it?
-
You should be able to paste that whole URL into a browser and it should download the zip file.. does it?
Yes as I wrote. When I paste it in a webbrowser a zip file is downloaded.
Zipfilename: GeoLite2-Country-CSV_20200121.zip
GeoLite2-Country-CSV_20200121
├── COPYRIGHT.txt
├── GeoLite2-Country-Blocks-IPv4.csv
├── GeoLite2-Country-Blocks-IPv6.csv
├── GeoLite2-Country-Locations-de.csv
├── GeoLite2-Country-Locations-en.csv
├── GeoLite2-Country-Locations-es.csv
├── GeoLite2-Country-Locations-fr.csv
├── GeoLite2-Country-Locations-ja.csv
├── GeoLite2-Country-Locations-pt-BR.csv
├── GeoLite2-Country-Locations-ru.csv
├── GeoLite2-Country-Locations-zh-CN.csv
├── LICENSE.txt
└── README.txt
I tried the recommended way to force download with recommended python 3 way but it immediately exited with:
{'address_count': 0, 'file_count': 0, 'timestamp': None, 'locations_filename': None, 'address_sources': {'IPv4': None, 'IPv6': None}}
Hmm Noticed that I get "invalid license key" in curl/other browsers where I didn't sign up for maxmind login.
-
Hmm.
Well generated a new Key at Maxmind but chose NO on "Will this key be used for GeoIP Update? " when generating key.
Which works for me now.
-
OK.. Think we need to change the docs again and advise to select no when selecting the licence type. :)
-
OK.. Think we need to change the docs again and advise to select no when selecting the licence type. :)
have the same problem but selecting "no" when i asked "Will this key be used for GeoIP Update?" doesn´t work for me.
the link itself works but not for opnsense. it always asked me to fill in the url which i did. but no success.
-
Take the url and use curl in the shell
curl -v url
see what happens.
-
Take the url and use curl in the shell
curl -v url
see what happens.
in the router shell i get an error: license_key=xxxxxxxxxxx: Command not found.
-
added a screenshot for more details
-
Look like curl is not installed. Just type curl, do you get a help prompt?
If not pkg install curl
-
Look like curl is not installed. Just type curl, do you get a help prompt?
If not pkg install curl
yes
-
Oops.. put your curl string in quotes i.e. curl -v "url"
-
Oops.. put your curl string in quotes i.e. curl -v "url"
doesnt work.- got a warning. see screenshot
-
Yes, that's working. It stops because its trying to output the binary file to the terminal. You would normally save it to file, but not interested in that. I think what it is maybe the internal timer that only calls the update once a day. Try the python commands I posted earlier in the thread and see if that unblocks it. Run the commands then go back to the Alias page and see if its updated. Look at msg #62 in this thread.
-
Yes, that's working. It stops because its trying to output the binary file to the terminal. You would normally save it to file, but not interested in that. I think what it is maybe the internal timer that only calls the update once a day. Try the python commands I posted earlier in the thread and see if that unblocks it. Run the commands then go back to the Alias page and see if its updated. Look at msg #62 in this thread.
yes, perfect. this works!! great support! thx for your help!
regards
rené
-
wouahhhh : your last update break all my rules with geoip !!!
Professional ?
not sure.
-
It does not break any rules. You have to have a key from Maxmind to use Geoip, it was not a choice made by the Opnsense devs, it was a fact of life. Once you have that set up, it works as it always did.
-
I think I may have discovered another reason why the GeoIP alias is not importing definitions after putting in a correct URL with a valid key from MaxMind.
During my setup I made another 'admin' user and disabled the 'root' user. As long as my root user was disabled I couldn't make the forced update through the shell and python3 working. I kept getting an error about permissions.
This made me think and enable the root user. Then switched to root user in shell and sure enough the forced update through python3 works like a charm.
Don't know if it will keep working (e.g. updating) when I disable the root user again.
But in my opinion this should also be possible with a disabled 'root' user, as long as the import/update can be done with another user in de admin group.
Best regards,
Joris.
-
Here's a way of forcing a download and seeing what is happening.
Go into the shell. Bold chars are what you must enter
root@gateway:~ # cd /usr/local/opnsense/scripts/filter/lib
root@gateway:/usr/local/opnsense/scripts/filter/lib # python3
You will now be seeing the Python interpreter.
>>> from geoip import download_geolite
>>> download_geolite()
Wait a few seconds and if you have got the correct url and licence you should see something like this:
{'address_count': 433499, 'file_count': 499, 'timestamp': '2020-01-06T23:45:56', 'locations_filename': 'GeoLite2-Country-Locations-en.csv', 'address_sources': {'IPv4': 'GeoLite2-Country-Blocks-IPv4.csv', 'IPv6': 'GeoLite2-Country-Blocks-IPv6.csv'}}....
Hi marjohn56,
thanks a lot, this pointed me to the right way. I configured the URL in opnsense with suffix=tar.gz. This URL in Browser was downloading the file very well, but it does not work in opnsense without any message. The python response was "File is not a zip file". So I changed the URL to MaxMInds Permalink to CSV-File, this is a zip. Now, after creating a new alias, my opnsense is blocking GeoIP traffic.
After weeks of head scratching it works!
Again: thanks a lot!
Regards
Roland
-
Here's a way of forcing a download and seeing what is happening.
Go into the shell. Bold chars are what you must enter
root@gateway:~ # cd /usr/local/opnsense/scripts/filter/lib
root@gateway:/usr/local/opnsense/scripts/filter/lib # python3
You will now be seeing the Python interpreter.
>>> from geoip import download_geolite
>>> download_geolite()
Wait a few seconds and if you have got the correct url and licence you should see something like this:
{'address_count': 433499, 'file_count': 499, 'timestamp': '2020-01-06T23:45:56', 'locations_filename': 'GeoLite2-Country-Locations-en.csv', 'address_sources': {'IPv4': 'GeoLite2-Country-Blocks-IPv4.csv', 'IPv6': 'GeoLite2-Country-Blocks-IPv6.csv'}}
Hit Ctrl-d to exit the Python interpreter.
This will download the data, and extract the lists to the /usr/local/share/GeoIP/alias folder and prove that your url is correct. If you run this at anytime it will update the files and you can prove this by looking at the changed date/time on the files in that folder. Note that the free Geolite files are only updated weekly, and according to the Maxmind website this happens on a Tuesday.
Thanks marjohn56, this works ! Do you know if there is an ETA when this will be eventually fixed ?
-
There's nothing to fix. If you just leave it after entering the details it will update... might take a few hours, but it will do so. The 'fix' is just for those with no patience. :)
With GeoIP it only gets updated once a week, updating it every time someone saves the info would really 'p*** ***' maxxmind, so hence the fetch and update is delayed, but it will get called.
-
I must be missing something: I did set up GeoIP Settings as per picture and before today the last update dated back to 3rd of June, so in my case the cron job it's not working... :o
-
It should update weekly, so there's an issue with that trigger. You might want to raise an issue on Github in that case.