Help with GeoIP and csv format

[ChrisC]

Hi everyone,
I just learnt of this service: https://ip66.dev/ which purports to be a drop-in replacement for the MaxMind service.
I downloaded the mmdb, and using mmdbinspect against it works fine.
There exists a github project to convert .mmdb files into .csv files, but I have no idea if this would be compatible.
Does anyone know the format of the CSV files that OPNsense will accept for the GeoIP function?
I assume it also has to be compressed? I have seen both .zip and .gz extensions mentioned in URLs.

Thanks!

[meyergru]

Interesting. Their database seems to be bigger at 17 MByte than Maxmind's GeoIP Lite at 11 MByte.

I just emailed them if they could provide a compressed CSV version like Maxmind.

That being said, if you need more accurate data than Maxmind's, which is also free, you can use ipinfo.io, like explained here.

[IPinfo]

I work for IPinfo. We are the most accurate IP geolocation data provider out there. The IP to Country data integration we have with Opnsense is the full accuracy version of it even though it is free. So, if you need country-level geolocation for free, it does not get better than this, which also includes paid data.

Another aspect is that we are active in the community. So, you can always ping me and ask questions about our data. Cheers.

— Abdullah | DevRel, IPinfo

[Patrick M. Hausen]

Hi!

Thanks for joining in the discussion. I use IPinfo with OPNsense, already.

Is the file format identical to Maxmind, so I can use it with software officially only supporting the latter? Like Elastiflow? Just point it at your files instead?

Thanks!
Patrick

[ChrisC]

Interesting. Their database seems to be bigger at 17 MByte than Maxmind's GeoIP Lite at 11 MByte.

I just emailed them if they could provide a compressed CSV version like Maxmind.

That being said, if you need more accurate data than Maxmind's, which is also free, you can use ipinfo.io, like explained here.

I did think about mailing them, but I thought I should at least try and figure out some of it myself!
Thanks, I'll check it out!

[meyergru]

@Patrick: You probably misunderstood, @IPinfo is not with IP66...

There are tools to extract CSV or JSON data from MMDB files, which seem to be the only ones that IP66 offers, unlike Maxmind and IPinfo.

However, I was unable to extract all data, including ASN with those, and I got errors all over the place during import to OpnSense, so I wrote my own: https://github.com/meyergru/opnsense-ip66

Later, I found the main problem is that for IPinfo-style .gz files, you need to send a Content-Disposition HTTP header - if you fail to do that, the file will get misinterpreted as a ZIP file in Maxmind format (with multiple files contained in it).

The number of entries is approximately (all have IPv4 and IPv6):

1. IPinfo: 3900000
2. IP66: 1200000
3. Maxmind (Geoip Lite): 1100000

So IPinfo has the clear lead in terms of detail.

[Patrick M. Hausen]

@meyergru If you run your own local Elastiflow instance you can enrich the data with Maxmind's GeoIP info by configuring the flow collector like so:

Code Select Expand

EF_PROCESSOR_ENRICH_IPADDR_MAXMIND_GEOIP_ENABLE: "true"
EF_PROCESSOR_ENRICH_IPADDR_MAXMIND_ASN_PATH: /etc/elastiflow/maxmind/GeoLite2-ASN.mmdb

As you can see the configuration points to the full path to the database file. The setup of acquiring that file and regular updates is completely outside of Elastiflow and in my case handled by the geoipupdate package on Ubuntu.

Question is: does IPinfo use the same file format and can I replace the Maxmind database with the IPinfo one?

If not, are there tools to convert?

I don't understand why you bring in IP66 - I don't even know what that is ;-)

Kind regards,
Patrick

[meyergru]

Because ip66.dev is what the OP mentioned. IPinfo was thrown into the mix only after that.

As I said, I only though that the ASN columns were needed. In fact, they are not from the GeoIP data at all. It was all down to having to set the HTTP header if you want to coerce OpnSense to read a .csv.gz file like the one from IPinfo.