Ten years ago I have published my IP ranges generator and few other articles how it can be used to generate mikrotik firewall address lists or Haproxy ACLs This articles gets many comments about missing or wrong IP ranges. Yes, I know about it, but I can’t do anything with that – source data delegated blocks have country assigned by RIRs. Manually doing exceptions is no way to go…
There are few companies doing exact geolocation like Maxmind, author of famous geoip database. Another player which I have found is IPInfo, which is providing free IP to Country database. After registration you can use your token to download it. According to their info, this should provide even better location data than Maxmind’s database. I tried using this database to generate accurate IP ranges for countries.
It seems accurate IPInfo data contains too many small networks. There are just too many subnets smaller than /27 and/or also hosts (/32) for ipv4 range, resulting to huge files. Files bigger than 1MB:
iwik@[avenger]:~/iwik.org/country_ranges$ du -hs *.cidr| sort -h | tail -n 5
908K FR.cidr
1.3M GB.cidr
1.4M DE.cidr
4.4M SA.cidr
6.4M US.cidr
iwik@[avenger]:~/iwik.org/country_ranges$ wc -l US.cidr SA.cidr DE.cidr GB.cidr
391721 US.cidr
254810 SA.cidr
82974 DE.cidr
77065 GB.cidr
For example SA.cidr contains networks like
8.31.163.51/32
8.31.163.54/31
8.31.163.56/31
8.31.163.68/31
8.31.163.70/32 For IPv6 it is even worse. Database contains too much small blocks and is huge. This can’t be used for mikrotik firewall access list, because big lists are slow to import and requires lot of disk space and memory.
iwik@[avenger]:~/iwik.org/country_ranges$ du -hs *.ipv6| sort -h | tail -n 5
3.1M CN.ipv6
3.3M HU.ipv6
3.8M JP.ipv6
15M US.ipv6
18M DE.ipv6
iwik@[avenger]:~/iwik.org/country_ranges$ wc -l DE.ipv6 US.ipv6 JP.ipv6 HU.ipv6
767643 DE.ipv6
656426 US.ipv6
183285 JP.ipv6
141488 HU.ipv6
I have decided to filter out smaller network blocks than /24 for ipv4 and /56 for IPv6. This resulted to more acceptable size, but still quiet big:
iwik@[avenger]:~/iwik.org/country_ranges$ du -hs *.cidr| sort -h | tail -n 5
188K RU.cidr
228K NL.cidr
296K GB.cidr
308K DE.cidr
1.5M US.cidr
iwik@[avenger]:~/iwik.org/country_ranges$ wc -l US.cidr SA.cidr DE.cidr GB.cidr
97220 US.cidr
1177 SA.cidr
20050 DE.cidr
19154 GB.cidr
iwik@[avenger]:~/iwik.org/country_ranges$ du -hs *.ipv6| sort -h | tail -n 5
1.9M NL.ipv6
2.3M DE.ipv6
2.9M CN.ipv6
3.2M JP.ipv6
7.2M US.ipv6
iwik@[avenger]:~/iwik.org/country_ranges$ wc -l US.ipv6 JP.ipv6 CN.ipv6 DE.ipv6
342059 US.ipv6
161283 JP.ipv6
138718 CN.ipv6
105985 DE.ipv6
With US.cidr as largest with nearly ~ 100 000 entries and 340 000 IPv6 networks. Can be used, but still lagre.
I have found mailfud.org is building (converting GeoLite2) to legacy geoip database for xt_geoip module building. This module needs GeoIP-legacy.csv.gz to build, but this csv data can be used for creating CIDR ranges.
Using this as source generates more accurate data as RIRs, but with not so many small networks as IPinfo data. Let’s see
iwik@[avenger]:~/iwik.org/country_ip_ranges$ du -hs *.cidr| sort -h | tail -n 5
296K NL.cidr
404K DE.cidr
448K FR.cidr
508K GB.cidr
1.8M US.cidr
iwik@[avenger]:~/iwik.org/country_ip_ranges$ wc -l US.cidr GB.cidr FR.cidr DE.cidr
111829 US.cidr
31968 GB.cidr
28007 FR.cidr
25358 DE.cidr
iwik@[avenger]:~/iwik.org/country_ip_ranges$ du -hs *.ipv6| sort -h | tail -n 5
340K CA.ipv6
820K NL.ipv6
852K DE.ipv6
908K FI.ipv6
2.3M US.ipv6
iwik@[avenger]:~/iwik.org/country_ip_ranges$ wc -l US.ipv6 FI.ipv6 DE.ipv6 NL.ipv6
111623 US.ipv6
41674 FI.ipv6
40663 DE.ipv6
36146 NL.ipv6
Much better source for IP ranges data, I will try at http://iwik.org/ipcountry using this method.
Code is available at https://github.com/iwikus/country_ip_ranges
If you like it, you can donate monero 🙂
monero:8ASwHRxipxS3eaZ3VnUAXcLYWAX3tgoQaEG9ArEUtQ4XBr1kMcqRbFvCcLBx3HdVWoX2PAKCLsxxAJYLSDKiCF1aAUrwSby
Hi Ivan,
Thank you for the article and for reviewing our free IP to Country database (https://ipinfo.io/products/free-ip-database). We are aware of the issue you mentioned where smaller ranges are resulting in an increase in the database size. This is a data operational issue that we are actively working on. This issue should be resolved soon without impacting the highest quality, accuracy, and reliability we promised, yet the database size will decrease.
However, please understand that the database will likely never have a similar size to comparable IP to country databases available out there. The reason, as you mentioned, is better location data. We stand by our promise of zero compromise and full accuracy of data, even though it is a free database. Other free databases may intentionally compromise on accuracy and use larger range aggregation and nearby location matching which consequentially results in smaller ranges. The idea behind our location data is based on active measurement through networking data (https://ipinfo.io/blog/probe-network-how-we-make-sure-our-data-is-accurate/). Our data accuracy promise will always make it the bigger database of the bunch.
As we continue to work on the database, we would like to keep you in the loop. You have my email, so please send me an email. This database is designed for community and open-source projects. We greatly appreciate user community evaluation and feedback from open-source projects, which we incorporate into our service. We will update you once we have continuously reduced the database size without compromising our data accuracy. Looking forward to hearing from you.
Thank you,
Abdullah
Developer Relations
IPinfo.io