Maxmind vs Logstash's GeoIP Filter

Summary

The Logstash GeoIP filter uses the a maxminddb-formatted database (that may or may not be provided by Maxmind) to populate a number of fields to enrich an event based on an IP address. As presently implemented in the Logstash filter, all fields need to be sub-fields of a single target (e.g, with target => client, we would have client.geo.*, client.as.*, etc.).

Motivation:

Clearly define destinations for all GeoIP Filter fields as a sub-field of a single target, so that users can enable ECS Compatibility Mode without losing metadata that they currently rely on.

Specifically, we have six fields that do not have direct analogues in ECS:

• timezone: the IANA name of the timezone e.g., America/New_York
• postal_code: a string postal code, length varies by country
• continent_code: “A two character continent code like “NA” (North America) or “OC” (Oceania)”
• organization: the name of the business or ISP associated with an address, reportedly available for ~40% of lookups.
• isp: the name of the ISP
• dma_code: us-only code representing Designated Market Area (~metro area)

Many of the existing ECS fields underneath geo.* are named aligning with Geo*2 like these here. I see some fields, such as timezone, being a good candidates as ECS additions, but I’m not so sure about others, such as the US-specific dma_code.

– @ebeahan

Detailed Design:

1. propose the addition of specs to ECS for
   • geo.timezone (IANA name, presently up to 30 characters, e.g., America/Argentina/Buenos_Aires),
   • geo.postal_code (freeform but relatively space limited, regulated by each country; see wikipedia),
   • geo.continent_code (docs define AF, AN, AS, EU, NA, OC, and SA)
2. guidance for namespacing the other fields so that they are usable and unlikely to present future conflict, bearing in mind that they must be sub-fields of the singular target that is the parent of the related geo and as fields.
   • (a) under a mmdb sub-key? e.g, ${target}.mmdb.organization
     • since the mmdb being used may or may not be provided by Maxmind, I’d like to avoid explicitly using “maxmind” in the key name
     • dma_code, while provided in mmdb, is a US-only Nielsen Ratings construct and may not be adequately described with a mmdb prefix.
   • (b) some other way?

Related: https://github.com/logstash-plugins/logstash-filter-geoip/issues/163

[nickpeihl]

Thanks for opening this issue @yaauie. Choropleth mapping in the Elastic Maps product could also benefit from some of these fields in ECS. We already support US postal codes and adding support for timezones and continents should also be possible.

cc @elastic/kibana-gis

[ebeahan]

We already have a geo.continent_name; did you mean geo.continent_code?

Yes, geo.continent_code. 🤦