Places/POIs layer

[iandees]

I'm interested in building a layer for "places" derived from OSM. In this case, "places" means places of interest.

Would there be interest in such a layer in layercake? If so, I'd like to use this as a place to discuss it.

---

There are a couple critical decisions to be considered:

1. Points or points+polygons? Points would probably make data consumers' lives easier, but including polygons for large places (airports, parks, shopping malls, etc.) would be really handy for constructing containment hierarchies.
2. Categories? Normalizing OSM tags to category would make data consumers' lives dramatically easier. Should we make our own schema? Use someone else's?
3. IDs? This is probably a stretch goal, but data consumers that need to do more than just render places on a slippy map would probably prefer identifiers that are at least somewhat persistent between node->way merges/splits, node->node moves, etc.

It's probably enough to start with a layer that extracts nodes with tags that match amenity=*, tourism=*, leisure=*, etc. and grow from there.

[jake-low]

Yes, I think this would potentially be a useful layer, and would welcome your collaboration on building it. Do you have a particular use case in mind? In my head there are a few different plausible use cases for a POIs layer:

• Geocoding: POIs could be ingested from Layercake into a database that supports full text search, then used to support searching for POIs by name in an application. The reverse direction (listing POIs near a location, e.g. to find nearby restaurants) can also be useful.
• Urban analysis: things like creating isochrones around supermarkets in order to identify food deserts, or analyzing changes over time in the density of different business types (which neighborhoods are restaurants opening and closing in?), or comparing different regions (what's the most common kind of restaurant cuisine in different cities?)
• Traffic analysis: similar to the above, POI density can be used to estimate how many people per day will be trying to travel to/from a particular area, which could be useful for urban planning (especially of the citizen science flavour where official measured traffic data may not be available to the public).

Regarding schema, POIs are interesting because OSM has such a complex folksonomy to describe them. It makes sense to build a large POIs (or places, I'm undecided on the name) layer as opposed to separate layers for amenities, shops, tourism features, etc, since each of the separate layers wouldn't be very useful on their own.

What I have in mind for the schema is something like this:

• type
• id
• tags
  • amenity
  • craft
  • leisure
  • shop
  • tourism
  • name
  • brand
  • opening_hours
  • website
  • ...
• bbox
• geometry

The amenity, craft, leisure, shop, tourism, etc columns would contain the value of that tag in OSM; in practice most POIs would have at most one of these columns populated, and the rest NULL, but sometimes a single POI can be in multiple of these categories.

Points or points+polygons?

Many POIs are mapped as areas, particularly businesses in standalone buildings (about 25% of amenity=restaurant are ways, not nodes, for example). We should definitely rows in the dataset for POIs mapped as areas, not just those mapped as nodes. As for the geometry column type, I'd lean towards mixed (points + polygons); data consumers can always take the centroid of the polygon or whatever if they want a single coordinate.

Categories? Normalizing OSM tags to category would make data consumers' lives dramatically easier. Should we make our own schema? Use someone else's?

I would prefer to stick to OSM's tags/categories, rather than invent our own or adapt an existing schema. Other projects, such as Overture, already exist which attempt to simplify OSM's complex and messy data model. On the one hand, this sort of simplifies things for data consumers, but on the other hand it means wrangling with a whole bunch of questions, like is a pizzeria a type of Italian restaurant?.

One of Layercake's design goals is to preserve the richness (and messiness) of OSM data, which lets consumers decide how they want to handle things, and avoids Layercake itself needing to deal with the complexity or make opinionated decisions on how to interpret various tag combinations. I think this goal is important both to keep the scope of the project manageable, and to fill a niche that hasn't already been filled by other data distributions.

IDs? This is probably a stretch goal, but data consumers that need to do more than just render places on a slippy map would probably prefer identifiers that are at least somewhat persistent between node->way merges/splits, node->node moves, etc.

I think this is probably out of scope for Layercake. Having worked on the problem of stable IDs for geodata features before, I'm not convinced that the problem is possible to solve in an automated way. Even if you correctly handle everything at the data level (node -> area remapping, etc), there are real-world edge cases. For instance, if a shopping mall with a Starbucks in it closes, but a new Starbucks opens a couple months later across the street, is that the "same" POI, or a new one? What about if a restaurant is sold, and the name and location stay the same but the menu changes completely? Google cares a lot about this issue (since they want a POI's pictures, reviews, etc to follow it around if it moves) and to my knowledge their solution is to incentivize business owners to manage their own listings, and to employ a bunch of customer support staff to help people navigate tricky cases like these.

OSM doesn't have a great solution to this today, but if it did, I would expect it to look similar to Wikidata IDs: tags on OSM objects that point to some stable entry in an external database, where (unlike OSM) the community cares a lot about keeping the ID of the "same" feature stable over time. This would have the advantage of benefiting all OSM data consumers, not just those who are using Layercake.

[ellenhp]

Geocoding: POIs could be ingested from Layercake into a database that supports full text search, then used to support searching for POIs by name in an application. The reverse direction (listing POIs near a location, e.g. to find nearby restaurants) can also be useful.

I would be cautious about designing a schema to facilitate geocoding, because while I don't know what it's like to build a "good" geocoder, I have written a mediocre one myself and I found that approximately half the work is just preprocessing data for indexing. A big part of this preprocessing that may be useful to call out specifically is that geocoders care immensely about a implicit data provided by the location of a POI within the planet's administrative boundaries. Augmenting the addr:* fields of a POI is hard and not something that I think layercake should endeavor to solve, at least from a cursory glance at the project.

[tannewt]

I'm working on a navigation app that could use POIs for showing you the closest X (toilet, starbucks, park, playground). It'd also use it for geocoding.

One way to avoid areas in the data is to only include access points for areas.

I think the simplest benefit is to shrink the amount of data to download and process for this. The highway and building data is huge.

[ianthetechie]

Thanks for starting the discussion @iandees! I have a motivating use case for our geocoder to get this finished within the next month and have already started on it locally ;)

Points or points+polygons? Points would probably make data consumers' lives easier, but including polygons for large places (airports, parks, shopping malls, etc.) would be really handy for constructing containment hierarchies.

I'd lean toward keeping the original geom rather than collapsing to points:

• It enables use cases like you mentioned where you want to do containment checks
• It's not going to make consumers' lives that much easier, is it? Curious what other people are using stack-wise, but I'd expect most people are using a query engine like DuckDB which can trivially transform (from the user's perspective) a polygon into a point, without us having to guess if they really wanted a centroid or a point on surface or something else.

As for the geometry column type, I'd lean towards mixed (points + polygons); data consumers can always take the centroid of the polygon or whatever if they want a single coordinate.

Agree on this as well @jake-low. Parquet consumers generally don't have the same challenges dealing with mixed geometry layers, and collapsing down / slicing some other way isn't a hard problem.

Schema / categorization

I agree we don't want to lose the components that go into any categorization. I do think it might be useful as an additive thing to include a category, but that's going to add a lot of complexity. I don't mind having this though if someone wants to build it and it's additive. It would probably make certain queries / visualizations easier.

As for which categorization we use, I don't really have an opinion. ID is the first one that pops into my head, and it's pretty well known and used by some other editors (EveryDoor? Probably more?).

Here is my WIP tag set (I'll mention this issue when I start committing):

    COLUMNS = [
        ("name", pyarrow.list_(pyarrow.string())),
        ("names", pyarrow.map_(pyarrow.string(), pyarrow.list_(pyarrow.string()))),
        ("amenity", pyarrow.string()),
        ("building", pyarrow.string()),
        ("shop", pyarrow.string()),
        ("office", pyarrow.string()),
        ("public_transport", pyarrow.string()),
        ("cuisine", pyarrow.string()),
        ("railway", pyarrow.string()),
        ("place", pyarrow.string()),  # e.g. place=square??
        ("sport", pyarrow.string()),
        ("natural", pyarrow.string()),
        ("tourism", pyarrow.string()),
        ("leisure", pyarrow.string()),
        ("historic", pyarrow.string()),
        # These two are extra WIP. I'm thinking of things like apartment complexes. Probably college campuses and other things present similar edge cases: https://www.openstreetmap.org/way/465285242
        # ("landuse", pyarrow.string()),
        # ("residential", pyarrow.string()),
        ("aerialway", pyarrow.string()),
        ("aeroway", pyarrow.string()),
        ("craft", pyarrow.string()),
        ("military", pyarrow.string()),
        ("healthcare", pyarrow.string()),
        ("brand", pyarrow.string()),
        ("opening_hours", pyarrow.string()),
        ("website", pyarrow.string()),
        # TODO: Other contact details
        ("wikipedia", pyarrow.string()),
        ("wikidata", pyarrow.string()),
        # TODO: Routing entrances???
        ("was", pyarrow.map_(pyarrow.string(), pyarrow.string())),
        ("disused", pyarrow.map_(pyarrow.string(), pyarrow.string())),
    ]

Routing entrances??

One way to avoid areas in the data is to only include access points for areas.

@tannewt are you referring to routing / site entrances? https://wiki.openstreetmap.org/wiki/Key:entrance. I think this would also be very useful to capture, but I am not totally sure how yet. I think the most useful thing for a data consumer would probably be to have these as a list (i.e. entrances). This will be a bit of work to process, but the entrances on their own are not very useful without the association back to the POI, and I think this is something we should try to solve in Layercake eventually.

I don't know stats offhand, but I would guess only a small minority of POIs have entrance info; most are just nodes or a way for the building/unit.

Stable IDs?

Yeah, I don't have any great ideas on this; it's a hard problem 🤷

On naming things

Classic hard CS problem :P

I vote for POI (though I recognize this is not what most datasets do; ATP, FSQ OS Places, and Overture are a clear majority). My rationale is basically that it's less confusing. "Places" are a bit of a nebulous concept. Foursquare actually embraces this a bit, and includes things like neighborhoods, "states" and the like. These are places in the vernacular, and OSMers would seem to be doubly confused since place is also an OSM tag that usually refers to a settlement of some sort (which I think is part of the reason that @jake-low eventually changed the name of that layer to settlements in layercake).

POI has none of this baggage.

[mvexel]

On category mapping:

I imagine there is a cozy coffeeshop in hell where folks socialize and try to figure out OSM key/value to POI classes
(me on mastodon, paraphrased)

I don't think we should attempt it, at least not a detailed mapping. I did this for a project I am working on. Basically a grouping of OSM k/v pairs into more general classes like fast food, grocery, specialty food etc. Anything more detailed than that will quickly become opinionated and culturally biased, and thus a source of noise. Someone already mentioned the Overture POI classes which has a sweet 2117 entries (meatball-restaurant anyone?). There is a valiant effort to map each and every one of them to OSM k/v pairs.

Related work by @1ec5 and others on OSM-to-NAICS mapping

Good discussion! Looking forward to helping make it happen.

[tannewt]

@tannewt are you referring to routing / site entrances?

Yup! That's what I'm referring to. I was thinking POIs like parks would have marked entrances. I agree we'd want to tie them to the original feature.

[mvexel]

FYI Overture is overhauling their places taxonomy

It's still a huge sprawling thing but this may be interesting:

Basic Category Mapping. All place categories are now mapped to a "Basic Level Category" (approx. 280 labels like restaurant, museum). Facilitates simplified map iconography, high-level filtering, and cross-taxonomy mapping.

280 basic categories is....still a lot, but I appreciate the effort!

[mvexel]

Perhaps of interest as well: over my extended break from full time work (which came to an end this week) I built a PBF-to-geoparquet cli with flexible yaml-based mapping. This is the POI mapping from the examples: https://codeberg.org/mvexel/cosmo/src/branch/main/examples/pois_annotated.yaml. More general info about the tool here.

It's fairly quick:

[~]$ time cosmo convert -i ~/data/osm-overture-poi-duckdb/utah-260126.osm.pbf -f ~/dev/cosmo/examples/pois_annotated.yaml -o utah-poi.parquet
Pass 1/2: indexing nodes: 24564930
Pass 2/2: blocks: 3321
cosmo convert -i ~/data/osm-overture-poi-duckdb/utah-260126.osm.pbf -f  -o   8.42s user 0.59s system 251% cpu 3.577 total

Depending on file size (or manual override) it will try and choose a node cache smartly.

Result.

This is before I discovered that this is sort-of a solved problem 🤷 :

• https://github.com/GIScience/ohsome-planet?tab=readme-ov-file (history only? aimed at data analysis - stores all tags as a MAP)
• QuackOSM - Python library but has CLI as well, supports filtering, not sure about flexibility)
• Planetiler supports flexible YAML based configuration as well.

So, cosmo is perhaps a solution in search of a problem but I learned some rust I guess.

[sfkeller]

Iandees wrote:

Would there be interest in such a layer in Layercake? If so, I'd like to use this as a place to discuss it.

There definitely is.

The lack of such a POI/Places layer was the main reason why I implemented the experimental student project, Cadence Maps (which is now partially obsolete because Overture Maps abandoned and revamped their categories/taxonomy after just a year...).

As you may know, Fred is thinking about to produce GeoParquet files as part of the well-known download service of Geofabrik: https://community.openstreetmap.org/t/osm-data-in-geoparquet-format/141690 . Note, that this is about downloads and not a GeoParquet delivery service over S3-compatible endpoints.

This means that decisions like ID are already taken (probably to be discussed later if needed). Regarding categories, we're speaking of the one documented in Geofabrik's "Layered GIS Format", which AFAIK contains 21 layers (including "pois"), plus 7 "associated area layers" (e.g. "pois_a"), which complement point with polygon geometries.

The "Layered GIS Format" POI layer has 9 top-level sub-categories (the 2x00-level groupings), including a miscpoi as catch-all. And it seems the schema handles places of worship in the separate layer, pofw.

My take for now regarding categories... I'm convinced that there need to be less than 20 top-level/base categories. So:

• I like the 9 top-level/base categories of Geofabrik's 'Layered GIS Format'.
• I like also the 19 base categories from Martjin, which are "Google-inspired" while still close to OSM main tags: https://gist.github.com/mvexel/2122956734bdb27b7ce7c707f4337711 .
• I dislike the abundance of 280 basic categories of Overture Maps Places (except when it comes to implementing an "pure-OSM" alternative!).

What's next from my standpoint?

• I'm supporting the idea of POI/Places layer in Layercake, in order to experiment with open source pipelines as well as categories, e.g. based on the 19 base categories from Martjin.
• I'm currently going to experiment with a delivery service of Geofabrik's GeoParquet over an S3-compatible endpoint under the working title "Cadence Maps Enhanced".

[sfkeller]

@mvexel / all: Rergarding categories.

Has anyone ever looked at “Place” categories according to schema.org; see also the schema.org docs - or even tried to map OSM tags to these?

Here's a preview of the top-level categories:

Thing
└── Place
    ├── Accommodation
    ├── AdministrativeArea
    ├── CivicStructure
    ├── Landform
    ├── LandmarksOrHistoricalBuildings
    ├── LocalBusiness
    │   ├── FoodEstablishment (z. B. Restaurant, Cafe)
    │   ├── Store (z. B. Einzelhandelsgeschäfte)
    │   ├── LodgingBusiness (z. B. Hotels, Hostels)
    │   ├── ProfessionalService (z. B. Anwaltskanzleien)
    │   ├── HealthAndBeautyBusiness (z. B. Friseure, Fitnessstudios)
    │   └── EmergencyService (z. B. Feuerwehr, Polizei)
    ├── Residence
    ├── TouristAttraction
    └── TouristDestination