Everything about OpenStreetMap.

Table of Contents

Data Structure

  1. Metadata: common attributes of OSM entities.
    • id: (not universally guaranteed) unique entity ID within an entity type.
    • visible: indicator of whether the entity is deleted or not; deleted entities are only returned by history calls.
    • version: edit version of the entity, in integers starting from 1.
    • changeset: an integer identifier for a group of changes made by a single user over a short period of time.
    • timestamp: time of the last modification in W3C date and time format (a subset of ISO 8601).
    • user: name of the last editor.
    • uid: OSM id of the last editor, integer.
  2. Type:
    1. Node node: geographic coordinates defining a point.
    2. Way way: an ordered list of node IDs representing a line connecting multiple nodes.
      • Closed way closed way: a way whose ends are identical.
      • Area area: a closed way without tags default to lines (highway, barrier, junction=roundabout) or explicitly tagged with area=yes.
    3. Relation Relation: an ordered list of member entity IDs with optional roles that defines their logical or geographic relationships.
  3. Tag: key-value pairs describing custom attributes of an entity.
    • Keys can be qualified with prefixes, infixes, or suffixes, forming namespace, e.g lanes:bus:forward.
    • Value can be a number, text, or categorical values delimited by semicolon.

A simplified example of OSM data represented in XML.

<node id="42437959" visible="true" version="12" changeset="13150866"
timestamp="2012-09-18T02:14:34Z" user="RoadGeek_MD99" uid="475877"
lat="40.7229823" lon="-73.9885488">
    <tag k="highway" v="traffic_signals"/>
</node>
<node id="42443513" visible="true" version="5" changeset="5155439"
timestamp="2010-07-07T00:47:51Z" user="Dylan Semler" uid="31855"
lat="40.7232770" lon="-73.9884510">
    <tag k="highway" v="traffic_signals"/>
</node>
<way id="5672851" visible="true" version="37" changeset="41811184"
timestamp="2016-08-30T20:49:09Z" user="infinitesunrise" uid="4434293">
    <nd ref="42437959"/>
    <nd ref="42443513"/>
    <tag k="highway" v="primary"/>
    <tag k="name" v="1st Avenue"/>
    <tag k="oneway" v="yes"/>
</way>
<relation id="1077653" visible="true" version="3" changeset="21582260"
timestamp="2014-04-09T04:29:57Z" user="lxbarth" uid="589596">
    <member type="way" ref="22898643" role="from"/>
    <member type="way" ref="34080174" role="to"/>
    <member type="node" ref="42428201" role="via"/>
    <tag k="restriction" v="only_straight_on"/>
    <tag k="type" v="restriction"/>
</relation>

Tags

OSM Cheat Sheet: tags and keyboard shortcuts.

Names:

  • name, the primary tag used for common names;
  • ref, reference numbers or codes.
  • alt_name, less common names.

Road classification: highway is the main tag (confusingly) used to classify any kind of road, street or path:

  • Restricted access road: motorway (one-directional by default);
    • ramps or motorway junctions: motorway_link (speed depends on curvature).
  • Standard road network: trunk (high capacity road without limited access), primary, secondary, tertiary;
    • link roads: trunk_link, primary_link, secondary_link, tertiary_link.
  • Smaller road: residential (in residential area), unclassified (minor public roads, non-residential);
  • Special type: living_street (slow traffic has absolute right-of-way), service (in private property or parking lots), pedestrian (normally forbidden for motor vehicles), road (classification unknown);
  • Tagged on node: traffic_signals, stop (stop signs), bus_stop, mini_roundabout;
  • May be tagged on areas, i.e. area=yes or type=multipolygon.

Other tags that can classify ways as part of a road network:

  • lanes (lanes:forward, lanes:backward) indicate number of lanes (in a given direction);
  • turn:lanes (left, through, right) and parking:lane indicate turning lanes and parking lanes;
  • bridge, tunnel for bridges and tunnels;
  • junction=roundabout, a single lane roundabout (one-directional);

Restrictions:

  • access additionally describes legal access to an entity via all or particular forms of transport.
  • maxspeed, maximum legal speed limit (default in kilometers per hour).
  • oneway (yes, no, -1).
  • hgv, heavy goods vehicle.

Spatial relation:

  • tagged as is_in;
  • inside an area tagged as place (continent, country, state, region, county, city, town, village, hamlet, suburb, island);
  • inside an area tagged as boundary=administrative and admin_level=8;

barrier, a physical structure that blocks or impedes movement (toll_booth, etc).

Relation

Turn restriction: member roles from, to, via, and tag type=restriction.

  • tag restriction
    • prohibitory: no_right_turn, no_left_turn, no_u_turn, no_straight_on;
    • mandatory: only_right_turn, only_left_turn, only_straight_on;
    • dead end: no_entry, no_exit;
  • tag except, exception by vehicle type.
  • tags for conditional restriction: day_on, day_off, hour_on, hour_off.

Multipolygon: member roles outer, inner, and tag type=multipolygon.

Other relations: route (a collections of ways) type=route, boundary.

File Format

OpenStreetMap.org provides weekly dump of its entire database in text and binary formats. Regional extracts are available and periodically updated from third parties. GeoFabrik.de provides continent, country and sub-country region extracts, also available in Shapefile. BBBike.org provides extracts for cities and regions as shapefile, CSV, SVG, OPL and formats for offline navigation (Garmin, Navit, maps.me, OsmAnd, mapsforge), also offers custom bounding box or polygon extracts within 24M sq km and 768MB file size. BBBike.org extracts set version of all map entities to -1 and all timestamps to 1969, which can cause trouble.

Formats:

  • .osm, an XML format.
  • .pbf (Protocol-buffer Binary Format): the primary binary format based on protocol buffers with <50% the size of XML format.
  • .o5m, a flat binary format that nobody uses.
  • .opl (Object Per Line): a preliminary format created by libosmium where each object is on its own line, about half the size of OSM XML files.

Protocol buffers is a language-neutral, platform-neutral extensible mechanism for serializing structured data. Delta coding and variable-byte coding are applied throughout.

Characteristics of OSM PBF:

  1. multi-level structure: compressed Blobs - PrimitiveBlocks - PrimitiveGroups - OSM entities
  2. mapping of one Protobuf type to each OSM entity type is difficult due to the “dense nodes” data type and parallel arrays in OSM data.
  3. String tables replacing repetitive strings are redundant as Gzip will do an equivalent job.
  4. Separate Protobuf specifications are used for file block structure and OSM data, which requires implementers write auxiliary code.
  5. The most effective Protobuf technique applied in PBF is varints for delta-coded fixed-precision coordinates.

For better compression and processing speed of OSM data, Conveyal invented a binary format called Vanilla Extract eXchange format (.vex).

Tools

Converting map data between OSM and external formats:

  • Import
    • JOSM plugin OpenData for Shapefile, KML, GML, CSV and others.
    • Merkaartor can directly import Shapefile (and write OSM).
    • GPSBabel can write to OSM file (including tags like created_by=GPSBabel-1.5.2).
    • osm-and-geojson for GeoJSON;
    • gml2osm for GML and csv2osm for CSV.
  • Export
    • GDAL/OGR has an OSM driver with read-only support, so ogr2ogr can write OSM data to any supported format.
    • QGIS can convert OSM file into a SpatiaLite DB file, and export selected tags and geometry types.
    • osmtogeojson and OSM2GEO can also export to GeoJSON.

Web Services

OpenStreetMap.org services:

  • Entity viewer via endpoint /<type>/<id>;
  • The OSM Editing API (API v0.6) is a RESTful HTTP API accessible via endpoint /api/0.6;
  • History via suffix /history.
  • iD is the default online map editor for OSM via endpoint /edit;

Overpass API is a read-only API for querying an OpenStreetMap database and extracting features over the web. Overpass Turbo is a Web frontend of Overpass API.

Metro Extracts by Mapzen: bounding box extracts of popular cities and regions as Shapefile and GeoJSON files split by geometry type of features (lines, points, polygons) or by logical groups (Roads, etc) of OpenStreetMap tags (no relation); coastline geometries of each area are also available in Shapefile.

OSM turn restrictions:

Standalone Executables

osmosis is a command line Java application for processing OSM data, which can extract data inside a bounding box or polygon.

osmconvert can convert and process OSM files faster than osmosis but has less functionality; it has a few special functions.

osmium also has a command line tool to convert file format, apply changeset, and extract historical data.

JOSM, Java OpenStreetMap, is the offline editor of OSM data, with many plugins. Features: validator feature can check and fix invalid data. JOSM plugins for fixing missing turn restrictions: ScoutSigns (road signs by Scout users), ImproveOsm (missing geometries and turn restrictions, by Scout), turnrestrictions (by skobbler).

Merkaartor is another OpenStreetMap editor, written in C++.

KeepRight, data consistency checks (quality assurance) for OSM.

PostgreSQL Loader: osm2pgsql; osm2pgrouting, imports OSM topology to PostgreSQL database; other java programs.

SQLite/SpatiaLite for OSM

Libraries & Frameworks

Programming Frameworks for accessing, processing, map rendering (static, interactive), geocoding, and navigation with OpenStreetMap data. Debian GIS Blend meta-package for OpenStreetMap already include many of these tools and libraries.

Accessing:

  • osm-common (Java): accessing, processing, geocoding; supports Overpass.
  • osmapi (Python): Python wrapper for the OpenStreetMap API.
  • osmaR (R): access OpenStreetMap data from file or API, and convert to other classes.
  • Data access APIs for Ruby and PHP are also available.

Data processing / parsing:

  • C/C++: pbf2osm, osmpbf; LibOsm (SQLlite); osmium;
  • GO: Gosmparse, osmpbf;
  • Java: osmosis, osm4j, BasicOSMParser, OSMemory;
  • Python: imposm (import OSM to PostgreSQL/PostGIS), osmread;

Navigation:

  • OSRM (C++), Open Source Routing Machine, used by Mapbox Directions.
  • Valhalla (C++), Mapzen Turn-by-Turn backend.
  • GraphHopper (Java), used by OpenStreetMap.org for bike and pedestrian routing.

Geocoding: Gisgraphy (Java)

Map rendering:

  • Web map interface: Leaflet, OpenLayers 3 (supports rotation);
  • WebGL (supports camera position (bearing, pitch) and lighting): Mapbox GL JS, Tangram;
  • HTML5 Canvas: Kothic JS, Cartagen
  • SVG: Kartograph (Python, JavaScript)

🏷 Category=Geographic Information System