OpenStreetMap

Everything about OpenStreetMap.

Table of Contents

Data Structure

OSM's data model can be split into OSM elements and OSM changeset.

OSM elements:

Type:
1. Node : geographic coordinates defining a point.
2. Way : an ordered list of node IDs representing a line connecting multiple nodes.
  - Closed way : a way whose ends are identical.
  - Area : a closed way without tags default to lines (highway, barrier, junction=roundabout) or explicitly tagged with area=yes.
3. Relation : an ordered list of member entity IDs with optional roles that defines their logical or geographic relationships.
Metadata: common attributes of OSM entities.
- id: (not universally guaranteed) unique entity ID within an entity type, integers starting from 1 (node id needs 64bit); OSM IDs are not permanent, every split of a way leaves one half with a different ID than the original way.
- visible: indicator of whether the entity is deleted or not; deleted entities are only returned by history calls.
- version: edit version of the entity, in integers starting from 1.
- changeset: integer id for a group of changes made by a single user over a short period of time.
- timestamp: time of the last modification in W3C date and time format (a subset of ISO 8601).
- user: name of the last editor.
- uid: OSM id of the last editor, integer.
Tag: key-value pairs describing custom attributes of an entity, both are character strings with maximum length 256.
- Keys can be qualified with prefixes, infixes, or suffixes, forming namespace, e.g lanes:bus:forward.
- Value can be a number, text, or categorical values delimited by semicolon.

Changeset is a special OSM entity for metadata about data edits, including id fields (id, user, uid), two timestamps (created_at, closed_at), special attributes (open, comments_count) and custom attributes via tags. Actual edits corresponding to a changeset is not part of the OSM data model, and is typically derived from OSM files using custom software.

A simplified example of OSM data represented in XML (the entire file should be enclosed in an <osm> tag pair).

<node id="42437959" visible="true" version="12" changeset="13150866"
timestamp="2012-09-18T02:14:34Z" user="RoadGeek_MD99" uid="475877"
lat="40.7229823" lon="-73.9885488">
    <tag k="highway" v="traffic_signals"/>
</node>
<node id="42443513" visible="true" version="5" changeset="5155439"
timestamp="2010-07-07T00:47:51Z" user="Dylan Semler" uid="31855"
lat="40.7232770" lon="-73.9884510">
    <tag k="highway" v="traffic_signals"/>
</node>
<way id="5672851" visible="true" version="37" changeset="41811184"
timestamp="2016-08-30T20:49:09Z" user="infinitesunrise" uid="4434293">
    <nd ref="42437959"/>
    <nd ref="42443513"/>
    <tag k="highway" v="primary"/>
    <tag k="name" v="1st Avenue"/>
    <tag k="oneway" v="yes"/>
</way>
<relation id="1077653" visible="true" version="3" changeset="21582260"
timestamp="2014-04-09T04:29:57Z" user="lxbarth" uid="589596">
    <member type="way" ref="22898643" role="from"/>
    <member type="way" ref="34080174" role="to"/>
    <member type="node" ref="42428201" role="via"/>
    <tag k="restriction" v="only_straight_on"/>
    <tag k="type" v="restriction"/>
</relation>

An OSM changeset represented in XML.

<changeset id="1077653" user="aeonesa" uid="61599" open="false" comments_count="0"
created_at="2009-05-04T18:36:38Z" closed_at="2009-05-04T19:36:38Z">
    <tag k="created_by" v="Potlatch 0.11a"/>
</changeset>

A simple OSM edit in osmChange format (.osc):

<osmChange version="0.6" generator="acme osm editor">
    <modify>
        <node id="1234" changeset="42" version="2" lat="12.1234567" lon="-8.7654321">
            <tag k="amenity" v="school"/>
        </node>
    </modify>
</osmChange>

Relation

Turn restriction: member roles from, to, via, and tag type=restriction.

tag restriction
- prohibitory: no_right_turn, no_left_turn, no_u_turn, no_straight_on;
- mandatory: only_right_turn, only_left_turn, only_straight_on;
- dead end: no_entry, no_exit;
tag except, exception by vehicle type.
tags for conditional restriction: day_on, day_off, hour_on, hour_off.

Multipolygon: member roles outer, inner, and tag type=multipolygon.

Other relations: route (a collections of ways) type=route, boundary.

File Format

File formats:

OpenStreetMap canonical formats
- XML: .osm, data file with at most one version per object (may also be history file); .osh, history file, multiple versions of an object are allowed; .osc, osmChange file.
- Protocol-buffer Binary Format (PBF): .pbf, the primary binary format based on protocol buffers.
- O5M/O5C: a flat binary format that nobody uses; .o5m for data files, .o5c for osmChange files.
Third-party specified formats
- Object Per Line: .opl or .osm.opl, a text format where one object per line, designed by Osmium for command line processing.
- Vanilla Extract eXchange format: .vex, a binary format for better compression ratio and processing speed, designed by Conveyal .

OSM data files are almost always ordered in a specific way: nodes, ways, relations; each group ordered by ascending ID. History files additionally order each element by ascending version. Change files are usually ordered by changeset ID.

Protocol buffers is a language-neutral, platform-neutral extensible mechanism for serializing structured data. Delta coding and variable-byte coding are applied throughout.

Characteristics of OSM PBF:

multi-level structure: compressed Blobs - PrimitiveBlocks - PrimitiveGroups - OSM entities
mapping of one Protobuf type to each OSM entity type is difficult due to the “dense nodes” data type and parallel arrays in OSM data.
String tables replacing repetitive strings are redundant as Gzip will do an equivalent job.
Separate Protobuf specifications are used for file block structure and OSM data, which requires implementers write auxiliary code.
The most effective Protobuf technique applied in PBF is varints for delta-coded fixed-precision coordinates.

Tools

Converting map data between OSM and external formats:

Import
- JOSM plugin OpenData for Shapefile, KML, GML, CSV and others.
- Merkaartor can directly import Shapefile (and write OSM).
- GPSBabel can write to OSM file (including tags like created_by=GPSBabel-1.5.2).
- osm-and-geojson for GeoJSON;
- gml2osm for GML and csv2osm for CSV.
Export
- GDAL/OGR has an OSM driver with read-only support, so ogr2ogr can write OSM data to any supported format.
- QGIS can convert OSM file into a SpatiaLite DB file, and export selected tags and geometry types.
- osmtogeojson (maintained by the overpass-turbo project) and OSM2GEO can also export to GeoJSON.

Web Services

OpenStreetMap.org services:

Entity viewer via endpoint /<type>/<id>;
The OSM Editing API (API v0.6) is a RESTful HTTP API via endpoint /api/0.6;
History via suffix /history.
iD is the default online map editor for OSM via endpoint /edit;

API v0.6 calls:

OSM and API Metadata
- API Capabilities: /capabilities (0.25 sq degree, 300 seconds timeout);
- API Connection Permissions: /permissions (not authorized, Basic Auth, OAuth);
- OSM User Info: /user query by id with endpoints for the current authenticated user (gpx_files, details, preferences);
- Geo-referenced text notes: /notes to edit (create, comment, close, reopen) and read (by bbox and text search);
OSM data:
- Elements: /<type>/<id> to edit (create, update, delete) and read (history or specific version) individual element; fetch referencing (ways, relations) or referenced (full) elements; fetch multiple elements with query "(node|way|relation)=id[,id][&...]".
- Extract: /map with query "bbox=left,bottom,right,top";
- Changeset: /changeset/<id> to edit (create, upload, update, expand bounding box, close), read (get, download), or discuss (comment, subscribe, unsubscribe) individual changeset; fetch multiple /changesets with query (user, display_name; bbox, time, open/closed; changesets)
Raw GPS Traces: /trackpoints to query GPS track points (bbox, page) as GPX files; /gpx to upload (create) and download (details, data).

Overpass API is a read-only HTTP API for extracting elements from an OpenStreetMap database. Two query languages are available for the Overpass API: Overpass XML and Overpass QL. Overpass Turbo is a Web frontend of Overpass API. Mapbox's list of useful queries.

planet.osm.org provides weekly dumps of the entire OpenStreetMap database in one file commonly referred to as Planet.osm. OSM data is provided in text (/planet/planet-latest.osm.bz2) and binary (/pbf/planet-latest.osm.pbf) formats. Diffs/change files produced with osmosis are organized under /replication/(day|hour|minute) where state.txt provides metadata about the most recent change file and /AAA/BBB/CCC.osc.gz is the change file with sequence number AAABBBCCC. You should use osmosis to retrieve and apply all replication diffs since its last run . Metadata of changes are at /planet/changesets-latest.osm.bz2 and under /replication/changesets respectively. Planet.osm has several mirrors.

Regional extracts are available and periodically updated from third parties. GeoFabrik.de provides daily extracts for continents, countries and sub-country regions; also available in Shapefile. BBBike.org provides weekly extracts for cities and regions as shapefile, CSV, SVG, OPL and formats for offline navigation (Garmin, Navit, maps.me, OsmAnd, mapsforge), also offers custom bounding box or polygon extracts within 24M sq km and 768MB file size. BBBike.org extracts set version of all map entities to -1 and all timestamps to 1969, which can cause trouble. Metro Extracts by Mapzen provides weekly bounding box extracts of popular cities and regions as Shapefile and GeoJSON files split by geometry type of features (lines, points, polygons) or by logical groups (Roads, etc) of OpenStreetMap tags (no relation); custom bounding box extracts with update support is available using Mapzen API; coastline geometries of each area are also available in Shapefile.

OSM turn restrictions:

Openstreetmap Navigation Data Map uses Mapillary vector layer to highlight detected traffic signs.
OSM Restriction Validator by Merten Peetz.
Turn restrictions with errors and warnings by Zartbitter.

Standalone Executables

osmosis is a Java program for processing OSM data, which can extract data inside a bounding box or polygon.

I/O: --read-pbf-fast; --write-xml;
extract: --boundig-polygon;

osm-c-tools by Markus Weber:

osmupdate can create planet change files and update OSM data files with them; faster than osmosis but skips history diffs and cannot update databases.
osmconvert can convert and process OSM files faster than osmosis but has less functionality; it has a few special functions (--all-to-nodes, --complex-ways and --out-statistics).
osmfilter filters OSM data files for specific tags.

osmium is the command line tool of the Osmium library. Besides all OSM file formats, osmium also supports a "debug" format (write-only) and the OPL format, and compression by gzip or bzip2. Osmium is memory conscious in design (uses index), and much faster than osmosis.

derive/apply/merge changeset, validate, and extract historical view.

view: fileinfo summarize header/content; show load content in pager;
subset: getid, get objects by ID; extract by bounding box, polygon, or multi-polygon; tags-filter, time-filter, changeset-filter, filter OSM files;
predicate: check-refs check referential integrity (whether all referenced objects are present in the file);
unary operation: sort by type (node, way, relation), ID, version (and timestamp for change files);
binary operation: diff display content differences; derive-changes create a change file;
n-ary operation: cat concatenate files; merge, merge OSM extracts (objects are unique by type, ID, and version; contents are not unchecked), do not work on OSM data files from different points in time; merge-changes, apply-changes, merge/apply change files;
forking: renumber regenerate ID; add-location-to-ways dereference way geometries; export OSM topology into geometry formats (currently supports GeoJSON);

JOSM, Java OpenStreetMap, is the offline editor of OSM data, with many plugins. JOSM plugins for fixing missing turn restrictions: ScoutSigns (road signs by Scout users), ImproveOsm (missing geometries and turn restrictions, by Scout), turnrestrictions (by skobbler).

Merkaartor is another OpenStreetMap editor, written in C++.

KeepRight, data consistency checks (quality assurance) for OSM.

PostgreSQL Loader: osm2pgsql; osm2pgrouting, imports OSM topology to PostgreSQL database; other java programs.

SQLite/SpatiaLite for OSM

Libraries & Frameworks

Programming Frameworks for accessing, processing, map rendering (static, interactive), geocoding, and navigation with OpenStreetMap data. Debian GIS Blend meta-package for OpenStreetMap already include many of these tools and libraries.

Data access:

osm-common (Java): accessing, processing, geocoding; supports Overpass.
osmapi (Python): Python wrapper for the OpenStreetMap API.
osmaR (R): access OpenStreetMap data from file or API, and convert to other classes.
Data access APIs for Ruby and PHP are also available.

Data processing / parsing:

C/C++: osmium by Jochen Topf, with bindings pyosmium to Python and node-osmium to Node.JS; pbf2osm, osmpbf;
GO: Gosmparse, osmpbf;
Java: osmosis (by Brett Henderson); osm4j, BasicOSMParser, OSMemory;
Python: imposm (import OSM to PostgreSQL/PostGIS), osmread; OSMnx, construct street networks and compute density/routing/topological measures [@Boeing2017];

Navigation:

OSRM (C++), Open Source Routing Machine, used by Mapbox Directions.
Valhalla (C++), Mapzen Turn-by-Turn backend.
GraphHopper (Java), used by OpenStreetMap.org for bike and pedestrian routing.
Pandana (Python): aggregate attributes to OSM nodes weighted by network distance, using OSRM.

Geocoding: Gisgraphy (Java)

Map rendering:

Web map interface: Leaflet, OpenLayers 3 (supports rotation);
WebGL: Mapbox GL JS, Tangram;
HTML5 Canvas: Kothic JS, Cartagen
SVG: Kartograph (Python, JavaScript)

Resources

Mapbox OSM Mapping Guide; Mapbox list of OSM resources; OpenStreetMap Help

🏷 Category=Geographic Information System