This provides a
GeoPandas-oriented API to OGR vector
data sources, such as ESRI Shapefile, GeoPackage, and GeoJSON. This converts to
/ from GeoPandas GeoDataFrames when the data source includes geometry and Pandas
DataFrames otherwise.
WARNING: this is an early version and the API is subject to substantial change.
Supports Python 3.6 - 3.9 and GDAL 2.4.x - 3.2.x (prior versions will not be supported)
Reading to GeoDataFrames requires requires geopandas>=0.8 with pygeos enabled.
This package is available on conda-forge for Linux and MacOS. Windows is not yet supported.
conda install -c conda-forge pyogrioThis requires compatible versions of GDAL and numpy from conda-forge for
raw I/O support and geopandas, pygeos and their dependencies for GeoDataFrame
I/O support.
This package is not yet available on PyPi because it involves compiled binary dependencies. We are planning to release this package on PyPi for Linux and MacOS. We are unlikely to release Windows packages on PyPi in the near future due to the complexity of packaging binary packages for Windows.
A driver error resulting from a NULL pointer exception like this:
pyogrio._err.NullPointerError: NULL pointer error
During handling of the above exception, another exception occurred:
...
pyogrio.errors.DriverError: Data source driver could not be created: GPKG
Is likely the result of a collision in underlying GDAL versions between fiona
(included in geopandas) and the GDAL version needed here. To get around it,
uninstall fiona then reinstall to use system GDAL:
pip uninstall fiona
pip install fiona --no-binary fionaThen restart your interpreter.
Clone this repository to a local folder.
Install an appropriate distribution of GDAL for your system. gdal-config must
be on your system path.
Building pyogrio requires requires Cython, numpy, and pandas.
Run python setup.py develop to build the extensions in Cython.
Tests are run using pytest:
pytest pyogrio/testsInstall GDAL from an appropriate provider of Windows binaries. We've heard that the OSGeo4W works.
To build on Windows, you need to provide additional command-line parameters because the location of the GDAL binaries and headers cannot be automatically determined.
Assuming GDAL is installed to c:\GDAL, you can build as follows:
python -m pip install --install-option=build_ext --install-option="-IC:\GDAL\include" --install-option="-lgdal_i" --install-option="-LC:\GDAL\lib" --no-deps --force-reinstall --no-use-pep517 -e . -vThe location of the GDAL DLLs must be on your system PATH.
Also see .github/test-windows.yml for additional ideas if you run into problems.
Windows is minimally tested; we are currently unable to get automated tests working on our Windows CI.
Full support:
Read support:
- ESRI FileGDB (via OpenFileGDB)
- above formats using the Virtual File System, which allows use of zipped data sources and directories
Other vector formats registered with your installation of GDAL should be supported for read access only; these have not been tested.
We may consider supporting write access to other widely used vector formats that have an available driver in GDAL. Please open an issue to suggest a format critical to your work.
We will most likely not consider supporting obscure, rarely-used, proprietary vector formats, especially if they require advanced GDAL installation procedures.
Based on initial benchmarks using recent versions of fiona, geopandas, and pygeos:
Compared to fiona:
- 1.6x faster listing of layers in single-layer data source
- 1.6x - 5x faster reading of small data sources (Natural Earth 10m and 110m Admin 0 and Admin 1 levels)
- 9 - 14x faster writing of small data sources
Compared to geopandas in native shapely objects, converting data frame here to pygeos objects:
- 6.5 - 16.5x faster reading of data into geometry-backed data frames
- 15 - 26x faster writing of GeoDataFrames to shapefile / geopackage
Use pyogrio.list_drivers() to list all available drivers. However, just
because a driver is listed does not mean that it is currently compatible with
pyogrio. Not all field types or geometry types may be supported for all drivers.
>>> from pyogrio import list_drivers
>>> list_drivers()
{...'GeoJSON': 'rw', 'GeoJSONSeq': 'rw',...}Drivers that are not known to be supported are listed with "?" for capabilities.
Drivers that are known to support write capability end in "w".
To find subsets of drivers that have known support:
>>> list_drivers(read=True)
>>> list_drivers(write=True)See full list of drivers for more information.
You can certainly try to read or write using unsupported drivers that are available in your installation, but you may encounter errors.
Note: different drivers have different tolerance for mixed geometry types, e.g., MultiPolygon and Polygon in the same dataset. You will get exceptions if you attempt to write mixed geometries to a driver that doesn't support them.
To list layers available in a data source:
>>> from pyogrio import list_layers
>>> list_layers('ne_10m_admin_0_countries.shp')
# Outputs ndarray with the layer name and geometry type for each layer
array([['ne_10m_admin_0_countries', 'Polygon']], dtype=object)Some data sources (e.g., ESRI FGDB) support multiple layers, some of which may
be nonspatial. In this case, the geometry type will be None.
To list information about a data layer in a data source, use the name of the layer or its index (0-based) within the data source. By default, this reads from the first layer.
>>> from pyogrio import read_info
>>> read_info('ne_10m_admin_0_countries.shp')
# Outputs a dictionary with `crs`, `encoding`, `fields`, `geometry_type`, and `features`
{
'crs': 'EPSG:4326',
'encoding': 'UTF-8',
'fields': array(['featurecla', 'scalerank', 'LABELRANK', ...], dtype=object),
'geometry_type': 'Polygon',
'features': 255
}To read from a layer using name or index (the following are equivalent):
>>>read_info('ne_10m_admin_0_countries.shp', layer='ne_10m_admin_0_countries')
>>>read_info('ne_10m_admin_0_countries.shp', layer=0)To read all features from a spatial data layer. By default, this operates on
the first layer unless layer is specified using layer name or index.
>>> from pyogrio import read_dataframe
>>> read_dataframe('ne_10m_admin_0_countries.shp')
featurecla ... geometry
0 Admin-0 country ... MULTIPOLYGON (((117.70361 4.16341, 117.70361 4...
1 Admin-0 country ... MULTIPOLYGON (((117.70361 4.16341, 117.69711 4...
2 Admin-0 country ... MULTIPOLYGON (((-69.51009 -17.50659, -69.50611...
3 Admin-0 country ... POLYGON ((-69.51009 -17.50659, -69.51009 -17.5...
4 Admin-0 country ... MULTIPOLYGON (((-69.51009 -17.50659, -69.63832...
.. ... ... ...
250 Admin-0 country ... MULTIPOLYGON (((113.55860 22.16303, 113.56943 ...
251 Admin-0 country ... POLYGON ((123.59702 -12.42832, 123.59775 -12.4...
252 Admin-0 country ... POLYGON ((-79.98929 15.79495, -79.98782 15.796...
253 Admin-0 country ... POLYGON ((-78.63707 15.86209, -78.64041 15.864...
254 Admin-0 country ... POLYGON ((117.75389 15.15437, 117.75569 15.151...You can read a subset of columns by including the columns parameter. This
only affects non-geometry columns:
>>> read_dataframe('ne_10m_admin_0_countries.shp', columns=['ISO_A3'])
ISO_A3 geometry
0 IDN MULTIPOLYGON (((117.70361 4.16341, 117.70361 4...
1 MYS MULTIPOLYGON (((117.70361 4.16341, 117.69711 4...
2 CHL MULTIPOLYGON (((-69.51009 -17.50659, -69.50611...
3 BOL POLYGON ((-69.51009 -17.50659, -69.51009 -17.5...
4 PER MULTIPOLYGON (((-69.51009 -17.50659, -69.63832...
.. ... ...
250 MAC MULTIPOLYGON (((113.55860 22.16303, 113.56943 ...
251 -99 POLYGON ((123.59702 -12.42832, 123.59775 -12.4...
252 -99 POLYGON ((-79.98929 15.79495, -79.98782 15.796...
253 -99 POLYGON ((-78.63707 15.86209, -78.64041 15.864...
254 -99 POLYGON ((117.75389 15.15437, 117.75569 15.151...You can read a subset of features using skip_features and max_features.
To skip the first 10 features:
>>> read_dataframe('ne_10m_admin_0_countries.shp', skip_features=10)NOTE: the index of the GeoDataFrame is based on the features that are read from
the file, it does not start at skip_features.
To read only the first 10 features:
>>> read_dataframe('ne_10m_admin_0_countries.shp', max_features=10)These can be combined to read defined ranges in the dataset, perhaps in multiple processes:
>>> read_dataframe('ne_10m_admin_0_countries.shp', skip_features=10, max_features=10)You can use the where parameter to define a GDAL-compatible SQL WHERE query against
the records in the dataset:
>>> read_dataframe('ne_10m_admin_0_countries.shp', where="POP_EST >= 10000000 AND POP_EST < 100000000")See GDAL docs
for more information about restrictions of the where expression.
You can use the bbox parameter to select only those features that intersect
with the bbox.
>>> read_dataframe('ne_10m_admin_0_countries.shp', bbox=(-140, 20, -100, 40))Note: the bbox values must be in the same CRS as the dataset.
You can omit the geometry from a spatial data layer by setting read_geometry
to False:
>>> read_dataframe('ne_10m_admin_0_countries.shp', columns=['ISO_A3'], read_geometry=False)
ISO_A3
0 IDN
1 MYS
2 CHL
3 BOL
4 PER
.. ...
250 MAC
251 -99
252 -99
253 -99Any read operation which does not include a geometry column, either by reading
from a nonspatial data layer or by omitting the geometry column above, returns
a Pandas DataFrame.
You can force a 3D dataset to 2D using force_2d:
>>> df = read_dataframe('has_3d.shp')
>>> df.iloc[0].geometry.has_z
True
>>> df = read_dataframe('has_3d.shp', force_2d=True)
>>> df.iloc[0].geometry.has_z
FalseSome data sources support NULL or otherwise unset field values. These cannot be properly stored into the ndarray for certain types. If NULL or unset values are encountered, the following occurs:
- If the field is a string type, NULL values are represented as None
- If the field is an integer type (np.int32, np.int64), the field data are re-cast to np.float64 values, and NULL values are represented as np.nan
- If the field is a date or datetime type, the field is set as np.datetime64('NaT')
To write a GeoDataFrame df to a file. driver defaults to ESRI Shapefile
(for now) but can be manually specified using one of the supported drivers for
writing (above):
>>> from pyogrio import write_dataframe
>>> write_dataframe(df, '/tmp/test.shp', driver="GPKG")The appropriate driver is also inferred automatically (where possible) from the
extension of the filename:
.shp: ESRI Shapefile
.gpkg: GPKG
.json: GeoJSON
It is possible to set GDAL configuration options for an entire session:
>>> from pyogrio import set_gdal_config_options
>>> set_gdal_config_options({"CPL_DEBUG": True})True / False values are automatically converted to 'ON' / 'OFF'.
You can display the GDAL version that pyogrio was compiled against by
>>> pyogrio.__gdal_version__see pyogrio.raw for numpy-oriented read / write interfaces to OGR data sources.
This may be useful for you if you want to work with the underlying arrays of
WKB geometries and field values outside of a GeoDataFrame.
NOTE: this may be migrated to an internal API in a future release.
Measured geometry types are not supported for reading or writing. These are not supported by the GEOS library and cannot be converted to geometry objects in GeoDataFrames.
These are automatically downgraded to their 2.5D (x,y, single z) equivalent and a warning is raised.
To ignore this warning:
>>> import warnings
>>> warnings.filterwarnings("ignore", message=".*Measured \(M\) geometry types are not supported.*")These geometry types are not currently supported. These are automatically converted to their linear approximation when reading geometries from the data layer.
pyogrio supports reading / writing data layers with a defined encoding. However,
DataFrames do not currently allow arbitrary metadata, which means that we are currently
unable to store encoding information for a data source. Text fields are read
into Python UTF-8 strings.
It does not currently validate attribute values or geometry types before attempting to write to the output file. Invalid types may crash during writing with obscure error messages.
Date fields are not currently supported properly. These will be supported in a future release.
pyogrio internally uses a numpy-oriented approach in Cython to read
information about data sources and records from spatial data layers. Geometries
are extracted from the data layer as Well-Known Binary (WKB) objects and fields
(attributes) are read into numpy arrays of the appropriate data type. These are
then converted to GeoPandas GeoDataFrames.
All records are read into memory, which may be problematic for very large data
sources. You can use skip_features / max_features to read smaller parts of
the file at a time.
The entire GeoDataFrame is written at once. Incremental writes or appends to
existing data sources are not supported.
Fiona is a full-featured Python library
for working with OGR vector data sources. It is awesome, has highly-dedicated
maintainers and contributors, and exposes more functionality than pyogrio ever will.
This project would not be possible without Fiona having come first.
pyogrio is an experimental approach that uses a vectorized (array-oriented)
approach for reading and writing spatial vector file formats, which enables faster
I/O operations. It borrows from the internal mechanics and lessons learned of
Fiona. It uses a stateless approach to reading or writing data; all data
are read or written in a single pass.
Fiona is a general purpose spatial format I/O library that is used within many
projects in the Python ecosystem. In contrast, pyogrio specifically targets
GeoPandas as an attempt to reduce the number of data transformations currently
required to read / write data between GeoPandas GeoDataFrames and spatial
file formats using Fiona (the current default in GeoPandas).
This project is made possible by the tremendous efforts of the GDAL, Fiona, and Geopandas communities.