Loading Skims from Files

Loading Skims from Files#

Sharrow can load skim data from several common file formats. This walkthrough uses the same small autotime skim to demonstrate three layouts:

  • Zarr stores the three dimensions natively.

  • OMX stores one matrix per time period, with the period encoded in the matrix name.

  • Parquet stores one column per time period, with named origin and destination columns identifying the matrix indexes.

The example creates temporary files so that it can be run without downloading any data.

from pathlib import Path
from tempfile import TemporaryDirectory

import numpy as np
import openmatrix
import pandas as pd
import xarray as xr

import sharrow as sh

Example skim data#

The source dataset has native otaz, dtaz, and timeperiod dimensions. The values are deliberately small and distinct so that the loaded data can be checked easily.

zones = np.array([101, 102, 103])
timeperiods = ["AM", "PM"]
autotime = np.array(
    [
        [[1.0, 1.5], [2.0, 2.5], [3.0, 3.5]],
        [[4.0, 4.5], [5.0, 5.5], [6.0, 6.5]],
        [[7.0, 7.5], [8.0, 8.5], [9.0, 9.5]],
    ],
    dtype=np.float32,
)
source_skims = xr.Dataset(
    {"autotime": (("otaz", "dtaz", "timeperiod"), autotime)},
    coords={"otaz": zones, "dtaz": zones, "timeperiod": timeperiods},
)
source_skims

<xarray.Dataset> Size: 136B
Dimensions:     (otaz: 3, dtaz: 3, timeperiod: 2)
Coordinates:
  * otaz        (otaz) int64 24B 101 102 103
  * dtaz        (dtaz) int64 24B 101 102 103
  * timeperiod  (timeperiod) <U2 16B 'AM' 'PM'
Data variables:
    autotime    (otaz, dtaz, timeperiod) float32 72B 1.0 1.5 2.0 ... 8.5 9.0 9.5
temporary_directory = TemporaryDirectory()
data_directory = Path(temporary_directory.name)

Zarr: native three-dimensional data#

Zarr preserves the dimensions and coordinates in the store. from_zarr reads the _ARRAY_DIMENSIONS metadata written by xarray, so no dimension mapping is needed when loading.

zarr_path = data_directory / "skims.zarr"
source_skims.to_zarr(zarr_path, mode="w")

zarr_skims = sh.dataset.from_zarr(zarr_path)
zarr_skims

<xarray.Dataset> Size: 136B
Dimensions:     (otaz: 3, dtaz: 3, timeperiod: 2)
Coordinates:
  * dtaz        (dtaz) int64 24B 101 102 103
  * otaz        (otaz) int64 24B 101 102 103
  * timeperiod  (timeperiod) <U2 16B 'AM' 'PM'
Data variables:
    autotime    (otaz, dtaz, timeperiod) float32 72B dask.array<chunksize=(3, 3, 2), meta=np.ndarray>

OMX: time period in matrix names#

The openmatrix (OMX) standard defines a data format that stores two-dimensional matrix data. To represent a third dimension, one approach is to store a matrix for each time period and include the time period after a double underscore in the matrix name, such as autotime__AM.

omx_path = data_directory / "skims.omx"
with openmatrix.open_file(omx_path, mode="w") as omx_file:
    for timeperiod in timeperiods:
        omx_file.create_carray(
            "/data",
            f"autotime__{timeperiod}",
            obj=source_skims.autotime.sel(timeperiod=timeperiod).values,
        )
    omx_file.create_carray("/lookup", "taz", obj=zones)
    omx_file.root._v_attrs.SHAPE = np.array([len(zones), len(zones)])

Sharrow includes the code needed to read this format of stored data into a three dimension dataset.

omx_skims = sh.dataset.from_omx_3d(
    str(omx_path),
    index_names=("otaz", "dtaz", "timeperiod"),
    time_periods=timeperiods,
)
omx_skims

<xarray.Dataset> Size: 136B
Dimensions:     (otaz: 3, dtaz: 3, timeperiod: 2)
Coordinates:
  * otaz        (otaz) int64 24B 101 102 103
  * dtaz        (dtaz) int64 24B 101 102 103
  * timeperiod  (timeperiod) <U2 16B 'AM' 'PM'
Data variables:
    autotime    (otaz, dtaz, timeperiod) float32 72B dask.array<chunksize=(3, 3, 1), meta=np.ndarray>

Parquet: named indexes and time-period columns#

Sometimes, skim data is stored as a table. This is especially convenient when passing data to tools which prefer tabular data and don’t natively handle multi-dimensional tensors. The parquet format is widely used for tabular data and can be used to store skims as a table. Here, origin and destination name the first two dimensions, while autotime__AM and autotime__PM encode the third dimension in their column names.

origin, destination = np.meshgrid(zones, zones, indexing="ij")
parquet_data = pd.DataFrame(
    {
        "origin": origin.ravel(),
        "destination": destination.ravel(),
        "autotime__AM": source_skims.autotime.sel(timeperiod="AM").values.ravel(),
        "autotime__PM": source_skims.autotime.sel(timeperiod="PM").values.ravel(),
    }
)
parquet_path = data_directory / "skims.parquet"
parquet_data.to_parquet(parquet_path, index=False)

Sharrow can also read this format of data back into the preferred three dimension dataset in memory.

parquet_skims = sh.dataset.from_parquet_3d(
    parquet_path,
    index_names=("origin", "destination", "timeperiod"),
    time_periods=timeperiods,
)
parquet_skims

<xarray.Dataset> Size: 136B
Dimensions:      (origin: 3, destination: 3, timeperiod: 2)
Coordinates:
  * origin       (origin) int64 24B 101 102 103
  * destination  (destination) int64 24B 101 102 103
  * timeperiod   (timeperiod) <U2 16B 'AM' 'PM'
Data variables:
    autotime     (origin, destination, timeperiod) float32 72B 1.0 1.5 ... 9.5
Contents