Shadow Pricing#

The shadow pricing calculator used by work and school location choice.

Structure#

  • Configuration File: shadow_pricing.yaml

Turning on and saving shadow prices#

Shadow pricing is activated by setting the use_shadow_pricing to True in the settings.yaml file. Once this setting has been activated, ActivitySim will search for shadow pricing configuration in the shadow_pricing.yaml file. When shadow pricing is activated, the shadow pricing outputs will be exported by the tracing engine. As a result, the shadow pricing output files will be prepended with trace followed by the iteration number the results represent. For example, the shadow pricing outputs for iteration 3 of the school location model will be called trace.shadow_price_school_shadow_prices_3.csv.

In total, ActivitySim generates three types of output files for each model with shadow pricing:

  • trace.shadow_price_<model>_desired_size.csv The size terms by zone that the ctramp and daysim methods are attempting to target. These equal the size term columns in the land use data multiplied by size term coefficients.

  • trace.shadow_price_<model>_modeled_size_<iteration>.csv These are the modeled size terms after the iteration of shadow pricing identified by the number. In other words, these are the predicted choices by zone and segment for the model after the iteration completes. (Not applicable for simulation option.)

  • trace.shadow_price_<model>_shadow_prices_<iteration>.csv The actual shadow price for each zone and segment after the of shadow pricing. This is the file that can be used to warm start the shadow pricing mechanism in ActivitySim. (Not applicable for simulation option.)

There are three shadow pricing methods in activitysim: ctramp, daysim, and simulation. The first two methods try to match model output with workplace/school location model size terms, while the last method matches model output with actual employment/enrollmment data.

The simulation approach operates the following steps. First, every worker / student will be assigned without shadow prices applied. The modeled share and the target share for each zone are compared. If the zone is overassigned, a sample of people from the over-assigned zones will be selected for re-simulation. Shadow prices are set to -999 for the next iteration for overassigned zones which removes the zone from the set of alternatives in the next iteration. The sampled people will then be forced to choose from one of the under-assigned zones that still have the initial shadow price of 0. (In this approach, the shadow price variable is really just a switch turning that zone on or off for selection in the subsequent iterations. For this reason, warm-start functionality for this approach is not applicable.) This process repeats until the overall convergence criteria is met or the maximum number of allowed iterations is reached.

Note that the simulation approach compares the modeled employment/enrollment share against the target employment/enrollment share for each zone, rather than the absolute number. Therefore, it does not require the absolute total workers/students to match the regional employment/enrollment totals.

Because the simulation approach only re-simulates workers / students who were over-assigned in the previous iteration, run time is significantly less (~90%) than the CTRAMP or DaySim approaches which re-simulate all workers and students at each iteration.

When choosing workers or students for re-simulation, the simulation approach selects individuals from overassigned zones by shadow pricing segment. In version 1.6.0 and prior, the segmentation was defaulted to use income groups for workers and school segments for students, and changable via the SEGMENT_TO_NAME setting in the shadow_pricing.yaml. However, this setting was easily overlooked or misconfigured, which could cause the shadow pricing to be accidentally skipped or cause a mismatch between the segmentation used for shadow pricing re-simulation and the segmentation used for work and school location choice simulation, if the latter uses a different segmentation scheme from the default income and school segments. In version 1.6.1, the segmentation used for shadow pricing re-simulation now inherits the segmentation used in the work and school location choice simulation, e.g., defined as CHOOSER_SEGMENT_COLUMN_NAME in the workplace_location.yaml. The legacy setting SEGMENT_TO_NAME in the shadow_pricing.yaml is now deprecated and ignored.

The segmentation in shadow pricing re-simulation, e.g., occupation or income groups, accommodates but does not require occupation- or income-specific land use employment controls. For example, the example SANDAG model segments workplace location choice by occupation, but all occupation segments share the same total_emp_revised target. Under the SANDAG configuration, ActivitySim combines the modeled workers across segments sharing that target when evaluating whether a zone is over- or under-assigned.

Configuration#

settings activitysim.abm.tables.shadow_pricing.ShadowPriceSettings#

Bases: PydanticReadable

Settings used for shadow pricing.

Config:

  • extra: str = forbid

Fields:
Validators:
field DAMPING_FACTOR: float = 1#

ctramp-style damping factor

Validated by:
field DAYSIM_ABSOLUTE_TOLERANCE: float = 50#
Validated by:
field DAYSIM_PERCENT_TOLERANCE: float = 10#
Validated by:
field FAIL_THRESHOLD: float = 10#

max percentage of zones allowed to fail

Validated by:
field LOAD_SAVED_SHADOW_PRICES: bool = True#

Global switch to enable/disable loading of saved shadow prices.

This is ignored if global use_shadow_pricing switch is False

Validated by:
field MAX_ITERATIONS: int = 5#

Number of shadow price iterations for cold start.

Validated by:
field MAX_ITERATIONS_SAVED: int = 1#

Number of shadow price iterations for warm start.

A warm start means saved shadow_prices were found in a file and loaded.

Validated by:
field PERCENT_TOLERANCE: float = 5#

zone passes if modeled is within percent_tolerance of predicted_size

Validated by:
field SCALE_SIZE_TABLE: bool = False#
Validated by:
field SEGMENT_TO_NAME: dict[str, str] | None = None#

Deprecated compatibility setting; chooser segmentation comes from model settings.

Validated by:
field SHADOW_PRICE_METHOD: Literal['ctramp', 'daysim', 'simulation'] = 'ctramp'#
Validated by:
field SIZE_THRESHOLD: float = 10#

ignore criteria for zones smaller than size_threshold

Validated by:
field TARGET_THRESHOLD: float = 20#

ignore criteria for zones smaller than target_threshold (total employmnet or enrollment)

Validated by:
field WRITE_ITERATION_CHOICES: bool = False#
Validated by:
field school_segmentation_targets: dict[str, str] | None = None#
Validated by:
field shadow_pricing_models: dict[str, str] | None = None#

List model_selectors and model_names of models that use shadow pricing. This list identifies which size_terms to preload which must be done in single process mode, so predicted_size tables can be scaled to population

Validated by:
field source_file_paths: list[Path] = None#

A list of source files from which these settings were loaded.

This value should not be set by the user within the YAML settings files, instead it is populated as those files are loaded. It is primarily provided for debugging purposes, and does not actually affect the operation of any model.

Validated by:
field workplace_segmentation_targets: dict[str, str] | None = None#
Validated by:
validator warn_deprecated_segment_to_name  »  all fields#

Examples#

Implementation#

activitysim.abm.tables.shadow_pricing.ShadowPriceCalculator(state: State, model_settings: TourLocationComponentSettings, num_processes, shared_data=None, shared_data_lock=None, shared_data_choice=None, shared_data_choice_lock=None, shared_sp_choice_df=None)#
activitysim.abm.tables.shadow_pricing.buffers_for_shadow_pricing(shadow_pricing_info)#

Allocate shared_data buffers for multiprocess shadow pricing

Allocates one buffer per model_selector. Buffer datatype and shape specified by shadow_pricing_info

buffers are multiprocessing.Array (RawArray protected by a multiprocessing.Lock wrapper) We don’t actually use the wrapped version as it slows access down and doesn’t provide protection for numpy-wrapped arrays, but it does provide a convenient way to bundle RawArray and an associated lock. (ShadowPriceCalculator uses the lock to coordinate access to the numpy-wrapped RawArray.)

Parameters:
shadow_pricing_infodict
Returns:
data_buffersdict {<model_selector><shared_data_buffer>}
dict of multiprocessing.Array keyed by model_selector
activitysim.abm.tables.shadow_pricing.buffers_for_shadow_pricing_choice(state, shadow_pricing_choice_info)#

Same as above buffers_for_shadow_price function except now we need to store the actual choices for the simulation based shadow pricing method

This allocates a multiprocessing.Array that can store the choice for each person and then wraps a dataframe around it. That means the dataframe can be shared and accessed across all threads. Parameters ———- shadow_pricing_info : dict Returns ——-

data_buffers : dict {<model_selector> : <shared_data_buffer>} dict of multiprocessing.Array keyed by model_selector

and wrapped in a pandas dataframe

activitysim.abm.tables.shadow_pricing.shadow_price_data_from_buffers_choice(data_buffers, shadow_pricing_info, model_selector)#
Parameters:
data_buffersdict of {<model_selector><multiprocessing.Array>}

multiprocessing.Array is simply a convenient way to bundle Array and Lock we extract the lock and wrap the RawArray in a numpy array for convenience in indexing The shared data buffer has shape (<num_zones, <num_segments> + 1) extra column is for reverse semaphores with TALLY_CHECKIN and TALLY_CHECKOUT

shadow_pricing_infodict
dict of useful info

dtype: sp_dtype, block_shapes : OrderedDict({<model_selector>: <shape tuple>}) dict mapping model_selector to block shape (including extra column for semaphores) e.g. {‘school’: (num_zones, num_segments + 1)

model_selectorstr

location type model_selector (e.g. school or workplace)

Returns:
shared_data, shared_data_lock

shared_data : multiprocessing.Array or None (if single process) shared_data_lock : numpy array wrapping multiprocessing.RawArray or None (if single process)

activitysim.abm.tables.shadow_pricing.shadow_price_data_from_buffers(data_buffers, shadow_pricing_info, model_selector)#
Parameters:
data_buffersdict of {<model_selector><multiprocessing.Array>}

multiprocessing.Array is simply a convenient way to bundle Array and Lock we extract the lock and wrap the RawArray in a numpy array for convenience in indexing The shared data buffer has shape (<num_zones, <num_segments> + 1) extra column is for reverse semaphores with TALLY_CHECKIN and TALLY_CHECKOUT

shadow_pricing_infodict
dict of useful info

dtype: sp_dtype, block_shapes : OrderedDict({<model_selector>: <shape tuple>}) dict mapping model_selector to block shape (including extra column for semaphores) e.g. {‘school’: (num_zones, num_segments + 1)

model_selectorstr

location type model_selector (e.g. school or workplace)

Returns:
shared_data, shared_data_lock

shared_data : multiprocessing.Array or None (if single process) shared_data_lock : numpy array wrapping multiprocessing.RawArray or None (if single process)

activitysim.abm.tables.shadow_pricing.load_shadow_price_calculator(state: State, model_settings: TourLocationComponentSettings)#

Initialize ShadowPriceCalculator for model_selector (e.g. school or workplace)

If multiprocessing, get the shared_data buffer to coordinate global_desired_size calculation across sub-processes

Parameters:
stateworkflow.State
model_settingsTourLocationComponentSettings
Returns:
spcShadowPriceCalculator
activitysim.abm.tables.shadow_pricing.add_size_tables(state: State, disaggregate_suffixes: dict[str, Any], scale: bool = True) → None#

inject tour_destination_size_terms tables for each model_selector (e.g. school, workplace)

Size tables are pandas dataframes with locations counts for model_selector by zone and segment tour_destination_size_terms

if using shadow pricing, we scale size_table counts to sample population (in which case, they have to be created while single-process)

Scaling is problematic as it breaks household result replicability across sample sizes It also changes the magnitude of the size terms so if they are used as utilities in expression files, their importance will diminish relative to other utilities as the sample size decreases.

Scaling makes most sense for a full sample in conjunction with shadow pricing, where shadow prices can be adjusted iteratively to bring modelled counts into line with desired (size table) counts.

activitysim.abm.tables.shadow_pricing.get_shadow_pricing_info(state)#

return dict with info about dtype and shapes of desired and modeled size tables

block shape is (num_zones, num_segments + 1)

Returns:
shadow_pricing_info: dict

dtype: <sp_dtype>, block_shapes: dict {<model_selector>: <block_shape>}

activitysim.abm.tables.shadow_pricing.get_shadow_pricing_choice_info(state)#

return dict with info about dtype and shapes of desired and modeled size tables

block shape is (num_zones, num_segments + 1)

Returns:
shadow_pricing_info: dict

dtype: <sp_dtype>, block_shapes: dict {<model_selector>: <block_shape>}