Repository navigation
Conversation
Location choice logsums run lot choice on one row per person and sampled destination. Preprocessing, skim setup and filtering copied that whole table several times before interaction_simulate chunking applied, exhausting memory. Run the lot choice in slices sized by explicit_chunk. The unique index used for random draws is still built once for the whole table, so choices are identical to the unsliced run. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
Author
|
Hi @dhensle, per our discussion, here are a bit more information about our tests: We ran our model to confirm the model result does not change with this update. We used chunk size of 200,000 for park and ride, and 40,000 for workplace location. Introduction of this change is needed to have our model run successfully in 2053 scenario under 256GB of memory. Reducing chunk size in workplace location without adding chunking for park and ride does not seem to resolve the memory issues we were encountering. Here are the impact on run time and memory on the 2023 scenario (~1.3 million HHs & 3.2 million Persons):
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Summary
run_park_and_ride_lot_choicenow processes choosers in slices sized by the park-and-rideexplicit_chunk(same sizing rule asactivitysim.core.chunk), instead of copying andpreprocessing the whole chooser table before
interaction_simulatechunks it.longer needs a full copy of the chooser table, so choices are identical to the unsliced run.
explicit_chunk: 0(or any non-explicit chunk mode) run as one slice,as before.
repeated index) and that check the slice size rules.
Changes
All code changes are in
activitysim/abm/models/park_and_ride_lot_choice.py, tests inactivitysim/abm/models/util/test/test_park_and_ride_models.py._choosers_per_slice(state, num_choosers, explicit_chunk_size)(new)Returns how many choosers to process at a time. It mirrors the explicit chunking rules in
activitysim/core/chunk.py:explicit, orexplicit_chunkis 0/negative: all choosers (no slicing),explicit_chunk < 1: that fraction of the choosers (ceil(n * fraction)),explicit_chunk / num_processes(total choosers per chunk across allprocesses), where
num_processesis 1 when not multiprocessing,_choose_lots_for_choosers(...)(new, code moved)The body that used to be inline in
run_park_and_ride_lot_choice(transit eligibilityfilter, skim setup, preprocessor TABLES clean up, chooser preprocessor, optional
CHOOSER_FILTER_EXPR, alternatives preprocessor and column trimming, estimation writes,and
interaction_simulate) moved into this helper unchanged, with two small additions:pnr_altsandlocals_dict, because preprocessing mutates themand that must not leak from one slice into the next,
Nonewhen no chooser in the slice is eligible, instead of returning fromthe outer function.
run_park_and_ride_lot_choice(changed)choosers.reset_index()(fulltable copy), then a new index
original_index * idx_multiplier + cumcount, thenadd_channel("pnr_lot_choice", choosers). Now the same index values are computed from theindex alone (
groupby(level=0).size().max()for the multiplier and.cumcount()), andthe rng channel is registered with an index-only frame (the channel only uses the index).
The index is computed once for the whole table so random draws do not depend on how it
is sliced. The
reset_index(drop=False)that adds the original index as a column is nowdone per slice.
ilocand passed to_choose_lots_for_choosers. Slice results are concatenated and reindexed to the fullindex with
-1for choosers without a lot, as before. The original index is restored and the rngchannel dropped at the end, as before.
unique, the caller's
choosersframe is used directly, so in-place column additions by thepreprocessor are preserved exactly as before.
returns -1 through
return_no_choices).Tests (
test_park_and_ride_models.py)test_run_pnr_slicing_does_not_change_choices(parametrized for unique and repeatedindex): 50 choosers, two lots with equal utility so every choice is a random draw. Runs
once unsliced and once with
explicit_chunk=8(7 slices of 8,8,8,8,8,8,2) and assertsthe choices and index are identical. It also asserts the unsliced run picks both lots,
so a difference in random draws would be visible.
test_choosers_per_slice_follows_explicit_chunk_rules: chunk mode disabled, 0, largerthan the number of choosers, fraction, and the division by
num_processes.Design decisions
Why slice inside
run_park_and_ride_lot_choicerather than incompute_location_choice_logsums?The expensive copies happen inside
run_park_and_ride_lot_choice, and it is also calledfrom tour mode choice, tour scheduling, at-work subtour mode choice and disaggregate
accessibility. Slicing at this level fixes all callers with one setting (the existing
park-and-ride
explicit_chunk) and keepslogsums.pyunchanged. For the unique indexcallers (for example tours) nothing changes unless
explicit_chunkis smaller than thenumber of choosers.
Why is the output identical?
Random numbers in ActivitySim are drawn per row from a seed based on the row index and a
per-row offset. They do not depend on which other rows are processed together. The only
chooser-set dependent value is
idx_multiplier(the maximum number of rows perperson), which is computed over the whole table before slicing, so the unique index
values are exactly the same as before. The channel is registered once with all of them.
Why reuse
explicit_chunkinstead of a new setting?The setting already documents "total choosers per chunk across all processes", and the
inner
interaction_simulatechunking uses the same value, so each slice is exactly onechunk of the inner step. A new setting would be one more number to tune that has to
agree with it.
Why not reduce the workplace location
SAMPLE_SIZEinstead?That would also reduce memory (rows scale with
SAMPLE_SIZE) but it changes the model:different sampled alternatives, more Monte Carlo noise and different random draws, so
results would no longer be comparable with earlier runs or calibration. This change
keeps results identical.
Trade-offs
interaction_simulateset up now run once perslice instead of once. The per-slice overhead is small compared with the memory saved,
but very small
explicit_chunkvalues (for example 20000 with 6 processes is about3,300 rows per slice, about 1,600 slices for
work_veryhigh) also produce largeDEBUG logs. The default of 200000 (about 33,000 rows per slice with 6 processes, about
160 slices) is recommended.
adaptive,production,training,disabled) are notsliced, because they do not use
explicit_chunk.location_sample_dfjoined topersons_merged) still exists inmemory once per process; only the multiple extra copies are removed.
Expected memory effect
Peak memory for this step should follow the slice size (about 33,000 rows x 7 KB, a few
hundred MB per process) rather than the table size. For the
work_veryhighsegment of the example model the roughestimate is about 100 to 120 GB peak (base plus one copy of the chooser table in each
process), compared with over 251 GB before. This is an estimate from the earlier logs;
it has not been measured on the 256 GB machine yet.
Validation
All run from
.py_envs\activitysimwith.\.venv\Scripts\python.exe:-m pytest activitysim/abm/models/util/test/test_park_and_ride_models.py -q -x -p no:cacheprovider-m pytest activitysim/abm/models/util/test -q -p no:cacheprovider -k "pnr or park or tour_mode or logsum"-m blackon the two changed files (black 22.12.0)Equivalence with the previous implementation was also checked with a temporary test
(not committed) that imported the original module from
git show HEAD:...and ran bothversions on the same fixture with 37-row slices, for a unique index (tours) and for a
repeated index (location logsums). Both produced identical Series (
assert_series_equal).TransLink has ran this code through our full regional model on the 256 GB machine, and the
ActivitySim example and estimation test suites.
We will provide memory profile in subsequent comments to this pull request.
Corresponding issue - #1138