Skip to content

mpi: Emit global index offsets on a single rank and rebuild Functions onto SubGrids - #3110

Open
mloubout wants to merge 4 commits into
mainfrom
mloubout/subgrid-rebuild
Open

mloubout wants to merge 4 commits into
mainfrom
mloubout/subgrid-rebuild

Conversation

@mloubout

Copy link
Copy Markdown
Contributor

Code generated with MPI on one rank (e.g. the DevitoPRO decoupler) now carries the global index and SubGrid offsets, and Function._rebuild(grid=subgrid) moves onto the SubGrid's Dimensions.

@codecov

codecov Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 72.27723% with 56 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.18%. Comparing base (1eb54aa) to head (952d17f).

Files with missing lines Patch % Lines
tests/test_subsampling.py 52.50% 38 Missing ⚠️
tests/test_mpi.py 71.05% 11 Missing ⚠️
devito/mpi/distributed.py 75.00% 4 Missing and 1 partial ⚠️
devito/types/dense.py 92.85% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3110      +/-   ##
==========================================
- Coverage   84.20%   84.18%   -0.02%     
==========================================
  Files         260      260              
  Lines       59481    59607     +126     
  Branches     5023     5031       +8     
==========================================
+ Hits        50085    50182      +97     
- Misses       8561     8588      +27     
- Partials      835      837       +2     
Flag Coverage Δ
pytest-gpu-aomp-amdgpuX 68.33% <62.96%> (+<0.01%) ⬆️
pytest-gpu-gcc- 78.94% <72.27%> (-0.01%) ⬇️
pytest-gpu-icx- 78.90% <72.27%> (+0.01%) ⬆️
pytest-gpu-nvc-nvidiaX 69.15% <62.96%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@mloubout mloubout added the API api (symbolics, types, ...) label Nov 12, 2026
@mloubout
mloubout force-pushed the mloubout/subgrid-rebuild branch 2 times, most recently from 563e80c to 3d7e391 Compare November 13, 2026 02:37
@review-notebook-app

Copy link
Copy Markdown

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

Comment thread devito/ir/equations/algorithms.py Resolved
Comment thread devito/ir/support/guards.py Resolved
Comment thread devito/ir/support/utils.py Resolved
@mloubout
mloubout force-pushed the mloubout/subgrid-rebuild branch 2 times, most recently from b6d2e36 to 4fedfd3 Compare November 13, 2026 14:58
@mloubout
mloubout force-pushed the mloubout/subgrid-rebuild branch 5 times, most recently from 6321cce to 0412530 Compare November 17, 2026 14:38
Comment thread devito/types/dense.py Resolved
Comment thread devito/types/dense.py Resolved
Comment thread devito/types/dense.py Resolved

@FabioLuporini FabioLuporini left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks OK, some minor nitpicks left but can go in

@mloubout
mloubout force-pushed the mloubout/subgrid-rebuild branch 2 times, most recently from 55c8670 to 90ea252 Compare November 19, 2026 17:16
@mloubout
mloubout force-pushed the mloubout/subgrid-rebuild branch from 90ea252 to 98490b4 Compare November 20, 2026 14:19
@mloubout mloubout added the no-pro-trigger Skip the devitopro submodule update on merge label Nov 20, 2026
Restore RankOffset and test_rank_offset_empty, dropped by the previous commit.
The Grid index offsets io_x are now RankOffsets on the LEFT side rather than
plain Scalars, so that a single symbol carries the global index of a rank's
first point. They are evaluated per rank against the runtime Grid and keep
their name through SubDimension concretization.

Strengthen test_halo_zero_subdomain (from #3117) to compile on the distributed
Grid and compare against an explicitly zeroed reference, serial and on 4 ranks,
with and without transpose. On main it fails on 4 ranks, where the RankOffset
bounds of the halo=0 indicator are added on top of the global index offset.
Comment thread devito/ir/equations/algorithms.py Resolved
Comment thread devito/mpi/distributed.py Resolved
Comment thread devito/mpi/distributed.py Resolved
Comment thread devito/types/dense.py Resolved
Comment thread devito/types/dense.py Resolved
Comment thread devito/types/dimension.py Resolved
The Grid index offsets io_x are now RankOffsets, which are uncached
DataSymbols, so test_solve expects one fewer cached symbol, as before #3106.

Review comments:
* Retrieve the Dimensions with mode='unique' in _make_global_index_map
* Return early from Distributor.subsample without domain decomposition
* Make Distributor.size_map a frozendict
* Make the Function size_map public
* Merge the Function and TimeFunction _is_on_grid

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

API api (symbolics, types, ...) no-pro-trigger Skip the devitopro submodule update on merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants