Skip to content

feat: the cache collects the work directories of plans that describe - #977

Open
NopeCharon wants to merge 2 commits into
mcpp-community:mainfrom
NopeCharon:feat/cache-collects-plan-work-dirs
Open

NopeCharon wants to merge 2 commits into
mcpp-community:mainfrom
NopeCharon:feat/cache-collects-plan-work-dirs

Conversation

@NopeCharon

Copy link
Copy Markdown
Contributor

Summary

mcpp emit build-database, mcpp pack --list-formats and the pin check of mcpp publish plan a project without writing into it. Each plans in a directory of the home, one per project: cache/build-database/<key>, cache/pack-formats/<key> and cache/publish-plan/<key>. Nothing removed these directories. mcpp cache gc, prune and clean read only the build cache, so the directory of a project that no longer exists stayed for ever. The issue measured 626 of them, 3.4 GB, in one home, and another 1.9 GB under pack-formats.

The three commands now get their directory from one function, mcpp::bmi_cache::plan_work_dir. It creates the directory and records each use on a .mcpp-plan record. A new directory gets the record with the project root in it. For an existing one, mcpp::home::mark_used sets the record's time, the same rule the build cache's entries follow.

The cache maintenance walks these directories as entries of kind work:

  • mcpp cache list shows each one, named by the command and the project its record names, e.g. emit build-database (work directory of /home/me/demo).
  • The collection after a build ([cache] unused_days), cache prune --older-than, cache gc --older-than and gc --max-size collect a directory by its record's time. A directory written by an earlier mcpp, without a record, is dated by its own time.
  • mcpp cache clean --work empties the three areas, and --all now includes them. A bare cache clean still drops package entries only.

Removing a directory costs the next plan of that project only writing it again. SPEC-005 R2.1 already calls the directory a cache that can be deleted at any time. docs/09 (store table and a paragraph) and docs/04 (the cache clean synopsis) are updated in English and Chinese.

Closes #969.

Criteria

  • e2e cache::entries/1119 the work directories of plans are cache entries is new and needs no toolchain. It seeds the three areas in a fresh home: one directory used 3 days ago, one used now, and one from before the record, 3 days old. It checks:

    • the three work rows of cache list --json and their labels;
    • that prune --older-than 1d and gc --older-than 1d remove the old directories and keep the fresh one;
    • that gc --max-size 1B removes the fresh one;
    • that a bare clean keeps all three, and clean --work and clean --all remove them.

    On a Linux aarch64 host I built mcpp with this branch (released 2026.11.19.1 as the bootstrap) and ran the case with the fresh binary as MCPP_E2E_MCPP: it passes. With the released 2026.11.19.1 as the binary under test it fails at its first check (cache list does not list the three work directories, "entries": []).

  • e2e cdb::sets/921 now also checks that emit build-database writes the .mcpp-plan record naming the project. It passes on the same host. I did not run it against the released binary. That binary writes no record, so the check cannot pass there.

  • On the same host, cache::entries/172, 174, 184, 959 and 1029 and cdb::sets/840 and 913 pass with the fresh binary. cache::entries/212 fails there with and without this change: it expects GCC's std.gcm, and this host's default toolchain produces std.pcm. That case is left to CI's GCC hosts.

  • I checked by hand with the fresh binary in a new project. emit build-database and pack --list-formats each created their directory with the record. cache list showed both as work rows. cache clean --work removed them (Cleaned work directories of plans (142.8 KiB)).

  • The Windows and macOS runs of 1119 are left to CI. The case uses only file-system calls that 1029 already uses on those hosts.

Intersections

New rule or feature Invariant it crosses Test at the crossing
Work directories are collected by last use A plan that describes never writes into the project and works in its own directory (SPEC-005 R2.1, R2.3) cdb::sets/921: its host lock under the work directory is not rewritten by a repeated emit, and the record names the project
Work directories are cache list entries cache list --json keeps its {root, entries} shape and each entry's keys cache::entries/1119 reads the work rows through the same keys 174 asserts
gc --max-size counts work directories std modules are never evicted for a size budget cache::entries/1029 (gc --max-size 0 keeps the std module)
cache clean --work, --all includes it A bare cache clean drops package entries only cache::entries/1119 (bare clean keeps the work directories), 174

Compatibility

cache list shows new work rows. Directories that existing homes already hold are listed and, by their own time, collected by the first build that runs the daily collection after 30 days unused, or by an explicit gc/prune. A removed directory is recreated by the next plan, which then re-runs that project's build programs once, as on a first run. The paths and keys of the work directories are unchanged. cache clean --all now also removes them. The gc summary still names its figure "package and tool entries", and that figure now includes the work directories, as it already included patched source trees.

Checks before merging

  • bash .github/tools/check_docs_style.sh, check_docs_structure.sh and check_version_pins.sh pass.
  • python3 .github/tools/check_workflow_assertions.py passes (a step asserts what its name says).
  • No commit on the branch carries an attribution trailer:
    git log origin/main..HEAD -i --grep='Co-Authored-By' prints nothing.
  • The squash merge is given an explicit subject and body, so GitHub does not
    compose one from the branch's commits. Suggested subject: feat: the cache collects the work directories of plans that describe.

`mcpp emit build-database`, `mcpp pack --list-formats` and the pin check of
`mcpp publish` plan a project without writing into it, each in a directory
of the home: `cache/build-database/<key>`, `cache/pack-formats/<key>` and
`cache/publish-plan/<key>`, one per project. Nothing removed one. `cache gc`,
`prune` and `clean` read only the build cache, so the directory of a project
that no longer exists stayed for ever: one home held 626 of them, 3.4 GB,
and another 1.9 GB under pack-formats (mcpp#969).

The three commands now take their directory from one function,
mcpp::bmi_cache::plan_work_dir. It creates the directory and records each
use on its `.mcpp-plan` record: the record is written with the project root
when the directory is new, and otherwise its time is set by
mcpp::home::mark_used, the same rule the build cache's entries follow.

The cache's maintenance walks these directories as entries of kind `work`:

- `mcpp cache list` shows each one, named by the command and the project
  its record names.
- The collection after a build, `cache prune --older-than`,
  `cache gc --older-than` and `gc --max-size` collect a directory by its
  record's time. A directory written before the record is dated by its own
  time.
- `mcpp cache clean --work` empties the three areas, and `--all` now
  includes them. A bare `cache clean` still drops package entries only.

Removing a directory costs the next plan of that project only writing it
again (SPEC-005 R2.1 already calls the directory a cache).

e2e 1119 seeds the three areas in a fresh home and checks the listing,
the collection by age and by size, and the clean flags. e2e 921 checks
that `emit build-database` records the project in its work directory.
docs/09 and docs/04 describe the store and the flag, in English and Chinese.

Closes mcpp-community#969.
…e return

The product writes and reads .mcpp-plan in binary; the e2e case seeded it
in text mode, so on Windows the project label ended in a carriage return.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The work directories of emit build-database are never collected: 3.4 GB in one home

1 participant