Skip to content

Fix wfdb_to_mat encoding, slicing, and header metadata - #633

Closed
goktugozkanmd wants to merge 1 commit into
MIT-LCP:mainfrom
goktugozkanmd:fix/wfdb-to-mat
Closed

goktugozkanmd wants to merge 1 commit into
MIT-LCP:mainfrom
goktugozkanmd:fix/wfdb-to-mat

Conversation

@goktugozkanmd

Copy link
Copy Markdown
Contributor

Fixes several crashes and data-corruption bugs in wfdb_to_mat (src/wfdb/io/convert/matlab.py), which previously had no unit tests:

  • Channel selection (channels): channels was not forwarded to rdrecord(), so all channels were written to .mat while record.n_sig was overwritten to len(channels), causing record.wrheader() to raise ValueError.
  • Sample slicing (sampfrom without sampto): desired_length = sampto - sampfrom subtracted sampfrom a second time after rdrecord() had already sliced the signal, corrupting the declared matrix dimensions or raising struct.error when sampfrom > sig_len / 2.
  • 8-bit (80) and 24/32-bit (24/32) binary encoding & MAT v5 padding: signal samples were always packed with struct.pack("<%sh", ...) regardless of bytes_per_element, corrupting format 80 files (TypeError in scipy.io.loadmat, missing the +128 offset-binary shift) and crashing on 32-bit values outside int16 range. Trailing 8-byte MAT v5 alignment padding (bytes_remain) was also accounted for in master_bytes but never written.
  • Missing samples & unset header fields: reading via physical=False (d_signal) avoids float round-trip errors and NaN-cast warnings; missing/invalid samples (_digi_nan) are mapped to the target format's sentinel instead of overflowing when subtracting adc_zero (e.g., sample-data/mixedsignals). Unset (None) adc_res, adc_zero, baseline, units, and sig_name fields (e.g., sample-data/n8_evoked_raw_95_F1_R9) no longer raise TypeError/AttributeError.
  • Companion .hea accuracy & cross-platform paths: clears already-applied skew, sets block_size to 0, sets init_value from signed decoded samples, recomputes checksum for the written output, and uses os.path.basename(os.fspath(record_name)) so forward-slash paths work on Windows as well as POSIX.

Changes

  • src/wfdb/io/convert/matlab.py: fix reading, binary serialization, padding, missing-sample preservation, and companion header metadata in wfdb_to_mat.
  • tests/io/test_convert.py: add TestWfdbToMat covering 16-bit padding and round-trip, channels + sampfrom slicing, 8-bit (format 80) offset-binary output, 24/32-bit records with unset header fields, missing-sample preservation and skew handling, and forward-slash paths (verified with both scipy.io.loadmat and wfdb.rdrecord).
  • changelog.d/wfdb-to-mat-fixes.rst: add towncrier changelog fragment.

Verification

  • uv run ruff format --check --diff . — passed
  • uv run ruff check . — passed
  • uv run towncrier build --draft — passed
  • uv run pytest tests/io/test_convert.py -v — 22 passed
  • uv run pytest -n auto -q -k "not url and not pn_dir" — 360 passed, 4 skipped (2 POSIX permissions, 2 remote PhysioNet 502)

@tompollard tompollard left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, it's a big improvement to wfdb_to_mat. Reading with physical=False, taking the length from record.sig_len, writing the padding, taking init_value from the signed values, and using adc_zero for a missing baseline all look right to me. The new tests pass locally, and five of the six fail with main's matlab.py, so they cover the fixes.

Valid samples can still overflow (matlab.py:246). The missing samples are handled now, but np.ascontiguousarray(..., dtype=out_type) still wraps silently when a valid sample doesn't fit. The output type is chosen from adc_res alone, so it doesn't allow for the adc_zero shift or for a header with a resolution that is too small:

Input (format 16) Written to the .mat Read back
adc_zero=1, samples [-32767, 0, 32767] [-32768, -1, 32766] the first sample becomes the invalid value (NaN)
adc_zero=-1, samples [-32767, 0, 32767] [-32766, 1, -32768] 32767 wraps to the invalid value
adc_res=8, samples [100, 300, -300] uint8 [228, 172, 84] 300 and -300 are corrupted

main does the same, so this PR doesn't cause it. Since the PR rewrites this line, could it be fixed here? Either check the min and max of shifted_data (leaving out the missing samples) against the output type and move up to the next size, or raise an error instead of writing wrong values. If you'd rather keep this PR as it is, I'm happy to open an issue for it.

Smaller things:

  • record.n_sig = num_channels (line 289) now does nothing, since num_channels = record.n_sig. desired_length (line 157) could also just be record.sig_len.
  • test_missing_samples_and_skew_preserved tests two unrelated things, so if the first part fails the skew check doesn't run. Could it be split into two tests?
  • test_forward_slash_path only tests something on Windows (it passes with main on Linux and macOS). A short comment saying so would stop someone removing it later.
  • Line 300 repeats the logic of calc_checksum. Fine as it is, just noting it in case they ever need to stay in sync.

@tompollard

Copy link
Copy Markdown
Member

Could you rebase this on main? #632 was merged and added tests for wfdb_to_wav at the end of tests/io/test_convert.py, so there's a conflict there. Keeping both sides resolves it: the wave and pathlib imports, and both test classes. I've opened #641 for the overflow and #642 for the smaller points from my review, so they don't need to hold up this PR.

tompollard added a commit that referenced this pull request Nov 21, 2026
## Summary

This branch brings the contributor's #633 commit onto current `main`,
preserving its authorship, and resolves the conflict with newer
converter tests. It adds the range safety fix from #641.

- Preserve #633's fixes for channel selection, slicing, MAT encoding and
padding, missing samples, and companion header metadata.
- Check the range of valid samples after the `adc_zero` shift. Promote
to a wider MAT type when needed, including when a shifted valid sample
would collide with the missing-value marker.
- Raise `ValueError` before writing an output file if the shifted
samples cannot fit in the 32-bit WFDB format.
- Add regression tests for both shift directions, understated `adc_res`,
missing samples, and the 32-bit limit.

## Verification

- Offline suite: 382 passed, 17 network tests deselected.
- Converter suite: 37 passed; MAT converter tests: 10 passed.
- Ruff format and lint passed.
- Sphinx HTML build passed with warnings treated as errors.

The first commit is from #633 by @goktugozkanmd and OpenBot; the second
commit addresses #641. This PR supersedes #633 for the release branch.

Closes #641.
@tompollard

Copy link
Copy Markdown
Member

Thank you for updating this so quickly. Our changes crossed: #650 merged just before your update. It includes your original commit with your authorship preserved, resolves the test conflict, and adds the range check from #641. I compared your updated commit with the version we incorporated and found no additional changes. I'm sorry I didn't let you know before taking this route, especially after asking you to rebase. Thank you for the fixes and tests. I'll close this PR as superseded by #650.

@tompollard tompollard closed this Nov 21, 2026
tompollard added a commit that referenced this pull request Nov 21, 2026
## Summary

Prepare wfdb 4.4.0 for release.

- Set the package version to 4.4.0.
- Generate the 4.4.0 release notes from all pending changelog fragments.
- Fill the previously empty ADC float32 overflow fragment before
consuming it.

This PR targets `main` now that #650 has merged. Thank you to
@goktugozkanmd for the `wfdb_to_mat` fixes and tests in #633. They were
incorporated into #650 with the original commit authorship preserved,
and are documented in these release notes.

## Checks

- Offline test suite on #650: 382 passed, 17 deselected.
- Ruff check and format check passed.
- Sphinx HTML build with warnings as errors passed.
- `uv build` produced the 4.4.0 sdist and wheel.

No release tag or package publication is included.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants