Repository navigation
Fix wfdb_to_mat encoding, slicing, and header metadata - #633
goktugozkanmd wants to merge 1 commit into
Conversation
tompollard
left a comment
There was a problem hiding this comment.
Thanks for this, it's a big improvement to wfdb_to_mat. Reading with physical=False, taking the length from record.sig_len, writing the padding, taking init_value from the signed values, and using adc_zero for a missing baseline all look right to me. The new tests pass locally, and five of the six fail with main's matlab.py, so they cover the fixes.
Valid samples can still overflow (matlab.py:246). The missing samples are handled now, but np.ascontiguousarray(..., dtype=out_type) still wraps silently when a valid sample doesn't fit. The output type is chosen from adc_res alone, so it doesn't allow for the adc_zero shift or for a header with a resolution that is too small:
| Input (format 16) | Written to the .mat | Read back |
|---|---|---|
adc_zero=1, samples [-32767, 0, 32767] |
[-32768, -1, 32766] |
the first sample becomes the invalid value (NaN) |
adc_zero=-1, samples [-32767, 0, 32767] |
[-32766, 1, -32768] |
32767 wraps to the invalid value |
adc_res=8, samples [100, 300, -300] |
uint8 [228, 172, 84] |
300 and -300 are corrupted |
main does the same, so this PR doesn't cause it. Since the PR rewrites this line, could it be fixed here? Either check the min and max of shifted_data (leaving out the missing samples) against the output type and move up to the next size, or raise an error instead of writing wrong values. If you'd rather keep this PR as it is, I'm happy to open an issue for it.
Smaller things:
record.n_sig = num_channels(line 289) now does nothing, sincenum_channels = record.n_sig.desired_length(line 157) could also just berecord.sig_len.test_missing_samples_and_skew_preservedtests two unrelated things, so if the first part fails the skew check doesn't run. Could it be split into two tests?test_forward_slash_pathonly tests something on Windows (it passes withmainon Linux and macOS). A short comment saying so would stop someone removing it later.- Line 300 repeats the logic of
calc_checksum. Fine as it is, just noting it in case they ever need to stay in sync.
|
Could you rebase this on |
## Summary This branch brings the contributor's #633 commit onto current `main`, preserving its authorship, and resolves the conflict with newer converter tests. It adds the range safety fix from #641. - Preserve #633's fixes for channel selection, slicing, MAT encoding and padding, missing samples, and companion header metadata. - Check the range of valid samples after the `adc_zero` shift. Promote to a wider MAT type when needed, including when a shifted valid sample would collide with the missing-value marker. - Raise `ValueError` before writing an output file if the shifted samples cannot fit in the 32-bit WFDB format. - Add regression tests for both shift directions, understated `adc_res`, missing samples, and the 32-bit limit. ## Verification - Offline suite: 382 passed, 17 network tests deselected. - Converter suite: 37 passed; MAT converter tests: 10 passed. - Ruff format and lint passed. - Sphinx HTML build passed with warnings treated as errors. The first commit is from #633 by @goktugozkanmd and OpenBot; the second commit addresses #641. This PR supersedes #633 for the release branch. Closes #641.
d2d89e2 to
0535939
Compare
|
Thank you for updating this so quickly. Our changes crossed: #650 merged just before your update. It includes your original commit with your authorship preserved, resolves the test conflict, and adds the range check from #641. I compared your updated commit with the version we incorporated and found no additional changes. I'm sorry I didn't let you know before taking this route, especially after asking you to rebase. Thank you for the fixes and tests. I'll close this PR as superseded by #650. |
## Summary Prepare wfdb 4.4.0 for release. - Set the package version to 4.4.0. - Generate the 4.4.0 release notes from all pending changelog fragments. - Fill the previously empty ADC float32 overflow fragment before consuming it. This PR targets `main` now that #650 has merged. Thank you to @goktugozkanmd for the `wfdb_to_mat` fixes and tests in #633. They were incorporated into #650 with the original commit authorship preserved, and are documented in these release notes. ## Checks - Offline test suite on #650: 382 passed, 17 deselected. - Ruff check and format check passed. - Sphinx HTML build with warnings as errors passed. - `uv build` produced the 4.4.0 sdist and wheel. No release tag or package publication is included.
Fixes several crashes and data-corruption bugs in
wfdb_to_mat(src/wfdb/io/convert/matlab.py), which previously had no unit tests:channels):channelswas not forwarded tordrecord(), so all channels were written to.matwhilerecord.n_sigwas overwritten tolen(channels), causingrecord.wrheader()to raiseValueError.sampfromwithoutsampto):desired_length = sampto - sampfromsubtractedsampfroma second time afterrdrecord()had already sliced the signal, corrupting the declared matrix dimensions or raisingstruct.errorwhensampfrom > sig_len / 2.80) and 24/32-bit (24/32) binary encoding & MAT v5 padding: signal samples were always packed withstruct.pack("<%sh", ...)regardless ofbytes_per_element, corrupting format80files (TypeErrorinscipy.io.loadmat, missing the+128offset-binary shift) and crashing on 32-bit values outsideint16range. Trailing 8-byte MAT v5 alignment padding (bytes_remain) was also accounted for inmaster_bytesbut never written.physical=False(d_signal) avoids float round-trip errors andNaN-cast warnings; missing/invalid samples (_digi_nan) are mapped to the target format's sentinel instead of overflowing when subtractingadc_zero(e.g.,sample-data/mixedsignals). Unset (None)adc_res,adc_zero,baseline,units, andsig_namefields (e.g.,sample-data/n8_evoked_raw_95_F1_R9) no longer raiseTypeError/AttributeError..heaaccuracy & cross-platform paths: clears already-appliedskew, setsblock_sizeto0, setsinit_valuefrom signed decoded samples, recomputeschecksumfor the written output, and usesos.path.basename(os.fspath(record_name))so forward-slash paths work on Windows as well as POSIX.Changes
src/wfdb/io/convert/matlab.py: fix reading, binary serialization, padding, missing-sample preservation, and companion header metadata inwfdb_to_mat.tests/io/test_convert.py: addTestWfdbToMatcovering 16-bit padding and round-trip,channels+sampfromslicing, 8-bit (format 80) offset-binary output, 24/32-bit records with unset header fields, missing-sample preservation and skew handling, and forward-slash paths (verified with bothscipy.io.loadmatandwfdb.rdrecord).changelog.d/wfdb-to-mat-fixes.rst: add towncrier changelog fragment.Verification
uv run ruff format --check --diff .— passeduv run ruff check .— passeduv run towncrier build --draft— passeduv run pytest tests/io/test_convert.py -v— 22 passeduv run pytest -n auto -q -k "not url and not pn_dir"— 360 passed, 4 skipped (2 POSIX permissions, 2 remote PhysioNet 502)