Skip to content

Fix RLE copy commands 0x2X/0x3X in sas7bdat (14-bit length) - #436

Open
hpoettker wants to merge 1 commit into
WizardMac:devfrom
hpoettker:fix-rle-compression
Open

hpoettker wants to merge 1 commit into
WizardMac:devfrom
hpoettker:fix-rle-compression

Conversation

@hpoettker

Copy link
Copy Markdown
Contributor

Resolves #307.

Problem

Reading RLE-compressed (COMPRESS=CHAR) sas7bdat files failed with Row #0 decompressed to -1 bytes (expected N bytes) when a row contained a non-repeating stretch longer than 8255 bytes, e.g. a long string of repeated "test". Repeated single characters were unaffected because they compress into insert commands instead of literal copies.

Root cause

SAS uses RLE commands 0x0X–0x3X as one copy command with a 14-bit length (up to 16447 bytes). ReadStat only decoded 0x0X and 0x1X this way, so longer copies broke the decoding of the whole row.

Changes

  • Decoder: decode commands 0–3 as one 14-bit copy, and fail on unknown commands instead of skipping them.
  • Writer: emit copies of up to 16447 bytes, like SAS does.
  • Test: round-trip test with long literal runs; it fails against the previous reader.

Validation

  • Read without errors: SAS 9.4 files (Linux and Windows) with literal runs from 4150 to 16450 bytes, long insert runs, mixed content and a 200-row multi-page dataset. Every row decodes to exactly the expected length.
  • Written: 32- and 64-bit files produced via ReadStat with this change read back in SAS 9.4 with matching lengths and MD5 checksums, covering every command boundary up to 30000-byte runs.

@hpoettker

Copy link
Copy Markdown
Contributor Author

Some Linux builds keep failing as the installation of dependencies runs into time-outs.

I'll push another empty commit later to restart the build.

@hpoettker

Copy link
Copy Markdown
Contributor Author

Thanks for the quick merge of #437! I've rebased on dev.

But the theory of constraints wins again. Now that the Linux builds are successful, it's the MacOS builds that fail due to a shortage of runners...

I'll push another empty commit later.

@hpoettker

Copy link
Copy Markdown
Contributor Author

And it's green! 🥬

@hpoettker
hpoettker force-pushed the fix-rle-compression branch 3 times, most recently from 24622fa to 80f0378 Compare November 19, 2026 13:33
@hpoettker

Copy link
Copy Markdown
Contributor Author

Sorry for the multiple pushes. But the small merge conflict with #438 is now fixed.

The Windows MSVC build got a runner with zlib 1.4, so it will unfortunately fail due to the problem described in #439.

Reading RLE-compressed sas7bdat files failed with "Row #0 decompressed
to -1 bytes" when a row held a non-repeating run longer than 8255
bytes. SAS-generated files show that commands 0x0X-0x3X form a single
copy command with a 14-bit length (64..16447 bytes); ReadStat decoded
0x2X as a short copy and ignored 0x3X.

- Decode commands 0-3 with the shared 14-bit length and declare their
  length byte for the input bounds check.
- Return -1 for unknown commands instead of skipping them.
- Let the compressor emit copies of up to 16447 bytes accordingly.

Add a round-trip test with long literal runs (a 0x2X copy and a split
0x3F 0xFF run), which fails against the previous reader.
@hpoettker

Copy link
Copy Markdown
Contributor Author

With the latest empty push, the build is now successful again.

The reliability of the Windows build has much improved with #426 and #439, and there is a meaningful speed-up in the Linux builds with #437. But I think the non-availability of ARM MacOS runners in the European late afternoon/evening is a problem that we won't be able to solve on a project level.

If there are concerns or questions about this PR or #434, please let me know. I'd be more than happy to implement requested changes or to generate test data with SAS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reading sas7bdat gets "A row in the file was not the expected length"

1 participant