Skip to content

fix(operator): Wait for the tee processes of capture_shell_output on exit - #1336

Open
lfrancke wants to merge 2 commits into
mainfrom
fix/capture-shell-output-wait-for-tee
Open

lfrancke wants to merge 2 commits into
mainfrom
fix/capture-shell-output-wait-for-tee

Conversation

@lfrancke

@lfrancke lfrancke commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Description

Note

Build fails until #1337 is merged

Side finding from my bug hunting/jig/kuttl investigation.
ZK operator has a bunch of random test failures in the logging test because of this bug so this should increase stability.

This affects mostly init containers which lose their captured output (stdout/stderr) sometimes (timing dependent) which means Vector doesn't ship anything, so the logs are missing and the test fails :(

capture_shell_output uses tee to capture the outptu but tee is a separate process and nothing waits for them to actually finish their job. It kinda looks like it's all in the bash process but it.s not. So when the PID 1 bash exits everything is torn down and this might happen before tee had time to write.

So this now sets an exit trap that waits for the tee processes. The wait is limited to 5 seconds (arbitrarily chosen). The reason is Trino's password-file updater which has a background process that keeps the pipe open and tee would never finish and we'd wait until the end of the grace period.

Tested using this:
Init container running the generated command plus echo hello && echo err >&2, using the ZooKeeper image, 50 pods each on k3d:

Prefix Pods with empty log files
main 49/50
this PR 0/50

Callers must not set their own EXIT trap but none of our operators currently do.

I know this is not super trivial to review due to all the bash-fu but it all makes sense to me.

Follow-up for spark-k8s-operator: the job container ends with sleep 1 to wait for the log file, which is no longer needed after this change.

Definition of Done Checklist

  • Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
  • Please make sure all these things are done and tick the boxes

Author

  • Changes are OpenShift compatible
  • CRD changes approved
  • CRD documentation for all fields, following the style guide.
  • Integration tests passed (for non trivial changes)
  • Changes need to be "offline" compatible

Reviewer

  • Code contains useful comments
  • Code contains useful logging statements
  • (Integration-)Test cases added
  • Documentation added or updated. Follows the style guide.
  • Changelog updated
  • Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

  • Feature Tracker has been updated
  • Proper release label has been added

@lfrancke
lfrancke force-pushed the fix/capture-shell-output-wait-for-tee branch from 0c971fd to 45b9ca8 Compare November 18, 2026 10:00
@lfrancke
lfrancke marked this pull request as ready for review November 18, 2026 10:15
@lfrancke lfrancke self-assigned this Nov 18, 2026
@lfrancke lfrancke moved this to Development: Waiting for Review in Stackable Engineering Nov 18, 2026
@adwk67
adwk67 self-requested a review November 19, 2026 16:47
@adwk67 adwk67 moved this from Development: Waiting for Review to Development: In Review in Stackable Engineering Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Development: In Review

Development

Successfully merging this pull request may close these issues.

1 participant