Skip to content

feat: add Stream Stores and the Stream calls of Workflow code - #384

Closed
moetemp wants to merge 6 commits into
moe/redo/regenfrom
moe/redo/streams-python
Closed

moetemp wants to merge 6 commits into
moe/redo/regenfrom
moe/redo/streams-python

Conversation

@moetemp

@moetemp moetemp commented Nov 22, 2026

Copy link
Copy Markdown
Owner

This PR adds the hand-written part of the Streams API, in six commits to read in order.

It's open on the fork so that CI runs on the whole stack. The base has the generated API. The Core submodule points at a commit of moetemp/sdk-rust, since the Stream calls of Core aren't merged yet.

  • StreamStore is the interface of a Store, and StreamConfig on the client says which Stores a client and its Workers have. MemoryStreamStore is a Store for tests.
  • Client and Activity code calls a Store directly. Workflow code sends an append, a close or a read to Core as a command, and Core calls the same Store through the bridge.
  • A Workflow read_page is one read command of Core with one job.
  • temporalio.contrib.streams has BufferedAppender, which batches the appends of client and Activity code.

The Workflow path of Streams needs the Stream commands and jobs of this Core.
The regenerated payload visitor reaches their payloads, so the rule for a Stream
body is in this commit.
A Stream Store is the handler side of the Stream service, which the generated
code doesn't have. The memory Store lets a test use Streams with no other
service, behind the checks that Core runs for a Store of Workflow code.
The server has no Streams yet, so the last client interceptor sends each
Stream call to a Store of the client's Stream config. The generated Stream
code also needs its helpers, apart from the two that build a Producer.
Workflow code can't call a Store itself, since a replay must get the same answers. Core makes the append and the close inside the Workflow Task and records each result in a marker. A create and a describe have no command in Core, so they run in a Local Activity of the SDK.
A read of Workflow code is one command of Core with one job, as a Local Activity is, so no state of a read depends on where an activation ends. A replay reads the Records again from the Store, so a replay Worker gets the event loop that its Python Stores need.
An Activity that appends each model token pays one Store call for each token. The appender batches them with one write in flight, and stops at the first failed batch so that a reader sees no gap.
@moetemp moetemp closed this Nov 22, 2026
@moetemp moetemp reopened this Nov 22, 2026
@moetemp

moetemp commented Nov 22, 2026

Copy link
Copy Markdown
Owner Author

Moved to temporalio#2129.

@moetemp moetemp closed this Nov 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant