A Rust client for Apache Pulsar scalable topics, built from the Scalable Topics client specification on the compio runtime.
Status: milestones 3 to 5. Producers publish to scalable topics, keeping per-key order across segment splits and
merges, and to regular persistent:// topics through the migration bridge. Queue consumers read every segment of a
topic, following its splits and merges, and acknowledge each message. Stream consumers read the segments the topic's
controller assigns them, each key in order, share segments by entry bucket when they outnumber them, and acknowledge
cumulatively. Checkpoint consumers read every segment, parents before their children, or as members of a group the
segments the controller assigns them, and resume from checkpoints the application keeps, in the Java client's bytes.
All work in the thread-safe, local and blocking APIs, with token and mTLS authentication, directly or through a proxy.
Producers batch messages by entry bucket and compress them, as Java's do. The design is in
DESIGN.md, the milestones in PLAN.md.
let client = PulsarClient::builder().service_url("pulsar://localhost:6650").build().await?;
let producer = client.new_producer(Schema::string()).topic("topic://public/default/orders").create().await?;
let id = producer.new_message().key("user-42").value("hello".to_string()).send().await?;
producer.close().await?;
let consumer = client.new_queue_consumer(Schema::string())
.topic("topic://public/default/orders")
.subscription_name("workers")
.subscribe().await?;
let message = consumer.receive().await?;
consumer.acknowledge(message.id()).await?;
client.close().await?;| Path | What it is |
|---|---|
crates/apache-pulsar |
The client: producers, Queue, Stream and Checkpoint consumers, layout watches, connections, the pool, lookup, authentication, TLS, the memory limit, and the thread-safe, local and blocking APIs. |
crates/apache-pulsar-protocol |
The sans-IO protocol core: generated protobuf code, framing, the connection, producer and consumer state machines, entry decoding, layouts and routing, message ids. Internal; not a stable API. |
crates/apache-pulsar-testkit |
The simulated broker and cluster, and the deterministic tests of the protocol state machines. Not published. |
checks/ |
Milestone 0 checks: compio behavior, and the handshake against a real broker. |
tools/java-vectors |
Generates test vectors from the Java client. |
xtask |
Repository tasks: protobuf code generation. |
cargo test --workspaceThe generated protobuf code is committed, so building needs no protoc. After changing the vendored
crates/apache-pulsar-protocol/proto/PulsarApi.proto, regenerate it (needs protoc 33.1, the version CI checks
with):
cargo run -p xtask -- codegenThe handshake against a real broker:
docker run -d --rm --name pulsar -p 6650:6650 -p 8080:8080 apachepulsar/pulsar:5.0.1 bin/pulsar standalonecargo run -p first-contact -- 127.0.0.1:6650The integration tests start a broker and a proxy in Docker, with TLS, token and mTLS authentication, and remove
them when they finish. A Java tool they run in the broker container is compiled in an eclipse-temurin JDK
container. cargo test skips them; they need Docker running:
cargo test -p apache-pulsar --test integration