Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions doc/rfc/stovepipe/steps/process.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,13 +60,13 @@ Strategy and baseline are persisted on the Request and are **immutable**: a rede

Validation is expensive and shares a baseline, so heads arriving while an earlier one runs `build → buildsignal → record` must not all start builds at once.

**Runtime state** lives on the Queue row; the **concurrency cap** does not — it is deployment configuration, resolved at gate-check time the same way SubmitQueue separates `storage` from `queueconfig` (see [submitqueue/extension/queueconfig/README.md](../../../../submitqueue/extension/queueconfig/README.md)): pipeline stages read mutable state from the store and read knobs like `max_concurrent` from a config `Store` (or a wiring default for MVP). Config is not written by ingest/process/record and does not need optimistic locking.
**Runtime state** lives on the Queue row; the **concurrency cap** does not — it is deployment configuration, resolved at gate-check time the same way SubmitQueue separates `storage` from `queueconfig` (see [submitqueue/extension/queueconfig/README.md](../../../../submitqueue/extension/queueconfig/README.md)): pipeline stages read mutable state from the store and read knobs like `max_concurrent` from a config `Store`. Config is not written by ingest/process/record and does not need optimistic locking.

| Source | Field | Meaning |
|---|---|---|
| Queue row | `last_green_uri` | Bookmark `buildsignal` advances on a successful build; empty until first green. |
| Queue row | `in_flight_count` | Requests past `process` and not yet terminal. `process` increments on admit; `record` (or DLQ reconciliation) decrements on terminal. |
| Queue config | `max_concurrent` | Cap on concurrent in-flight validations. **Default 1** (global wiring default for MVP; per-queue override when a Stovepipe `queueconfig` extension lands). |
| Queue config | `max_concurrent` | Cap on concurrent in-flight validations. **Default 1** in the example queue configuration file; shared defaults and per-queue overrides live under the process stage configuration. |

A slot is held from admit until the build goes terminal (`process → build → buildsignal`), not just while `process` runs. It is released when the Request reaches **any** terminal state and `in_flight_count` is decremented — `buildsignal` recording the build's outcome, success *or* failure, or the DLQ reconciler forcing a terminal `failed` (see [integrity](#in_flight_count-integrity)). A build *failure* frees the slot just like a success; only a Request that never terminates keeps its slot.

Expand Down
8 changes: 7 additions & 1 deletion service/stovepipe/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,12 @@ The Stovepipe controllers live under [`stovepipe/controller/`](../../stovepipe/c
| `PORT` | no | gRPC listen address | `:8083` |
| `HOSTNAME` | no | Subscriber name for the queue consumers | `stovepipe-<unix_ts>` |

## Queue and stage configuration

The example server reads `queues.yaml` from its working directory by default; the normal and debug Docker images include [`server/queues.yaml`](server/queues.yaml). Select another complete file with `-config`. No embedded baseline or cross-file merging is involved.

The file contains shared settings under `defaults.stages` and explicitly named `queues` with optional stage overrides. Queue names must match `MQ_TENANTS`. Invalid configuration fails startup; changes require a restart. See the [configuration provider](../../stovepipe/extension/queueconfig/README.md) for inheritance and validation rules.

## Running

### Docker Compose (recommended)
Expand Down Expand Up @@ -92,7 +98,7 @@ History lookup is defined by retained `request_log` rows. A request with no reta
bazel build //service/stovepipe/server:stovepipe
bazel build //service/stovepipe/client:stovepipe

go run ./service/stovepipe/server
go run ./service/stovepipe/server -config service/stovepipe/server/queues.yaml
```

## Testing the Ping RPC
Expand Down
7 changes: 6 additions & 1 deletion service/stovepipe/server/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,8 @@ go_library(
"//stovepipe/extension/fixer/noop:go_default_library",
"//stovepipe/extension/projectissues/noop:go_default_library",
"//stovepipe/extension/projectstatus/noop:go_default_library",
"//stovepipe/extension/queueconfig/default:go_default_library",
"//stovepipe/extension/queueconfig:go_default_library",
"//stovepipe/extension/queueconfig/yaml:go_default_library",
"//stovepipe/extension/sourcecontrol:go_default_library",
"//stovepipe/extension/sourcecontrol/fake:go_default_library",
"//stovepipe/extension/storage:go_default_library",
Expand Down Expand Up @@ -80,6 +81,7 @@ filegroup(
testonly = True,
srcs = [
"Dockerfile",
"queues.yaml",
":stovepipe_linux",
],
visibility = ["//test:__subpackages__"],
Expand All @@ -88,6 +90,7 @@ filegroup(
go_test(
name = "go_default_test",
srcs = ["main_test.go"],
data = ["queues.yaml"],
embed = [":go_default_library"],
deps = [
"//api/base/hook:go_default_library",
Expand All @@ -101,6 +104,8 @@ go_test(
"//stovepipe/core/requestlog:go_default_library",
"//stovepipe/entity:go_default_library",
"//stovepipe/extension/buildissue/noop:go_default_library",
"//stovepipe/extension/queueconfig:go_default_library",
"//stovepipe/extension/queueconfig/yaml:go_default_library",
"//stovepipe/extension/storage:go_default_library",
"//stovepipe/extension/storage/mock:go_default_library",
"@com_github_stretchr_testify//assert:go_default_library",
Expand Down
1 change: 1 addition & 0 deletions service/stovepipe/server/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ WORKDIR /root/
# Copy pre-built Linux binary.
# Built via: make build-stovepipe-linux
COPY .docker-bin/stovepipe ./stovepipe
COPY service/stovepipe/server/queues.yaml ./queues.yaml

EXPOSE 8080

Expand Down
1 change: 1 addition & 0 deletions service/stovepipe/server/Dockerfile.debug
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ WORKDIR /root/
# Copy pre-built Linux debug binary.
# Built via: make build-stovepipe-linux-debug
COPY .docker-bin/stovepipe-debug ./stovepipe
COPY service/stovepipe/server/queues.yaml ./queues.yaml

EXPOSE 8080 2345

Expand Down
59 changes: 49 additions & 10 deletions service/stovepipe/server/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ import (
"context"
"database/sql"
"errors"
"flag"
"fmt"
"net"
"os"
Expand Down Expand Up @@ -66,7 +67,8 @@ import (
fixernoop "github.com/uber/submitqueue/stovepipe/extension/fixer/noop"
projectissuesnoop "github.com/uber/submitqueue/stovepipe/extension/projectissues/noop"
projectstatusnoop "github.com/uber/submitqueue/stovepipe/extension/projectstatus/noop"
queueconfigdefault "github.com/uber/submitqueue/stovepipe/extension/queueconfig/default"
"github.com/uber/submitqueue/stovepipe/extension/queueconfig"
queueconfigyaml "github.com/uber/submitqueue/stovepipe/extension/queueconfig/yaml"
"github.com/uber/submitqueue/stovepipe/extension/sourcecontrol"
sourcecontrolfake "github.com/uber/submitqueue/stovepipe/extension/sourcecontrol/fake"
"github.com/uber/submitqueue/stovepipe/extension/storage"
Expand Down Expand Up @@ -114,8 +116,10 @@ func (hookResolver) For(*basehook.HookEvent) []hookext.Hook {
}

func main() {
configPath := flag.String("config", "queues.yaml", "path to the Stovepipe queue configuration file")
flag.Parse()
code := 0
if err := run(); err != nil {
if err := run(*configPath); err != nil {
if errors.Is(err, context.Canceled) {
fmt.Println("Stovepipe server stopped by signal")

Expand All @@ -131,7 +135,7 @@ func main() {
os.Exit(code)
}

func run() error {
func run(configPath string) error {
// Set up signal handling early so retry loops can respond to SIGTERM
ctx, cancel := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer cancel()
Expand All @@ -143,6 +147,32 @@ func run() error {
}
defer logger.Sync()

tenants, err := servicemq.ParseRequiredTenants(os.Getenv("MQ_TENANTS"))
if err != nil {
return fmt.Errorf("failed to configure queue subscribers: %w", err)
}

configs, err := queueconfigyaml.NewStore(configPath)
if err != nil {
return fmt.Errorf("load queue configurations: %w", err)
}

resolved, err := configs.List(ctx)
if err != nil {
return fmt.Errorf("list queue configurations: %w", err)
}

names := make([]string, 0, len(resolved))
for _, cfg := range resolved {
names = append(names, cfg.Name)
}

if err := servicemq.ValidateTenantSetsEqual("queue configuration", names, "MQ_TENANTS", tenants); err != nil {
return fmt.Errorf("configure queue subscribers: %w", err)
}

logger.Info("queue configurations loaded", zap.Any("queues", resolved))

// Initialize metrics scope
scope := tally.NewTestScope("stovepipe", nil)
metricsStopCh := make(chan interface{}, 1)
Expand Down Expand Up @@ -205,11 +235,6 @@ func run() error {
return fmt.Errorf("failed to configure queue database pool: %w", err)
}

tenants, err := servicemq.ParseRequiredTenants(os.Getenv("MQ_TENANTS"))
if err != nil {
return fmt.Errorf("failed to configure queue subscribers: %w", err)
}

mysqlQueue, err := queueMySQL.NewQueue(queueMySQL.Params{
DB: queueDB,
Logger: logger,
Expand Down Expand Up @@ -263,7 +288,20 @@ func run() error {

storageFty := storageFactory{backend: store}
materializer := requestlog.NewMaterializer(scope)
primaryCount, err := registerPrimaryControllers(primaryConsumer, logger.Sugar(), scope, storageFty, materializer, registry, sourceControl, brf, hookResolver{}, detectorFactory, repairExecutors)
primaryCount, err := registerPrimaryControllers(
primaryConsumer,
logger.Sugar(),
scope,
storageFty,
materializer,
configs,
registry,
sourceControl,
brf,
hookResolver{},
detectorFactory,
repairExecutors,
)
if err != nil {
return err
}
Expand Down Expand Up @@ -385,6 +423,7 @@ func registerPrimaryControllers(
scope tally.Scope,
store storage.Factory,
materializer requestlog.Materializer,
configs queueconfig.Store,
registry consumer.TopicRegistry,
sourceControl sourcecontrol.Factory,
brf buildrunner.Factory,
Expand All @@ -399,7 +438,7 @@ func registerPrimaryControllers(
scope,
store,
materializer,
queueconfigdefault.NewStore(),
configs,
sourceControl,
registry,
stovepipemq.TopicKeyProcess,
Expand Down
49 changes: 44 additions & 5 deletions service/stovepipe/server/main_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,8 @@ import (
"github.com/uber/submitqueue/stovepipe/core/requestlog"
"github.com/uber/submitqueue/stovepipe/entity"
buildissuenoop "github.com/uber/submitqueue/stovepipe/extension/buildissue/noop"
"github.com/uber/submitqueue/stovepipe/extension/queueconfig"
queueconfigyaml "github.com/uber/submitqueue/stovepipe/extension/queueconfig/yaml"
"github.com/uber/submitqueue/stovepipe/extension/storage"
storagemock "github.com/uber/submitqueue/stovepipe/extension/storage/mock"
"go.uber.org/mock/gomock"
Expand All @@ -53,11 +55,25 @@ func (c *recordingConsumer) Start(context.Context) error { return nil }

func (c *recordingConsumer) Stop(int64) error { return nil }

func testQueueConfigs(t *testing.T) queueconfig.Store {
t.Helper()

configs, err := queueconfigyaml.NewStore("queues.yaml")
require.NoError(t, err)

return configs
}

// registeredControllers runs the host's registration exactly as run() does and
// returns the registry it registers against, the primary controllers, and the
// DLQ controllers.
func registeredControllers(t *testing.T) (consumer.TopicRegistry, []consumer.Controller, []consumer.Controller) {
t.Helper()
return registeredControllersWithConfig(t, testQueueConfigs(t))
}

func registeredControllersWithConfig(t *testing.T, configs queueconfig.Store) (consumer.TopicRegistry, []consumer.Controller, []consumer.Controller) {
t.Helper()

registry, err := newTopicRegistry(nil, "subscriber")
require.NoError(t, err)
Expand All @@ -67,8 +83,20 @@ func registeredControllers(t *testing.T) (consumer.TopicRegistry, []consumer.Con
primary := &recordingConsumer{}
deadLetter := &recordingConsumer{}

_, err = registerPrimaryControllers(primary, logger, tally.NoopScope, store, requestlog.NewMaterializer(tally.NoopScope), registry,
fakeSourceControlFactory{}, fakeBuildRunnerFactory{}, hookResolver{}, buildissuenoop.New(), noopRepairExecutorFactory{})
_, err = registerPrimaryControllers(
primary,
logger,
tally.NoopScope,
store,
requestlog.NewMaterializer(tally.NoopScope),
configs,
registry,
fakeSourceControlFactory{},
fakeBuildRunnerFactory{},
hookResolver{},
buildissuenoop.New(),
noopRepairExecutorFactory{},
)
require.NoError(t, err)

_, err = registerDLQControllers(deadLetter, logger, tally.NoopScope, store, requestlog.NewMaterializer(tally.NoopScope), registry, noopRepairExecutorFactory{})
Expand Down Expand Up @@ -214,9 +242,20 @@ func TestRegisteredFixControllerRetainsPerExecutorDisposition(t *testing.T) {
registry, err := newTopicRegistry(queue, "subscriber")
require.NoError(t, err)
primary := &recordingConsumer{}
_, err = registerPrimaryControllers(primary, zaptest.NewLogger(t).Sugar(), tally.NoopScope, stores,
requestlog.NewMaterializer(tally.NoopScope), registry, fakeSourceControlFactory{}, fakeBuildRunnerFactory{},
hookResolver{}, buildissuenoop.New(), noopRepairExecutorFactory{})
_, err = registerPrimaryControllers(
primary,
zaptest.NewLogger(t).Sugar(),
tally.NoopScope,
stores,
requestlog.NewMaterializer(tally.NoopScope),
testQueueConfigs(t),
registry,
fakeSourceControlFactory{},
fakeBuildRunnerFactory{},
hookResolver{},
buildissuenoop.New(),
noopRepairExecutorFactory{},
)
require.NoError(t, err)
var fixController consumer.Controller
for _, controller := range primary.controllers {
Expand Down
17 changes: 17 additions & 0 deletions service/stovepipe/server/queues.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
defaults:
stages:
process:
maxConcurrent: 1
gateWaitDelay: 5s
buildSignal:
acceptedPollDelay: 5s
runningPollDelay: 2s
buildAnalysis:
pollDelay: 2m
issueTrack:
refreshDelay: 30s
recoveryWaitDelay: 30s
queues:
- name: monorepo/main
- name: monorepo/release
- name: monorepo/slow?buildrunner-fake=build-slow
2 changes: 1 addition & 1 deletion stovepipe/controller/process/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ go_test(
"//stovepipe/core/requestlog:go_default_library",
"//stovepipe/core/requestlog/mock:go_default_library",
"//stovepipe/entity:go_default_library",
"//stovepipe/extension/queueconfig/default:go_default_library",
"//stovepipe/extension/queueconfig/mock:go_default_library",
"//stovepipe/extension/sourcecontrol:go_default_library",
"//stovepipe/extension/sourcecontrol/mock:go_default_library",
"//stovepipe/extension/storage:go_default_library",
Expand Down
4 changes: 2 additions & 2 deletions stovepipe/controller/process/process.go
Original file line number Diff line number Diff line change
Expand Up @@ -229,8 +229,8 @@ func (c *Controller) coalesce(ctx context.Context, store storage.Storage, reques
// admitLatestHead holds the delivery while the build-slot gate is closed; otherwise it marks the
// request processing, counts its build slot, and publishes it to build.
func (c *Controller) admitLatestHead(ctx context.Context, store storage.Storage, delivery consumer.Delivery, request entity.Request, queueRow entity.Queue, cfg entity.QueueConfig) error {
if queueRow.InFlightCount >= cfg.MaxConcurrent {
return c.holdForBuildSlot(ctx, delivery, request, queueRow.InFlightCount, cfg.GateWaitDelayMs)
if queueRow.InFlightCount >= cfg.Stages.Process.MaxConcurrent {
return c.holdForBuildSlot(ctx, delivery, request, queueRow.InFlightCount, cfg.Stages.Process.GateWaitDelayMs)
}

strategy, baseURI, err := c.deriveBuildStrategy(ctx, queueRow, request)
Expand Down
9 changes: 7 additions & 2 deletions stovepipe/controller/process/process_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ import (
"github.com/uber/submitqueue/stovepipe/core/requestlog"
requestlogmock "github.com/uber/submitqueue/stovepipe/core/requestlog/mock"
"github.com/uber/submitqueue/stovepipe/entity"
queueconfigdefault "github.com/uber/submitqueue/stovepipe/extension/queueconfig/default"
queueconfigmock "github.com/uber/submitqueue/stovepipe/extension/queueconfig/mock"
"github.com/uber/submitqueue/stovepipe/extension/sourcecontrol"
sourcecontrolmock "github.com/uber/submitqueue/stovepipe/extension/sourcecontrol/mock"
"github.com/uber/submitqueue/stovepipe/extension/storage"
Expand Down Expand Up @@ -104,12 +104,17 @@ func newControllerWithScope(t *testing.T, ctrl *gomock.Controller, scope tally.S
})
require.NoError(t, err)

configs := queueconfigmock.NewMockStore(ctrl)
cfg := entity.QueueConfig{Name: testQueue}
cfg.Stages.Process = entity.ProcessConfig{MaxConcurrent: 1, GateWaitDelayMs: 5000}
configs.EXPECT().Get(gomock.Any(), gomock.Any()).Return(cfg, nil).AnyTimes()

c := NewController(
zap.NewNop().Sugar(),
scope,
staticStorageFactory{store: m.store},
m.materializer,
queueconfigdefault.NewStore(),
configs,
m.sourceFactory,
registry,
stovepipemq.TopicKeyProcess,
Expand Down
Loading
Loading