Skip to content

WIP - Cloudstack AI Services plugin - #14733

Open
kiwiflyer wants to merge 1 commit into
apache:mainfrom
kiwiflyer:Cloudstack-AI-Plugin
Open

kiwiflyer wants to merge 1 commit into
apache:mainfrom
kiwiflyer:Cloudstack-AI-Plugin

Conversation

@kiwiflyer

Copy link
Copy Markdown
Contributor

Description

This is a work in progress

This PR adds CloudStack AI Services: a native API plugin (plugins/api/ai-services)
and a ui/ section that let an operator source open-weight LLMs, deploy them on NVIDIA
GPU capacity, tune them through profiles, and sell them to tenants as OpenAI-compatible
endpoints.

The plugin is the front door only — auth, RBAC, async jobs, events, usage and limits. It
delegates to an external reconciler over mTLS, which is the component that touches GPU
resources, CKS and the serving layer. That engine is not in this PR, which is the main
thing I would like feedback on; see "Open questions" below.

What it contains -

  • 58 new API commands — 38 v1, registered always, and 20 v2, registered only when the
    global setting ai.services.v2.enabled is on (default false), so a GA install does
    not advertise an API for features it is not running.
  • A new UI section of 19 screens — model catalogue, endpoints with scoped API keys and
    copy-ready snippets, a playground, generation profiles and effective-parameter
    resolution, usage and budgets, plus admin screens for setup preflight, GPU capacity and
    the model registry. Routes carry a permission, so CloudStack drops any screen whose
    command the caller cannot call, and the Tier 2 screens appear and disappear with that one
    setting.
  • Ten ai.services.* ConfigKeys, including the engine URLs and its mTLS material.
  • Default role access is declared on @APICommand(authorized = …), so there is no
    migration, no commands.properties and no operator step. Explicit role_permissions
    rules still win.

The change is additive: 158 files, 19,207 insertions and a single deletion (a comma, to
append to en.json). Only four existing files are touched — plugins/pom.xml,
client/pom.xml, ui/src/config/router.js, and ui/public/locales/en.json, into which 150
keys are appended without reordering or reformatting anything already there.

Where the rest of it lives

Everything this plugin talks to is Apache-2.0 and public, so the serving layer can be built
and the design read is available here:

https://github.com/kiwiflyer/CloudStack_AI_Plugin

SPEC.md The technical specification. Section 9 is the command map this plugin is generated from, section 10 the engine REST contract it calls.
docs/installation.md How to stand the whole thing up: the four pieces and their order, every config key and environment variable, and an honest list of what has never been run.
aisvc/ The reconciler itself — the piece that is not in this PR. observe() → diff() → plan() → execute(), one leader per zone.
gateway-hooks/ The enforcement hooks the gateway loads for keys, scopes, profiles, quotas, budgets and guardrails.
install.py Scripts everything outside CloudStack — virtualenv, dependencies, migrations, mTLS material, reachability. Standard library only.

For the serving layer specifically, the two decisions that shape it are
ADR-0001
(plain Kubernetes Deployments rendered by the engine, with blue/green weighting at the
gateway rather than in the serving layer) and
ADR-0002
(LiteLLM Proxy, MIT side only). Where a CKS cluster comes from is
ADR-0007,
and all eight ADRs
record what was decided and what is still open.

Also worth reading:
why this exists and what it lets an operator sell,
the limits-and-defaults sheet
(every threshold and TTL read from the code, with gaps marked ·not built rather than
omitted), a walkthrough of the UI section
screen by screen, and the
third-party licence position
— the dependency tree follows ASF Category A/B/X, GPL/LGPL/AGPL are excluded including for
Python imports, and CI fails on anything it cannot classify.

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

How Has This Been Tested?

Against a live 4.23.0.0 management server (the simulator image), with the plugin built,
installed and loaded, and the UI section compiled into that server's own ui/ tree.

  • 481 JUnit 5 tests in the module. mvn -pl plugins/api/ai-services test is green and
    checkstyle is clean against cloud-style.xml.
  • Command registration measured, not assumed: listApis goes from 865 to 903 with the
    plugin loaded — exactly the 38 v1 commands — and to 923 with ai.services.v2.enabled on,
    exactly the 20 v2 ones. The gate holds on a real server in both directions.
  • Marvin integration tests (test/integration/plugins/ai-services/) cover all 58
    commands across three default roles, allowed and denied. They call each command with no
    arguments, because access is checked before parameters are: a role that may call one
    fails on its arguments (431) and one that may not fails on the name (432), so nothing is
    executed either way.
  • Playwright, 40 passed / 7 skipped, against the plugin and the UI in the same
    management server: the section visible per role, every admin screen absent for a domain
    admin and a tenant, and a tenant typing an admin screen's URL getting a 404 rather than a
    refusal. The 7 skips are the ai.services.v2.enabled-off group, which cannot run while
    the setting is on.
  • Tenant isolation measured end to end: the same listAiModels returns two models to
    root admin, two to an account in /Customers/Acme, and none to one in /ROOT.
  • The UI also goes through CloudMonkey and a Terraform provider, so nothing in the
    section is reachable only by clicking.

How did you try to break this feature and the system with this change?

The failure mode I worried about most is CloudStack dropping something in silence, so
most of the hardening is aimed there:

  • A plugin that fails to wire is not an optional plugin, it is a dead management server.
    One missing bean in the Spring context took the whole /client context down — no API at
    all, for anything — while 451 unit tests stayed green, because they construct the manager
    directly. There is now a test that walks every @Inject field and fails on any type
    without a bean.
  • CloudStack ignores an unknown request parameter. A dialog or client can therefore
    collect a value that goes nowhere — a key "restricted" to two models and a CIDR was being
    created unrestricted. Tests now compare every parameter each caller sends against the
    command that receives it, in both directions, including required parameters never offered.
  • A response field the far side sends but the response class does not declare is dropped
    the same way.
    Ten were, including a guardrail profile's checks. A test walks each
    engine payload against its response class; a field may be omitted only with a recorded
    reason.
  • Role denial, not just role visibility: a tenant is refused all 38 commands without an
    explicit grant, verified on a live server rather than in a unit test.
  • Engine failover is exercised in all four branches, including the one that matters:
    a replica that answered is never retried, so a POST cannot be applied twice.
  • Deliberate negatives are asserted too — a refused command must surface the engine's own
    sentence rather than a bare status code, and an async command must not report success for
    work no job ever did.

What has not been tested, and I would not want a reviewer to assume otherwise:

  • Nothing here has run against a live main/24.0.0-SNAPSHOT server. It compiles and its
    tests pass against this tree, and none of the ~45 CloudStack APIs it imports has moved,
    but compiling is not loading — and the one defect only a real server could show took the
    management server down.
  • The GPUs are simulated. Request signing, the API, RBAC and the schema are real; no
    physical GPU capacity has been allocated by this code.
  • Only the module itself has been built here, with
    mvn -pl plugins/api/ai-services test. A full reactor build has not been run, and this
    PR edits plugins/pom.xml and client/pom.xml, so CI is the first thing that will
    compile the whole tree with the module in it. I would not be surprised by a packaging
    problem there.
  • The UI section has not been built with npm run build in this tree. A build reporting
    Compiled successfully says nothing about whether labels resolve anyway — that has to be
    checked on a rendered screen.

@boring-cyborg boring-cyborg Bot added component:build component:UI Python Warning... Python code Ahead! labels Nov 19, 2026
@codecov

codecov Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 48.72100% with 1724 lines in your changes missing coverage. Please review.
✅ Project coverage is 20.97%. Comparing base (6a0fcd6) to head (898608c).

Files with missing lines Patch % Lines
...rg/apache/cloudstack/ai/AiServicesManagerImpl.java 51.69% 95 Missing and 5 partials ⚠️
...ack/ai/api/response/AiPrivateEndpointResponse.java 40.29% 37 Missing and 3 partials ⚠️
.../apache/cloudstack/ai/dao/AiDeploymentDaoImpl.java 0.00% 40 Missing ⚠️
.../cloudstack/ai/api/response/AiAdapterResponse.java 46.57% 34 Missing and 5 partials ⚠️
.../cloudstack/ai/api/response/AiHfCheckResponse.java 46.57% 34 Missing and 5 partials ⚠️
...e/cloudstack/ai/api/response/AiBudgetResponse.java 44.11% 34 Missing and 4 partials ⚠️
.../cloudstack/ai/api/command/UploadAiAdapterCmd.java 0.00% 37 Missing ⚠️
...he/cloudstack/ai/api/response/AiBatchResponse.java 41.26% 34 Missing and 3 partials ⚠️
...stack/ai/api/response/AiReconcilerRunResponse.java 43.07% 33 Missing and 4 partials ⚠️
...ck/ai/api/response/AiGuardrailProfileResponse.java 40.00% 30 Missing and 3 partials ⚠️
... and 92 more
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #14733      +/-   ##
============================================
+ Coverage     20.81%   20.97%   +0.16%     
- Complexity    20923    21356     +433     
============================================
  Files          6053     6172     +119     
  Lines        548451   552076    +3625     
  Branches      73279    73626     +347     
============================================
+ Hits         114138   115781    +1643     
- Misses       420948   422750    +1802     
- Partials      13365    13545     +180     
Flag Coverage Δ
uitests 3.85% <ø> (-0.01%) ⬇️
unittests 22.42% <48.72%> (+0.17%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@kiwiflyer
kiwiflyer force-pushed the Cloudstack-AI-Plugin branch from 47a6c4e to 8efcb65 Compare November 19, 2026 14:15
Adds a native API plugin (plugins/api/ai-services) and a ui/ section
that let an operator source open-weight LLMs, deploy them on NVIDIA GPU
capacity, tune them through profiles, and sell them to tenants as
OpenAI-compatible endpoints.

The plugin is the front door only — auth, RBAC, async jobs, events,
usage and limits. It delegates over mTLS to an external reconciler,
which is the component that touches GPU resources, CKS and the serving
layer; that engine is not in this change.

  * 58 API commands: 38 v1, registered always, and 20 v2, registered
    only when the global setting ai.services.v2.enabled is on (default
    false), so a GA install does not advertise an API for features it
    is not running.
  * A UI section of 19 screens. Each route carries a permission, so
    CloudStack drops any screen whose command the caller cannot call,
    and the Tier 2 screens appear and disappear with that one setting.
  * Ten ai.services.* ConfigKeys, including the engine URLs and its
    mTLS material.
  * Default role access declared on @APICommand(authorized = ...), so
    there is no migration, no commands.properties and no operator step.
    An explicit role_permissions rule still wins.

The change is additive. Four existing files are touched: plugins/pom.xml
and client/pom.xml to build and bundle the module, ui/src/config/
router.js to register the section, and ui/public/locales/en.json, into
which 150 keys are appended without reordering or reformatting anything
already there.

Tested against a live 4.23.0.0 management server with the plugin loaded:
481 JUnit tests and checkstyle clean; listApis goes 865 -> 903 with the
plugin and -> 923 with v2 enabled, exactly the declared commands; Marvin
covers all 58 commands across three default roles, allowed and denied;
and the Playwright suite passes 40 with 7 skipped against the plugin and
the UI in the same management server.

Not yet exercised: a full reactor build, any live 24.0.0-SNAPSHOT
server, and real GPU capacity — the simulator's cards are simulated.
@kiwiflyer
kiwiflyer force-pushed the Cloudstack-AI-Plugin branch from 8efcb65 to 898608c Compare November 19, 2026 14:31
@kiwiflyer kiwiflyer removed the Python Warning... Python code Ahead! label Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant