Skip to content

Add backup file extraction feature - #13935

Open
JoaoJandre wants to merge 8 commits into
apache:mainfrom
scclouds:add-kboss-file-extraction
Open

JoaoJandre wants to merge 8 commits into
apache:mainfrom
scclouds:add-kboss-file-extraction

Conversation

@JoaoJandre

Copy link
Copy Markdown
Contributor

Description

This PR adds backup file extraction feature. Currently only supported by KBOSS.

This feature must be enabled in the backup offering through the allowextractfile parameter. If it is set to true, it will be possible to extract files from backups created with that offering.

The listBackupFilesystems API was added to allow listing file systems inside backup volumes:

Parameter Description Required?
backupId ID of the backup for which the file systems should be listed Yes
volumeId ID of the volume belonging to the backup for which the file systems should be listed. If not provided, the file systems of all backup volumes will be listed No
Example execution of the listBackupFilesystems API
(labinterno) 🐱 > list backupfilesystems backupid=ecb15182-d6a8-4044-8251-6d437b27b971 
{
  "count": 1,
  "datastoreobject": [
    {
      "format": "ext4",
      "isdirectory": false,
      "isfilesystem": true,
      "name": "/dev/sda1",
      "size": 0,
      "volumeid": "4af9cee2-6014-44fa-ac86-09a649b67363",
      "volumename": "ROOT-198"
    }
  ]
}

The listBackupFiles API was added to allow listing files inside backup volumes:

Parameter Description Required?
backupId ID of the backup for which the files should be listed Yes
volumeId ID of the volume belonging to the backup for which the files should be listed Yes
filesystem Name of the file system that should be mounted in order to list the files Yes
path Path to be listed inside the mounted file system Yes
issymlink If the directory is a symbolic link to another directory. If not informed, ACS will check it No

The filesystem and path parameters expect that the files do not contain special characters such as (, |, or >, for example. Both are expected to receive values in the format /path/to/something.

Example execution of the listBackupFiles API
(labinterno) 🐱 > list backupfiles backupid=ecb15182-d6a8-4044-8251-6d437b27b971 volumeid=4af9cee2-6014-44fa-ac86-09a649b67363 filesystem=/dev/sda1 path=/tmp
{
  "count": 8,
  "datastoreobject": [
    {
      "isdirectory": true,
      "lastupdated": "2026-05-26T13:26:39+0000",
      "name": ".ICE-unix",
      "size": 4096,
      "volumeid": "4af9cee2-6014-44fa-ac86-09a649b67363",
      "volumename": "ROOT-198"
    },
    {
      "isdirectory": true,
      "lastupdated": "2026-05-26T13:26:39+0000",
      "name": ".Test-unix",
      "size": 4096,
      "volumeid": "4af9cee2-6014-44fa-ac86-09a649b67363",
      "volumename": "ROOT-198"
    },
    {
      "isdirectory": true,
      "lastupdated": "2026-05-26T13:26:39+0000",
      "name": ".X11-unix",
      "size": 4096,
      "volumeid": "4af9cee2-6014-44fa-ac86-09a649b67363",
      "volumename": "ROOT-198"
    },
    {
      "isdirectory": true,
      "lastupdated": "2026-05-26T13:26:39+0000",
      "name": ".XIM-unix",
      "size": 4096,
      "volumeid": "4af9cee2-6014-44fa-ac86-09a649b67363",
      "volumename": "ROOT-198"
    },
    {
      "isdirectory": true,
      "lastupdated": "2026-05-26T13:26:39+0000",
      "name": ".font-unix",
      "size": 4096,
      "volumeid": "4af9cee2-6014-44fa-ac86-09a649b67363",
      "volumename": "ROOT-198"
    },
    {
      "isdirectory": true,
      "lastupdated": "2026-05-26T13:26:45+0000",
      "name": "netplan_a2jbieem",
      "size": 4096,
      "volumeid": "4af9cee2-6014-44fa-ac86-09a649b67363",
      "volumename": "ROOT-198"
    },
    {
      "isdirectory": true,
      "lastupdated": "2026-05-26T13:26:42+0000",
      "name": "systemd-private-5f39ee6fdaa3434caa2eb42ad39db34f-systemd-resolved.service-jLhWCz",
      "size": 4096,
      "volumeid": "4af9cee2-6014-44fa-ac86-09a649b67363",
      "volumename": "ROOT-198"
    },
    {
      "isdirectory": true,
      "lastupdated": "2026-05-26T13:26:39+0000",
      "name": "systemd-private-5f39ee6fdaa3434caa2eb42ad39db34f-systemd-timesyncd.service-3aOZJF",
      "size": 4096,
      "volumeid": "4af9cee2-6014-44fa-ac86-09a649b67363",
      "volumename": "ROOT-198"
    }
  ]
}

To download files, the downloadBackupFile API was added:

Parameter Description Required?
backupId ID of the backup from which the file should be downloaded Yes
volumeId ID of the volume belonging to the backup from which the file should be downloaded Yes
filesystem Name of the file system that should be mounted in order to download the file Yes
path Path to be downloaded inside the mounted filesystem. If it is a file, it will be compressed using gzip; if it is a directory, it will be added to a tar archive and compressed using gzip Yes

The filesystem and path parameters expect that the files do not contain special characters such as (, |, or >, for example. Both are expected to receive values in the format /path/to/something.

Example execution of the downloadBackupFile API
(labinterno) 🐱 > download backupfile backupid=ecb15182-d6a8-4044-8251-6d437b27b971 volumeid=4af9cee2-6014-44fa-ac86-09a649b67363 filesystem=/dev/sda1 path=/tmp/systemd-private-5f39ee6fdaa3434caa2eb42ad39db34f-systemd-resolved.service-jLhWCz
{
  "downloadbackupfileresponse": {
    "name": "systemd-private-5f39ee6fdaa3434caa2eb42ad39db34f-systemd-resolved.service-jLhWCz",
    "state": "DOWNLOAD_URL_CREATED",
    "url": "http://192.168.123.142/userdata/215a45db-70e1-4649-93cb-258abe572573/5967473b-29b1-4057-85a3-cd924a458033.gz"
  }
}

The list and download file workflow was also added to the GUI.

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

image

How Has This Been Tested?

Test descriptions

Test Result
List filesystems for a volume containing filesystems Filesystems returned
List filesystems for a volume without filesystems Empty list returned
List filesystems for a backup without specifying the volume List of all filesystems across all backup volumes returned
List files in a standard directory List of files returned, with correct file types and sizes
List files in a symlink to a directory, specifying the isSymlink=true flag List of files returned, with correct file types and sizes
List files in a symlink to a directory, without specifying the isSymlink flag List of files returned, with correct file types and sizes
List files in an empty directory Empty list returned
List files in a directory containing symlinks List of files returned, including the symlink file, with its canonical path information
List files in a directory containing a symlink pointing to a non-existent path List of files returned, excluding the symlink file
List files in a non-existent directory Error (expected)
List files in a symlink pointing to another symlink Error (expected)
Download a standard file File downloaded successfully
Download a standard directory File downloaded successfully
Download a symlink pointing to a file File downloaded successfully
Download a symlink pointing to a directory File downloaded successfully
Download a symlink pointing to another symlink Error (expected)

All the tests above were executed for both Linux ROOT volumes (ext4) and Windows ROOT volumes (NTFS).

@weizhouapache weizhouapache added this to the 4.24.0 milestone Aug 20, 2026
@codecov

codecov Bot commented Aug 20, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 3.85%. Comparing base (2f34b0c) to head (1d950da).
⚠️ Report is 9 commits behind head on main.

❗ There is a different number of reports uploaded between BASE (2f34b0c) and HEAD (1d950da). Click for more details.

HEAD has 1 upload less than BASE
Flag BASE (2f34b0c) HEAD (1d950da)
unittests 1 0
Additional details and impacted files
@@              Coverage Diff              @@
##               main   #13935       +/-   ##
=============================================
- Coverage     20.77%    3.85%   -16.92%     
=============================================
  Files          6053      495     -5558     
  Lines        548222    42979   -505243     
  Branches      73236     8151    -65085     
=============================================
- Hits         113896     1658   -112238     
+ Misses       420984    41089   -379895     
+ Partials      13342      232    -13110     
Flag Coverage Δ
uitests 3.85% <ø> (-0.01%) ⬇️
unittests ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@JoaoJandre

Copy link
Copy Markdown
Contributor Author

It would be nice if someone could test this using RPM, since I have not tested installing the RPM packages.

@github-actions

Copy link
Copy Markdown

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

@apache apache deleted a comment from blueorangutan Aug 23, 2026
@apache apache deleted a comment from blueorangutan Aug 23, 2026

@GaOrtiga GaOrtiga left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@DaanHoogland DaanHoogland moved this from Backlog to conflict/waiting for author in CloudStack Testing Aug 31, 2026
@JoaoJandre

Copy link
Copy Markdown
Contributor Author

@blueorangutan package

@DaanHoogland

Copy link
Copy Markdown
Contributor

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

@DaanHoogland

Copy link
Copy Markdown
Contributor

It would be nice if someone could test this using RPM, since I have not tested installing the RPM packages.

I can give it a quick look, @JoaoJandre . Do you need more then just a monkey test, i.e. specific functional tests? Or do you have a scripted version of the tests to perform?

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19791

@JoaoJandre

Copy link
Copy Markdown
Contributor Author

It would be nice if someone could test this using RPM, since I have not tested installing the RPM packages.

I can give it a quick look, @JoaoJandre . Do you need more then just a monkey test, i.e. specific functional tests? Or do you have a scripted version of the tests to perform?

@DaanHoogland
A simple test would work:

  1. Install packages
  2. Set the backup provider to KBOSS
  3. Create a backup offering with Allow file extraction
  4. Take a backup using such offering
  5. Check if it is possible to browse/download a file

Honestly my main concern is just if the guestfish is being correctly installed when the RPMs are installed. The rest should work.

@DaanHoogland

Copy link
Copy Markdown
Contributor

@blueorangutan test keepEnv

@DaanHoogland DaanHoogland self-assigned this Oct 28, 2026
@blueorangutan

Copy link
Copy Markdown

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian Build Failed (tid-17312)

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian Build Failed (tid-17315)

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian Build Failed (tid-17317)

@DaanHoogland

Copy link
Copy Markdown
Contributor

@JoaoJandre , not sure if this is because the lab is to busy or a real failure. I'll have to abandon it for a while, sorry.

@github-actions

Copy link
Copy Markdown

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

@RosiKyu

RosiKyu commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@RosiKyu a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 20080

@RosiKyu

RosiKyu commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

@blueorangutan test

@blueorangutan

Copy link
Copy Markdown

@RosiKyu a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

@RosiKyu
RosiKyu self-requested a review November 17, 2026 18:23
@RosiKyu RosiKyu self-assigned this Nov 17, 2026
@blueorangutan

Copy link
Copy Markdown

[SF] Trillian Build Failed (tid-17586)

@RosiKyu

RosiKyu commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

@JoaoJandre I tested this on a KVM lab (Oracle Linux 9, advanced zone) with KBOSS, first on 24.0 without this PR, then the PR packages. The agent package installs guestfs-tools and libguestfs on the hosts by itself, and listing file systems and files, downloading a file, a folder and a symlink, the "Browse backup files" tab and KBOSS backup/restore all work for the root admin. Bad paths and an offering without allowextractfile are refused on download.

Two questions:

  1. Which roles should be able to use this? With the PR packages, listBackupFilesystems, listBackupFiles and downloadBackupFile are only in the root admin's listApis. For a user and a domain admin they are not listed and the call gives "unknown command", while createBackup, listBackups and restoreBackup are listed for both. Is the feature meant to be root admin only, or should users be able to get files back from their own VM backups?

  2. What exactly should allowextractfile control? On a KBOSS backup whose offering does not have allowextractfile, downloadBackupFile is refused ("... as its backup offering does not allow it."), but listBackupFilesystems and listBackupFiles still list the contents (for example the file names under /root), and the backup shows browsable: true, so the "Browse backup files" tab is offered for it too. Is listing meant to be allowed without the setting, with only the download blocked?

Also noticed:

  • listBackupFiles returns all entries whatever keyword, page and pagesize are set to (for example keyword=host on /etc returns all 218 entries).
  • A missing folder or file, or bad input, gives "Unable to list/download files of backup [...]. Please check the logs."

Thanks!

@JoaoJandre

Copy link
Copy Markdown
Contributor Author

@JoaoJandre I tested this on a KVM lab (Oracle Linux 9, advanced zone) with KBOSS, first on 24.0 without this PR, then the PR packages. The agent package installs guestfs-tools and libguestfs on the hosts by itself, and listing file systems and files, downloading a file, a folder and a symlink, the "Browse backup files" tab and KBOSS backup/restore all work for the root admin. Bad paths and an offering without allowextractfile are refused on download.

Two questions:

1. Which roles should be able to use this? With the PR packages, `listBackupFilesystems`, `listBackupFiles` and `downloadBackupFile` are only in the root admin's `listApis`. For a user and a domain admin they are not listed and the call gives "unknown command", while `createBackup`, `listBackups` and `restoreBackup` are listed for both. Is the feature meant to be root admin only, or should users be able to get files back from their own VM backups?

2. What exactly should `allowextractfile` control? On a KBOSS backup whose offering does not have `allowextractfile`, `downloadBackupFile` is refused ("... as its backup offering does not allow it."), but `listBackupFilesystems` and `listBackupFiles` still list the contents (for example the file names under `/root`), and the backup shows `browsable: true`, so the "Browse backup files" tab is offered for it too. Is listing meant to be allowed without the setting, with only the download blocked?

Also noticed:

* `listBackupFiles` returns all entries whatever `keyword`, `page` and `pagesize` are set to (for example `keyword=host` on `/etc` returns all 218 entries).

* A missing folder or file, or bad input, gives "Unable to list/download files of backup [...]. Please check the logs."

Thanks!

@RosiKyu Regarding your first question, these APIs were not included in the default roles, but custom roles can add permission for the APIs. Regarding your second question, you got it correctly, you can list the filesystems/files of a backup regardless of the configuration, but you cannot download a file unless the offering allows it, this is intended.

For the page, pagesize and keyword, I believe these parameters should not be available in these APIs. Since guestfish does not support pagination, we would have to get everything from the backup anyway and filter it inside ACS. I will remove these parameters from these two APIs.

For your last note, could you give me an example (with reproducible steps) of when does this happen?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: conflict/waiting

Development

Successfully merging this pull request may close these issues.

6 participants