Skip to content

Migrate/minio to boto3 - #1790

Open
L-Agbaria wants to merge 5 commits into
openml:mainfrom
L-Agbaria:migrate/minio-to-boto3
Open

L-Agbaria wants to merge 5 commits into
openml:mainfrom
L-Agbaria:migrate/minio-to-boto3

Conversation

@L-Agbaria

Copy link
Copy Markdown
Member

First iteration of the change, created a new S3Client class which is meant to take over the MinIOClient class.
Added a very basic test that will need to be further developed, ensured that all tests point to the same S3 client instead of MinIO.

There are still some failing tests, this is not the final version but is a partial update to allow for incremental checking and verification.

Metadata

  • Reference Issue: Change MinIO to boto3 #1789
  • New Tests Added: Yes
  • Change Log Entry: Created S3Client and migrated MinIOClient uses to it.

@codecov-commenter

codecov-commenter commented Nov 12, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 67.54386% with 37 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.10%. Comparing base (ccb5244) to head (4dca23a).

Files with missing lines Patch % Lines
openml/_api/clients/s3.py 63.04% 34 Missing ⚠️
openml/_api/resources/dataset.py 71.42% 2 Missing ⚠️
openml/_api/setup/backend.py 50.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##             main    #1790       +/-   ##
===========================================
+ Coverage   49.48%   80.10%   +30.61%     
===========================================
  Files          63       63               
  Lines        5565     5539       -26     
===========================================
+ Hits         2754     4437     +1683     
+ Misses       2811     1102     -1709     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@PGijsbers PGijsbers left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! First off: good job! You may see a lot of comments in the pull request, but don't let that scare you. They are mostly some feedback on how to write clearer/more useful comments, and some minor refactoring suggestions. The refactoring suggestions are actually mostly talking about code that already existed before and you merely migrated. I thought in those cases it may be a good opportunity to address those concerns now if they are easy to pick up. There are also some comments that simply ask some questions and/or leave a note on something we probably should look at later (it wasn't really in scope for the PR).

Comment thread openml/_api/clients/s3.py

# Handle the endpoint
endpoint = f"{parsed_url.scheme}://{parsed_url.netloc}"
if parts[0] == "s3": # URL has mount in first index

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The expected path prefix "s3" is now hardcoded, it makes sense to have this as a function parameter. This allows us to more easily change it later (the place that will call this function could provide the allowed prefix e.g., from a configuration file).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would also help document the intention, e.g., parts[0] == path_prefix or (following advise below) parsed_url.path.startswith(path_prefix). I would go so far as to say that it makes the in-line comment redundant.

On the inline comment (which I think isn't necessary): "# URL has mount in first index" talks about what the code is doing, but also very much in code terms rather than human terms. For example "first index" is obvious from [0], but we would have to look up how the list is constructed to find out what the first index holds. If we wanted to explain what the code does, something like "the path starts with a known path prefix" would be much easier to read.

Comment thread openml/_api/clients/s3.py Resolved
Comment thread openml/_api/clients/s3.py Resolved
Comment thread openml/_api/clients/s3.py Resolved
Comment thread openml/_api/clients/s3.py
# Handle the bucket and object key
bucket = parts[0]
object_key = "/".join(parts[1:])
if not object_key:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You split up the path into a list, perhaps based on the previous code. This works, but it might be nicer to keep instead the path as a whole. You'd something like:

...
path = parsed_url.path
if path.startswith(path_prefix):
  path = path.removeprefix(path_prefix)
  endpoint += path_prefix

bucket, object_key = path.split("/", maxsplit=1)

(I didn't run this code, but it should be more or less correct)

Comment on lines 395 to 399
def test__get_dataset_parquet_file_does_not_exist(self):
description = {
"oml:parquet_url": "http://data.openml.org/dataset20/does_not_exist.pq",
"oml:parquet_url": "https://data.openml.org/dataset20/does_not_exist.pq",
"oml:id": "20",
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thought:
Seems like there is overlap with test__download_s3_file_object_does_not_exist. This tests the higher level and the other the lower level. Not sure if we need to keep both, but if we do we probably want to mock the lower level behavior in the test of the higher level so they test different behaviour.

def test__get_dataset_parquet_file_does_not_exist(self):
description = {
"oml:parquet_url": "http://data.openml.org/dataset20/does_not_exist.pq",
"oml:parquet_url": "https://data.openml.org/dataset20/does_not_exist.pq",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar to test__download_s3_file_object_does_not_exist, we do not care about the server state so there is no need to use a production url. Let's also improve this test by targetting the test server.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(and let's update the URL to be the modern format e.g. [prefix?]/datasets/0000/0020/does_not_exist.pq.

def test__get_dataset_parquet_not_cached():
description = {
"oml:parquet_url": "http://data.openml.org/dataset20/dataset_20.pq",
"oml:parquet_url": "https://data.openml.org/dataset20/dataset_20.pq",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does load the file. But the test server (both at TU/e and through services) also offers files which can be used instead - there is nothing special about this specific dataset file. So let's update this test to access a test server file.

Comment thread openml/_api/clients/s3.py

# Create S3 client and paginator
client = self._make_s3_client(endpoint_url=endpoint)
paginator = client.get_paginator("list_objects_v2") # PageSize caps at 1_000

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
paginator = client.get_paginator("list_objects_v2") # PageSize caps at 1_000
# Calls `list_objects_v2`, a method that is defined on the `boto3.client`.
# It returns some or all (up to 1,000) of the objects in a bucket with each request.
paginator = client.get_paginator("list_objects_v2")

For me a comment would have been helpful here, because I didn't know where to look for "list_objects_v2" or what it meant (though I obviously had a hunch, the v2 threw me off since we also have a bunch in our codebase now), my journey was:

  • read get_paginator docs to find out it references a function defined on the client
  • grep our codebase assuming it was something we added, 0 results
  • realize that the client we talk about it the boto3.client and the function is defined in the dependency, look up documentation in boto3

I think the first sentence of the proposed comment makes all of that clear. In this case, rewriting the code to be clearer isn't possible because it's a third party module1, so a comment is suitable (in my opinion).
The second sentence (explaining what the function does) could be dropped since it's now easier to look up, but since the reference-by-string doesn't allow a lookup of the function in the IDE directly I think this is useful context to embed.

Footnotes

  1. In a more extreme case, it may be worth to write a wrapper class around a cryptic dependency to make it easier to use and understand. But I don't think this is worthwhile here since we only use a few functions and they are mostly clear. ↩



@mock.patch.object(minio, "Minio")
def test_download_all_files_observes_cache(mock_minio, tmp_path: Path) -> None:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a particular reason for dropping this test (and the one below) instead of converting them?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants