Repository navigation
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1790 +/- ##
===========================================
+ Coverage 49.48% 80.10% +30.61%
===========================================
Files 63 63
Lines 5565 5539 -26
===========================================
+ Hits 2754 4437 +1683
+ Misses 2811 1102 -1709 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
PGijsbers
left a comment
There was a problem hiding this comment.
Thanks! First off: good job! You may see a lot of comments in the pull request, but don't let that scare you. They are mostly some feedback on how to write clearer/more useful comments, and some minor refactoring suggestions. The refactoring suggestions are actually mostly talking about code that already existed before and you merely migrated. I thought in those cases it may be a good opportunity to address those concerns now if they are easy to pick up. There are also some comments that simply ask some questions and/or leave a note on something we probably should look at later (it wasn't really in scope for the PR).
|
|
||
| # Handle the endpoint | ||
| endpoint = f"{parsed_url.scheme}://{parsed_url.netloc}" | ||
| if parts[0] == "s3": # URL has mount in first index |
There was a problem hiding this comment.
The expected path prefix "s3" is now hardcoded, it makes sense to have this as a function parameter. This allows us to more easily change it later (the place that will call this function could provide the allowed prefix e.g., from a configuration file).
There was a problem hiding this comment.
It would also help document the intention, e.g., parts[0] == path_prefix or (following advise below) parsed_url.path.startswith(path_prefix). I would go so far as to say that it makes the in-line comment redundant.
On the inline comment (which I think isn't necessary): "# URL has mount in first index" talks about what the code is doing, but also very much in code terms rather than human terms. For example "first index" is obvious from [0], but we would have to look up how the list is constructed to find out what the first index holds. If we wanted to explain what the code does, something like "the path starts with a known path prefix" would be much easier to read.
| # Handle the bucket and object key | ||
| bucket = parts[0] | ||
| object_key = "/".join(parts[1:]) | ||
| if not object_key: |
There was a problem hiding this comment.
You split up the path into a list, perhaps based on the previous code. This works, but it might be nicer to keep instead the path as a whole. You'd something like:
...
path = parsed_url.path
if path.startswith(path_prefix):
path = path.removeprefix(path_prefix)
endpoint += path_prefix
bucket, object_key = path.split("/", maxsplit=1)(I didn't run this code, but it should be more or less correct)
| def test__get_dataset_parquet_file_does_not_exist(self): | ||
| description = { | ||
| "oml:parquet_url": "http://data.openml.org/dataset20/does_not_exist.pq", | ||
| "oml:parquet_url": "https://data.openml.org/dataset20/does_not_exist.pq", | ||
| "oml:id": "20", | ||
| } |
There was a problem hiding this comment.
thought:
Seems like there is overlap with test__download_s3_file_object_does_not_exist. This tests the higher level and the other the lower level. Not sure if we need to keep both, but if we do we probably want to mock the lower level behavior in the test of the higher level so they test different behaviour.
| def test__get_dataset_parquet_file_does_not_exist(self): | ||
| description = { | ||
| "oml:parquet_url": "http://data.openml.org/dataset20/does_not_exist.pq", | ||
| "oml:parquet_url": "https://data.openml.org/dataset20/does_not_exist.pq", |
There was a problem hiding this comment.
Similar to test__download_s3_file_object_does_not_exist, we do not care about the server state so there is no need to use a production url. Let's also improve this test by targetting the test server.
There was a problem hiding this comment.
(and let's update the URL to be the modern format e.g. [prefix?]/datasets/0000/0020/does_not_exist.pq.
| def test__get_dataset_parquet_not_cached(): | ||
| description = { | ||
| "oml:parquet_url": "http://data.openml.org/dataset20/dataset_20.pq", | ||
| "oml:parquet_url": "https://data.openml.org/dataset20/dataset_20.pq", |
There was a problem hiding this comment.
This does load the file. But the test server (both at TU/e and through services) also offers files which can be used instead - there is nothing special about this specific dataset file. So let's update this test to access a test server file.
|
|
||
| # Create S3 client and paginator | ||
| client = self._make_s3_client(endpoint_url=endpoint) | ||
| paginator = client.get_paginator("list_objects_v2") # PageSize caps at 1_000 |
There was a problem hiding this comment.
| paginator = client.get_paginator("list_objects_v2") # PageSize caps at 1_000 | |
| # Calls `list_objects_v2`, a method that is defined on the `boto3.client`. | |
| # It returns some or all (up to 1,000) of the objects in a bucket with each request. | |
| paginator = client.get_paginator("list_objects_v2") |
For me a comment would have been helpful here, because I didn't know where to look for "list_objects_v2" or what it meant (though I obviously had a hunch, the v2 threw me off since we also have a bunch in our codebase now), my journey was:
- read
get_paginatordocs to find out it references a function defined on the client - grep our codebase assuming it was something we added, 0 results
- realize that the client we talk about it the
boto3.clientand the function is defined in the dependency, look up documentation in boto3
I think the first sentence of the proposed comment makes all of that clear. In this case, rewriting the code to be clearer isn't possible because it's a third party module1, so a comment is suitable (in my opinion).
The second sentence (explaining what the function does) could be dropped since it's now easier to look up, but since the reference-by-string doesn't allow a lookup of the function in the IDE directly I think this is useful context to embed.
Footnotes
-
In a more extreme case, it may be worth to write a wrapper class around a cryptic dependency to make it easier to use and understand. But I don't think this is worthwhile here since we only use a few functions and they are mostly clear. ↩
|
|
||
|
|
||
| @mock.patch.object(minio, "Minio") | ||
| def test_download_all_files_observes_cache(mock_minio, tmp_path: Path) -> None: |
There was a problem hiding this comment.
Is there a particular reason for dropping this test (and the one below) instead of converting them?
First iteration of the change, created a new S3Client class which is meant to take over the MinIOClient class.
Added a very basic test that will need to be further developed, ensured that all tests point to the same S3 client instead of MinIO.
There are still some failing tests, this is not the final version but is a partial update to allow for incremental checking and verification.
Metadata