Repository navigation
Add Business Search and structured errors to main - #67
Merged
Merged
Conversation
BrightDataError gains status_code/url/method/retry_after/retryable/raw (all keyword-only, safe defaults). Adds RateLimitError, reparents DatasetError, exports both levels. No behavior change.
datasets filter/status read the body as text first, so a non-JSON error (429 returns a bare string) surfaces as RateLimitError instead of a content-type parse error. api_client.get_status raises instead of collapsing every non-200 into the status string "error", which made an expired token look like a failed scrape.
Retryability no longer follows from the exception class. An APIError with no status code is raised locally, sometimes after the server already accepted the work, so repeating it can create a duplicate billed job - those are now never retried. Explicit 5xx still is. 429 never is, since those responses consume quota and retrying extends the lockout.
Every request already passes through ResponseContextManager, so classify there instead of leaving each subsystem to improvise: 429 -> RateLimitError with retry_after, other 4xx/5xx -> APIError, all carrying status_code, url, method and a bounded body. 202 deliberately passes through - it means success for scraper_studio.trigger_immediate and drives DataNotReadyError recovery elsewhere. Repairs the two call sites whose contract would otherwise change: crawler crawl() keeps returning CrawlResult on HTTP errors, and the unlocker's async get_status keeps returning a status string for its poll loop.
Results are the shape most callers actually receive, and their error field is a string - so every structured attribute was being discarded at the boundary where an exception became a ScrapeResult or CrawlResult. Adds an optional cause field, populated wherever that conversion happens, so callers can branch on the failure instead of parsing its message. Also resolves retryable from status_code when not given explicitly, so the attribute and is_retryable() can no longer disagree.
The engine raises before the per-endpoint checks run, which silently changed two contracts: dataset filter failures surfaced as APIError rather than DatasetError, and error messages lost the endpoint label. The datasets layer now re-types engine errors while letting RateLimitError through, and engine messages carry a bounded body excerpt so a bare print(exc) stays useful.
Add Business and E-commerce search categories
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR brings Business Search and E-commerce search to the Python SDK's
mainbranch. Applications can find company records, professional profiles, and products through the SDK, with the same interface in asynchronous and synchronous code.It integrates the work already reviewed and merged into
devin #56 (structured errors) and #66 (Business and E-commerce search).What users get
client.search.business.companies()andclient.search.business.people()client.search.ecommerce.products()Company and People searches accept structured Ludicrous queries or natural-language Instant/Smart queries. Product search accepts structured Ludicrous queries. Each call returns one
BrightSearchPage, preserving records, request identity, paging information, and index coverage.README examples, an executable example, and a notebook demonstrate the API. The notebook prints company records and metadata as readable text; its saved outputs are cleared.
Behavior and compatibility
RateLimitErrorwithretry_after. Scrape/crawl results retain the originating exception where failures become result objects.retryable; an explicitretryable_exceptionslist remains a caller override.Integration with current main
The integration branch starts from official
mainatf024dfband merges officialdevat69eb736, preserving both histories.Validation
src/,tests/, and the public example.This PR targets
main; it does not publish a package or create a release.