Skip to content

Add Business Search and structured errors to main - #67

Merged
romanv-brd merged 16 commits into
brightdata:mainfrom
karaposu:feat/business-search-main
Nov 19, 2026
Merged

romanv-brd merged 16 commits into
brightdata:mainfrom
karaposu:feat/business-search-main

Conversation

@karaposu

Copy link
Copy Markdown
Contributor

This PR brings Business Search and E-commerce search to the Python SDK's main branch. Applications can find company records, professional profiles, and products through the SDK, with the same interface in asynchronous and synchronous code.

It integrates the work already reviewed and merged into dev in #56 (structured errors) and #66 (Business and E-commerce search).

What users get

Category Methods
Business client.search.business.companies() and client.search.business.people()
E-commerce client.search.ecommerce.products()

Company and People searches accept structured Ludicrous queries or natural-language Instant/Smart queries. Product search accepts structured Ludicrous queries. Each call returns one BrightSearchPage, preserving records, request identity, paging information, and index coverage.

README examples, an executable example, and a notebook demonstrate the API. The notebook prints company records and metadata as readable text; its saved outputs are cleared.

Behavior and compatibility

  • Search requests reuse the client's HTTP engine without requiring zones. Pagination remains explicit, and Bright Search failures opt out of the generic helper's default retry policy because the search may already have executed.
  • Structured errors from v2.6.0: structured errors — translate non-2xx at the engine #56 are included because Business Search depends on them. HTTP errors now carry status, request context, and bounded response text; HTTP 429 raises RateLimitError with retry_after. Scrape/crawl results retain the originating exception where failures become result objects.
  • Search-specific handling retains available diagnostic headers and request identity, preserves timeout context, rejects invalid JSON payloads before dispatch, and does not follow redirects. Existing search-engine and platform methods remain available.
  • Callers that match error-message text should use structured attributes instead. The generic retry helper honors retryable; an explicit retryable_exceptions list remains a caller override.

Integration with current main

The integration branch starts from official main at f024dfb and merges official dev at 69eb736, preserving both histories.

Validation

  • 558 tests passed on each of Python 3.10.15, 3.11.10, and 3.12.7, including 175 indexed-search cases and the existing CSV security regressions. The adjusted recursion fixtures were also rechecked on 3.10/3.11 after their full-suite runs.
  • Ruff and Black passed for src/, tests/, and the public example.
  • Wheel and source distribution built successfully. Installed-wheel smoke checks passed for imports, async/sync search categories, response serialization, and timeout compatibility.
  • Notebook schema and code cells validated; no saved outputs. Verified all 89 files changed only on main remain byte-for-byte unchanged.
  • The repository's non-blocking mypy check reports the same 82 diagnostics as current main, with no new diagnostics.
  • Prior live API qualification is recorded in Add Business and E-commerce search categories #66. This integration used deterministic tests and local HTTP peers; it did not rerun billed live searches.

This PR targets main; it does not publish a package or create a release.

karaposu and others added 16 commits August 22, 2026 09:42
BrightDataError gains status_code/url/method/retry_after/retryable/raw
(all keyword-only, safe defaults). Adds RateLimitError, reparents
DatasetError, exports both levels. No behavior change.
datasets filter/status read the body as text first, so a non-JSON error
(429 returns a bare string) surfaces as RateLimitError instead of a
content-type parse error. api_client.get_status raises instead of
collapsing every non-200 into the status string "error", which made an
expired token look like a failed scrape.
Retryability no longer follows from the exception class. An APIError with
no status code is raised locally, sometimes after the server already
accepted the work, so repeating it can create a duplicate billed job -
those are now never retried. Explicit 5xx still is. 429 never is, since
those responses consume quota and retrying extends the lockout.
Every request already passes through ResponseContextManager, so classify
there instead of leaving each subsystem to improvise: 429 -> RateLimitError
with retry_after, other 4xx/5xx -> APIError, all carrying status_code, url,
method and a bounded body. 202 deliberately passes through - it means
success for scraper_studio.trigger_immediate and drives DataNotReadyError
recovery elsewhere.

Repairs the two call sites whose contract would otherwise change: crawler
crawl() keeps returning CrawlResult on HTTP errors, and the unlocker's
async get_status keeps returning a status string for its poll loop.
Results are the shape most callers actually receive, and their error field
is a string - so every structured attribute was being discarded at the
boundary where an exception became a ScrapeResult or CrawlResult. Adds an
optional cause field, populated wherever that conversion happens, so
callers can branch on the failure instead of parsing its message.

Also resolves retryable from status_code when not given explicitly, so the
attribute and is_retryable() can no longer disagree.
The engine raises before the per-endpoint checks run, which silently
changed two contracts: dataset filter failures surfaced as APIError rather
than DatasetError, and error messages lost the endpoint label. The datasets
layer now re-types engine errors while letting RateLimitError through, and
engine messages carry a bounded body excerpt so a bare print(exc) stays
useful.
Add Business and E-commerce search categories
@romanv-brd
romanv-brd merged commit c7e56f3 into brightdata:main Nov 19, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants