Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,13 @@
archive/
tests

# Track the Business Search contract tests; keep local test artifacts ignored.
!tests/
tests/*
!tests/unit/
tests/unit/*
!tests/unit/test_bright_search.py

# Byte-compiled / optimized / DLL files
__pycache__/
*.py[codz]
Expand Down
23 changes: 23 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,28 @@
# Bright Data Python SDK Changelog

## Version 2.6.0 - Business Search and structured errors

- **Business Search**: `client.search.business.companies()` and `.people()` search indexed company and professional records using structured Ludicrous queries or natural-language Instant/Smart queries. Both async and sync clients return one `BrightSearchPage` with records, request ID, paging metadata, and index coverage.
- **E-commerce search**: `client.search.ecommerce.products()` supports structured Ludicrous queries. Search calls use the shared engine without requiring zones; pagination and retries are explicit caller decisions.
- **Search request and response handling**: validate JSON payloads before dispatch, preserve available HTTP diagnostics, and retain timeout context through `RequestTimeoutError`, also catchable as `TimeoutError`.
- **Examples**: README usage, an executable example, and a notebook with simple and paginated company searches and plain-text results.
- **HTTP errors now raise typed exceptions.** `AsyncEngine` previously converted only 401/403 and handed every other failure back as a normal response, leaving each subsystem to improvise. Classification now happens once, at the single point every request passes through: `429` → the new **`RateLimitError`** (with `retry_after` parsed from the header), other 4xx/5xx → `APIError`.
- **Exceptions carry data, not just prose.** `BrightDataError` gained `status_code`, `url`, `method`, `retry_after`, `retryable` and `raw` (the response body, bounded to 4 KB). Callers can branch on the failure instead of parsing its message.
- **Fixed**: a rate-limited dataset request surfaced as an aiohttp *content-type* error, because the 429 body is a bare string and the response was parsed as JSON before its status was checked. It now raises `RateLimitError`.
- **Fixed**: a token expiring mid-poll was reported as `"Job failed with status: error"` — blaming the user's scrape for an authentication problem. `DatasetAPIClient.get_status` no longer collapses every non-200 into the status string `"error"`.
- **Results carry the failure too.** `ScrapeResult` / `CrawlResult` gained `cause`, the originating exception, populated wherever one is converted into a result. `error` remains the human-readable message:
```python
result = await client.scrape.x.posts(url)
if not result.success and isinstance(result.cause, RateLimitError):
await asyncio.sleep(result.cause.retry_after or 60)
```
- **Retry decisions respect service policy.** `retry_with_backoff` honors SDK errors' `retryable` flag. Outside Bright Search, HTTP 5xx, network errors, and plain timeouts retain their default retries; 429 is excluded from the default policy. Bright Search sets `retryable=False` because an unsuccessful response can follow an executed search. An explicit `retryable_exceptions` list overrides the default policy.
- `DatasetError` now subclasses `BrightDataError` (still catchable as before). `RateLimitError` and `DataNotReadyError` are exported from `brightdata` top level.
- **Note on messages**: error messages are shorter and more uniform, with detail moved to attributes. Code matching on message *text* may need updating; use `status_code` instead.
- **Known limit**: this is status-based, so it cannot see HTTP 200 responses carrying an error in the body (SERP's inner envelope, Web Unlocker error content). Those remain string-shaped.

---

## Version 2.5.2 - Safer CSV exports

- **Security fix**: `export_csv()` (and `export()` for `.csv` files) now automatically escapes cell values that spreadsheet apps like Excel, Google Sheets, or LibreOffice could otherwise misinterpret as formulas, so scraped data can't unexpectedly trigger links or formulas when the exported file is opened.
Expand Down
123 changes: 123 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,34 @@ export BRIGHTDATA_API_TOKEN="your_api_token_here"
**Already logged in with the CLI?** The SDK works with no configuration — it automatically
falls back to the credentials stored by `brightdata login`.

## Handling errors

Failures carry structured data, not just a message:

```python
from brightdata import BrightDataClient, RateLimitError, APIError

async with BrightDataClient() as client:
try:
data = await client.datasets.instagram_profiles.download(snapshot_id)
except RateLimitError as e:
await asyncio.sleep(e.retry_after or 60) # the API told us how long
except APIError as e:
print(e.status_code, e.raw) # not a message to parse
```

Methods that return a result instead of raising expose the same information on `cause`:

```python
result = await client.scrape.x.posts(url)
if not result.success and isinstance(result.cause, RateLimitError):
await asyncio.sleep(result.cause.retry_after or 60)
```

Use `e.retryable` to decide whether repeating the call is safe. Rate limits are never
retryable — a 429 response itself consumes quota, so retrying extends the lockout; wait
`retry_after` instead.

## Quick Start

This SDK is **async-native**. A sync client is also available (see [Sync Client](#sync-client)).
Expand Down Expand Up @@ -184,6 +212,101 @@ async with async_playwright() as pw:

**When to use:** sites that require full browser automation — JS rendering, login flows, interactive clicks. For plain HTML fetches, prefer `client.scrape_url()`.

### Indexed Search: Business and E-commerce

Business Search groups company and professional records under
`client.search.business.companies()` and `client.search.business.people()`. Use
structured queries with `mode="ludicrous"`, or natural language with
`mode="instant"` / `mode="smart"`. Natural-language queries allow up to 200 characters.

Product search is a separate category: `client.search.ecommerce.products()`.
It accepts structured queries with `mode="ludicrous"`. Each method returns one page.

```python
from brightdata import BrightDataClient

# Direct Bright Search requests do not need Web Unlocker or SERP zones.
async with BrightDataClient(auto_create_zones=False) as client:
page = await client.search.business.companies(
query="US software companies with 50 to 500 employees",
mode="instant",
view="summary",
limit=10,
)
for document in page.documents:
print(document["bright_id"], document["data"].get("name"))
print(page.req_id, page.meta["matched"], page.meta["coverage_percent"])

products = await client.search.ecommerce.products(
query={"and": [
{"equals": {"retailer": "Argos"}},
{"equals": {"sku": "4326522"}},
]},
mode="ludicrous",
view="full",
)
```

The same methods return `BrightSearchPage` from the synchronous client:

```python
from brightdata import SyncBrightDataClient

with SyncBrightDataClient(auto_create_zones=False) as client:
page = client.search.business.people(
"Product managers in the United States",
mode="instant",
view={"fields": ["name", "current_title", "url"]},
)
```

`source=None` uses the category's server default; `view=None` uses `id_only`.
E-commerce supports only `ludicrous`, and its default `retail_products_live` source
supports `id_only`, `full`, or a custom field list, but no `summary` view. Every
document includes `bright_id`; fields inside `data` are optional. `page.raw`,
`page.to_dict()`, and `page.to_json()` preserve the complete response body,
including additional fields. Response headers are separate in `page.response_headers`.

Use explicit `offset` and `limit` for paging. Ludicrous allows limits up to 100
and offsets up to 1000. Instant and Smart expose a 100-record window:
`offset + limit` must not exceed 100, with a maximum page size of 100 for Instant
and 10 for Smart. `matched` is generally a reached-count lower bound, not a total, and partial
coverage is reported in `coverage_percent`. Pages are separate searches against
the current index and are billed separately, including served empty pages.

The SDK does not automatically paginate these calls. Bright Search failures set
`retryable=False`, so the generic `retry_with_backoff` helper does not replay them
under its default policy, including HTTP 5xx and response-body failures. A failed
response can follow an executed search. Callers can make another explicit request;
supplying `retryable_exceptions` to the helper is an explicit policy override.

Redirects raise `APIError` without following another URL. HTTP errors retain
available `req_id`, `response_json`, and selected response headers. A
`RateLimitError` exposes `retry_after` for deciding when to make another request.
Interrupted successful response bodies raise `NetworkError`; malformed text or
JSON raises `APIError`. If an error body has invalid text encoding, `raw` contains
readable diagnostic text, while `response_json` and `req_id` remain unset.

Request and body timeouts raise `RequestTimeoutError`, which is also catchable
as the built-in `TimeoutError`. It retains the URL, method, and any known HTTP
status, selected headers, and `retry_after`. Unread bodies do not supply a request
ID or parsed JSON. Legacy engine calls keep their existing timeout behavior.

The complete request payload must be JSON-compatible. Non-finite numbers,
non-string object keys, unsupported objects, circular references, and structures
the JSON encoder cannot serialize raise `ValidationError` before dispatch. Valid
nested values and unknown field names are passed through without changing them.

For a debug-enabled account, the optional `debug` mapping accepts the wire keys
`explain`, `benchmark`, `trace`, and `__overrides__`. These keys are omitted unless
explicitly supplied; nested null, false, and zero values are preserved. Diagnostic
output is available in `page.explain`, and its internal structure is unstable.
Flights is experimental and is not exposed by these methods.

See [the Bright Search example](examples/bright_search.py) for async and sync usage.
The [Business Search notebook](notebooks/10_bright_search.ipynb) walks through a
simple company search and a structured search with selected fields and pagination.

### Discover API

AI-ranked web search. Unlike SERP (which returns engine-ordered results), Discover takes a `query` plus an `intent` phrase and re-ranks by relevance. Optionally extracts full page content as markdown.
Expand Down
54 changes: 54 additions & 0 deletions examples/bright_search.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
"""Bright Search examples using the configured Bright Data API key.

Run this file to make one Company search. The other functions demonstrate
People and structured E-commerce search when called explicitly.
"""

import asyncio

from brightdata import BrightDataClient, BrightSearchPage, SyncBrightDataClient


async def company_search() -> BrightSearchPage:
"""Make one request, leaving paging and recovery under caller control."""
async with BrightDataClient(auto_create_zones=False) as client:
page = await client.search.business.companies(
"US software companies with 50 to 500 employees",
mode="instant",
view="summary",
limit=10,
)
print(f"Request: {page.req_id}")
print(f"Reached: {page.meta['matched']}; coverage: {page.meta['coverage_percent']}%")
for document in page.documents:
print(document["bright_id"], document["data"].get("name"))
return page


def people_search_sync() -> BrightSearchPage:
"""Use the same page contract from an ordinary synchronous script."""
with SyncBrightDataClient(auto_create_zones=False) as client:
return client.search.business.people(
"Product managers in the United States",
mode="instant",
view={"fields": ["name", "current_title", "url"]},
)


async def product_search() -> BrightSearchPage:
"""Send the native structured query without a separate query-builder API."""
async with BrightDataClient(auto_create_zones=False) as client:
return await client.search.ecommerce.products(
query={
"and": [
{"equals": {"retailer": "Argos"}},
{"equals": {"sku": "4326522"}},
]
},
mode="ludicrous",
view="full",
)


if __name__ == "__main__":
asyncio.run(company_search())
Loading
Loading