← back to catalog

commoncrawl hosted

CommonCrawl web + Wikipedia, April-2026 snapshot (frozen). General web lens, not a news or scholarly index.

tier

hosted

category

web

coverage

2026-04

ranking

bm25

knobs

q, limit, fetch_urls, fetch_snippets, min_quality, max_host_rank, min_length, max_per_host, host_boost, slug_boost, sort_by, blend_weight

Guidance

Default sort_by=bm25 here, but sort_by=quality ranks by authority NOT relevance (can surface high-authority off-topic pages); blend_weight mixes both. No date filtering on this corpus (crawl_date is metadata, not a filter knob). max_per_host=1 dedups hosts (by literal host string, so 'www.x.com' and 'x.com' count separately). text is a snippet (median_snippet), not the full page. Web index has no titles: title falls back to host.

Usage

The public endpoint is not live yet. Keys are invite-only during beta; contact shapeandfact+takedown@gmail.com.

Plain HTTP:

curl -s https://bipsearch.ai/search \
  -H "Authorization: Bearer YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"corpus": "commoncrawl", "q": "climate policy briefing", "limit": 5}'

Or via MCP, after adding the server once (see quickstart):

# after: claude mcp add --transport http bipsearch https://bipsearch.ai/mcp \
#   --header "Authorization: Bearer YOUR_KEY"
list_corpora()
search(corpus="commoncrawl", q="climate policy briefing", limit=5)