Skip to Content
Baseera Docs

Network and fetching

How Baseera guards the outbound requests it makes for crawling, url items, webhooks and model calls.

Baseera makes outbound requests on your behalf: it fetches web pages, calls your webhook, and calls your model provider. Each kind goes through a guard that limits where it can connect.

Fetching URLs, crawls and webhooks

Knowledge url items, crawled websites and webhook deliveries all use one fetcher (apps/runtime/src/knowledge/fetch-url.ts). It allows only public https addresses:

  • https only, with no credentials in the address;
  • port 443 or 1024 and up;
  • the runtime resolves the name itself, and every returned address must be public;
  • loopback, private, link-local (including the cloud metadata address), carrier-grade NAT, multicast, reserved and documentation ranges are refused, for IPv4 and IPv6, including IPv4-mapped, NAT64 and 6to4 forms;
  • the connection goes to the address that was checked, so a name cannot change its answer between the check and the connection;
  • at most 3 redirects, each checked again, and a webhook never follows one;
  • no cookies and no Authorization header;
  • caps on size, time and content type: for a url item, 20 MB and 20 seconds, and only HTML, plain text, Markdown, CSV, PDF and DOCX.

This is the protection against server-side request forgery (SSRF). A webhook address is checked when you save it, by the same code, and the save fails closed if the runtime cannot be asked. It is checked again at every delivery, so a name that turns private later is refused.

Crawled websites

A knowledge base can follow a website. The limits are constraints in the database, not only in the form:

  • https and a public-looking name only: no IP address, no local name, port 443 or 1024 and up;
  • depth 0 to 5, 1 to 500 pages, 20 include and 20 exclude paths;
  • 5 websites per knowledge base and 20 per organization;
  • a 2 MB page, 15 seconds per page, no compression;
  • a page outside the crawl's own origin is refused, and so is a redirect that leaves the origin or robots.txt;
  • one crawl per site at a time, with a pause between requests (1 second, or the site's Crawl-delay up to 10 seconds).

robots.txt is read each time. A missing file allows everything. A restricted file, or one over 500 KiB, forbids everything. A failing one stops the run. noindex is honored. A run that read nothing, or whose removals would be more than half the pages, removes nothing.

Logs hold ids and codes, never an address, a header or page text.

Webhook deliveries

Webhooks are signed with HMAC-SHA256 over ${t}.${body} and carry ids and a status, never document content. The finish webhook uses the same fetcher with a 5-second timeout and one retry. See Webhooks for verification.

Model calls

Model calls go through the gateway with a guarded fetch that can only reach each provider's own endpoint over https:

  • the exact host and path of each provider, nothing else;
  • redirects are refused;
  • the provider's key goes in a header, never in a URL;
  • response bodies are capped (20 MB), as are stream lines (1 MB), tool input (64 KB per call) and tool calls (32 per response);
  • timeouts are the runtime's own: 30 seconds to connect, 60 seconds of silence between chunks, 5 minutes in total.

Provider errors are treated as untrusted. They are mapped to a closed set of codes, and nothing from a provider's body reaches an end user.

Next steps

On this page