Skip to Content
Baseera Docs
Knowledge

Crawl a website

Read a public help center or docs site into a knowledge base, keep it in sync, and let answers link to the live page.

Add a website and Baseera reads its public pages in the background. Only pages that changed are embedded again, and each page keeps its own address, so a citation opens the live page.

You need the Owner, Admin or Developer role, and the knowledge base needs a working embedding key.

Add a website

Open the knowledge base and press Add website in the bar above the page.

Enter the Address, a public https address. Only this site is read, never other sites its pages link to.

Choose How to find pages:

  • Follow links: starts at the address and follows links to other pages on the same site.
  • Read the sitemap: reads sitemap.xml (or the address, if it ends in .xml) and fetches the pages it lists.

Set the limits:

  • Link depth: how many links away from the start to follow, 0 to 5.
  • Page limit: at most 500 pages. Pages beyond it are not read.
  • Only these paths and Skip these paths: one per line, each starting with /. * matches any text. A skipped path wins over an included one. Leave Only these paths empty to read the whole site.

Choose Sync: Only when I ask, Every day or Every week.

Optionally set Who can read its pages (see Knowledge access), then press Add website.

Pages are read in the background, and you can leave the page.

What you see

The Websites section shows each site's status (Waiting, Reading, Idle), the result of the last run (Synced, Synced in part, Failed), Last sync and Next sync. The counts Pages, Updated, Unchanged, Skipped, Not reached and Removed describe the last run, and Pages lists the pages found. Each page is an ordinary source in the Sources table.

Press Sync now to read again. It is refused while a sync is queued or running, and within 5 minutes of the last start. Options changes the settings, and Remove deletes the website with all its pages, chunks and embeddings.

How the crawler behaves

  • It identifies itself in its requests, follows robots.txt, waits between requests, stays on the one site and never signs in. Only public pages are read.
  • It honours nofollow, and noindex pages are not stored.
  • Sites that need JavaScript to show their text are not read. A page with almost no text is skipped.
  • Navigation, footer and sidebar text is left out, so a change there does not count as a change to the page.
  • A page that disappears is removed only on proof (the address answers 404 or 410) or after two finished runs in a row did not see it. If more than half of the pages could not be found, none are removed and the run shows Synced in part.

You can add up to 5 websites per knowledge base and 20 per organization. Deleting a single page by hand does not exclude it: the next sync that finds it adds it again. Use Skip these paths instead.

What can go wrong

MessageCause and fix
"Enter a valid https address." or "Only https addresses can be read."Use a full https address.
"Use a public website name. IP addresses and local names aren't allowed."Only public sites can be read.
"Only port 443 or a port from 1024 up is allowed."Change the port.
"Remove the part after # from the address."Remove the fragment.
"This address is already in the knowledge base."It exists. Press Sync now on it instead.
"Choose a working embedding key in the knowledge base settings first."Pages cannot be embedded without a key.
"The site's robots.txt does not allow this address to be read."The site owner has blocked it. Baseera respects that.
"The site's robots.txt couldn't be read, so nothing was fetched."Retried automatically.
"The sitemap couldn't be read." or "The sitemap lists no pages that can be read."Check the sitemap address, or switch to Follow links.
"No page could be read, so nothing was changed. Check that the site is up and public."The site is down, private or needs JavaScript.
"The site is limiting requests. It will be tried again later."Retried automatically.
"More than half of the pages could not be found this time, so none were removed."If the site was restructured, remove the pages you no longer want.
"A sync is already queued or running." / "A sync started less than 5 minutes ago."Wait and retry.
"The limit is 5 websites per knowledge base and 20 per organization."Remove a website you no longer use.

Next steps

On this page