Page Comparison

...

1. Name: Enter a unique name for your knowledge base in the “Knowledge Base“ text field.

2. Description: Provide a brief description of your knowledge base in the “Description” text field.

3. URL: Input the starting URL of the web page you want the crawler to analyze in the “Initial URL” text field.

...

Configuring Your Crawler

1. Privacy Settings: Choose between “Private” or “Public” toggle. A “Public” knowledge base is accessible to all users, while a “Private” knowledge base is only accessible to the owner.

2. Bypass Settings: Decide whether to “Comply” or “Bypass” websites that normally block crawlers.

   • Comply: The crawler will respect website settings that block crawling and won’t be able to crawl blocked pages.

   • Bypass: The crawler attempts to access and crawl even those websites that have anti-crawling measures. (*bypass of the blocked pages consumes more tokens)

...