Web

Index public or internal web pages

How it works

The Web Connector scrapes sites based on a base URL.

• It only indexes files from the same domain and containing the same base path.
• It will index pages reachable via hyperlinks from the base URL.
• The text contents are cleaned up via some heuristics and some metadata such as the page Title is extracted.

Setting up

Authorization

• As long as the page is reachable, no additional authorization is necessary.

Indexing

1. Open Web connector

   Navigate to the Admin Panel and select the Web Connector.

2. Enter base URL and index

   Input the base URL to index and click on Index.

   

To see the status of the indexing, visit the Connectors Status page (top left).