Content Source Crawling issues

Problem Statement

What are the possible causes of crawl failure for a new content source?

Causes

1. Invalid credentials

2. Incorrect content source name

3. Incorrect client URL

4. Incorrect crawling Start Date

5. Overlap of crawling operations with patch deployment

6. Mismatch between the article fields and fields configured in the admin panel

7. Expiration of password used to authenticate a content source

8. User used to authenticate the content source does not have Admin access

9. Incorrect sitemap in case of website type content sources

Solution

1. Ensure that the login credentials of the content source are the same as the ones used to connect the content source in the SU admin panel.

2. Check if the content source name contains any non-ASCII characters. Only ASCII characters should be used to set the content source name.

3. Only baseURL should be used while configuring content sources in the SU admin panel.

4. The crawling start date should not be a current or future date but should be in the past, from when you would like crawling to start.

5. If the crawl fails in between, check with your CSM if there was any patch deployment happening while crawling was in progress.

6. If you have updated the article structure for one of your content sources by deleting a field, make sure to update the SearchUnify admin panel object fields for a given content source.

7. Content source authentication - Check if the user password used for content source authentication has expired. The content cannot be auto-crawled on the set frequency on an old password.

8. Check if the user used to authenticate the content source has admin access or not. Only users with admin access will be able to authenticate and crawl the content source.

9. In case of website-type content sources, if the sitemap is not in XML format or if the URLs in the sitemap are directed to 404 then crawling will fail. So, it is recommended either to correct the URLs or remove those from the sitemap.

For more information on how to configure a new content source please visit: https://docs.searchunify.com/Content/Content-Sources/Content-Source.htm