Drawbacks of Increased Crawling Frequency?

Leigh Leigh

• April 26, 2022 13:27

Hi there,
If we set the recrawling frequency to the most frequent it can go (2 minutes), are there any disadvantages to this higher frequency? Slowness, higher chance of crawl failure, etc? Documentation doesn't say anything, and while I figure there aren't any serious negatives, I want to be sure. Thanks!

Related to

• content sources
• crawling

0

• 

  ashishprasher

  • April 28, 2022 12:17

  Hi @... interesting question indeed.
  
  Since you haven’t specified a content source, I cannot give you a concrete answer. Generically speaking, there are some content sources that don't fetch and index delta data. So, we have to fetch all the data. For these sources, we don't support updates in minute/hourly manner.
  
  However, for content sources that provide delta data, the frequency can be set in minutes (e.g. every 15 minutes).
  
  Setting frequency any lower than 15 minutes will impact the search performance and may cause overheads due to increased API costs.
  
  I hope this answers your query.

  0

Please sign in to leave a comment.