There is a possibility that the site will be crawled

minhaz59

Optimize URL parameters. Suppress the generation of parameters that do not change the content, and avoid unnecessary URL generation due to the order of parameters etc. ・Use 404 and noindex appropriately 404 and noindex will greatly reduce the crawl frequency, but URLs that are not set even though they are unnecessary pages will waste your crawl budget. Avoid soft 404s and set status codes appropriately. ・Use robots.txt Block URLs that you do not want to be crawled.


However, it may still be indexed, especially if it is referenced by an external site. - Set nofollow If you are linking to a non-regular page that will not be indexed, it is a good idea to set nofollow. However, nofollow is not an instruction that Google always follows, and if there are links from other pages that do not have nofollow, .  Bahamas Email List The basic policy is to control pages that lead to unnecessary crawling based on their importance: do not generate unnecessary pages > do not link to unnecessary pages (do not show them to Google) > normalize unnecessary pages.





Particularly on large-scale sites, there are cases where a huge number of URLs are generated depending on the URL parameters, so measures must be taken to ensure that only necessary URLs are crawled. Reference: Best and 5 worst practices for faceted navigation Improve content quality Also, improve the quality of the content of the URLs you want to crawl. In addition to the quality itself, it can also be helpful to let Google know that the URL is well-viewed and popular.