Managing Crawl Rules in SharePoint 2013

It's very simple to configure crawling rules in SP 2013 as it was in 2010.
Lets have a quick walk through of it. Crawl rules are used to specfiy detailed logic to the crawler while content crawling. There are situations where we don't want to crawl a particular content or we want exceptional content to be crawled inside that. There are many option available.

• Prevent content on a site from being crawled. For example, if you created a content source to crawl http://www.ks.com, but you do not want the search system to crawl content from the subdirectory http://www.ks.com/downloads, create a crawl rule to exclude content from that subdirectory.
• Crawl content on a site that would be excluded otherwise. For example, if you excluded content from http://www.ks.com/downloads from being crawled, but you want content in the subdirectory http://www.ks.com/downloads/content to be crawled, create a crawl rule to include content from that subdirectory.
• Specify authentication credentials. If a site to be crawled requires different credentials than those of the default content access account, create a crawl rule to specify the authentication credentials.

You can use wildcard(*) characters to include or exclude any content for crawl.Also, Crawl rules can be used to crawl content which is confidential or you may not want to crawl it or any oother account is required to crawl that with different credentials, you can specify all these in Crawl Rules.
How to create a new Crawl Rule:

• Go to SharePoint Administration, make sure you are login with Search Admin Account.

• Application Management section, click Manage Service Application.

• On Manage Service application Page, click on Search Service Application. You will see the above Screen for Search Administration.
• Click on Add New Crawl Rule.

• Now as explained above you can specify what to include and what to exclude in this new crawl rule.

• If you have selected " Include all items in this path" in Crawl Configuration, then only you can specify the authentication for this new Crawl Rule (in case if special authentication is required to implement the new rule).

Testing Crawl Rule

1. On the Search Administration page, in the Crawling section, click Crawl Rules.
2. On the Manage Crawl Rules page, in the Type a URL and click test to find out if it matches a rule box, type the URL that you want to test.
3. Click Test. The result of the test appears below the Type a URL and click test to find out if it matches a rule box.

Happy SharePointing.......

Ref: MSDN Link