Blog

Adding a Dynamic Robots.txt file to an ASP.NET MVC site

Ted Anderson

Ted Anderson

— 2 min read

Robots.txt is required to allow search engines to properly index your site, and more importantly not index it. If you have a public-facing staging or preliminary site that you don’t want to show up in Google results, you need to make sure that it returns the correct robots.txt with the

Disallow: /

line to prevent indexing. However, manually adding robots.txt files to staging and production environments as a manual process can be improved with the process below – the same code can serve up a locked down robots.txt in staging or internal URLs, and allow indexing in production.

First, add a route that responds to /robots.txt in either Global.asax.cs or RouteConfig.cs before your Default routes:

<pre class="csharpcode">routes.MapRoute(
<span class="str">"Robots.txt",
<span class="str">"robots.txt",
    <span class="kwrd">new
    {
        controller = <span class="str">"Robots",
        action = <span class="str">"RobotsText"
    }
);

You’ll also need to make sure that runAllManagedModulesForAllRequests is true in web.config as normally text files bypass the ASP.NET pipeline:

<pre class="csharpcode"><span class="kwrd"><<span class="html">system.webServer<span class="kwrd">>
    <span class="kwrd"><<span class="html">modules <span class="attr">runAllManagedModulesForAllRequests<span class="kwrd">="true"<span class="kwrd">></<span class="html">modules<span class="kwrd">>
    ...
    <span class="kwrd"></<span class="html">handlers<span class="kwrd">>
  <span class="kwrd"></<span class="html">system.webServer<span class="kwrd">>

The create a new controller called “RobotsController” with a single action “RobotsText”. All requests to /robots.txt will go here:

<pre class="csharpcode"><span class="kwrd">    public <span class="kwrd">class RobotsController : Controller
    {

        <span class="kwrd">public FileContentResult RobotsText()
        {
            var contentBuilder = <span class="kwrd">new StringBuilder();
            contentBuilder.AppendLine(<span class="str">"User-agent: *");

            <span class="rem">// change this to however you want to detect a production URL
            var isProductionUrl = Request.Url != <span class="kwrd">null && !Request.Url.ToString().ToLowerInvariant().Contains(<span class="str">"elasticbeanstalk");

            <span class="kwrd">if (isProductionUrl)
            {
                contentBuilder.AppendLine(<span class="str">"Disallow: /elmah.axd");
                contentBuilder.AppendLine(<span class="str">"Disallow: /admin");
                contentBuilder.AppendLine(<span class="str">"Disallow: /Admin");
                contentBuilder.AppendLine(<span class="str">"Sitemap: http://www.mysite.com/sitemap.xml");
            }
            <span class="kwrd">else
            {
                contentBuilder.AppendLine(<span class="str">"Disallow: /");
            }

            <span class="kwrd">return File(Encoding.UTF8.GetBytes(contentBuilder.ToString()), <span class="str">"text/plain");

        }
    }

You can try a number of ways of detecting a production environment, from the naïve URL checking above to environment variables in your application container.

.

Read more