A while ago I got a lot of error when trying to diagnose my site with Google webmasters tools. One of the core problems was the fact that Googlebot was trying to index AJAX links, which just can't be good.
We start simply by adding a route for our robots.txt
// ~/app/config/routes.php Router::connect( '/robots.txt', array( 'controller' => 'seo', 'action' => 'robots' ) );
After that, we need to type some code in our SeoController:
// ~/app/controllers/seo_controller.php class SeoController extends AppController { var $uses = array(); var $components = array('RequestHandler'); function robots() { if (Configure::read('debug')) { Configure::write('debug', 0); } $urls = array(); // ...snip... // fill the $urls array with those you don't // want to be indexed/crawled // for example $urls[] = '/articles/view/some-article/commentpaging'; $this->set(compact('urls')); $this->RequestHandler->respondAs('text'); $this->viewPath .= '/text'; $this->layout = 'ajax'; } }
Of course, the URLs should not be hardcoded like this.
Important bits:
- response type is text
- layout is ajax (simply because it's empty)
After that, in your view:
// ~/app/views/seo/text/robots.ctp User-agent: * <?php foreach ($urls as $url) { echo 'Disallow: '.$url."\n"; } ?>
And that's it! Of course, this can be easily extended to disable access only to certain robots, but that's normally not a very good idea (except for those bastard bots which don't even bother checking robots.txt).
To see this in action, simply see my robots.txt:
http://lecterror.com/robots.txt
The URLs for it are generated like this:
$articles = $this->Article->getSitemapInformation(); $urls = array(); foreach ($articles as $article) { $urls[] = Router::url( array ( 'controller' => 'articles', 'action' => 'view' ) ).'/'.$article['Article']['slug'].'/page'; }
Easy, right?
Happy baking!
Article comments — View · Add
var $uses = array();
var $components = array('RequestHandler');
So this way you won't get errors like:
- Error: Database table seos for model Seo was not found.
- Undefined property: SeoController::$RequestHandler
Gr, Paul Edenburg