Robots.txt question

Started by Ender, October 27, 2010, 05:31:26 PM

Is the robots.txt file set to disallow all spidering purposefully?

It would be really really handy to be able to do site: searches for posts using a for real search engine rather than the crummy built in search.  I can understand if it's for game advertising reasons and only wanting official stuff showing up in search, but I don't think opening up to search engines would be a bad thing at all.



1.   A biped, ungrateful.

Personally, I find the GDB search functions highly useful and accurate, with only very occasional limitation. If you'd like to discuss your specific issues with the coders and/or make suggestions about internet or software issues, then the better way to do that would be through the request tool.
Quote from: Decameron on September 16, 2010, 04:47:50 PM
Character: "I've been working on building a new barracks for some tim-"
NPC: "Yeah, that fell through, sucks but YOUR HOUSE IS ON FIREEE!! FIRE-KANKS!!"

The robots.txt stopped the spidering because the archives are so huge, and there are so many different search engines/spiders, that the server was getting completely buried by search robots.