miniBB ® miniBB®
miniBB Support Forums
 | Forums | Register | Reply | Search | Statistics | Manual |
Suggestions miniBB Support Forums / Suggestions /   

Extend robots.txt

Author kuopassa
Registered
#1 | Posted: 1 Aug 2014 09:33 
Currently miniBB's robots.txt file has by default this:

User-agent: *
Disallow: /templates

I recommend changing this to:

User-agent: *
Disallow: /templates/
Disallow: /img/
Disallow: /lang/

Author tom322
Registered
#2 | Posted: 1 Aug 2014 17:31 
Good idea.. I'd also include /fonts/ folder.

Author Paul
Lead Developer
#3 | Posted: 1 Aug 2014 23:03 
Thanks for your suggestions.

If you care about your server's security, it's much better not to put any of these folders under robots.txt, but to protect them on the server's level.

I suppose, /lang/ should never be indexed anyway, 'cause any of the files it has, will provide an empty output (language packs do have just definitions, not the output of something). I think SE will not index an empty output, because there's nothing to index.

Also, I'm not sure if in any place of miniBB public sources you could find a reference to the /lang/ folder. It should not just appear in any of the public output HTML. Correct me if I'm wrong.

/img/ folder - you'd laugh, but it brings some traffic. I see few of users come to this forum just to see what the BOLD ICON or PICTURE ICON are about. Also as I know from my experience, many users prefer to use the same folder to keep extra images for the forum, like banners. I'd say it's a matter of taste if you disable this from robots or not. I think, it wouldn't secure anything significantly, but it may bring a little something.

/fonts/ folder comes only in a case if you have the Captcha module installed. It should not appear in any of the public outputs as well by default, it's used only as the system folder. If you are afraid of stealing your sacred private fonts, then putting it on noindexing mode of robots is not the proper solution. Then you'd better set up another permissions for that folder, which would allow system PHP scripts to read from that, but no other scripts having public access to that.

Preventing displaying of the contents of the folder on the server's level would also cover all cases, incl. SE robots or public access. That means, if you'd open the folder directly, you'd see a 403 Error or something similar. DirectoryIndex directive in Apache is responsive for that. This is used for all the folders you mention on miniBB forums; for templates, we've put another exclusion under .htaccess.

Suggestions miniBB Support Forums / Suggestions / Extend robots.txt Top

Your Reply Click this icon to move up to the quoted message

 Short link for this topic:

 ?
Only registered users are allowed to post here. Please, enter your username/password details upon posting a message, or register first.


Before posting, make sure your message is compliant with our forum posting rules. If not, it may be locked or deleted with no explanation.
 
miniBB Support Forums Powered by Forum Software miniBB ® Features  Requirements  Demo  Download  Showcase  Gallery of Arts
Compiler  Premium Extensions  Premium Support  License  Contacts
Proceed with the Captcha add-on: protect your miniBB-forums from the automated spam and flood.
Captcha Addon for miniBB