miniBB ® miniBB®
miniBB Support Forums
 | Forums | Register | Reply | Search | Statistics | Manual |
The Other miniBB Support Forums / The Other /   

GoogleBot hitting the forum - hard

Author radeldudel
#1 | Posted: 16 Jan 2008 04:25 
I've got a slow-moving site with not so much traffic (50-100 Visits a day).

Some time after I got miniBB installed, the GoogleBot started to go crazy on the forum. Traffic in kb and page hits are ten times more than before, but all the added traffic is caused by GoogleBot.
Funny thing is, the forum is not really in use (yet).

I'd guess that the forum software does somehow convince GoogleBot to come and look often for changes, which is a good thing for a lively forum, but a bit overboard for a forum that is not used (yet).

Any idea what causes GoogleBot to love my site? Does MiniBB somewhere randomly change content so Google believes the site changed every time they went looking?


Author Paul
Lead Developer
#2 | Posted: 16 Jan 2008 07:03 | Edited by: Paul 
I suppose you could put NOINDEX,NOFOLLOW attribute for ROBOTS meta tag and it will avoid Google indexing your pages at all. Read also related threads:

Why my miniBB forum is not searched friendly on Google?

How do I avoid GOOGLE finding my board?

There is no randomly changed content in miniBB and no kind of other tricks. It's just has a well-generated plain easy valid XHTML pages which are not the miracle to love by the indexing engine. Miracle? I believe not - just a good layout prepared for the content.

We are also not affiliated with Google anyway, however we love Google and Google loves us :-)

Author radeldudel
#3 | Posted: 16 Jan 2008 07:26 

Not being in the Google index is not really an option for me. I'm just a bit put off by the large amount of useless traffic since nothing has changed.
I want to be found by Google, and I think miniBB forum is very search friendly for Google, btw.

I checked my forum, and found there is a bottom line on every page that is changing with every reload. Possible Contents are:

"Powered by bulletin board script miniBB™ © 2001-2008"
"Powered by online community software miniBB™ © 2001-2008"
"Powered by light forum script miniBB™ © 2001-2008"

Random content like this can make GoogleBot believe that the pages are really changing, and it tries to keep its index up to date by checking them as often as stuff is changing. Since stuff is changing on nearly every reload Google keeps upping its crawl speed to match the change speed.
Googles Webmaster Tools even opt me to allow even higher crawling speeds since my site is changing so much. This is marvellous for a high-traffic website, but not for a site with few changes.

I can't be sure this is the cause of GoogleBots high traffic, it is just the first thing I noticed that might have this effect.

It might even have nothing to do with MiniBB, since I added some other stuff to this website lately, I gotta check the logs some more to be sure where exactly Googlebot is looking for changes.

Thanks so far,

Author radeldudel
#4 | Posted: 16 Jan 2008 07:50 
Woops, I checked my logfiles, and it seems Googlebot is not hitting the forum, but some other new stuff on my site. Funny thing is, the tool to analyse the log file does not correctly show much traffic in this other area, but instead shows more traffic in the forum, misleading me into thinking the forum was causing more traffic.
Checking the raw logfiles showed where the traffic is really flowing.

Sorry to stir up questions without really checking all my facts!!!

Sam (lovely software, btw)

Author Paul
Lead Developer
#5 | Posted: 17 Jan 2008 03:58 
I agree I've missed some way this copyright notice :-) True, it's randomly changed every time, but I doubt it can be the reason of heavy Google visits. If we talk about the first page, then for example amount of Topic Views is changed every time too, but the content itself may stay the same (the same for internal topic listing pages as well).

I think if your board is live there are small changes everyday. Even if somebody registers on your forums, there is a change; not saying if somebody posts a two-letters message, there are big changes around :-)

So if Google would be hooked up on something like this, it would kill itself even on amateur sites.

It's ok for the logs :-) No problems, this is always an interesting question and I hope it will be forever while we develop miniBB.

Author Danny2000
#6 | Posted: 18 Jan 2008 12:13 
you can register in google sitemap and define your "google indexing speed" over there ?(set it on slower speed)

Author radeldudel
#7 | Posted: 18 Jan 2008 13:20 

I know, as you can read in my post above Google Webmaster Tools even allow me to select higher crawling speed for my site since Google thinks it is highly changing.

But as I, too, wrote above, I already found the source of my problem, and it has nothing to do with MiniBB.


The Other miniBB Support Forums / The Other / GoogleBot hitting the forum - hard Top

Your Reply Click this icon to move up to the quoted message

 Short link for this topic:

Only registered users are allowed to post here. Please, enter your username/password details upon posting a message, or register first.

Before posting, make sure your message is compliant with our forum posting rules. If not, it may be locked or deleted with no explanation.
miniBB Support Forums Powered by Forum Software miniBB ® Features  Requirements  Demo  Download  Showcase  Gallery of Arts
Compiler  Premium Extensions  Premium Support  License  Contacts
Proceed with the Captcha add-on: protect your miniBB-forums from the automated spam and flood.
Captcha Addon for miniBB