search index (database tweak)

the default database setup for MiniBB does not include any index for minibb_posts.post_text ... I don't know how else to put this but, isn't that sort of insane? :)

I mean, the search function uses this field for every query, and I would imagine it's EXTREMELY ineffective to not use an index for something that gets used as frequently as a simple search function?

it's of course no problem, since it takes like five seconds for me to add it myself, I'm just concerned that people with zero to little database knowledge won't detect or know about this...

mp303
what kind of index field do you need? do you want to copy all posts for the trhread into one index field, doubling the database size? like some other forum scripts do?

copy? ... ehm, no - I just want to add a full-text search index for the post_text field ... you'll need more disk space, yeah, but at what cost will you be saving a few megabytes of diskspace, or even a few hundred megabytes for forums with extremely heavy traffic? - your system is going to be doing serious overtime if it has to search through all of the posted text on every search query without any kind of optimization...

mp303
Remember that not each mysql version supports full-text index, thus for now this is experimental option, not fully supported yet.

Plus, if you have a huge forums with overloaded traffic, miniBB is not the best solution for this (as it said in our concepts).

yes, I read that, and frankly I didn't get it - how is it NOT the best solution for huge forums?

You claim it's the smallest and fastest, so that should make it IDEAL for boards with huge traffic, since you can host a larger database of posts, with more frequent access, for more users, without spending as much CPU/DISK capacity as any of the other boards ... that is what you designed it for, right?

Keeping CPU and disk requirements to a minimum, has to be criteria for every board, regardless of size ... I mean, everyone wants to make the most of their server - everyone wants the maximum amount of CPU/disk/bandwidth capacity left over for other tasks on their site ... or not?

What we are talking about here is that miniBB is ideal for small/medium sites, when you have not so many visitors per day. When you have huge traffic, you must think not about the scripts themselves, but also about complicated servers. If the motor is fast by its nature, it will not work if you have not enough fuel.

That doesn't really answer my question ... how is a fast engine LESS suitable for a larger site, e.g. on a machine with more "fuel"? :)

mp303
miniBB is still being used for forums with 100 - 200 thousands of posts and dozens of thousands of members and it works ok. mySQL is generally not suitable for more than 400 000 records in one table with thousands of selects per day when we're not using replication and all that stuff as practice shows, so full-text search won't help in this case, it will only add compatibility issues to our forum. What good is it, if you still won't be able to handle huge communities on mysql without constantly deleting old posts? I don't get it ;) It's still fast and small and full-text search won't help in practice. For now the feature you're referring to is only suitable for single projects not available for the masses. So for example if we build our own site, of course we will tune everything considering the db and server and use most features currently installed versions provide.. for miniBB we use everything common and make scripts which use common functions as fast as possible.

in real life miniBB can handle more posts than most other forum scripts do and post/topic pages load a lot faster. I also have no example of miniBB community where search function is slow although personally I run several heavy-loaded communities with ~40k registered members and more than one hundred thousand of posts.




thanks,

sounds good - can I see one of these huge forums running somewhere on the net? I'd like to see with my own eyes how fast it is :)

statistics page -> 38 seconds to load

global keyword search with a single term -> made two attempts, waited for about 4 minutes both times, and finally got "the page can not be displayed".

on my third attempt, I waited for about one minute, then gave up, closed my browser and opened a new one - for the next 3-4 minutes, all of the pages each took about 40 seconds to load, strongly suggested that the server is still gagging on the aborted search while trying to process new requests at the same time ... it returned to normal after about 3-4 minutes.

I realize this site has 200.000 posts, but ... that is incredibly slow. Do you know what kind of server it's running on? 100 MHz? heh ;)

Let's hope nobody actually uses the search function ... if I had been the webmaster of this site, I think I would simply remove - it's not like anyone is going to wait several minutes for a search result before giving up and aborting, so all the search function does now is choke the CPU ... most pages take at least 20 seconds to load, even if you're not choking the poor server with keyword searches ...

But seriously though, even if our forum only reaches maybe 20.000 posts, I already fear for the impact that searches and page views will have on the response time of other pages on the server :/

I understand what you meant now - MiniBB is great for small and medium size sites, because of it's small footprint, but it should definitely not be used for sites of this magnitude...

dunno, think it depends on the server. haven't done any volume testing for over a year but when i did minibb seemed fine. that was on a P4 2.6 Ghz/1 Gb RAM.

not sure what they're running over at that harry potter site, but my laptop could server pages faster than that :)

mindplay 4days
Of course, it depends on the server itself, as far as I know, harryjezakon uses PostgreSQL to handle requests, and it is already optimized with some kind of PHP optimizer, and there is no such good hardware for handling these requests (this is the fuel I am talking about).

But you know, we can take responsiblity for these results. As I have already said, we have still not designed miniBB for those large projects BECAUSE we have no place to test it enough. On our site, there are about 5,000 posts, and it works like there are 5 posts (the same time) - and we have not the most powerful server here.

I know about the most of the largest forums community, it's on forums.ixbt.com (Russian) - they have thousands visitors each day, and thousands posts each day. Guess, mySQL isn't the best solution for them, so they wrote special software for themselves, and this software uses some kind of database only for temporary data; all data is saved in STATIC HTML files. And this is only the best solution for huge forums - use something archived and static.

..all data is saved in STATIC HTML files. And this is only the best solution for huge forums - use something archived and static.

definitely :)

here's a direct link for the lazy:
http://forum.ixbt.com/

btw, speaking of dodgy solutions, check out: http://forums.urbanterror.net

custom asp forum running on a fairly beefy wink2k box afaik, but it's struggling all the time (if it's up at all).

getting back on topic, mp303 - you could maybe post up optimisations for flavours of minibb (e.g: mysql) here in the forums or submit them as an add-on or solution.

@Team:

how about writing a simpe performance test script? e.g. one that generates a hundred thousand random posts - this would be easy to implement, and would instantly show you how well it performs ... well, if I can find the time, I think I'll do some testing myself - I'll need such a script anyways, for performance comparison during optimization. I'll see if I can find the time, and post my results here :)

mindplay
We already wrote such kind of script at the very beginning, when we released 1st version of miniBB. But this kind of script means nothing. You will have large database - that's true, but one is to HAVE database and another is to OVERLOAD it. I can not emulate 10 visitors on my page in one second, for example. And it all depends again on the server, OS, hardware I run - it could be run OK on my own server, but it can stuck in on another server. When we talk about such kind of overloading here, software itself - I repeat - means nothing. You can have ruppa-dubbaling Porsche Cayenne Turbo, but without fuel it's NOTHING.