maybe I need to removing and testing some add-ons to find out what causes this
Not maybe. Definitely :) Try to switch off the add-ons one by one and you'll find the reason.
Recently I've got a report from one of the biggest miniBB forums owners too that his hoster has updated the filesystem to its latest release, and this helped to improve to load pages times faster. Just to mention...
On the sites you've mentioned everything runs quickly probably because there are not so many posts as it may be on your forum. When there are millions of posts, it would be worth to make a deep investigation on that :) there are lot of ways to improve performance of mySQL - but I think switching to innodb would not be the best solution!