You should always warn us if you've made a custom modification.
It's not our free task to investigate custom problems, and it seems now it was completely your mistake of improper configuration.
All I could say admin's email specified in $admin_email of setup_options.php is the common email used as "Reply-To" in all notifications to the users; also as if you set $emailadmin or $emailadmposts, this email will be used also. However, for individual subscribings on forum, admin can specify his own email address and his Profile. In most common cases it is fully enough, however I will put your suggestion into "tasks for thinking" and may be figure it all out for the default version.
By this time, if you look at sendMail() function of bb_functions.php, you will notice it is possible to simply change all return and reply-to addresses to whatever you want.
miniBB provides much more convenient tools as Checker
for example, which avoid using of email notifications for admin and keep your resources.
I didn't understand all the hassle around the SMTP security and the things you've done, and how it can be related to miniBB in general (there is a possibility to use PhpMailer