hi folks - just reading thru this thing -
I'm an EE - and can assure you that there's no problem hooking all the - poles of your batteries together !

only concern I have is making sure that you DO NOT HOOK UP THE 4 BATTERIES TO THE CHARGER AT THE SAME TIME - this might create a problem for your charger since you put 4 batteries in parallel which decreases the resistance you put on that charging circuit - that might be overloading the charging circuit and burn it..
e.g. if you normally run 2 batteries on the charger and you add another 2 you basically half the resistance to the charging circuit.
So make sure, that the way you configure the switches to the batteries, do not allow you to connect the 4 batteries to the charging circuit at the same time.

something like this using 3 switches would work: make sure switch1 is not a (1,2,or both) switch, but a 1 or 2 switch !! switch2 and 3 can be (1,2, or both) switches..

charger+ --->in switch1>output1>in switch2>output1>batt1
>switch2output2>batt2
>in switch1>output2>in switch3>output1>batt3
>switch3output2>batt4