check out this site, they have a good method of removing the rotten wood with the auger bit, I haven't tried their repair stuff( it is expensive).
http://transomrepair.com/zk/
Years ago I talked to a guy who ran a big glass shop in west Fl(Porta Products). he used the same method to dig out the old transom, but he left the inner skin in place, rather than making a mold on the inside. He said they would take a 5 gal bucket, pour 3 gal of resin in it, then mix in what ever scrap chop glass, grinding dust, saw dust, milled fibers, cabosil, micro baloons, or what ever they swept up off the floor of the shop, till it got to the right thickness, then pour it in the boat where the transom was. The inner skin would make the inside of the mold to hold the mix( just make suire to tape any holes to keep the stuff from running out). He showed me a picture of a big go-fast with three merc outboards on it, on brackets. He said they had repaired the transom in that boat the same way and had no problems with it. I have never tried to repair a transom this way, but it is tempting to try it out.