That portion of the starter can cause the problem, I just replaced mine cause I could see a little smoke come off it when it would sorta drag on occasion. Lithium actually gets sticky over a period of time, I stopped using it.
The starter itself could be the problem, over time and use the particles from the brushes and windings can coat or build up in the starter causing some insulating qualities between the brushes and their contact surfaces. Just disassemble it, brush it out, reassemble and test. If it improves or stops the problem, buy a new starter and replace at your convience.
Don't rule out the starter solonoid, if it keeps dragging, have someone hold the key switch and you tap on it to see if it makes the starter improves.
Obviously check all the connections.
First and most important, after charging the batteries, do a load test on each one to see just how good they are. A regular voltmeter won't do it, something like this from Harbor Freight or the like will pay for itself after just a few uses and make you look smarter.
