Looks like it is the ESC. I performed the bypass that I found online here:
http://www.73-87.com/7387garage/drivetrain/escrepl.htm (Not the replacement, but just splicing the wires as detailed at the bottom)
And it immediately seems to have helped. Also, I had an issue with the engine bogging a little under very light throttle, this seems to be gone as well. I'll drive it around for a few days and see if that was it.