Power Window...Regulator or Wiring Issue

[crazeetxn]

Trying to figure out my problem here and having no luck. Passenger window motor was gone so I replaced it. One of the wires to the switch was broke, so I got a new switch from classic. Wiring colors were different, but I think I figured it out using a pattern and the letters on the back.

Problem is, it will either roll down fine, or if I reverse wires, roll up fine from the drivers side. But it won't do both. I can hear the motor engage, but it won't do anything...unless you wait. Then it goes up a little. Wait...goes up a little more...etc.

Anywhere else I can troubleshoot?