Headers vs Cast Manifolds for Low RPM

[Chris in Idaho]

I have a 1986 short bed 4x4. The previous owner pulled the original 4.3 v6 out and installed a Goodwrench 350 with a Quadrajet on top of a square to spread bore adapter on an Edelbrock Performer EPS intake manifold, and it has full length headers. It idles smoothly, so I assume the cam hasn't been changed. It has a SM465 transmission and NP208 transfercase which give it a nice low gear ratio for crawling/trails.

My question is this:
For what I like to do with this truck I need smooth controllable throttle response and good bottom end torque (I need to be able to lug the engine way down... like 300 RPM without dying). Right now the engine runs more like a hotrod than a truck. It has a lot of power, but the throttle response is touchy... it kinda explodes into action, and below 1300-1500 rpm it doesn't really have the torque to lug the truck around. I think the intake manifold (especially with the adapter) has too much volume to keep the intake charge velocity up at low RPM. I just got back from the junkyard where I paid $13 for a non-EGR Quadrajet intake manifold, and I'm wondering if I should have grabbed a pair of cast exhaust manifolds too. The headers are kinda cool, but I'm not really concerned with top end horsepower. My uncle always tells me that headers destroy bottom end torque... is he right? Would it work better with stock manifolds?