The only part you mentioned that sounds scary is the long haul annually. I say if you have enough brakes, you drive like a sane person, and you load appropriately you'll be fine.
For comparison: This is why I'm rehab'ing my family's old Jimmy. The Subaru makes ~300hp, has upgraded brakes (factory ABS is awesome too), has a factory tow rating of 2000 lbs/200 tongue weight, but is really pretty much limited to those figures. In this pic:
The trailer axle tipped the scales at 2600 and I figure I had at least 300 on the tongue. Car was sluggish from a stop (not a healthy idea), but did fine on a 700 mile trek to Idaho Falls and back from the Portland area. Some of the mountain passes were kinda sketchy for the drivetrain, but I slowed down and all was well. Soon I'll be needing to tow my baja bug to the beach and back, which is way more than I want to have the subaru do. ATVs are one thing, complete cars are entirely another.
So I'm with the Suburban camp. If we didnt already have my Jimmy in the family awaiting a use, I was shopping for a 3/4 ton Burb. Plus in WA if you can find a decent one 85 and older (next year 86 and older) you're emissions exempt.
Erik