Just got back from chasing Bruce's RV around this weekend.
Short version: A lateral angle in your pivot axis will indeed cause an off center behavior. I'm surprised it is too much to correct with brakes, which makes it sound more like a brake problem. At your low taxi speeds, small changes in caster angle will change steering effort more than centering behavior and I would not expect your problem is due to caster angle.
The high speed stability of a castering nosewheel is sensitive to "trail," or the (horizontal) distance the steering axis intersection is in front of the wheel's ground contact point. Think of this as the tire patch "trailing" behind the pivot axis intersection.
For most applications, Q included, we want positive trail, tire contact point trailing the pivot axis intersection. There are several geometric ways to add mechanical trail : the big ones are steering angle and offset.
Bicycles and motorcycles add trail by adding rake angle, or inclining the pivot axis. Big wheels, slower speeds, and they run 2-3 inches of trail, but they also can apply more steering effort and want some self-steering for stable turns.
Our fork already builds in some mechanical trail, about 7 inches if I recall. Most Tri Q installations benefit from adding more. The plans call for about 2 degrees. A degree or two more reduces shimmy tendency, a degree or 2 less increases shimmy tendency. Steering efforts are inverse. It looks to me like Scott Swing picked 2 degrees as an acceptable compromise between shimmy avoidance and steering effort.
However, I have not yet inspected an aircraft that is operating with that low a breakout force, which has, among other things, to do with the fact that most of our aircraft are not mythical 500lb factory demonstrators (more on weight effects in a minute). The Kantrowitz method applied to a more typical weight indicates a kinematic shimmy bound breakout force somewhat higher, at 16.4lbs (see https://quickheads.com/index.php?option=com_content&view=article&id=2387&catid=48&Itemid=144
I run a little bit lower, probably around 10.5-11.This produced a setup that is well-behaved in most conditions.
One thing that has not been quantified is the change in steering angle under load transfer. Under heavy braking after the wing has quit lifting, load transfers to the nosegear, which changes the caster angle (more vertical-->reducing trail) and can cause shimmy (see Fig 6(b) in last link for an illustration of the bend). If you are heavy, have a forward CG location, or set your damper to the 6lbs breakout, you might occasionally see mild shimmy when braking hard somewhere between 21-28mph. For those reasons, if I were building today, I would investigate setting my caster angle to 2.5 degrees. I'd expect that load transfer will make your lateral angle more problematic when you are heavy or under load...might be best to address your lean now.
Great to know that Kevin is able to run more friction. Kevin, have you ever experimented with other breakout forces?