Yes, caster helps a little bit, but if the toe-in is wrong, it will not be enough to keep the car going straight. That’s why you sometimes have cars that drift either left or right, it is almost always the toe-in which is wrong, usually on one side (after a shock or a bump).
And you’re right, power steering goes with a bit higher caster angle.
On my previous ’oldie’, a 1962 Jaguar MkII (with an extremely heavy cast iron V6), the original steering was not assisted (very rare option). I made the mistake of having an electric power steering installed, not adjustable. For parking and city driving, a dream. For high (or even moderate) speed driving, not exactly a nightmare, but much too sensitive for ease of mind. You had to concentrate all the time to keep the car in a straight line.