Ideally a used camshaft should come with the towers and the rockers it was run in with.
Though not ideal, I already have replaced camshafts only with no problems (when going for a performance grind for example).
In any case, towers should be replaced as a set as they are machined together. Mismatching them may have the camshaft binding due to tower journals slight misalignement.
In your case, you should look into the condition of your tower journals. Given that the rest has been damaged by oil starvation, I wouldn't be surprised if they are scored.