Indeed, that's totally possible as in the first case the RMS is computed on the few points you have picked (4 or 5 I guess) while in the other case all the points of the aligned cloud will be used. Therefore, if some parts are not perfectly overlapping, they will increase the RMS a lot.
If you want to estimate the RMS on all the points in the first case, you can compute the distances between the aligned cloud and the reference one just after point-picking. You won't get the RMS as is, but the std. dev. should be equivalent (I believe it is output in the Console, but you can use the Gaussian Distribution fit also).
Fine registration-problems
Re: Fine registration-problems
Daniel, CloudCompare admin