What version(s) are you observing this with? We pushed a tweak that might greatly reduce this error for Build 63 and above.
In more general terms, 5 images should be totally sufficient. If you aren’t being super-precise when tagging the pixel locations on each image, that will create and compound error. Using natural features or other objects where determining an exact position to tag is difficult, you’ll likely have more error. Low-quality GPS position data when recording the location of a GCP location is also a potential source of error.