Generally 3 positions should give you a fairly accurate result. Having more positions may give you slighlty better accuracy, but it can also lead to more error too.
How far off are your positions? What are you georeferencing?? An air photo?? A scanned map?What kind of GPS are you using? How accurate is it? There are many possible reasons why your position could be off.