Paint flakes off because the car usually not sanded well. You can do it this way. I would dry sand the whole car with 320, getting in the cracks and edges well (might have to hand sand these areas). Then go over it with 600-800 wet sand to knock down the deeper sand scratches from the 320. Lastly, scuff the car with the scotch bright pad. This way you will have the car nice and ruffed up so that the new paint will have something to adhere to.
Throw on a coat of sealer and go to town on your new paint job. Should last a long time if properly sanded and cleaned before painting. Primer is used to cover body work, bare metal, and fill rock chips.