Paint flakes off because the car usually not sanded well. You can do it this way. I would dry sand the whole car with 320, getting in the cracks and edges well (might have to hand sand these areas). Then go over it with 600-800 wet sand to knock down the deeper sand scratches from the 320. Lastly, scuff the car with the scotch bright pad. This way you will have the car nice and ruffed up so that the new paint will have something to adhere to.
Throw on a coat of sealer and go to town on your new paint job. Should last a long time if properly sanded and cleaned before painting. Primer is used to cover body work, bare metal, and fill rock chips.
I did that on my Cutlass when I had it as a rush job for Senior Prom and paint started peeling off a few months later. Ended up spending twice the money I wanted to get it stripped all the way down and repainted
If your woried there are many adhesion promotores out there today that will basically stick to anything and never come off you just apply a coat of this first then spray .. Lack of adhesion is usually in the prep ..