Wow. You pretty much nailed that frisketted airbrush look. Well done!

Photoshoppery tips: masking just doesn't quite work like AI, yeah. Try using layer opacity masks to control your paint. Or ask Ashy, she knows PS better than I do and I've seen her do this thing with 'linking' layers to create what looks like easily-editable masks. Also play with the little buttons near the top of the layer palette, below the opacity controls - one of them is 'lock transparency', which means you can't lay down new areas of paint, just paint over existing stuff. Depending on your blend modes this may be a solution.