I can empathize.
I've been doing fashion since 2011. It's a hobby inasmuch as I'm not making money at it. Yet. I've shot professional models, some for money, some TFP (time for photos), as well as my daughter who is about to turn 18. Once I did a fashion shoot with my daughter and one of her friends in preparation for an American Girl Fashion Show shoot. Both girls were 13 at the time. As it was about lighting, outfits weren't all that important. Since both my wife and I knew the other girl's Mom, it was just the three of us. That said, the other Mom had to sign a release. The girls' payment was ice cream at their favorite ice cream shop.
Generally speaking I will not shoot minors for fashion. I'm paranoid.
I am the chair of our high school band boosters photography & media group, which means I'm shooting lots of kids. The band has ~160 kids, almost all of whom are minors. In the context of a marching band or concert band event, I'm fine with photographing them. They're great kids, and their parents are equally good. Our photos go to a closed website. I also shoot high school girls varsity soccer. It's a sporting event, so it's public.
This last spring I was going to NY and was trying to coordinate a fashion shoot through Model Mayhem (MM). One option would have been with two young girls and we were going to be on the streets of Manhattan. They were good, one a ballet dancer and the other just starting out in modeling. When I learned one of them was only 17 and that her Mom would send a signed release with her, I said no. No parent in person handing off the model? No shoot. They seem to be nice girls, and probably everything would have been fine. It's the unpredictable that worries me. Tentatively we've planned to shoot next year, and that would be great. For now however, no. I have far too much to lose.
Regarding the "P" word, that's why I won't do fashion with minors. I've shot nudes but really prefer my models clothed. There's a photo of a girl on a premiere artistic photo website where she was modeling as implied nude and appears to be a minor. She's beautiful. She's physically mature. It's a stunning photograph, fully tasteful. Regardless, I never would have taken the photo, and had I done so I never would have posted it publicly.
So there's all of the negativity.
Were I in your shoes, would I do it?
I'd be open to it.
The model and her Mom are good with it. As long as you've communicated everything, once they say they're good with everything, stop worrying and don't give them a reason to worry either. Get them to sign a release, and be prepared to have them say they don't want the photos made public. Should they do that, be gracious and comply. That's your reputation, and you don't want to ruin that.
Screw those who call you pervert. If they have a problem with your photos, tell them to make their own bloody photos!
As long as you have a mutual understanding between the model and the Mom, and everyone is comfortable with the shoot, go for it.