I'm with Scott Spellman on this one, so I won't repeat in my own words what he said.
However, I don't know why you are reluctant to give the client the other images. And this is something you need to explain to him.
In my case, I simply tell them that my package conditions are so and so, or that the contract specifies so and so and I don't take kindly to scope creep. Scope creep is a no-no, simply because it requires more work.
Many clients have this notion that all we have to do is click a button. Thus, when we're reluctant to shoot more they think we're lazy or that we want to nickel-and-dime them. They fail to see, or choose to ignore, many times that each shot requires preparation: this is all the more true with commercial shots, which are mostly staged and require careful placement of subjects, lights, and fine tuning of big and small details. We have to explain all this to them and sometimes it's not easy, and some still remain upset even when they can't deny the evidence.
Same thing goes with post-production. I tell the clients that my rate includes minor retouching of only a number of images, and because retouching takes time, any major retouching of a final image, or additional images with minor retouching will cost extra. This is clearly spelled in my communications under Terms & Conditions. Again, some will still remain upset: so let them. It is what it is. If we were to a car dealership and asked for an extra car just because we're buying their flagship SUV, they wouldn't give it to us, would they? So why should you?
Do note, however, that at length my reasoning behind my actions relates to work (fairness) and the client's interests (good service); MY personal image factors very little –if at all– into all that. If I were to tell a client that I don't want to give him his favourite photo from the shoot because it is not 'consistent with my branding', or 'doesn't really shew "my style" ', or that it plainly makes me look bad, he'll conclude that I'm a self-conceited egoist and he will have a genuine and perfectly justifiable reason to be upset.