Just tell him the agreement was for a disc of JPEG files, which is what you told him, and you both agreed to. Advise him he is asking for more than he paid for, which seems unusual to ask and pay for X, then expect X, Y, and Z to be delivered. Let him know your standard charge is x times the standard fee when the RAW files are to be delivered, and many photographers won't even release the RAW files. If he really wants them, you will be happy to provide them at your standard rates.
Personally, I don't see any issue with giving clients RAW images, any more than giving them JPEG's on disk. Once you give them a file of sufficient size to reproduce, whether it be RAW or JPEG, you have already given them the farm so to speak. So whether its RAW or JPEG, if you release the digital images just make sure you get a reasonable fee to cover your loss of re-prints.
For commercial clients, I deliver DNG files as my RAW images since I can embed the RAW file as well as a correct JPEG proof. I also ask those clients to sign a waiver releasing me or any quality related issues. If they mess up the file and blow their $30K print job, I'm not responsible since I have no control over the quality. I don't do much retail work, so take this with a grain of salt. Two different worlds when it comes to pricing, workflow, delivery, and procedures.