You can reject a man, or you can tell him how attracted you are to him. But you can't do both; it's not fooling anybody, least of all him.
You certainly aren't going to be able to say anything that will keep him from feeling inadequate. He's not suffering some form of clinical depression that gives him irrational feelings of inadequacy; he is suffering utterly rational feelings of inadequacy stemming from the fact that you have told him he is inadequate, at least in your eyes. He didn't measure up, and that's what "inadequacy" means. If you'd thought he was such hot stuff, you'd have wanted him.
Rejection in any form sucks. Rejection from someone you love sucks more. Rejection coupled with well-meaning condescension is irritating as well.