A guy from the gym asked me out, but should I go?

He's not someone you've had a crush on but you're impressed with his directness in saying he'd like to take you to dinner and may I have your phone number. And then after you say yes, you're wishing you hadn't. It doesn't feel right for some reason so do you go anyway, or just cancel?