Dealing with a creepy stranger I gave my phone number to?

You did good. If he asks why you wont respond, tell him the truth. Tell him straight up, "I don't want to talk to you"