He's a cutie, that's for sure. I think the best thing you can do is just sit still and let him come to you. Let him sniff you over, but don't try to pet him, no matter how hard this will be to resist. I think the action of your hand reaching towards him scares him a bit and makes him back off, so if you don't do this he may come closer to you. Calves are naturally much more curious than an adult cow would be, so you just sitting there will invite him to investigate.
Has he started eating a little of what his dam eats? If he has, feed him that. Or at least try. Just hold your hand out and let him sniff what's in your hand. Keep repeating this until he's confident about you doing this and a bit more braver. Once you've got that, then you can try to pet him.
DO NOT PET THE HEAD. This will teach him he can get away with butting you, especially if you "reward" him by petting him everytime he bunts you or pushes against you with his head. To start petting him, pet on the cheek, under the chin or on the neck, just like with a horse.
You may also want to consider start halter-training him soon too, as that will really help teach him both the fact that you are boss over him and to start getting him used to your touch.
Is he a miniature breed by any chance? Like Mini Angus or something similar?