This is driving me crazy :(?

You are driving yourself crazy. If you really *love* him, then trust him. He has apparently been honest with you so far, definitely a good sign. Just try to show that you are a caring/attentive girlfriend and he will stay with you. If you become crazy or paranoid about this, he won't be so keen on seeing you.