There's no point pulling away if all you two are going to do is just constantly think of the other. It defeats the purpose and it's making it worse even. So, break up with him if you think that's what's best, but stick around so that he doesn't have to wonder.