You shouldn't go back if you are not in love with him and don't think you could fall in love with him and even if you think you could fall in love with him there are many other obstacles standing in your way, the fact that he is married to someone else and the fact that you have someone your in love with. The reason you are up in arms about this is because of the guilt that you feel about leaving the first time. If your leaving didn't have much of a negative affect on your children then it wasn't a bad thing and the fact that your children left the church and agree with your teachings shows that they have your best interest at heart and understood why you had to leave. Although you think that your going back may benefit your children all they want is for you to be happy. Love is one of the main ingredients that makes a happy home without love you won't really have a relationship but co existing and that won't be much of a life. You will be a prison within your marriage without any escape.
Would you rather be with someone that you know you love and that loves you back or be with someone that you do not love but are with for your children's sake? I think your children would rather you be happy than be miserable to make them happy although you are trying to put your children first and being unselfish you still deserve to be happy with someone you love and care about. Another obstacle is the fact that you have different religious beliefs, it will play the most important role in your lives. You really have no obligation to go back especially if your children adjusted pretty well to you divorcing it may be best that you do not take that road again. I suggest you explore your relationship with the other man that you love and loves you back and leave well enough alone.