You are at way different places in your lives.
You've never had an adult relationship. He's been married.
You have no children. He has FOUR.
Your future is an open book. He has ties to an ex and a life that doesn't include you. He was probably marrying and having kids when you were in middle school.
This isn't the recipe for a successful relationship.
When I was about 21, I had a man who was in his early 30s, divorced with two little kids, who was very interested in me. We were close and he was a good person and in many ways, we had a lot in common, but I didn't pursue a relationship with him for the reasons discussed above. It might be temporarily painful for one or both of you to break it off but it's ultimately in both of your best interests.
I think 15 years is more significant when you are younger. If you were 43 and he were 58 and you were both divorced with kids it might not be as much of a problem as it is now.
Get used to me. Black, confident, cocky; my name, not yours; my religion, not yours; my goals, my own; get used to me. -Muhammad Ali