Is there any reason to go to the doctors?

I stepped on a (small) lacrosse ball 6 weeks ago and hurt my ankle really badly. It swelled so the outside of my ankle was the size of a baseball, it really hurt to point my toe, and walking was extremely painful. After 2 weeks of playing lacrosse on it, I went to the doctor, and was put in an air cast stirrup for a week and a half. They took xrays and saw no bone damage. After the week and a half I went back to playing lacrosse. The swelling never went completely down and when i stretch my ankle to the side, extending the muscles on the outside, i get a sharp pain on the side of the round bone there. I'm planning on taking 2 weeks off to let it heal, but my question is is there any damage I could have done that will take longer than 2 weeks to heal? Should I go to the doctor again?