AnnieBear,
There is nothing for you to be ashamed about. This man is a disgrace to be calling himself a man of God. This is why most women never report rape. I was raped at 15 (three weeks before my 16th birthday) and the boy who raped me was my first. I had sex him before b/c I was a virgin and wanted to know what's it's like to have sex. After that, he came to me and asked me to be his girlfriend, I said no. So we were at my friends house setting up for my party, it was me and him left there; the subject of us came up again, stuff was said, and that night he raped me. I was so ashamed. I blamed everyone for what happened expect him. I felt like a hoe, a LovePeddler, I felt dirty. I remember going home, taking a shower and putting my clothes in the washing machine. I then went to the mall because I didn't want to be alone but I didn't want to be with people. It felt like everyone knew, like everyone was laughing and pointing at me. It felt like everyone saw that I walked differently, that something happened, like I wanted or asked for it. I never want to experience that again. I wouldn't wish that feeling on my worse enemy.

My advice to you, since it has been 10 yrs, confront him. Don't do it to get back at him, do it to heal yourself. This shouldn't be about revenge. If he won't see than you confront him in front of people if you have to. Let him recognize and remember what he did to you. A rapist would walk by his victim and even speak to her like nothing happened. Do not be ashame, it is not your fault what he did to you. Tell your brother also cause he is going to find out and it is best to be from you.

@AnnieBear:
My dear, please accept my sincerest sympathies over this ordeal.  And, please read the next sentence carefully, for it is true:

YOU HAVE ABSOLUTELY NO REASON TO FEEL SHAME OR DISGRACE, AS YOU DID NOTHING WRONG!

Allow me to suggest that you seek professional counseling to help you with closure.  What was done to you cannot, unfortunately, be undone.  However, it must be dealt with.  You have given ten years of your life to somebody who is not worthy of it.  Give him no more. 

Please, seek the help of a professional.  And, please let us know how things work out for you.