being a Cohen in regard to relationships and dating

I was raised by two Jewish parents and growing up we went to a conservative temple. After my Bar-mitzvah my parents stopped going to synagogue and were non observant except for Hanukkah. I’m sharing this as background because I believe this and the fact that there were not many Jewish girls in my high school or younger days led me to date non-Jews and eventually marry one. I was married for 20 yrs and since the divorce I have reconnected with my Jewish roots. I actually dated one Jewish woman who led me to meeting a Chabad Rabbi and since then I go to Chabad services from time to time. I still consider myself more conservative than Orthodox and yet I have learned that my family lineage is that I am a Cohen. As such the Rabbi’s have told me (according to orthodox Jewish laws) that a Cohen cannot marry a divorced woman, a non Jew and I think it even says a non virgin. So at my age, this seems pretty unrealistic to me to find a woman (even a Jew) meeting these criteria. I am told that if I did marry a woman who does not meet the criteria for Jewish law on this that I would lose my Cohenian rights or privilege’s and would not be recognized by orthodox Rabbi’s as such. I am in conflict with this because I really feel connected with the spiritual aspects of being a Cohen but don’t believe that I will ever be able to be in a relationship if I adhere to the orthodox or what is considered Jewish law about this. Since I didn’t know about this before my first marriage and I’m more conscious of it now it makes thinking about relationships harder for me. Anybody who has experienced something like this or has some light to shed please do so. thank you.