Could be a simple matter of "Port is not for you." I happen to like a lot of different ports, including six grapes, but my lovely bride doesn't like port at all, even though we both like very similar red wines (cab, pinot noir, barolo).
I do think it might be worth your while to try a tawny port to see if a little less sweetness does the trick, but in the end, drink what you like and like what you drink.
