Agree, in fact I've found that the brandy never makes it better. I use Grain. It has no flavor profile, and leaves the ports to taste like the flavor they're supposed to be.
Black Forest, Toasted Caramel, Coffee, Vanilla Fig all are very good. Probably in that order for me.