I think you will want to keep all the acid you can with this to offset the sweetness of the port. Of course that is if you plan on keeping this with residual sweetness as well. Ports typically stop fermentation due to high alcohol while they still have plenty of sugars to ferment. You can do this 2 ways:
1.) Keep feeding your port sugar until the yeast die, then fortify with your chosen Brandy
or
2.) Start with a high gravity must and when the SG gets to the point that you have enough residual sugar, add a neutral extremely high alcohol brandy to kill the yeast! They use about 190 proof to do this in portugal. This is the traditional method for making port.
Personally, I would choose option #1! Again, this all assumes that you are going to make a sweet port, at which point you need the acid to brighten up the sweetness of the wine.