Cavic said in Trebinje in southeastern Bosnia-Herzegovina that reasons for his decision were not of a political nature, but that he had previously arranged engagements.
He reiterated that Milosevic was "an undisputed political notable with accompanying controversies".
"At one stage he represented the highest authority for the Serbs and he then he lived to see his authority lost in his own country," Cavic said.
He added that he personally could not forgive Milosevic the blockade of Republika Srpska in 1994 and the embargo "which was ten times worse than the embargo imposed on us by the entire world".
In the summer of 1994 Milosevic, at the time president of the Federal Republic of Yugoslavia, made a decision which isolated the Bosnian Serb entity and its leadership for rejecting the Vance-Owen peace plan to stop the war in Bosnia-Herzegovina.