Trump suggests his opinion of Taylor Swift has made her less popular