Because when one minority group gets recognized as being so "special" that it warrants a law expressly for its protection, that means that it is frequently and almost inescapably seen as denigrating other minority groups that have not been so designated.
Thus, although it may be counter-intuitive, we should repeal all "hate crime" laws. Our laws should not put one group into competition with another on the issue of who is more special, important or vulnerable.
Simply stated, things like an assault and battery should be crimes regardless of the identity of the victim, and those crimes should be prosecuted. Then only after the perpetrator has been convicted, the particular circumstances in that individual case — including possible situations of hate and prejudice — should be explored and argued to the judge for an appropriate sentence.
All cases are unique in some fashion, and basically this approach is designed to deal well and appropriately with each of them.
Yes, to assault a person who is black, Hispanic, gay, elderly, or a fan of the San Francisco Giants simply because of that condition is reprehensible, and the perpetrator should be punished. But is that assault necessarily worse than if the victim were to be Asian, Italian, straight, young, or a fan of the Boston Red Sox? I hope the answer to that question is plain. Besides, we can't continue to address every bad situation by simply passing another law.
Fundamentally we must understand that we will never run out of minority groups that could be victimized, so why should we go down that road at all? This approach simply pulls us apart at the exact time that we so badly need to be drawn together.