Actually, the people I have seen refer to him as “off his rocker” are not white. Part of his reputation came from Michael Eric Dyson’s attack on him which focused on his ego. It stated that he was only upset with Obama because he had not been invited to an inaugural ball. West has also made some very racist-sounding attacks on Barack Obama including invoking the n-word to refer to him.
This was during some of his moments he may wish he could take back. I don’t know. I am not involved in that level of the argument, but I will say that an attack that tries to make me sound racist for invoking language widely used to refer to someone is not very objective.