Well, let's go back to what President Woodruff said:
So, the removal doesn't come from being innocently wrong about something per se. The removal comes when the prophet is so wrong--and, apparently, deliberately so--that the incorrect teaching either separates people from the channels of revelation, or else stops them from doing the specific things that God actually wants them to do.
Brigham Young's having taught "Curse of Cain" and other similar ideas, while certainly discomforting, doesn't require me to reject him under the Woodruff analysis; because it has yet to be shown that this idea was the sole and proximate cause to any inhibiting of revelation or otherwise wrongful action; let alone that Young was acting out of any form of mens rea. Now, the idea of the priesthood ban being contrary to God's instructions would, I think, would be more problematic. But then, the Church has never conceded that the ban was inappropriate under the circumstances or that Young exceeded his authority in enacting it.