The "Jared Diamond is a racist" argument is, as I understand it, based on some rather circuitous reasoning:

    Diamond argues that environmental/ecological/geographical circumstances are why the "West" conquered "the Rest"

    In doing so, he argues that ideology/religion/racism were not the determining causes for why this happened — expansionist ideologies and religions only result in conquering other people if the resources are there to do so

    In so arguing, he lets Europeans off the hook, making it seem "natural" that they ended up taking over the Americas, etc., because it was really just the geography that did it

    Thus it is racist, because he should really be acknowledging that the Europeans did some terrible things, and were motivated by lots of unpleasant ideologies, and were kind of bad people

Or something along those lines. OK, I sort of see it, maybe? You could, I guess, read GG&S as a European-descendant and say, "ah, my ancestors were just playing the role that geography set up for them, they weren't so terrible after all!" That's the fear, anyway: that white people will read GG&S as comforting, because it absolves them of conquest, slavery, genocide, etc.

From Diamond's point of view, this is a completely incorrect inference. He is not trying to pass moral or ethical judgment on what happened, just trying to articulate why it happened (and why the reverse conquest did not happen). He thinks it is silly to imagine that the Europeans had any monopoly on expansionist ideologies or religions (see, e.g., the Aztecs, who were an expansionist, bloody, slaver, empire-building culture, albeit one with very low-tech materials). What they had, he would argue, is the guns, germs, steel, etc., which he sees as being environmentally/ecologically/geographically determined. Furthermore, he would argue, the whole point of his book (very explicitly) is that lots of people think the Europeans "won" because they had better genes, and he is constructing an argument against that racist argument. He is very clear that he thinks that humans are pretty much the same critters everywhere — that only the circumstances change.

As you can tell, I think the "Diamond is a racist" argument misses the mark a bit in many ways. I agree with Diamond that "Europeans had more dangerous ideas/culture/politics/religion" is a rather silly argument (I mean, they had these things, sure, but they were not unique in this). I think you can make a nice statement about the intersection of human culture with the kinds of environmental factors Diamond talks about, and in some of his later postscripts to GG&S he tries to gesture towards that (specifically in talking about China, who had all the same resources but didn't conquer people outside of their own geographical orbit for the most part). So I think you can say that Diamond's argument is a incomplete and mono-causal; it is a very "rough" resolution of big shifts in human history. But I think concluding that his environmental argument is itself "racist" or that his entire approach should be thrown out the window (and we should find recourse in, I don't know, intellectual history?) is silly.

I also think that the implicit argument that is usually just out of sight here — that the reason Europeans did these unpleasant things to the peoples of the Americas (and not vice versa) is because they were just culturally bad in some way — is itself laughably simplistic, both in its views of the Europeans (evil bad technology people) and in its views of the Native American peoples (pastoral friendly peaceful types).

When it comes to popular "West vs. the Rest" kinds of narratives, I admit I prefer the work of Charles Mann (i.e. 1491 and 1493) because he manages to talk about some of the same sorts of factors as Diamond, but does more effort to weave them into human culture in a more plausible way. Obviously anything that happens on that scale of history is going to be a complicated and messy interaction of both natural and cultural factors. And the desires of a historian (who want to deal with human choices and decisions) and someone taking a more anthropological/biological, long-scope view of humanity (who wants to see humans as essentially a very productive animal) are probably generally going to be at loggerheads no matter what (which is to say, I doubt there is going to be a narrative that can satisfy both goals).

Historians can tend to neglect natural factors (like climate and geography), and Diamond arguably swung too far in that direction. But I don't think that makes him or his argument racist.