It’s not simply the toxin, it’s the dose. Bit of a dose-response lesson in the Talmud: Rav Bivi’s daughter was given lime as a depilatory. The lime was applied gradually, one limb per session. However, a gentile neighbor tried lime for his daughter. She was given the lime all at once and she died. (Shabbat 80b) Don’t try this at home.
Lime aka calcium hydroxide is still used today as a depilatory. Depending on the dilution, calcium hydroxide can cause serious alkali burns and poisoning. But it’s generally safe, if you’ve got a Talmudic sense of the dose-response curve.
Conversely, some hazards are not safe at any level.
So, does "the dose make the poison?" Some scientists argue: “Yet new evidence emerging from modern scientific research that combines toxicology, developmental biology, endocrinology and biochemistry is demonstrating that this assumption is wrong, at least in its simplest and most-widely used form. And the implications for this new realization are profound, because it means that the safety standards used to protect public health are built upon false assumptions and likely to be inadequate.” As a result, the 'safe' or 'acceptable' thresholds keep getting lowered for lead, mercury, etc. (You could chalk up phthalates to this list, were they better regulated!)