I agree that if we should avoid all toxic plants, then there would not be much interesting left for the garden, or in the nature, but accidents do occur. I remember a case in Norway when a horse died from being poisoned by eating oleander! A farmer was tired of her potted oleander and dumpted outside, where the horse found it (it had never been taught to stay away from unknown plants).