I wouldn't. From what I just quickly Googled, the danger comes from vermiculite that is contaminated with asbestos. Repeated or prolonged inhalation can give you lung cancer. The pure mineral is inert. I didn't find much about ingestion, but, in my opinion, the small amount of exposure that you describe doesn't warrant medical intervention.