Both are dangerous, really.  If mold has become a problem in your house by growing in a wet spot, you can contract the allergens by inhaling spores or by touching them.

Touching and inhaling the actual mold growth can be harmful, causing allergic reactions, but the air around the mold can be just as dangerous because the spores are airborne and can be everywhere, producing the same reactions.