Towels are such great bacteria traps because every time you use a towel, you transfer your natural skin bacteria, and any other germs you’re carrying, onto their surface. Most of these germs won’t have any negative health effects because they’re coming from you. “Our bodies are adapted to being able to live in this environment with all of these microbes around,” says Emily Martin, an assistant professor of epidemiology at the University of Michigan School of Public Health.

But towels offer the perfect environment for bacteria and other microorganisms to grow because they’re often damp, warm and absorbent, and they hang in dark bathrooms.

Because your hands can transfer microbes to your towels from whatever they touch, the bathroom is an especially threatening place for a towel to spend most of its time. There, your hands touch fecal organisms like coliform bacteria—organisms that can indicate the presence of disease-causing bacteria in water—Escherichia coli or even Salmonella, according to Charles Gerba, a microbiologist at the University of Arizona. In one study (which has not yet been published) Gerba says he found that nearly 90% of bathroom towels were contaminated with coliform bacteria and about 14% carried E. coli. His published research has also found these bacteria on kitchen hand towels, and the numbers of E. coli correlated to how frequently towels were washed.

Many people don’t wash their hands thoroughly, he says, so when they dry their hands on a towel, they’re rubbing bacteria into a place it’s likely to grow. “After about two days, if you dry your face on a hand towel, you’re probably getting more E. coli on your face than if you stuck your head in a toilet and flushed it,” Gerba says.

Even outside of the home, towels have indeed been shown to help spread infection. “I worry about towels that remain damp or moist,” says Susan Whittier, director of clinical microbiology at New York-Presbyterian and Columbia University Medical Center. “That’s a breeding ground for MRSA,” an infection caused by antibiotic-resistant Staphylococcus aureus. Some people carry it on their skin or in their nose without getting sick. But research has shown that passing the bacteria along can infect others. Sports teams, the members of which tend to share things and have frequent skin contact, are especially vulnerable. After a 2003 outbreak of MRSA among college football players in Los Angeles, a study found that sharing towels on the field helped facilitate the transmission of the infection.

Even so, it’s unlikely you’ll get sick just from touching a used towel, according to Whittier. The bigger risk comes if you have cuts or abrasions (as many athletes do) or dry skin that can let the microbes into your system, she says.

The best way to protect yourself is to wash kitchen and bathroom towels often. People should wash any bathroom towels after about two days of use, Gerba says, particularly if you have young children in the house. And not just any rinse will do. Bacteria can survive regular detergent, according to Gerba, so use hot water and a product with activated oxygen bleach to thoroughly clean towels.

You may be able to go slightly longer before washing if you’re careful about keeping towels very dry, Whittier says. “As long as it’s drying completely between use, there’s almost no chance of passing bacteria from one person to another,” she says. Still, you’ll want to wash them at least once a week. And if they start to smell, Whittier says you know you’ve let them go too long.

The bottom line: if you stay on top of cleaning both your hands and your towels regularly, experts say it’s very unlikely they’ll make you sick.