Robots.txt is a file on a web sites root directory which spiders are supposed to read to determine which parts of a website they may or may not visit.