# check for bad robots User-agent: * Disallow: /email-addresses/ # index the site by all means, but try not to do any of the big media files # unless you wish to cache them and serve them up locally User-agent: * Disallow: .wav Disallow: .mp3 Disallow: .flac Disallow: .ogg Disallow: .m4a Disallow: .m4p Disallow: .mov Disallow: .mpg Disallow: .mp2 Disallow: .mpeg # you ignore the above directive: go directly to jail, do not collect 200 # or maybe it'll be ok now it's a series of lines ? #User-agent: msnbot #Disallow: /