User-agent: googlebot Disallow:/aspnet_client/ Disallow:/data/ Disallow:/Templates/ Disallow:/Tools/ Disallow:/Webcards/ User-agent: webcrawler Disallow:/aspnet_client/ Disallow:/data/ Disallow:/Templates/ Disallow:/Tools/ Disallow:/Webcards/ User-agent: msnbot Disallow:/aspnet_client/ Disallow:/data/ Disallow:/Templates/ Disallow:/Tools/ Disallow:/Webcards/ User-agent: askjeeves Disallow:/aspnet_client/ Disallow:/data/ Disallow:/Templates/ Disallow:/Tools/ Disallow:/Webcards/ User-agent: * Disallow:/ #Lines, starting with '#', specify a comment # #Relative URL's on a server start with '/', this means the entire site is closed off. # #Lines that start with /folder_name indicates that all other robots should not visit URLs starting with /folder_name #Note the '*'is a special token, meaning "any other User-agent"; you cannot use wildcard patterns or regular expressions in either User-agent or Disallow lines. # #Two common errors: # #Wildcards are _not_ supported: instead of 'Disallow: /tmp/*' just say 'Disallow: /tmp/'. #You shouldn't put more than one path on a Disallow line (this may change in a future version of the spec)