# wget is allowed to get nearly everything User-agent: wget Disallow: /cgi-bin/ # Search engines are disallowed User-agent: * Disallow: /cgi-bin/ Disallow: /mp3/ Disallow: /webalizer/