rogue crawler

Noun phrase. Any software that retrieves Web documents for unauthorized processing, such that it does not honor robots.txt exclusion standards (or equivalent robots meta tags) by identifying itself with a unique user-agent and respecting all normal directives assigned to that user-agent, or when spoofing other user-agents the crawler does not honor the directives assigned to its own unique user-agent.