The process of sending search engine spiders to a tailored page, yet directing your visitors to what you want them to see. This is done using server side includes (or other dynamic content techniques). SSI, for example, can be used to deliver different content to the client depending on the value of HTTP_USER_AGENT. Most normal browser software packages have a user agent string which starts with "Mozilla" (coined from Mosaic and Godzilla). Most search engine spiders have specific agent names, such as "Gulliver", "Infoseek sidewinder", "Lycos spider" and "Scooter".
By switching on the value of HTTP_USER_AGENT (a process known as agent detection), different pages can be presented at the same URL, so that normal visitors will never see the page submitted to search engines (and vice versa).
In practise this is somewhat simplistic. Some search engines pretend to be "plain mozilla" browsers to prevent use of agent name delivery. Effective use of agent name delivery can be very difficult, and may not even work.
How do you spot agent name delivery at work? This is quite difficult, as the owners of web pages using agent name delivery can control what you see! You may be able to guess that a page is using this technique if it appears to be indexed incorrectly or the title or description don't match the page you see, but this could also have been achieved by switching pages after the relevant search engine has indexed it. If you really want to see the search engines' tailored version of a page, write a program (e.g. a Perl script) to retrieve the URL with HTTP_USER_AGENT set to each of the strings used by the search engine spiders. If agent name delivery is in use, one or more of the retrieved pages will be different to the others!
See also hidden text and IP delivery.
From: Search Engines