Does anyone know what robots.txt is? My site was analazied and they said I need robots.txt in the HTML code. Can anyone help or advise? Thanks! B.R.
www.videodocprod.com

That rather surprised me. Because while there are reasons you could want to use a robots.txt file as a means to an end, I couldn't understand why your page would need one.

However, having viewed your site, I suspect it is "needed" so that the search engines can discover you actually have more pages.

Instead of a robots.txt, I suggest you scrap your present menu. Replace it with one that works in a search engine spider and with all browsers. I only found the thing when I checked your site in a second browser.



While you are at it, delete the stupid barts1000 script that is trying to sabotage normal browser functions. Immature 12 years old sometimes find i funny to have some popup jumping up like a raised middle finger when someone tries to bookmark, reload, open in new window, or whatever rightclick function they are using. You seem to be trying to run a business. Insulting potential customers is a very bad idea.

Oh, forgot:
Try Google on robots.txt

robots.txt is a text file that contains instructions on which pages of your site the search engines should list/ignore. It effectively does the same thing as the robots meta tag but all in one place for the whole site and giving the opportunity to give different instructions to different search engines.

Not having a robots.txt file with valid entries in it tells MSN search not to bother with listing your site.

Certainly sounds useful. It recently occurred to me that there actually are pages on my site that might be better ignored, so I ought to look into it. But surely the default behaviour of MSN like most other search engines is that if you lack one, they follow the links? The lack of robots.txt certainly hasn't prevented me from getting the no. 1 spot in MSN for the search terms I'd expect that from.

All the major search engines happily spidered and indexed my site without having a robots.txt file. When I restructured part of the site and replaced old pages with placeholders that forwarded to the new structure I discovered that some search engines will not recognise "noindex" when put into the header.

Well I have seen several SEO articles that stated that MSN doesn't index sites that doen't have a robots.txt. As they had not indexed my site at that point I added a robots.txt file and was indexed soon afterwards. The articles that I read may have been wrong and it may just have been a coincidence that my site got indexed soon after adding that file.

Mine doesn't and is indexed by MSN. Actually they make a better job of it than Yahoo.