I was looking for web spider script for use on my friend’s website, i found a great Php Spider and Search Engine script named Sphider.
It is a lightweight web spider and search engine written in PHP, using MySQL as its back end database. It is suitable for adding search functionality to small or medium sites (up to around 100,000 pages). It also works great as a tool for site analysis - finding broken links, gathering statistics about the site etc.
Features:-
-
Full text indexing.
-
Can index both static and dynamic pages.
-
Finds links in <a href=…>, <frame …>, <area …> and <meta …> tags, and can also follow links given in javascript as strings via window.location and window.open.
-
Respects robots.txt protocol.
-
Follows server side redirections.
-
Allows spidering to be limited by depth (ie maximum number of clicks from the starting page), by (sub)domain or by directory.
-
Supports indexing of pdf and doc files (using external binaries for file conversion).
-
Allows resuming paused spidering.
-
Possbility to exclude common words from being indexed.
-
Sophisticated administrator interface
Visit this page to see the list of some websites using Sphider.
You can download Sphider from here.
Technorati Tags: script, php, sphider, search, index
If you enjoyed this post, make sure you subscribe to my RSS feed!
Related Posts: