Last update : 2001-05-06
Note on this translation : Spelling corrections by Brien Louque, thanks to him.
Prerequisites
PhpDig requires a Web server (Apache is my preference) with Php (module or cgi), and a MySql database server.
The following configurations were tested:
PHP 4.0.3pl1 cgi, Apache 1.3.14, MySql 3.23.28, Windows 2000 ;
PHP 4.0.5rc1 module, Apache 1.3.14, MySql 3.23.28, Windows 2000 ;
PHP 3.0.18 module, Apache 1.3.14, MySql 3.22.32, Linux kernel 2.4.
In the php.ini file, safe_mode must be OFF on the server which does the indexing. This permits PHPDig to set the time limit in the spidering/indexing scripts. On the other hand, this parameter does not affect the server when it returns results for the search queries.
Install PhpDig
1. Unzip the archive in a directory and configure Apache to serve it.
2. Create a MySql database or choose an existing one in which to insert the 5 PhpDig tables.
3. Run the init_db.sql script to create the tables. Use the following shell commands.# mysql > create database phpdig ; (or any other name) > \q # mysql phpdig < init_db.sql
You have to edit two files:
config.php3
This file configures the engine general options. All the variables to be edited are commented.
connect.php3
This file configures the connection parameters to MySql. If your web hosting allows you directs connections to MySql, you can run the indexing scripts on a local machine rather than on the server.
Search form
Extremely simple, with a few options. The text field takes one or more words separated by spaces and is case insensitive.
You can choose to perform the search on beginning, whole or parts of words. This option applies to all the search terms.
To exclude a search key, you just have to put a "-" character before.
Lastly, a select list allows the user to choose listing the results in groups of 10, 30 or 100.
Results browsing
The results pages show a list of documents matching the search keys, ranked by "relevance". The more the search keys occurs in a document, the higher the page appears in the results.
A navigation bar at the bottom of the page can be used to go to the next group of results.
Below the title of the found pages are two links making it possible to limit the results to a given site or a sub-folder within this site.
Put the form anywhere
Just call the phpdig_form() function (in the function_phpdig_form.php3 file. The variable $result_page contains the URI of the page which performs the search and displays results.
Running site indexing (or just indexing a sub-folder) is done by the admin.php3 script. This admin page provides maintenance functions for the engine too.
Site spidering
Just fill in the url field, select the maximum search depth in levels and click on the "Dig This !" button.
For each level, PhpDig displays the number of new links it has found. If no new link is found, PhpDig stops its browsing and displays the list of the documents.
During spidering, the MySql "tempspider" table is filled by the links. At this time, only the existence of html content in the document is tested.
Exclude paths from indexing
Once the page displays the documents discovered, the user has the possibility of excluding sub-folders from indexing.
Indexing documents found
This process is time-consuming. For my configuration, the browsing of Apache server documentation takes 01:19 for 101 URIs distributed out over 4 levels. The indexing of these pages takes 08:27. For each new indexed page, PhpDig returns to the navigator the result of the indexing (time, word count).
Many processes are carried out for each page indexed to avoid too big a dictionary :
- Extraction of title, description and keyword metatags ;
- Extraction of images Alt attributes ;
- Htmlentities coded string are translated to ascii strings ;
- HTML code is deleted ;
- The first 600 characters of the document are stored as a summary ;
- A hash string is computed to detect duplicates of the same document with different filenames (i.e. "/" and "index.html") ;
- All text is lowercased, without accents or signs ;
- Distinct words are extracted with occurrence ;
- The "engine" table is filled with document/word/occurrence records;
- The "spider" table is filled with document extra info.
From the admin page, you can reach the update interface by choosing a site and clicking on the "update form" button.
Update a branch
On the left side of the screen is the client-side folder structure of the site.
The red cross deletes all the selected branch, including sub-folders, from the engine.
The green check mark reindexes the selected branch documents if they were indexed for more than 10 days (default value).
The blue arrow displays the "folder" content, in order to reindex the documents individually.
Update one document
The document's listing of a folder is on the right side of the screen.
The red cross deletes the document from the engine.
The green check mark forces reindexing of this document.
The blue arrow runs a new spidering starting from this page. The depth is 3 levels in this case (default value).
Cleaning scripts
3 scripts are used to delete useless data in the PhpDig database. The links are in the admin page.
Clean index deletes index records not linked to any page. Useful if manual deletes are done in the database.
Clean dictionary deletes keywords which are not used by the index. Useful for reducing the size of the dictionary, particularly when a large site contains a great deal of technical words.
Clean common words must be run when new common words are added in the common_words.txt file. It deletes all reference to those common words.
Auto update
You must have an executable cgi of Php to use auto update.
Put a command in crontab like :php -q [absolute_path_to_script]auto_indexing_cgi.php3 [site_url] [path][site_url] is the url to update, [path] limits update to this path and sub-path.
auto_indexing_cgi.php3 must be in the same directory as the other scripts.