The SEARCpHp searchengine class (class.searchengine.php) is provided
under the same license as PHP. The PHP license differs from other 
free software licenses.  Make sure you have read it!

You are free to modify and use this class in any way you wish,
but there are NO IMPLIED OR EXPRESSED WARRANTIES.

This class was written from scratch by 
	Hans Anderson
	me@hansanderson.com
	http://www.hansanderson.com/php/

	 ----------------------------------------
	| 1.  Installing/Setting Up		 |
	| 2.  Use / Documentation		 |
	| 3.  Support			         |
	| 4.  How SEARCpHp Works		 |
	| 5.  Changes				 |
	 ----------------------------------------

	1. INSTALLING / SETTING UP
	 NOTE: you will need to add the path to the 
	       search/ directory to your php_include_path
	       or else do a lot of hacking to have all the
	       files found!

	 You should have the following files:

		* chmod.sh		 - shell script to help
					   set up some permissions

		* stopwords.php		 - list of words you don't
				 	   want to index or search

		* class.searchengine.php - main class

		* config.php 		 - configuration directive.
					   EDIT THIS! You will need to
					   edit the paths of directorys
					   and the unix program "sort"
					   (use 'which sort' to find path)

		* README 		 - this file

		* class.timer.php 	 - class to time searches
					   written by David Sklar
					   <sklar@student.net>

		* filesystem.example	 - an example of how to index files
					   on your hard drive.

		* search.db		 - example db for indexing, testing.
					   There are 27 jokes, some funny, some
					   not.  First create the database search,
					   then import this file. You might need 
					   root privileges.

						prompt$> mysqladmin create search
						prompt$> mysql -pyourpasswd search < search.db

		* index.php		 - sample indexing script using provided
					   jokes database.  You need to be able
					   to use PHP from the command line to 
					   run it (or modify it to work with your
					   web server).  Edit about line 11 to
					   your database username and password.	

		* search.php		 - test search script from command line
					   You need to have the cgi version of
					   php to run it.  See the www/ directory
					   for samples of searching via a web
					   page.

		And these directories
		 data/ 			 - where the data lives
		 tmp/			 - used for indexing/merging 
					   (temporary files)
		 www/			 - sample search form and 
					   results page using provided
					   jokes database.  Copy these into
					   your web server htdocs in a place
					   you can test them.

	2. USE / DOCUMENTATION

		a. User Functions
			i.   search()
			        Search Engine Math
				  - normal searching can just be keywords,
				    ie 'cool php scripts' (returns any record
				    with one of those words)
				  - for words you *must* have, add a '+',
				    ie 'cool +php scripts' (returns any record 
				    with php.  Records with cool and scripts will
				    be returned first.  If 'php' is not found in 
				    any records, no results will be returned.)
				  - for words you *can't* have, add a '-',
				    ie 'cool +php -scripts' (returns any record
				    with php, but not scripts.  If a 'good' record
				    also has cool, it's returned first.  If there
				    are no records *with* 'php' but *without* 'scripts',
				    nothing will be returned, even if 'cool' is found.)

			ii.  index()
				 Call this function to do the indexing.
				 It takes two arguments, Title and Words,
				 $idx->index("$Title","$Words").  Title is what
				 will be returned on matches, words are what
				 are indexed.  You'll probably want to pass
				 the title as a fully completed href link.

				 :wf: (Weigh Factor) is the syntax for weighing words higher.
				 For instance, if you have two fields, Field1
				 and Field2, and you want Field1 words to be weighed
				 higher, do something like this:
				 $idx->index("$Title",":wf3: $Field1 :wf1: $Field2");
				 Default Weigh Factor is set in config.php

			iii. merge()
				 Call this after you have finished
				 indexing (index()).  This sorts and
				 optimizes the index.  If you don't
				 do this, it won't work.  See the
				 sample index.php script.

			iv.  purge_index()
				 Deletes the files used to populate
				 the index.  Use before re-indexing,
				 IF YOU WANT A NEW INDEX.  If you only
				 want to add or append to the index,
				 don't do this.  Don't do this until you
				 re-index, or else you won't be able to
				 search it.

			v.   stopwords.php
				 Stopwords (words you don't want to index, 
				 even if they pass the other requirements
				 (length, especially)).  It's a hash, so you
				 need to define it as a hash.  There are a
				 few examples in the file stopwords.php
		
		b. Example Usage
			$idx = new Idx("jokes");

			i.   prototype: array search(str searchterms,int start, int numresults)
			      usage: $idx->search("bill -gates +heaven",41,20);

			      explanation: 
				will display all matches to search 'bill -gates +heaven'
				starting at record 41, showing 20 records.  This
				effectively shows records 41-60.

			ii.  prototype: bool index(str title, string words)
			      usage: $idx->index("Title","Words to Index :wf3: More Words to Index");

			      explanation:
				title is what gets written to a file for later display
				in the results.  I usually make the title into a link
				pointing at a record, so the user can click on it
				to view the result.  Like this:
				$idx->index("<a href=\"/db.php?ID=4\"","words words words");

				Of course, the ID=4 would probably be something like ID=$ID
				the title parameter is *not* put in the index, it's for
				record display only.  All words you want indexed need
				to be in the second argument.

			iii. prototype: bool merge(void)
			      usage: $idx->merge();

			      explanation:
				Merges the files in the tmp/ directory into a searchable index

			iv.  prototype: bool purge_index(void)
			      usage: $idx->purge_index();

			      explanation:
				Basically, this just deletes the 4 index files from the
				data/ directory.

			v.   stopwords 
				See the file stopwords.php.

	3. SUPPORT

		You can email me whenever you want for advice or
		help.  This doesn't mean I'll be getting back to
		you at that very instant, but I'll do my best.
		You may also contact the normal PHP user mailing
		list for help.  There is nothing contained in the
		class that isn't standard PHP.


	4. HOW SEARCPHP WORKS

		This isn't an easy thing to explain, but I want
		to give you an idea of what goes on here, so you
		can come up with ideas to make it more efficient.
		Bear with me, please.

		Indexing:

		A title and string of indexable words are given
		to the index function.  index() first writes
		the Title to the titles.data file and receives
		the byte number of the start of the Title, and
		the length of the Title.  These functions also
		check to make sure the words are indexable, that
		each is not a stop word or too long or too short,
		etc.  If a word it passes, it's put into an
		associative array.  This array fills up with words from
		the *current* iteration of the function.  That is,
		all the words for *the current record*, if indexable,
		are placed there.  If there is more than one occurence
		of a word, the associative array increments by one,
		noting how times each word is found.  (when searching,
		we'll sort based on how many times each word is found
		in a record).

		When we've run out of words for that record, we
		write the array to the harddrive, freeing up memory.
		Each word is put into a tmp/ file based on it's first
		character (a, b, 3, 4, etc).  They are not sorted.  Each
		word written to a tmp/ file is followed by the byte number
		of where that record's Title was put in the titles.data file.
		An entry in tmp/a might look like this:

		anderson 345:50:2

		That means the word 'anderson' was in the record whose title
		is found in titles.data starting at byte 345 and ending at
		byte 395 (345+50).  There were two (2) occurrences of 
		'anderson' in that record.
		
		After all the items in that array are written to the harddrive,
		the array is destroyed and index() returns TRUE.  If there was
		an error, it would return FALSE.  Errors are usually unwritable
		directories.

		That was just one record.  If you are pulling information from
		a database, you are probably looping over the result set and 
		call index() many times, each resulting in a Title being written
		to the titles.data file, and the words being stored in the tmp/
		directory.

		Merging:

		The merging system simply opens up each file in the tmp/ directory
		via PHP's file() function, putting each line into an array.  Then
		the array is sorted alphabetically and we build a huge associative
		array, with each key being a word and each value being an array
		of the offsets for the title file (the 345:50:2 stuff).  Then,
		for each word, we write a line to the title_offsets.data file.
		That line is a serialize()-ed version of the array of title_offsets.
		My Brain Hurts!  The function that writes this line to the title_offests
		file tells us the byte start and length (like the titles 345:50 thing),
		and we write the word and that byte start/length to the index.data
		file. The index file contains a space, the word, another space,
		and the offsets for where we wrote the title_ offsets. Confusing!
		Wait until the Search explanation, which will help you understand
		this better. This can take some memory, depending on the size of each
		file in tmp/, which is mostly due to how many *different* words
		you indexed.

		When one tmp/ file is done, say 'm', store the byte start and
		byte ends in an array.

		When the merging is done, we serialize() the list of byte
		starts and byte ends and write them to word_offsets.data.

		Searching:

		When you pass words to the search() function, it splits on the
		space in the word, makes sure it's searchable (basically, indexable,
		because if it's not indexable, we aren't going to find it).  Then
		we decide if it's a must('+') word, not('-') word or can word.  We
		pull the first letter of the word (if anderson, then 'a'), open
		the index starting at where the 'a' words start (based on the 
		word_offsets.data file), reads in all the words starting with
		'a' (instead of the whole index), and looks for the word.  If
		it finds it, it returns the line with the word on it.  That 
		line is the byte start and length for the title_offsets.data file.
		So we access that, getting the serialize()ed array of bytes/lengths
		for the titles.data file.  We unserialize() this and do some sorting
		and checking to make sure we should return that record (based on the
		search engine math (+,-) the user used), removing duplicates, etc.

		All the results are placed in an associative array and number of total
		results and the actual results (an array of titles) starting at 
		$start and going for $numresults are returned to the calling
		script (sample: search.php).

		The calling script probably would each() through the array, displaying
		the results.

		Whew.  Told you it was confusing.  I'm a poor explainer, too.  If
		you have any questions, let me know.  If you have any suggestions,
		I could go for that, too.

	5. CHANGES

		I don't do a very good job of tracking changes.
		You can probably compare me more to an absent-
		minded professor than a programmer.  Hence,
		this change log isn't totally accurate.  Sorry.

	       12/04/00:
			At line 1026 added "if(!$w) continue"
			to fix problem that added blank records
			to the index. 

	       10/16/00:
			At line 783, added more to split()

	       10/10/00:
			Fixed medium sized bug in search().  On
			index, it would check word against 
			is_indexable(), but wouldn't upon search.

		9/12/00:
			Fixed a bad bug in in_common(). Can't
			believe it took this long to find it.  If
			you are using a version downloaded before
			this date, please upgrade.

		8/24/00:
			Added support for forced phrase matching.
			Now you can force phrases to match, but not
			words in those phrases.  This was a client
			request, but probably isn't very useful
			for most.  Check out the function factor()
			in class.searchengine.php for more information.

              c. 5/1/00:
			Added function plural_stem().  It is used by
			default when searching and indexing.  If you
			want to use plural_stem(), simply copy new
			class.searchengine.php over the old one and
			reindex.  If you don't want to use it, comment
			out the two calls to it (search for $this->plural_stem)
			plural_stem() removes all plurals for words 
			while indexing and stores the word sans plurality.
			Upon searching, it also removes plurality, allowing
			plural words to match non-plurals.  Thus horse and
			horses match, as does guesses and guess and pansy
			and pansies.

	        c. 1999:
			Began writing SEARCpHp
