Search Engines.

How to get the best out of the Internet.

Right - you've spent some time wandering around the Internet in a pretty aimless fashion, and you've got bored with looking at people's home pages, and you've had enough of looking for popgroups or football teams; its time to start doing some work.
Whats the best place to start? There are a large number of different search engines out there on the Internet, and you need to know what they do, how they do it and which is the best for the purpose you have in mind. There are essentially THREE types of search engine which are available to you.

Free text search engines

Index search engines

Multi-search engines.

Each of these works very differently, and you need to be aware of when its best to use one, and when best to use another.

Free text search engines
- This type of search engine allows you to simply type in any term, phrase or word which takes your fancy. The search engine will then go away and attempt to match your term(s) with sites that it has listed in its database. It will then give you a list of sites which match your criteria. Most of the search engines will, unless you tell them differently, use an implied OR, which means you will get a lot of hits - sometimes many thousands. They will then attempt to relevance rank them for you, which means that you only have to check the first page or so of hits in order to locate sites which are most appropriate. Each of the search engines works slightly differently however, so you'll need to try them out and find which works best for you.
- These engines work best if you know exactly what you're looking for, and have some key words which neatly define what you are looking for. Its up to you to choose the most appropriate terms, and its worth using more terms, rather than less. Rather than search for car also search for cars vehicles automobiles and so on.
- These search engines are of little value if you're not sure of exactly what you want, or simply wish to see what is available on the Internet in the subject area which interests you.

Index search engines.
- These engines will guide you through your search by providing you with a series of headings, and subheadings until you finally reach a list of sites.
- Very useful if you want a broad overview of what is available within a specific subject area, although you are constrained by the classification scheme which is in use, so you may have to go up and down the subject trees a few times until you find exactly what it is that you're after.

Multi-search engines.
- These search engines are bolted on front ends to a number of other Internet search engines. They have advantages and disadvantages which are discussed later, but if you can't wait, you can visit the page of information on multi-search engines now.

How do they work?

All search engines have what are called 'robots' or 'spiders' which spend their time going from link to link across the Internet. When they find a new site, or an updated site, they will copy some information about the site back to their home database. It is this database which is interrogated when you run a search. People can register their web pages with search engines, which means that they usually get listed much more quickly than waiting for the spiders to come across them.

Are they free?

Generally speaking, yes they are. They make their money in one of two ways; either to promote software/hardware such as with Altavista, or they will sell advertising space on their systems, which you'll get to see when you go to the search page. One or two search engines do charge for their services, but personally I see little value in paying money to search the Internet when it can be done for free.

Things to remember when searching using these engines.

If you are used to online database searching, or using CD-ROM products, you'll find that these engines are an annoying mix of the very skillful and the very primative. Altavista will, for example, search through a very large database in a matter of a few seconds, and while it is reasonably sophisticated does not stand up well in comparison to some of the advanced features offered by CD-ROM publishers.
Don't forget that there is more to searching the Internet than just looking at WWW pages. An increasing number of search engines will allow you to use their facilities to search newsgroups, or perhaps people's email addresses.
The databases which the search engines use may well be out of date; new material may have been added, other material may have been deleted. You will only discover this when you actually click to go to a particular site.
If you don't find what you want from one, try using another.
Use a variety of different engines, appropriate to your needs. If for example you just want information on UK or European resources, it makes more sense to go to a search engine which focuses on that region, rather than use a search engine which is global in approach.
There are a great many search engines, but for a list of those which I find particularly useful, please visit my Search Engines page. (URL http://www.venus.co.uk/philb/engines.htm)

Search engines - a tabular comparison.

	Altavista	InfoSeek	Excite	Lycos	Magellen	Webcrawler	Hotbot	Yahoo	Northern Light
WWW	YES	YES	YES	YES	YES	YES	YES	YES	YES
Usenet	YES	YES	YES	NO	NO	NO	YES	YES	NO
URL	YES	YES	NO	YES	YES	YES	NO	NO	YES
Boolean	YES	NO	YES	YES	YES	YES	YES	YES	YES
Proximity	YES	YES	NO	YES	NO	YES	NO	NO	NO
Wildcards	YES	NO	NO	YES	NO	NO	NO	YES	YES
Inc/Exclude	YES	YES	NO	YES	YES	NO	YES	NO	YES
Implied OR	YES	YES	YES	NO	YES	YES	YES	YES	YES
Search fields	YES	NO	NO	YES	NO	NO	YES	YES	YES
Capitalisation?	YES	YES	NO	NO	NO	NO	YES	NO	NO
Meta tags?	YES	YES	NO	NO	N/A	NO	YES	N/A	NO
	Altavista	InfoSeek	Excite	Lycos	Magellen	WebCrawler	Hotbot	Yahoo	Northern Light
Truncation	YES	NO	NO	YES	YES	NO	YES	YES	YES
Phrase Search	YES	YES	NO	YES	YES	YES	YES	YES	YES
Relevence Rank	YES	YES	YES	YES	YES	YES	YES	NO	YES
Preview Doc. size	YES	YES	NO	YES	NO	NO	NO	NO	NO
Date doc updated	YES	NO	NO	NO	NO	NO	NO	NO	YES
Summary of doc	YES	YES	YES	YES	YES	YES	YES	VARIED	YES
Refine searches?	YES	YES	YES	YES	YES	YES	YES	NO	YES
Group results?	NO	Yes, by site	YES	YES	NO	NO	NO	NO	YES
Frequency of database update	1 day - 1 months	1 day - 2 months	1-3 weeks	2-3 weeks	N/A	Weekly	1 day - 2 weeks	N/A	N/A
Indexed pages	140 million	30 million	55 million	25 million	50 million	2 million	110 million	N/A	67 million
Portal?	NO	YES	YES	NO	NO	NO	NO	YES	NO
Geographic Specific?	Yes, with the domain: qualifier	YES	YES	YES	NO	NO	YES	YES	NO

This chart was current at September 1998. You are encouraged to check the various search engines yourself to ensure the validity of the data. Please email me any updated information to include!

The league table

Having compared the different search engines I thought it might be a bit of fun to rank them. This is very unscientific (I mainly added up the Yes's and took account of updating and the size of the database), but I think its reasonably accurate and reflects my own experiences. So, for what its worth, here's the league table, with points assigned to give you an indication of how they match up.

Altavista	21
Infoseek	17
Hotbot	16
Northern Light	15
Lycos	15
Yahoo	11
Excite	10
Magellan	10
WebCrawler	9

Relevence Ranking.

How does this work?
Search engines will take a number of different things into account when assigning the relevence of a particular web page in conjunction with your search criteria. These are as follows:

How many words appear?
- Most search engines will use an implied OR statement. If any of the terms you have chosen appear in a web page, that is considered to be a hit. Most search engines will increase the relevence if all of the terms you chose appear within the page, and rank those higher.

Any words in the title
- If one or more of the terms you chose appear in the title of the page (note that this is not the same as the URL) these will be given a higher ranking than a web page in which the term simply appears in the text.

Keyword or Description
- Some search engines will take into account the META-TAG element, although others will ignore it.

The number of times the term appears on the page
- If a term appears 5 times that may be considered to be more relevent than a page where the term only appears 4 times. However, since some web authors have taken to including a paragraph which simply repeats certain terms very often (100 times is not uncommon) some search engines will deliberately ignore this element.

Stop words
- Some of the search engines will look for all the words that you have asked for, while others may well ignore some common words.

Spam blockers
- Some search engines will ignore pages which attempt to confuse or fool the engine into believing that a page is more relevant than it actually is, by including strings of words repeated literally hundreds of times. If a robot recognises this, it may ignore that page entirely or downgrade the results.

Multi-search engines.

Now you've had a chance to look at some single search engines, you might want to have a look at some multi-search engines by going to http://www.philb.com/msengine.htm

Back to Phil Bradley's home page.