UK Search Engines

Summary

This article provides an overview of existing UK search engines, together with a comparison chart. It was written to provide searchers with a little more information about which search engine(s) to consider using when they have a UK specific query. I have also included my own personal view of which are the best ones to use, and why.

Why use a UK specific search engine?

The simple answer is that using a global search engine such as Altavista even if you choose very specific search terms you will end up with literally thousands of matches. It is of course possible to limit this with some search engines, but there are times when a smaller, geographically specific engine may make more sense. To provide you with a quick demonstration of this, I ran two searches on Altavista, the first time to just give me a total result, the second time to limit the results to UK domains. I chose a phrase "Phil Bradley", partly because I've a fairly good idea when that name occurs on the web (can't imagine why!), and a single term "Everton" since it is a reasonably unusual term and is very UK based, being as it is the greatest football team in the country; well, I like to think so anyway! For the sake of consistency I also ran searches on the UK engines that I looked at to compare the results.
Searching on "Phil Bradley" gave me 1,082 results and restricting to the UK gave me 103 results. Searching "Everton" gave me 89,965 responses, and when I limited to the UK I got 14,717 results. Now, I certainly could have then waded my way through the results, but in this case, as with many others, it is clearly going to take a lot more time than I have available. Using a more specific search engine may well give a tighter and more focussed result.

Which search engine to use?

I am not going to try and pretend that this is a scientific study, since it isn't. I located a number of search engines which were specific to the UK, but which were also general in nature. Consequently, I ignored some very useful engines which were not only geographically specific, but also subject specific, such as the excellent EEVL search engine.
All of the information that I have included was obtained directly from the sites themselves, however in some instances the details about the engines was so poor that I was unable to find all the answers that I wanted. If you find out more information yourself, I'd be delighted if you could email me and let me know. However, a lack of answers is in itself quite useful; the more blanks against a search engine, the more they tell you inadvertently, and I for one am very wary of using a piece of software which does not clearly state its purpose, methodology, background, or in some instances does not even provide a help screen! The first time that I ran this comparison, in January 1998 I included G.O.D. (Global Online Directory) and UKSearch, but unfortunately, both of these seem to have either fallen by the wayside, or were down on the two days that I ran my tests. In their place I have included Lycos UK and the UK Directory. Running this search again in October 2000 I dropped Euroferret, and the UK Cybersearch engine seemed to have vanished. In their place I put in the UK version of Altavista and UK Max.
The search engines I explored are:

• Excite UK

• UKIndex

• UKPlus

• Altavista UK

• UK Max

• Mugomilk

• Yahoo UK

• The UK Directory

• LycosUK

The comparative chart

For ease of reading, and printing out, I have divided the chart into two sections; 3 and 5 search engines each. The grouping is entirely arbitrary, and should not be taken to imply any kind of ranking system; you'll need to read my conclusion to find that out! If you want more information about the individual features themselves I have put an anchor link in each - just click on it and you will be taken to my definition and commentary on them.

This table updated 19/20 October 2000. Figures in brackets May 1999. Please let me know of any errors and I'll correct them!

Explanation of features

Search Newsgroups

Quite obviously, all the search engines searched the WWW, but that is only part of the Internet. Usenet Newsgroups can also turn out to be a useful source of information, and any search engine which also gives you the option of searching them has to be more flexible.

Relevance ranking

This is another reasonably obvious one; there is no use in getting hundreds of hits if the search engine isn't then going to put them into some sort of order for you. Unfortunately, not all do, and even those which do this, its very difficult to obtain much, indeed any, information on exactly how their algorithms work.

Focus search

A searcher is limited by their ability to tighten up on the results that they get, choose new keywords and so on. If they can't do that, or do not know a subject in enough detail the result is going to be a set of results which are often too broad, or which do not fully match the concepts that the searcher is looking for. As a result a number of search engines, such as Altavista are introducing ways of focussing the search further, using a variety of different methods to do this. Once again, its a valuable tool, but is only offered by a small number of engines.

Reviews, Abstracts, Summaries

Reviews are generated by real people, surprisingly enough. Personally, I have little or no use for them; the reviewers generally seem to take the opportunity of indicating how clever, witty or erudite they are. They may however be of some use if you are keen to avoid sites with a controversial subject matter, but since we never know who the reviewers actually are, the review is only ever going to be a general guide.
Abstracts or summaries are usually computer generated, and their effectiveness relies on the web designer being effective, and ensuring that their opening sentence clearly defines the page content, or that the summary they produce when registering their URL with search engines includes appropriate terms and keywords.

Boolean operators

AND, OR, NOT are the usual ones, as I'm sure you're aware. The existence of this option tends to indicate that the developers of the search engine are doing their best to provide us with a more sophisticated and useful engine. A lack of operators drastically limits the searchers ability to find things quickly and easily. Some search engines will use a '+' or '-' to include/exclude words instead of actual terms.

Proximity searching

There is little point in retrieving a site using the terms 'Cattle breeding' and getting 'cattle' on the first line of the page and 'breeding' on the last line. Proximity allows you to further narrow your search, to just retrieving those pages on which the two terms are reasonably close together. Another valuable feature, and one we're used to with online and CD-ROM search engines, but sadly lacking in the majority of engines I looked at here.

Wildcards and Truncation

Two more useful features. None of the engines used wildcards, and only a few used truncation, which was a disappointing result.

Implied OR

Straightforward really.

Search by category

I use this to mean the difference between a free text search engine of the Altavista kind, and the index type used by Yahoo. I have also included the number of different categories that you see on the opening screen in brackets, but not sub categories to be found 'below' the initial opening screen.

Use Metatags?

Metatags are becoming increasingly common, and are a useful way for a web designer to increase the probability of a page being found by a search engine. However, the vast majority of search engines did not explicitly say if they used them or not; only Excite was quite clear in saying that it did not. Although I don't agree with their reasoning, it was good to see that they had directly considered the possibility.

Phrase searching

This is obviously an important feature, since it allows the user to focus searches tightly. It is a shame that more engines do not consider it important enough to include.

Document size/date

Again, useful pieces of information. If I am searching for current information there is no point in looking at a page which was last updated 6 months ago. Similarly, it is useful to know how big a page is going to be before going to it. It is a shame that more engines have not incorporated these features.

Database update

The more frequently a database is updated the less chance you will have of finding out of date links, or pages which have changed since the last database update. Unfortunately, many search engines omit to tell users how often they update; I suspect because they are not as frequent as they should be!

Geographic limitation

Since this is an article about UK databases, I wanted to make sure that it was possible to limit searches to just the UK, but also if they could be expanded out to Ireland, Europe or the rest of the world.

Advanced search

Is there an option to do an advanced search at the site.

Portal

Does the site have a variety of other options, such as free email, chat facilities, personalised pages and so on.

Refine

Can the search be refined once it has been run to narrow down the number of results obtained?

Parentheses

A sophisticated search engine should be able to run a search such as apple and (orange or pear)

Sort results

Can the results returned be sorted into any other order than the given default?

Number of hits returned

This should be expanded out to 'per page'. Most search engines as a default provide you with ten references, and you then have to click to get the next ten and so on. Some do however give you a little more flexibility in formatting the screen according to your own wishes. This is much more sensible than just attempting to show you all of the hits, since it will take a very long time for the page to download.

.com?

Many sites will restrict themselves to just those which are in the .uk domain, but an increasing number of UK based sites (such as mine for example) are .com sites. If a search engine excludes .com sites it is not going to be fully comprehensive.

Conclusions.

The two engines that came head and shoulders above the rest wre AltaVista and Lycos. They both had more features than any of the others and proved to be more flexible and sophisticated in their approach to searching. Interestingly, with the exception of Yahoo! the UK specialist search engines fared worse than those engines that were global to start with, and which added in a UK based component.
However, I also noticed that almost without exception the amount of information it was possible to find out about the search engines was very limited indeed. In most cases, if a 'Help' option was given it was barely adequate. As for getting more indepth information (such as the way they relevance ranked for example) it was almost impossible, and I would have needed to contact the companies directly to obtain this. I don't find that a satisfactory approach, and I think that all of them could do more to assist us in this respect.

A league table

More for my own fun than anything else I worked out my own ranking system based on the information retrieved from my survey and have ranked the sites in order. The number in brackets is the points I gave to each site, so you can see that Excite is way out in front of the others, both in terms of sophistication, but also in terms of the amount of information that it makes available about itself. When there was a draw I took into account the number of hits each engine retrieved on my two terms.

If you want, you can take a look at the previous version of this page, written in May 1999.

If you are reading this article as a printout, the URL is http://www.philb.com/ukengine.htm

This article last updated 20th October 2000. Please send me any updates or amendments. This article is © Phil Bradley, 2000.