Introduction

This article provides information on searching the Internet using Multi-search engines, rather than individual search engines, discusses the theory of how these things work, their advantages and disadvantages and finally presents the reader with a small tabular comparison of some major Multi-search engines.

Why use a Multi-search engine?

At the time of updating (February 2003) there are at least 150,000 different search engines (you can view the ones which I use most by visiting my search engine page) available which cover both general and specific subjects, or which search specific elements of the Internet such as Web pages or Usenet.
While some of these are particularly effective and sophisticated none of them are entirely comprehensive. They may only use a small database from which to create your set of results (Yahoo for example only indexes a very small proportion of the 3 billion pages indexed by Google), or they may not be updated particuarly quickly (Alltheweb is updated every fortnight or so, while Google is updated monthly). Their spider programs may not be very fast, which means that their currency might not be a real reflection of the state of play on the Internet.
Consequently, even if you have a favourite search engine, or even several of them, to ensure anything like a comprehensive search you may need to use several of them before you are satisfied that you have found everything you require on a particular topic. A Multi-search engine may save you the trouble of going to a variety of different sites in order to run your search, or it may suggest a search engine which you had not considered, or perhaps did not even know about!

How do Multi-search engines work?

From the explorations that I have undertaken, there appear to be 3 different approaches which are in operation at the moment:

• A straightforward list of different search engines

• Searches which take place one after another.

• Searches which take place simultaneously

Each of these has their own advantages and disadvantages, so lets examine them in a little more detail.

• A straightforward list of different search engines
  • These work by simply copying the appropriate URL for the cgi script onto the web page. This is not particularly difficult to do and the user then simply inputs the appropriate search term(s) into the dialogue box and submits the search. This is then run by the search engine in particular and the user is presented with a list of results in exactly the same way that they would if they had gone to visit that particular site directly.
  • The advantage of this approach is simply that you can reduce the amount of time you spend going from one site to another in order to complete your search.
  • It might also suggest other search engines for you which you had not considered using before. It is almost impossible to keep up with all of the different search engines which are available, and if someone is happy to do this on your behalf, it makes sense to take advantage of it.
  • However, the disadvantage of this approach is that, strictly speaking, these sites are (in my opinion) misleading the user. They are not offering a multi-search engine, but have simply collated the work of others onto a new home page. Thats not to say there is anything intrinsically wrong with this, and the Webmaster will have had to have done a reasonable amount of work to set the page up, but its really nothing more than a slightly more sophisticated list of links.
  • These are by far the most common sites offering a multi-search facility and examples of sites which take this approach are:
    • Find-It at http://www.itools.com/search/

• Searches which take place one after another.
  • This is much closer to the concept of a multi-search engine. A site of this nature will usually have a single entry line where you input the search just as you would with a single search engine interface. You may then have the opportunity of deciding which search engines you the search to run under (usually from a list given in a check-box type situation) and the multi-search engine then transmits the search simultaneously to all of the search engines you have indicated.
  • Once the search has been run, the results will be displayed on the screen for you in a list, commonly separated into the results as provided by the different search engines. However, the main disadvantage of this type of approach is that before the list can be generated on the screen all the different search engines have to have sent their results back to the multi-search engine site. Consequently the speed of the search is dictated by the speed of the slowest search engine.
  • An example of this type of search engine can be found at:
    • Dogpile at http://www.dogpile.com/

• Searches which take place simultaneously
  • This type of search engine is very similiar to the previous approach, the main difference being that searches do not have to wait until each search engine has completed its work - as soon as results are available from one search engine, they are displayed on the screen for you to view. The result is therefore that much faster, and while you are browsing down through the list of hits, others are being added to the page even as you view.
  • An example of the multi-search engines which take this approach are:
    • Ixquick at http://www.ixquick.com/

Characteristics of Multi-search engines.

I have tried to put together a list of the different elements which one might expect to appear on a multi-search engine page. Unfortunately few, if any of the multi-search engines exhibit all of these elements, and indeed some will have very few of them.

• The number of search engines that a Multi-search engine will use
  • The number of search engines which are used varies dramatically - the smallest in the sample that I looked at only referred to half a dozen, while the largest in the sample gave me access to over one thousand search engines, or database front ends. This is no real indicator of quality however; it depends much more on the search engines which are used (and also the variety) rather than the sheer number. I would much prefer to use a Multi-search engine which referenced a small number of what I would consider to be high quality search engines than a much larger number of engines that I did not really know or trust.
  • Nonetheless, I think it is an acceptable criteria to use when evaluating the effectiveness of a Multi-search engine; while more is not necessarily better, less could certainly be considered worse. While I would not normally evaluate success in terms of the number of hits this is one of the reasons why one would use a Multi-search engine, so I feel that it is justified.

• The elements of the Internet which are searched.
  • It seems almost automatic these days to regard the phrase 'Search the Web' as a synonym for 'Search the Internet'. Of course, while that is understandable, given the hype and attention surrounding the World Wide Web, it behoves us to remember that there are a number of other aspects of the Internet which deserve consideration as well, such as Usenet newsgroups, or individuals email addresses and so on. Multi-search engines are at the mercy of the search engines they choose to reference, but given that there are a good number of these which concentrate on specific aspects such as those just mentioned there is no reason why they should not be made available for searching as well.

• Any words, all words, phrase searching.
  • Again, there is little that the Multi-search engine can do directly about this since they are unable to affect the internal workings of individual search engines. However, it is an option that should be offered to the end user; if one search engine can search on a phrase out of the list available it seems short sighted not to offer this. Other engines on the list will simply ignore the phrase aspect and search on the words using an implied OR. If this is not given as an option though, it reduces the effectiveness of those search engines which can undertake phrase searching. It seems to be an obvious point, but is one which was overlooked by some of the Multi-search engines which I looked at

• Boolean operators, truncation and proximity searching.
  • The very same comments can be made here as just given above. A Multi-search engine should provide and reflect the variety of approaches made available by the engines it references, but all too often this was not the case.

• Focussing a search.
  • There are of course many occasions when the user will not wish to do a global search, but will want to focus on one aspect specifically, such as searching in a specific domain (such as .com) or a geographical area (such as Europe). Yet again some search engines allow the user to focus the search this way, but this is not always reflected in the interface offered by the Multi-search engine.

• Choice of subject area.
  • This approach is very familiar to anyone who has ever used the Index approach to search engines, by taking a broad subject area and choosing various subheadings until the specific subject is reached. It is well known to all of us in the information profession that much of the time we will not want everything on a subject, but will wish to focus on the medical or legal aspects for example. Some Multi-search engines did offer the ability to choose a specific subject focus or indeed provide a subset of search engines which cover a particular subject area, and the The Big Hub deserves to be singled out here for being quite superb in this approach.

• Time taken and hits returned.
  • Two important elements here which both approach the problem of the amount of time that it takes to run some of these searches. A major disadvantage of using a Multi-search engine is that you are very much left in the hands of the engines referenced. If they decide to take a long time to return a result, or they are particularly comprehensive you may be sat twiddling your thumbs while they work. By limiting the search either by number of hits or by getting the search to end after a particular period of time you are able to exert at least a little control over the whole process. The danger here of course is that you are not going to get the same level of comprehensiveness that you might get otherwise, but at least you are being given the choice

• Display
  • Many search engines will allow you one of three display modes; brief, normal or verbose. Your choice is likely going to depend on what you actually want from the search, and the variety can be quite helpful in some circumstances.

• Collate results.
  • When using a single search engine it is not uncommon to find the same site turning up as a hit several times - de-duping does not seem to be that much of a priority for a lot of developers! This problem is exacerbated when you do a multi-search; it can be annoying to retrieve what appears to be a reasonable number of hits, only to find that there are a great many duplicates in the list. In my opinion, one of the keys strengths of a good Multi-search engine should be that it is able to collate the results, de-dupe and then display. Unfortunately however, it appeared to be very uncommon to find one which did.

• Help screens/FAQ's
  • It is slightly distressing to see so many Internet retrieval engines attempt to give users the impression that searching is a very simple process, when of course we know that its rather more complex than that. All too often I would log onto a Multi-search engine page to find no instructions, no hints on how to create a better or more effective search, and no way of identifying how many search engines were used. Indeed, the lack of such information was quite astonishing. Perhaps its just me, but if I'd taken a lot of time to create a Multi-search interface, I'd want everyone to know who I was, how I had done it and why it was effective. Its possible that the producers of these facilities have rather smaller egos than I, but I tend to think that it is related rather more to laziness than anything else.

Comparison of the search engines.

I am not going to pretend that this comparison is the result of highly scientific work; I have also not checked my results with the producers of the different engines, though I have tried to ensure accuracy. Due to the fast changing nature of the Internet however, you are advised to check the information for yourself, since it will doubtless become out of date quickly.

I chose Meta-search engines which offered as much variety as possible, and gave me examples of list, consecutive and simultaneous searches, and obtained the data for all of them on 25th February 2003. According to Yahoo there are at present a total of over 100 different sites offering engines of this type, so for a full list I would point you to them.

My conclusions, which are my own personal impressions, nothing more, are as follows.

I see no value whatsoever in the list approach. These are not examples of what I would regard as 'true' multi-search engines; anyone could put a list of these together and claim that they had created a multi-search engine when in actual fact all that they have done is be slightly creative with cut and paste facilities. The possible exception is Metasearch, which does automatically put your desired search terms into the appropriate places on the cut and paste search engines they have referenced. This approach also is unable to properly provide boolean operators etc, since the page does not interact with the search engines themselves, simply providing this front end cut and paste job. Worse, they are usually unable to offer much by the way of help screens, since this is dependant on the search engines themselves.

Consecutive multi-search engines were however much better. They did make attempts to integrate their page into the search engines, and so are generally better at providing a wider range of functions, although I still found that help screens and guides to searching were very limited. The major disadvantage of this approach is that it can take considerable time for the search to be completed, and the weak link is always going to be the slowest engine that they reference. However, they do seem to work reasonably well, and are certainly worth experimenting with.

Simultaneous search engines seem to be few and far between, but they are without a doubt the most effective. Superseek uses the Frames approach to overcome the problem of obtaining and displaying results on the screen, but this approach does mean that you have to have a frames compatible browser available, which not everyone will have. The search results screen also looks as though its come straight out of an aeroplane cockpit and is a little daunting when you first view it. However, it does not take long to get used to. They do also have a non-frames approach, but I did not try this out. Worth experimenting with.

My two favourites however are the Internet Sleuth and Savvy Search. Both were helpful, fast and efficient. I would be quite happy to use either or both of these to run a multi-search, and I would recommend them.

I would welcome comments, additions, updates and so on; please feel free to email me.

Tabular comparision of Meta-search engines.

Summary

The search engines in this category are almost all equally as good - it's very difficult to point at one and say that it's the best. The only one lagging behind is Dogpile, but the other four are virtually equal on merits. They're certainly a lot more powerful than they used to be. The previous version of this page is archived at mseng00.htm If you want to see the original version of this report it's archived at msengarc.htm

Single search engines.

If you are interested in having a look at a number of different single search engines, please visit my page which offers a comparison of these, and an appraisal of how they are used.

This page last updated on 26th February 2003 ©Phil Bradley.