Technology for finding information on the Internet. Types of Search Tools

1. Introduction

2. Search technology

2.1 Search tools

2.2 Search Engines

2.3 Directories

2.4 Link Collections

2.5 Databases of addresses (addresses database)

2.6 Search the Gopher archives (Gopher archives)

2.7 FTP File Search System (FTP Search)

2.8 Usenet News Conferencing Search Engine

2.9 Meta-search systems

2.10 People Search Systems

3. Conclusion

Application . Brief information   about search engines

1. Introduction

Every year, the volume of the Internet is increasing at times, so the probability of finding necessary information   increases dramatically.

The Internet unites millions of computers, many different networks, the number of users increases by 15-80% annually. And, nevertheless, more and more often when accessing the Internet the main problem it turns out not the lack of the required information, butthe opportunity to find her. As a rule, an ordinary person, for various reasons, cannot or does not want to spend more than 15-20 minutes searching for the answer he needs. Therefore, it is especially important to correctly and competently learn, it would seem, a simple thing - where and how to look in order to receive the DESIRED answers.

To find necessary information, you need to find her address. There are specialized search servers   (index robots (search engines), thematic Internet directories, meta-search systems, people search services, etc.).

The main technologies for searching for information on the Internet are described below, general features of search tools are provided, and structures search queries   for the most popular Russian-language and English-language search engines.

2. Search technologies

Web technology World Wide Web (WWW) is considered a special technology for the preparation and placement of documents on the Internet. WWW includes web pages, electronic libraries, catalogs, and even virtual museums! With such an abundance of information, the question arises sharply: “How to navigate in such a huge and large-scale information space?” People come to the rescue in solving this problem search tools.

2.1 Search Tools

Search tools are a special software, the main purpose of which is to provide the most optimal and high-quality information search for Internet users. Search tools are hosted on special web servers, each of which performs a specific function:

1. Analysis of web pages and recording the results of the analysis at one or another level of the search server database.

2. Search for information at the request of the user.

3. Providing a convenient interface for searching information and viewing the search result by the user.

The methods of work used when working with certain search tools are almost the same.

First, consider the following concepts:

1. The interface of the search tool is presented in the form of a page with hyperlinks, a query string (search string) and query activation tools.

2. Search Engine Index- this information basecontaining the result of the analysis of web pages, compiled according to certain rules.

3. A query is a keyword or phrase that the user enters into the search bar. To form various queries, special ("", ~) and mathematical symbols (*, +,?) Are used.

Information retrieval scheme is simple. The user types a key phrase and activates the search, thereby receiving a selection of documents according to the formulated request. This list of documents is arranged according to certain criteria so that at the top of the list are those documents that most closely match the user's request. Each of the search tools uses different criteria for ranking documents, both in the analysis of search results and in the formation of the index (filling the index database of web pages).

Thus, if you specify a query in the search bar for each search tool of the same design, you can get different search results. It is of great importance for the user which documents will appear in the first two or three dozen documents according to the search results and how much these documents meet the user's expectations.

Most search tools offer two search methods - simple search (simple search) and advanced search (advanced search) using a special request form and without it. Consider both types of search using the example of an English-language search engine.

For example, AltaVista is convenient to use for arbitrary queries, “ Something about online degrees in information technology”, While the search tool Yahoo allows you to receive world news, information about exchange rates or weather forecasts.

Mastering the criteria for refining the query and advanced search techniques allows you to increase the efficiency of the search and quickly find the necessary information. First of all, you can increase the search efficiency by using the logical operators (operations) Or, And, Near, Not, mathematical and special characters. Using operators and / or symbols, the user associates keywords in the desired sequence to get the most relevant search result. English request forms are given in table 1.

Table 1

A simple request gives a certain number of links to documents, as the list contains documents containing one of the words entered upon request, or a simple phrase (see table 1). The and operator allows you to specify that all keywords should be included in the content of the document. However, the number of documents may still be large, and their viewing will take enough time. Therefore, in some cases it is much more convenient to use the context operator near, indicating that the words should be located in the document in sufficient proximity. Usingnear greatly reduces the number of documents found. The presence of the symbol "*" in the query string means that the search for a word by its mask will be carried out. For example, we get a list of documents containing words starting with "gov", if we write "gov *" in the query string. It can be the words government, governor, etc.

The most developed service for searching for Russian-language information is provided by the Yandex search server.

In Yandex, you can simply write a phrase in Russian that describes what you want to find, and the system will analyze and process your request, and then try to find everything that relates to a given topic.

You can, using special operators, make a line explaining to the search engine what information your interest should meet. Some of the Yandex query language operators can be viewed here: http://help.yandex.ru/search/?id\u003d481939

The equally popular search engine Rambler maintains statistics on the attendance of links from its own database, supports the same logical operators AND, OR, NOT, metacharacter * (similar to the * character extension in AltaVista), coefficient symbols + and -, to increase or decrease the significance words entered into the query.

Let's look at the most popular technologies for finding information on the Internet.

2.2 Search engines

Web search engines are servers with a huge database of URLs that automatically access WWW pages at all of these addresses, examine the contents of these pages, generate and write keywords from the pages into their database (indexes pages).

Moreover, search engine robots follow the links found on the pages and reindex them. Since almost any WWW page has many links to other pages, with this kind of work the search engine can theoretically crawl all sites on the Internet.

This type of search tools is the most famous and popular among all Internet users. Everyone has the names of well-known web search engines (search engines) – Yandex,

Rambler, Aport.

To use this type of search tool, you need to go to it and type in the search term your keyword.

To make your search more effective, pay attention to the following points in advance:

determine the subject of the request. What exactly do you want to find in the end?

pay attention to language, grammar, the use of various non-letter characters, morphology . It is also important to correctly formulate and enter keywords. Each search engine has its own form of compiling a request - there is only one principle, but the symbols or operators used may vary. Required request forms also vary depending on complexity software   search engines and the services they provide. Anyway, each search engine has a "Help "(" Help "), where all syntax rules, as well as recommendations and tips for searching, are easily explained (screenshot of search engine pages).

use the capabilities of different search engines . If not found on Yandex, try on Google. Use advanced search services.

to exclude documents containing certain terms, use the "-" sign before every such word. For example, if you need information about Shakespeare’s work, with the exception of Hamlet, enter the query in the form: Shakespeare Hamlet.   And in order to include certain links in the search results, use the symbol "+ ": links about the sale of cars specifically - request" sale + car ".

each link in the list of search results contains a snippet - several lines from the found document, among which your keywords are found. Before clicking on the link, evaluate the snippet's compliance with the query subject. Following the link to a specific site, carefully look around the main page. As a rule, the first page is enough to understand whether you came to the address or not. If yes, then conduct further searches for the necessary information on the selected site (in the sections of the site), if not, return to the search results and try the next link.

remember that search engines do not produce independent information (except for explanations about themselves). Search system

– it is only an intermediary between the owner of the information (site) and you. Databases are constantly updated, new addresses are entered in them, but the lag behind the information that actually exists in the world still remains. This is simply because search engines do not work at the speed of light.

The most famous web search engines include Google, Yahoo, Alta Vista, Excite, Hot Bot, Lycos. Among the Russian-speaking can distinguish Yandex, Rambler, Aport.

Search engines   are the largest and most valuable, but far from the only sources of information on the Web.

Information search on the Internet is carried out in two main ways - using directories (they are also called directories) and using search engines.

Directories provide contextual searches for structured browsing, while search enginesas their name implies, they do not provide a context, but they allow you to find specific words or phrases.

Directories can be likened to the table of contents of the book, and search engines to the subject index.

Often, search engines combine both a search engine and directories.

This is clearly seen on the example of the first page of Yandex, where under search bar   placed a list of directories that allow the user to refine the request as you move deeper into each of them.

Due to the fact that the principle of organizing directories is clear to everyone who used the library catalog - and among the readers of such, we dare to suppose the vast majority - we will not dwell on the technique of working with directories and pay more attention to working with search engines. At the end of the conversation about directories, we give an example of a “chain” by which a search in the Yandex catalog is performed: Business\u003e Advertising\u003e Advertising on the Internet.

All search engines work on the same algorithm and are based on the same principles. Differences between them arise only at the level of technical implementation of these principles in work.

To understand the principle of the search engine, let's try to divide the question into two parts: what is the basis of the search and how it is implemented.

All search engines are based on three basic operators that underlie Boolean algebra (also called Boolean logic or Boolean).

These are logical operators “AND”, “OR” and “NOT”. They work as follows.

1. The logical "And." If between the two words in the query there is an “AND” operator, then as a result of the search only those documents will be found that contain both words. So, for example, at the request of a dog and a cat, a document containing an offer will be found "The dog was chasing the cat,"documents consisting of text “The cat was resting”or "Dog food" we will not see.

2. Logical "OR".If the OR operator is between the words, then the search result will be documents that contain at least one of these words.

If we do not make special restrictive reservations, then materials in which both these words are present will also be found.

At the request of a dog OR cat, we will receive documents excluded in the previous request and containing the text “The cat was resting”or "dog food",as well as material with the proposal "The dog was chasing the cat."

3. Logical "NOT."If the two previous operators described those words that you want to include in the query, the “NOT” operator excludes words from the query. Users who first encounter query operators often express surprise: they say, isn’t it easier to not include an unnecessary word in the query? Why introduce an additional operator? Unfortunately no. Not easier.

In fact, in order to understand the importance of the logical operator “NOT”, it makes sense to recall that our request does not create anything new on the Internet. We only fish out what we need from the existing huge, but still finite array. In this case, it is necessary to cut off informational garbage. We cut it off using the “NOT” operator. Unfortunately, it is not for us to decide whether we will see this garbage in the extradition. So, for example, at the request of information about the roof ridge, informational garbage invariably appears in the form of documents about the Humpbacked Horse, figure skating, hockey, horses, etc. There is no way to do without a logical “NOT”.

Let's look at examples of the logical operator "NOT".

Upon request, a dog NOT a cat will find a document containing the text "dog food",but the documents with the words “The cat was resting”or "The dog was chasing the cat,"and even "Food for dogs and cats"will be excluded from the issuance.

Often found a little over difficult option   writing a query that contains all or almost all of the above operators. In this case, it is better to use an element such as parentheses. Brackets allow you to separate the same type of query words from the rest. In addition, the compiler himself is visually much more convenient to distinguish individual fragments of the request. We will not be too theorizing about brackets, but simply demonstrate the operation of the specified element on concrete examples. In our opinion, it will be more clear how and why brackets are used.

So, request fluffy AND (dogs OR cats)

will allow you to get documents related to both fluffy dogs and fluffy cats - individually or together. The brackets at the same time “open” according to the usual arithmetic rules common factor   (for those who do not like mathematics, we hasten to say that we will not go into it more deeply). And here's the request for fluffy AND (dogs OR cats) NOT (dogs AND cats)

will give out documents that say about fluffy dogs or fluffy cats, but will not contain texts where both cats and dogs will be mentioned at the same time.

Once again, all search engines today work based on the analysis of these three operators, although the nuances of writing them in different search engines   may vary.

Each full-fledged search engine has its own staff of robots, or spiders. They are also called crawlers and spiders. These are programs that jump from page to page and scan the texts on them, without delving into their contents. Then they dump the documents to the servers of their hosts and go to the next pages. How does a spider determine where to go? He finds the so-called hyperlink (the very one, when you hover over it, the cursor takes the form of an open palm, and when you click on it, it goes to another page) and goes on it. That is why, if no link leads to the page, the spider will not come to it either. The exception is the situation when the page owner manually informs the search engine about it by filling out a special form on the website of the search engine.

On the server of the search engine, the text is divided into separate words, each of which is assigned coordinates, after which they are entered in the server table with a link to the Internet address at which the text was placed at the time of its visit by the spider.

The search engine itself is a big local networkconsisting of powerful computers with a huge volume disk memory. These machines are divided into subgroups (the so-called clusters), between which information collected by spiders is distributed.

When a search engine receives a query, it searches for an answer in its table, and not on the Internet.

It’s important to understand how the spider decides how often it should visit this or that page. This algorithm looks approximately as follows. After working with the page, the spider returns to it, but, for example, in two weeks. And if he sees that no changes have occurred, he plans the next visit in a longer period - say, in a month. And if even then he doesn’t find anything new, then he will visit here even later, in a month and a half or two.

That is why it often happens that a search engine returns a result on request, and an attempt to go to a page using the link received is unsuccessful - most likely, no page just exists in the same place, but the spider has not visited it for a long time, and, accordingly, the search the system does not know about its removal.

The whole complex of processes described above is called indexing.

The history of the evolution of search engines is most fully, in our opinion, presented in the book of recognized experts in the field of invisible Internet Chris Sherman and Harry Price "Invisible Internet".

Until the mid-1960s, there were few computers. Isolated from each other, they could not exchange information.

In 1962, Professor Licklider from the leading technical university in the United States - the Massachusetts Institute of Technology - formulated the concept of the global computer network "Galactic Network". The idea began to be implemented by an employee of the US Department of Defense Larry Roberts (Larry Robberts), who four years after the publication of the article professors proposed to combine separate computers   ministries to the network described by Liklider. These are the prerequisites for the emergence of the ARPANET network, which then turned into what is now called the Internet. The first ARPANET node appeared in 1969, and for the next several years universities and various counterparties connected to it were commissioned by the US military.

In 1973, the U.S. Department of Defense initiated a new program that envisaged providing reliable communications between computers using a very large number of different connections. The purpose of this solution was to increase the system's resistance to attempts to massively disrupt electronic communications. Since all this happened during the Cold War, it was about resilience to the terrifying consequences of a strategic nuclear confrontation. Since ARPANET was a single network, which at the system level reduced its ability to resist damage, the idea arose of creating a “network of networks”, which theoretically could be infinitely large. This project was called "Internetting", and the network itself is "Internet".

As the number of machines connected to the Internet increased, the question of the need for tools to easily find text and other files on remote computer, ideally - on anyone, wherever it is located on the Web.

Access to files at the earliest stages of the development of the Internet was carried out in two stages, each of which was carried out manually: special commands were entered from the keyboard. By the way, then computers could only be controlled by specialists who were able to enter commands on the appropriate line. Graphical interface, which allows an untrained person to work comfortably with a machine, has not yet been invented. So, first of all, using the Telnet program, a direct connection was established with the computer on which the desired file is located. At this stage, only communication was established, nothing and nowhere was transmitted at that moment. And only then with special program   - FTP - could this specific file   to take.

Obviously, it took a lot of time to find the right document: it was necessary to know the exact address of the computer on which it is located.

Meanwhile, there were more and more files, interest in them was constantly growing, and in order to find the address of one of them, we usually had to turn to discussion groups asking for help and in the hope that one of the interlocutors would tell me the cherished address, on which the necessary information is stored.

As a result, special FTP servers began to appear, which were a repository of files organized in a directory, on the principle of storing information on personal computer. Such servers exist to this day.

The first workable, publicly accessible tool for finding files stored on FTP servers was called Archie and was created in 1990 by the group system administrators   and undergraduate students at McGill University in Montreal. Archie was the prototype of today's search engines, but much more primitive and limited in its capabilities. He surfed the Internet, searched for files on different FTP servers and uploaded the list of directories of each found server to his own, forming a common directory.

This directory was then processed and stored in a central database, within which a search could be organized. Search on your own computer had already existed for a long time and, despite the fact that it also required the input of commands, did not create difficulties in work. However, without special preparation, a person could not use the computer fully. The database was at McGill University and updated monthly.

In 1991, the team of Mark McCahill of the University of Minnesota created the Golden Gopher program (Golden Gopher - translated from English as “gold digger” or “prospector”), which combined both protocols - Telnet and FTP. All that the user had to do to gain access to the necessary information was to click on the hyperlink in the menu. Thus, for the first time in history, it was no longer necessary to enter any commands, so from now on people could “roam” on Internet resources without special training.

The program showed the user sequentially appearing step-by-step menus, which allowed him to easily go into the depths of the directory base, more and more approaching specific documents, which were the purpose of the search. This algorithm, in fact, is preserved today in the Directories located on the Internet.

It became possible to receive as text documentsboth graphic and musical, without reference to any particular format. And most importantly, in principle, it has become possible to easily find and obtain the necessary information on the Internet.

However, problems still remained. One of them, and quite serious, was due to the fact that computers were built on different platformsah, who sometimes did not understand each other. Here you can draw an analogy with people who speak completely different languages \u200b\u200band therefore can not build a more or less meaningful conversation. In those days, not operating systems competed with each other, as they are now, but manufacturers of computer hardware. Today, it is less important who produced the computer. It is much more significant that it is installed on it: Windows, Linux, Mac OS or some other system. And then it was the hardware manufacturers that determined the face of the Internet.

Objectively, the idea was brewing, according to which computers of different platforms should be able to work in one protocol, which allows viewing pages regardless of which particular machine these pages are created on. It was necessary to come up with such a universal protocol and make it user-friendly. The first who guessed to combine the well-known by that time simple form   Hypertext with universal communication protocols, was Tim Berners-Lee.

In order for the user to get his hands on a platform-independent and yet simple tool, Berners-Lee created HTML (HyperText Markup Language, that is, Hypertext Markup Language). All Web documents formatted with hTML tags, are seen exactly the same throughout the world, regardless of the type of computer on which the person opened the page of the site. Therefore, even today, when converting a file to HTML format, for example, on a machine running operating system   MacOS, you can be sure that this file will look exactly the same on a computer running Windows.

Berners-Lee then came up with the Universal Resource Identifier, a standardization method for addresses that assigns unique addresses to computers on the Internet (today we call them URLs, which usually starts with “www” in the user's usual way). Finally, the inventor brought together all these elements, creating a system in the form of Web servers that store HTML documents and provide them to other computers, creating HTML requests for documents at specific URLs.

But Berners-Lee wanted to see the Internet as an information space in which you can access freely any type of data. In the early stages of the development of the global Web, plain HTML text documents prevailed. By that time, there were information retrieval systems on local machines, so several servers appeared that tried to index some part web pages   and before setting off on the Internet for something, they suggested looking for the necessary information on these servers.

In this case, the main problem was to find pages that, in principle, could be indexed. Since the Internet is devoid of a centralized structure and general table of contents, the only way to achieve this was to search for a link to the page and click on this link, followed by adding the found resource to the index.

However, another problem soon arose. The most popular pages were visited by spiders more often than others, since the maximum number of links pointed to them. Spiders, the number and capabilities of which were limited, “hung” on such pages and wasted resources, leaving many other addresses, still less popular, unvisited. To solve this problem, it was necessary to create a program that would allow you to ignore already indexed pages and focus on finding new ones. Otherwise, it threatened a resource problem.

In 1993, a physics student at the Massachusetts Institute of Technology, Matthew Gray, created the first well-known Web robot called the “World Wide Web Wanderer,” or simply “Wanderer,” which means “wanderer” or “wanderer” in English. . The fact is that Gray became interested in statistics. The result of this hobby was the emergence of a “wanderer”: the invention was intended to help the student analyze the size of the Internet and its growth rate. "Vanderer" simply came to the page and determined the fact of its existence, without entering the contents of the found address into the database. Despite the fact that the creator of the robot did not pursue any other goals, his brainchild, which actually debuted in the “race” of progressive Internet discoveries, formed the basis of more complex programs, which added the ability to save the contents of pages in the database to the wanderer’s ability to navigate the Web data after visiting them.

It so happened that 1994 was a watershed in the history of the creation of search engines. University of Washington graduate student Brian Pinkerton is tired of an endless series emailsfriends sent to him with information about the good sites they found on the Internet. Of course, he needed sites, but the flurry of messages with their addresses was annoying, and visiting all pages took a lot of time. However, Pinkerton found a solution to the problem - he created a robot that he called WebCrawler (something like "an all-terrain vehicle for the Internet"). WebCrauler, like Vanderer, crawled from page to page, remembering all the text of a Web document and storing it in a database that was accessible to search words. The inventor introduced his brainchild to the public in April 1994, and did it virtually - through a Web interface. The database at that moment contained information from 6,000 different servers. A week later, it began to expand, with a daily increase of more than 100 new servers. So the first search engine was born.

At that time, the term “crawler” or “spider” was introduced into the everyday life of Internet users, which is used, as we have said, to this day.

1994 - WebCrawler, Lycos, Yahoo!

1995 - Infoseek, SavvySearch, AltaVista, MetCrawler, Excite. The advent of metasearch engines.

1996 - HotBot, LookSmart.

1997 - NorthernLight.

1998 - Google, InvisibleWeb.com.

Russian-language search engines appeared in this sequence:

2004 - Russian version of google (www.google.com) and the Russian version of Yahoo! (http://en.yahoo.com).

Before proceeding to the description of the search engine query language, let us consider what elements the spider will work with usually consists of a website.

I must say that the HTML language is quite simple and logical. It is a way to break text using special elements - tags that define the structure and appearance   text when viewing it in a browser. You should know about tags that they are always paired and that they happen opening(indicate the beginning of a specific formatting) and closing(indicate its end). The closing tag is the same in spelling as the opening tag, but it has a slash in front of it.

We give an example of a very simple site (Fig. 1).

Fig. 1. An example site, as seen in the Mozilla Firefox browser.

At the top of the page shown in the figure, that is, not in the text of the site, but on the upper margin of the page frame, next to the round browser icon, is the inscription: “Showing the site’s device”. It is located in the so-called page title (which is enclosed between the opening tag ) We draw your attention to the fact that this is the title of the whole page, not text.

In the middle of the picture in bold italics it is displayed: "This is a simple site." This inscription is the title of the text. The font of the phrase "This is a simple site" exceeds the font size of the text on the site in size, it is specially highlighted as the title of the text. When marking with using HTML   this text is located below the tag