Search Engine Concepts and Techniques
[Pages:9]Search Engine Concepts and Techniques
SWE 642, Spring 2008 Nick Duan
February 27, 2008
1
Overview
Information Retrieval on the Internet
Types of digital information ? data and data format Types of delivery mechanism - Information flow How the game is played: From producer to consumer
Robots, Crawlers, Spiders, Agents
The discovery process Robot exclusion protocol
Search Engine Concepts
Types of Search Engines How Indexing and search works? Introduction to Apache Lucene and Nutch A Lucene example
Summary
February 27, 2008
Nick Duan
2
1
Information Retrieval on the Internet
Internet - The fourth industrial revolution
The exponential growth of the web has created unprecedented abundance of information ? Information overflow
The ultimate quest ? Retrieving the right information at the right time Computer tools are needed to help people with various IR tasks
February 27, 2008
Nick Duan
3
Types of Information &
Delivery Methods
Textual
Structured: Relational, SGML/XML, "strongtyped" documents Unstructured: free text, HTML, "weak-typed" documents
Binary
Images, Videos, Sounds, digitized sensor data Encoded text information: PDF, Word, etc.
Info delivery mechanism over the Internet
Unrestricted access: Pull (search) vs. Push (channel feeds) Restricted access: User security required
February 27, 2008
Nick Duan
4
2
How the Game is Played
Producers
Authors/initial content publishers/web-site hosts
Service Providers
Web-site hosts
Search engines (e.g. Google), Content aggregators (e.g. AOL), application-specific service providers (e.g. on-line merchants)
Consumers
Internet users, value-added re-publishers/content aggregators
Our focus is on discovery and search
Discovery: Web crawling to retrieve web content, and save the content in repository
Search: Index the web page repository, processing search queries, presenting the results
February 27, 2008
Nick Duan
5
Robots, Crawlers, Spiders, Aggregators
Different terminologies but the same information discovery function: Collecting HTML or other documents (textual or encoded textual) by following the URL links of each HTML page Recursive exploring of URLs or fixed URL set
You may specify the depth of the URL tree Keep track of URLs to prevent circular links Multi-threaded or clustered configuration to boost performance
Robot name to be specified in User-Agent:
Googlebot, MSNBot, Infoseek Sidewinder...
February 27, 2008
Nick Duan
6
3
The Web Discovering Process
Web sites
URL queue
For each of the URLs in the queue Is robot exclusion on? if yes, continue to next URL retrieve the web page parse its HTML content collect URLs ( ................
................
In order to avoid copyright disputes, this page is only a partial summary.
To fulfill the demand for quickly locating and searching documents.
It is intelligent file search solution for home and business.
Related download
- googling social interactions web search engine based
- how to search the internet effectively
- the consequences of search bias how application of the
- a reflection of search engine strategies
- united states of america smartphones visual exploratory
- search engine concepts and techniques
- web searching for systematic reviews a case study of
- trademarks as search engine keywords who what when
- mulinex multilingual indexing navigation and editing
Related searches
- uncensored search engine 2019
- what search engine is best
- best image search engine uncensored
- best search engine 2019 reddit
- different search engine names
- different search engine than google
- duckduckgo search engine download free
- most secure search engine 2019
- best search engine for flights
- ask search engine free download
- search engine with least censorship
- search engine that doesn t filter results