+

WO2010060117A1 - Procédé et système visant à améliorer l’utilisation de chercheurs humains - Google Patents

Procédé et système visant à améliorer l’utilisation de chercheurs humains Download PDF

Info

Publication number
WO2010060117A1
WO2010060117A1 PCT/US2009/066406 US2009066406W WO2010060117A1 WO 2010060117 A1 WO2010060117 A1 WO 2010060117A1 US 2009066406 W US2009066406 W US 2009066406W WO 2010060117 A1 WO2010060117 A1 WO 2010060117A1
Authority
WO
WIPO (PCT)
Prior art keywords
query
request
search
inquiry
user
Prior art date
Application number
PCT/US2009/066406
Other languages
English (en)
Inventor
Michael Burroughs
Thomas Cooper
John Ellis
Original Assignee
Chacha Search, Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US12/275,864 external-priority patent/US8301651B2/en
Priority claimed from US12/434,385 external-priority patent/US8719256B2/en
Application filed by Chacha Search, Inc. filed Critical Chacha Search, Inc.
Priority claimed from US12/629,500 external-priority patent/US20100138402A1/en
Publication of WO2010060117A1 publication Critical patent/WO2010060117A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2452—Query translation
    • G06F16/24522—Translation of natural language queries to structured queries
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33—Querying
    • G06F16/3331—Query processing
    • G06F16/3332—Query translation
    • G06F16/3334—Selection or weighting of terms from queries, including natural language queries

Definitions

  • Embodiment(s) described herein relate to search engine technologies including human-assisted search engines and, more particularly, to a method and system of processing a request for information using a combination of automated processing and human intelligence.
  • a human assisted search service can provide a more natural and effective response to a request for information. Natural language queries may be interpreted, and a response to a query may be obtained by a person. This may be particularly desirable if a query is submitted using a mobile device which may have limited capabilities. In particular, the use of SMS or voice to submit a request for information may be more readily accepted when a human-assisted response is provided.
  • Human assisted search services have found greater acceptance than purely automated services due at least in part to a higher answer quality, increased ease of use, and generalized response capability.
  • human-assisted search systems must generally have a low cost to serve a user in order to find broad acceptance. For example, pay-per-use systems such as any question answered (AQA), or Infonxx Knowledge Generation Bureau have answered a few million queries per year, while free services such as ChaCha® have answered tens of millions of queries per month.
  • a method and system of improving utilization of human searcher(s) including using Natural Language Processing to recognize an intent of a user request, provide information to an expeditor to respond to queries, and automatically route a query to a relevant searcher.
  • a computer-implemented method executes an operation including receiving a request for information, executing a processing with respect to the request including comparing a predefined query pattern, correcting spelling, substituting words, removing signatures using information of stored requests, creating automatically an annotation graph for the request, selecting a category, obtaining queries determined to match the annotation graph using an index of queries, calculating a matching function comparing the queries to the annotation graph, calculating a rating of a search result associated with the queries, ranking the queries based on the matching function, ranking the search results based on the rating and providing a highest ranked search result responsive to the request.
  • Figure 1 is a block diagram of a system embodiment.
  • Figure 2 illustrates a flow diagram of a process of performing a search.
  • Figure 3 illustrates a flow diagram of a process of responding to a request.
  • Figure 4 illustrates a flow diagram of a process of responding to a search result.
  • Figure 5 illustrates a database record for a raw query.
  • Figure 6 illustrates a database record for a vetted query.
  • Figure 7 illustrates a flow diagram of a process of performing a search.
  • Figure 8 illustrates a flow diagram of a process of processing a search request.
  • Figure 9 illustrates a flow diagram of a process of reusing a search result.
  • Figure 10 illustrates a flow diagram of a process of routing a search request.
  • Figure 11 illustrates a flow diagram for vetting a request.
  • Figure 12 illustrates a query processing GUI for preparing a search for a searcher.
  • Figure 13 illustrates a query processing GUI for selecting a search result.
  • Figure 14 illustrates a GUI for selecting a search result provided by a resource.
  • Figure 15 illustrates a GUI for selecting a search result provided by a resource.
  • a method and system for improving utilization of human searcher(s) using natural language processing includes using NLP to process an initial query. Should the NLP be unable to determine an intent of a customer, a system may utilize information from the NLP process to determine if a request may be handled by a human assistant specializing in query processing or 'expediter' or may be routed directly to a search specialist or 'searcher' to perform a search. NLP results may be used to assist an expediter in using a toolset which may enable a rapid and repeatable response to various user requests. Subsequent to processing by an expediter, NLP may be applied to a request in order to determine if a previous answer may be provided and/or if a searcher should obtain a search result.
  • NLP natural language processing
  • Processing such as algorithmic processing of a request is applied in order to optimize the use of human guides.
  • a request is first compared to standard templates to determine if the request conforms to an automated response format. If a query is not found to conform to a recognized format, the query is analyzed using NLP. NLP may be used to remove information from and/or augment a query such that the query may more easily be categorized and/or associated with a recognized format.
  • a request is compared to a database of requests which may be used to determine whether a search result is available which may be used to provide a response to the request. If it is determined that a search result is not available, the results of NLP may be used to determine if an ambassador guide or a searcher should receive the request.
  • a method and system of improving utilization of human searcher(s) is disclosed.
  • a query is received by a search system, which may apply NLP to the query. If a query is received, it may be processed to determine if the query is a formulaic or "keyword query".
  • a "keyword query” is a query which begins with and/or includes a word and/or phrase which indicates that the query is intended for automated processing. For example, a query which begins with the word "weather” and includes a noun phrase which is recognized as a location may be determined to be a keyword query.
  • a request may be determined to be a "keyword query" based on one or more factors including when the request contains at least one word pre- identified by a user (who may be an administrator) as being a cause for an automated processing, based on previous association of the query with automated processing, etc.
  • NLP of a query may be applied to make the query easier to interpret and/or to remove extraneous information. For example, spelling correction, grammar correction, word substitution, and signature removal may be applied to improve the probability that a query may be recognized automatically and/or using a human assistant.
  • NLP may include named entity analysis.
  • a 'named entity' may be any noun or noun phrase which is associated with a proper noun. For example, a database of celebrities, sports figures, cities, neighborhoods, etc. may be used to obtain a lexicon for named entity analysis.
  • NLP of a query may include categorization and reformatting of a query. For example, a Bayesian model for categorization may be applied, based on factors such as noun phrases associated with a query which may be associated with an index of categories. In at least one embodiment, grammar of a query may be examined to apply parts of speech tags to the query.
  • the query may be compared to templates which may allow a query to be answered automatically. If an answer can not be provided programmatically (or automatically), a processed query may be compared to a database of stored queries, and if it is determined that a query is a match to a stored query, an answer associated with the stored query may be provided responsive to the query. While a query is being described as being compared to a template, the present invention is not limited thereto. For example, a query may be compared to one or more formatted, predefined or standardized queries including to determine whether the query can be automatically answered using one or more of the responses of matching quer(ies).
  • a query may be analyzed to determine whether an expediter is to be presented with the query. If an expediter is presented with the query, a processed query and a user query maybe presented to the expediter. An expediter may reformat a query, associate a location with a query and/or may categorize a query, and a resulting query may be processed to obtain queries which may match the resulting (revised or reformatted) query. If an expediter has modified and/or augmented a query, parameters for selection of an answer and/or matching of a query to a stored query may be modified. An expediter may override a system matching.
  • a query is directed to a searcher
  • the searcher may receive information of an unprocessed, a NPL processed, and an expediter processed forms of a query.
  • a categorization associated with a query may indicate whether an expediter has selected the categorization.
  • a "user" may be any person or entity which may submit a request or search request.
  • a request or search request or query is information which may be submitted using any communication service which may be able to access a search system as further described herein below.
  • a query may include text, a word, keywords, and/or a fully formed question.
  • a query may include any media which are compatible with an interface to a search service such as text, images, audio, video, URL's, etc.
  • a "guide” may be any person who may be compensated and/or may be a volunteer who may respond to a request.
  • An "ambassador” or “expediter” is a guide who may perform processing of a request and/or a search result(s).
  • a “searcher” is a guide who may perform an information search responsive to a request.
  • a "raw query” is a request submitted by a user, which may include any type of information provided by a user and/or associated with a user.
  • a "vetted query” includes a request which is associated with a category, a structured query, or otherwise qualified query.
  • a “structured query” is a question formulated according to a structured grammar.
  • a grammar construction required for a structured query is a question.
  • a structured query may also be referred to as a "succinct query”.
  • An "identifier" or ID is a group of characters, numbers and/or other types of information which may be used to identify an item.
  • a "guided request” is a request which uses the assistance of one or more guides.
  • a "result” or “search result” is any information which may be provided responsive to a request.
  • a search result includes but is not limited to any of an advertisement(s), a link to a web page, a message of any sort, image, audio, text, games, interactive media and/or software of any sort.
  • a "search resource” is any source of information which may be used to obtain a search result.
  • a search resource includes automated and/or human-assisted systems, and any type of media and/or systems which may provide information. For example, web pages, search engines, data feeds, databases, a private archive, printed materials, etc. may be a search resource.
  • a "profile” is a group of one or more characteristics which may be associated with a person. Profile characteristics include but are not limited to demographic, geographic, personality, affiliations, areas of interest, historical actions, preferences, memberships, associations, etc.
  • NLP Natural Language Processing
  • NLP may be used to correct errors, detect the language of a query, and to make other corrections including as is well established in the art.
  • NLP is further used to estimate a match between a user request and a structured query.
  • NLP is used to evaluate a quality of stored search results associated with a query in order to determine whether a stored result may be provided to a user.
  • NLP is further used to optimize the use of ambassadors. If NLP determines a most probable type of query, an ambassador may be provided with tools associated with the most probable type of response. Likewise if a request is categorized with a high level of confidence and a structured query has been determined to match a user request with high probability, a request may be provided to a searcher without utilizing an ambassador.
  • system 100 includes searcher systems 105, 110, a network 115 such as the Internet, a search system 130, user systems 135, 140, a database 120, which may comprise various records, and ambassador system 145, 150.
  • Each of the guide systems can be operated by a guide to obtain a search result on behalf of a user located at a user system (e.g., the user systems 135, 140). While only a limited number of searcher (also referred to as a human searcher), ambassador (also referred to as a vetting guide or query vetting guide or expediter) and user (also referred to as an information seeker or requester) systems are depicted in Figure 1 , it is within the scope of the disclosure for multiple searcher, ambassador and user systems to be utilized.
  • searcher also referred to as a human searcher
  • ambassador also referred to as a vetting guide or query vetting guide or expediter
  • user also referred to as an information seeker or requester
  • the network 115 may be a global public network of networks (the Internet) and/or consist in whole or in part of one or more private networks and communicatively couples the searcher computer systems 105, 110, the ambassador systems 145, 150 and user computer systems 135, 140 with the other components of the system 100 such as the search system 130, and the database 120.
  • the network 115 may for example include any or all of a wireless network(s) such as a cellular phone network, a WiMax network, a WiFi network, etc. a network such as the Public Switched Telephone Network (PSTN), and/or any other type of communication network.
  • PSTN Public Switched Telephone Network
  • the search system 130 allows interaction to occur between the searcher systems 105, 110, the ambassador systems 145, 150 and the user systems 135, 140.
  • a request can be transmitted from the user systems 135, 140 to the search system 130, where a request can be accessed by the searcher systems 105, 110 and/or the ambassador systems 145, 150.
  • a search result produced using the searcher systems 105, 110 or the ambassador systems 145, 150 in response to a request produced by the user systems 135, 140 may be transmitted to the search system 130, where it may be stored by the search system 130 and/or may be transmitted to the user systems 135, 140 and/or the ambassador systems 145, 150.
  • the user systems 135, 140 may include within the scope of the disclosure, any device through which an information seeker can submit a request to and/or receive information or search result from the search system 130.
  • the user system 135 may be a device configured for connection to a network and may run web browser software.
  • the user systems 135, 140 may be a portable or laptop computer, a personal data assistant (PDA), a desktop computer such as a PC or Mac®, a workstation or a terminal connected to a mainframe, a specialized search apparatus, or a smart phone, etc. within the scope of the disclosure.
  • PDA personal data assistant
  • Any device which can communicate using Internet Protocol (IP) may be a user system, a searcher system, and/or an ambassador system within the scope of the disclosure.
  • IP Internet Protocol
  • the search system 130 may include a gateway for voice communication and a speech- to-text system or other transcription device and/or personnel to facilitate access to the search system via voice communications such as through a land line phone, cellular phone, Voice over Internet Protocol and/or other telephonic device(s). Any device(s) which may be used to communicate using voice (speech) may be a user system, a searcher system and/or an ambassador system.
  • the search system 130 may include hardware and/or software interface to a system which provide communication services such as Instant Messaging (IM), email, Short Messaging Service (SMS), Enhanced Messaging Service (EMS), Multimedia Messaging Service (MMS), and/or other forms of messaging services.
  • IM Instant Messaging
  • SMS Short Messaging Service
  • EMS Enhanced Messaging Service
  • MMS Multimedia Messaging Service
  • Any device which may communicate using such services may be a user system, a searcher system, and/or an ambassador system within the scope of the disclosure herein.
  • a request may be submitted to the search system 130 using any or all communication services which are able to communicate with the search system 130.
  • the search system 130 may include any number of hardware systems, which may function as servers or other elements.
  • the search system 130 may include one or more servers, computers, etc.
  • servers such as the PowerEdge® 2900 by Dell, or the BladeCenterJS22 by IBM, or equivalent systems might be used to implement elements of the search system 130.
  • the search system may utilize an operating system (OS) such as Microsoft Windows XP, or Linux, etc. Voice routing and packet switching may be accomplished using well established technologies such as those provided by Cisco, or other networking companies.
  • OS operating system
  • Voice routing and packet switching may be accomplished using well established technologies such as those provided by Cisco, or other networking companies.
  • a server functionality associated with any of the communication services identified herein above may be implemented to allow messages to be transmitted between the elements of the system 100.
  • the search system 130 is communicatively coupled with the database 120.
  • the database includes data that is processed during operation of the embodiments.
  • Figure 1 illustrates the database 120 as a separate component of the system
  • the database 120 may be integrated with the alternate search system 130.
  • the records maintained in the database 120 may be stored in any typical manner, including in a Network Attached Storage (NAS), a Storage Area Network (SAN), etc. using any typical or proprietary database software such as DB2® , Informix®, Microsoft® SQLServerTM, MySQL®, Oracle®, etc., and may also be a distributed database on more than one server.
  • Elements of the database 120 may reside in any suitable elements of the system 100.
  • elements of the database 120 may be stored in any storage media accessible to a user system, a searcher system, and/or an ambassador system.
  • guides searchers
  • ambassadors exiters
  • a prospective guide may also be invited to register as a guide with the search system 130.
  • a previously registered guide and/or an affiliate group may initiate the invitation process and thereby be designated as a sponsor for the invited guide.
  • a prospective guide is sent an email invitation to register as a guide with the search system 130.
  • the email invitation in one embodiment contains a link to a guide registration page and some indicia recognizable by the search system 130 as to the person or entity that invited a prospective guide to register so that upon registration the person or entity may be accredited with sponsorship of the invited guide.
  • the prospective guide may for example initiate a registration process by selecting a link that directs the guide's web browser to a URL of a guide registration page generated by a web server function of the search system 130 (Fig. 1 ). While registration of a prospective guide is described with respect to the search system 130 providing pages to a browser, the present invention is not limited to any particular configuration for registering guides. For example, a system dedicated to registration of guides may be provided which is separate from the search system 130.
  • a prospective guide may elect to be a searcher and/or an ambassador, and/or to perform other activities.
  • a prospective guide may undergo training and/or testing which may be used to determine whether a guide is competent to perform a task. After completion of testing and/or training, a guide may be allowed to perform a task.
  • a registration and qualification process for a guide(s) is further described herein below.
  • a guide may act as an ambassador, and/or as a searcher, etc. based on completion of training and/or testing.
  • An ambassador processes a user search request and/or a search result(s).
  • a searcher may receive a processed request and may perform a search responsive to the processed request.
  • a guide may act as an ambassador and/or a searcher including but not limited to time spent as a guide, level of knowledge in particular area, etc.
  • the role of ambassador and searcher may be mutually exclusive.
  • the performance of an ambassador may be monitored. Multiple feedback mechanisms may be utilized including ratings by one or more users, guides, or other persons, time required to complete a task(s), complexity rating of a query, and/or other speed and/or quality metrics.
  • a rating of an ambassador may be based on a percentage of queries provided to an ambassador which are answered without utilizing a searcher.
  • a process 200 for performing an information search in which one or more ambassadors act as an intermediary is provided.
  • the process 200 may be operative on any suitable elements of the system 100.
  • the process 200 is operative on a server associated with the search system 130.
  • operation 205 (Fig. 2) a determination is made as to whether a request is received. If it is determined in operation 205 that a request is not received, control remains at operation 205 and process 200 continues. If it is determined in operation 205 that a request is received, control is passed to operation 210 and process 200 continues.
  • the determination in operation 205 may be made using various criteria.
  • a message is received at a server associated with the search system 130, it may be determined that a request is received. For example, if an email message, an SMS, EMS, and/or MMS message, an IM, an IP message, and/or a voice message is received at an address associated with the search system 130 (Fig. 1 ) it may be determined that a request is received.
  • the search system 130 may determine whether the assistance of a guide will be used to respond to a request as further described herein.
  • an ambassador processes a request.
  • a process for augmenting, modifying or processing information of a request using an ambassador is further described herein with respect to Fig. 3. Control is passed to operation 215 and process 200 continues.
  • the rating and/or compensation of an ambassador may be affected by acceptance of a request processed by the ambassador. If a vetted query is not accepted by a searcher, it may be assigned to a second ambassador. Actions of guides may be utilized as a form of quality control. For example, if two ambassadors agree on the content of a vetted query, and a second searcher accepts a vetted query, the rejected vetted query may not be counted towards the rating and/or compensation of an ambassador. The compensation and/or rating of a searcher who rejected a vetted query may be affected if a rejected vetted query is accepted by an ambassador and/or a different searcher.
  • an information search is performed by a searcher responsive to a vetted query and a search result is returned.
  • An information search may be performed by any guide.
  • a searcher and/or an ambassador may perform a search.
  • Control is passed to operation 225 and process 200 continues.
  • an automated search without requiring a searcher may be performed responsive to the modified request.
  • a search result is (optionally) reviewed by an ambassador and may be presented to a user.
  • a process for review and presentation of a search result is further described herein below with respect to Figure 4.
  • a search result may be sent to a user without requiring review by an ambassador. Control is passed to operation 230 and process 200 continues.
  • rating information is obtained.
  • a rating may be based on user acceptance, and/or a review by a guide, and/or review by a system administrator.
  • Information of the process 200 is recorded.
  • Information such as ratings of a search result, ratings of a guide, ratings of a vetted query, time to perform operations, etc. may be recorded.
  • the database 120 (Fig. 1 ) may be updated to indicate actions of a user, a searcher, an expediter, processing of a query, etc. Control is passed to operation 205 and process 200 continues.
  • an ambassador who performs the processing in operation 210 and an ambassador who performs the review in operation 225 may be the same person. In other embodiments, a different ambassador may perform the processing and the review operations. In at least one embodiment, a searcher may perform the review operation.
  • a process 300 for providing a response to a request is provided.
  • the process 300 may be operative on any suitable elements of the system 100.
  • the process 300 is operative on a server associated with the search system 130.
  • operation 305 a determination is made as to whether a guided request is received. If it is determined in operation 305 that a guided request is not received, control remains at operation 305 and process 300 continues. If it is determined in operation 305 that a guided request is received, control is passed to operation 310 and process 300 continues.
  • the determination in operation 305 may be made based on various criteria. For example, a user preference may indicate that a guided search is to be performed.
  • the search system 130 (Fig. 1 ) may determine whether a guided request is received. Content of a message may be used to determine whether a guide is to respond to a request. A target address such as a telephone number, a short code, an IM credential, etc. may be used to determine whether a guide is to respond to request.
  • An automated analysis of a query may be performed as further described herein which may in part determine whether a guide will respond to a request.
  • a request (raw query, or original) is processed to determine relevant information.
  • a raw query may be processed to improve matching of the raw query to a database of queries.
  • a raw query may include text, audio, and/or other media.
  • a raw query may be processed in order to determine one or more keywords, categories and/or vetted queries which may be associated with a raw query.
  • a vetted query is further described herein below.
  • a user ID may be associated with a raw query, and a unique ID may be assigned to a raw query.
  • a raw query may be compared to structured queries using stemming, word counts, and/or other forms of processing which are further described herein. Control is passed to operation 315 and process 300 continues.
  • operation 315 a determination is made as to whether an ambassador is available to respond to a guided search request. If in operation 315 it is determined that an ambassador is not available to respond to a guided search request, control is passed to operation 315 and process 300 continues. If in operation 315 it is determined that an ambassador is available to respond to a guided search request, control is passed to operation 320 and process 300 continues.
  • the determination in operation 315 may be based on various criteria.
  • any available ambassadors are notified, and a first ambassador to respond is selected. If an ambassador is notified and does not respond, the ambassador may be recorded as unavailable.
  • an ambassador may be selected based on information associated with a raw query. For example, an ambassador may be selected based on a type of device a requester is utilizing to submit a request, and/or to receive a search result. An ambassador may be selected based on a competency, a training status, a keyword and/or category, geographic information, demographic information, personality information, interest area information, affiliate group information, a contractual relationship, a language skill, etc., which may be associated with a raw query and/or an ambassador.
  • an ambassador may not be available to process another request. If an ambassador is not logged-in the ambassador may not be available. Availability of an ambassador may be determined dynamically. For example, if no ambassador who meets a group of criteria is available, one or more criteria may be modified so that an ambassador who meets a different group of criteria is available. Alternately, a system may wait until an ambassador who meets a group of criteria is available. An ambassador may be notified of a request via various forms of communication such as IM, a voice message, a pop-up in a browser, an email message, an SMS, EMS or MMS, etc. which may be used to communicate with a system associated with the ambassador.
  • operation 320 information of a raw query and/or associated information such as a keyword, category, profile, vetted query, etc. identified in operation 310 are provided to an ambassador.
  • An ambassador may be presented with a GUI such as the GUI 1200 (Fig. 12) which may provide information of a request. Control is passed to operation 325 and process 300 continues.
  • an ambassador constructs a structured query.
  • a structured query may be based on a keyword, category, image, audio, video, a profile, a structured query, a vetted query and/or any information associated with a raw query.
  • An ambassador may select an advertisement which may be transmitted to a user. Control is passed to operation 330 and process 300 continues.
  • a vetted query be constructed by various construction methods.
  • a structured grammar is employed which includes phrasing a request in the form of a question.
  • a structured query construction may require standard elements to be selected. Using the example illustrated in Fig. 5, the structured query 'Who is the starting quarterback for the Chicago Bears' is a structured query associated with the raw query 'Who is the #1 quarterback for the Bears?'.
  • An ambassador may communicate with a user using a real-time interface such as a chat or IM session, voice connection, text-to-speech, etc.
  • Interaction with a user may allow an ambassador to determine information of a raw query including intended meaning of a keyword, a geographic location, and/or other information which may be used to form a structured query and/or which may be associated with and/or contained in a vetted query.
  • An ambassador may confirm user acceptance of a structured query which is associated with a raw query.
  • An ambassador may modify, adjust, qualify and/or supplement a keyword and/or other information of a raw query.
  • an ambassador may construct the structured query 'what is the best apple computer available?', or the structured query 'what is the best apple for baking a pie?', or the structured query 'what is the best apple?'
  • a vetted query may be ambiguous, it is structured to conform to a standard format, which may improve matching of raw queries to structured queries.
  • An ambassador may be presented with a query template which requires selection of a number of elements of a structured query from a list of available alternatives.
  • an interrogatory word or phrase such as 'what, who, when, where, how many', etc., may be selected from a list, and/or a verb or verb form such as 'is, was, will be', etc., may be selected from a list.
  • a structured query may be analyzed automatically to check for conformance to spelling and grammatical conventions of a target language.
  • a structured or vetted query may allow a searcher to receive a query which contains a standard set of information from an ambassador.
  • a vetted query may format a request for submission to an automated search system.
  • An ambassador may select elements of a structured query based on a keyword and/or category associated with a raw query.
  • a raw query submitted by a user such as 'bear's quarterback', or 'QB for the bears' may for example be converted to a succinct or structured query such as 'Who is the starting quarterback for the Chicago Bears?'.
  • One or more previously constructed structured queries associated with a raw query may be presented to an ambassador as further described herein.
  • a request is associated with a number of categories.
  • a categorization may be automatically associated with the raw query.
  • An ambassador may be presented with a list of categorizations which may be associated with a request.
  • an ambassador is not allowed to change a categorization associated with a structured query. For example, a categorization of a query reviewed by a number of ambassadors and/or searchers may not be permitted to be changed by a particular ambassador. Alternately, an ambassador may be allowed to change a categorization associated with a structured query based on various criteria.
  • An ambassador may be allowed to change a categorization of a structured or vetted query based on a ranking, rating and/or other characteristic associated with an ambassador and/or a structured query. For example, an ambassador who has correctly categorized a number or a percentage of succinct or structured queries may be allowed to modify a categorization associated with a structured query. If a structured query has not been categorized more than a number of times by an ambassador, an ambassador may be allowed to modify a categorization associated with the structured query. For example, if a structured query is associated with a category three times a change of categorization may not be allowed subsequently. A structured query which is brief such as 'Who are the Bears?' may be allowed to be associated with multiple categories.
  • a structured or "succinct" query which is more specific such as 'Who is the current quarterback for the University of California Bears?' may be allowed to be associated with only one category. Selection or creation of a structured query may change the ranking of a category associated with a request.
  • An ambassador may be presented with a taxonomy tree structure which allows an ambassador to modify a categorization of a request.
  • a categorization selected by an ambassador may be used to determine a most likely categorization of a vetted query. Control is passed to operation 335 and process 300 continues.
  • an ambassador selects information (a search result and/or other response) which may be presented to a user.
  • an ambassador may select an advertisement to be presented to a user.
  • a selection may be based on an explicit selection by a guide, and/or may be based on a number of categories selected by a guide.
  • a category association may be used to select a guide and/or information to be presented to a user and/or a guide such as an advertisement, a search resource, and/or a search result.
  • an ambassador may be presented with one or more search results which have been associated with a vetted query based on a ranking of a search result.
  • a search result associated with a vetted query may be selected in various ways.
  • a vetted query may be associated with a human-touched (reviewed) search result which has been judged superior to one or more algorithmic search results based on voting by one or more guides.
  • a vetted query may be associated with a search result produced by an algorithmic search engine.
  • a vetted query may be associated with a search result produced by an affiliate group.
  • An ambassador may select a search resource which may be provided to a guide and/or a user.
  • a search result may be automatically provided to a user based on the association of a search result with a vetted query.
  • a search result may be selected by an ambassador.
  • An ambassador may select to transmit any or all information associated with a vetted query to a user.
  • An exemplary GUI 1400 for selection of a response is illustrated in Figure 14. Control is passed to operation 340 and process 300 continues.
  • operation 340 a determination is made as to whether a response is accepted. If in operation 340 it is determined that a response is accepted, control is passed to operation 355 and process 300 continues. If in operation 340 it is determined that a response is not accepted control is passed to operation 345 and process 300 continues. In at least one embodiment, acceptance of a response or result may be determined based on user actions such as activating an action button, clicking on a link associated with a result, and/or sending a text message. In at least one embodiment, if an ambassador selects a search result, it is determined that a response is accepted.
  • a rating of an ambassador is determined.
  • a rating of an ambassador may be determined based on factors such as user acceptance of a search result provided, time required to send a search result to a user, a number of structured queries submitted to a user for review, click-through of an advertisement, a rating by a user a guide, a system administrator, etc. For example, if a user accepts a first vetted query proposed by an ambassador, an ambassador may receive a high rating, or if a user 'clicks through' a search result or advertisement provided by an ambassador, the ambassador may receive a high rating.
  • an ambassador rating may be based on acceptance of a vetted query by a searcher.
  • an ambassador rating may be based on the time between receiving a query and when a search result is accepted. In at least one embodiment, an ambassador rating may be based at least in part on a percentage of search queries which require a search by a guide other than the ambassador. A rating of an ambassador may be based on a combination of ratings from a number of search sessions, and/or rating factors. An ambassador rating may be based on any information indicated in the database 120 (Fig. 1 ). Information of the process 300 is recorded. Information of a structured query, a raw query, a vetted query, a category, a keyword, an ambassador, a searcher, a search result, an advertisement, and/or a user may be recorded and/or modified. Control is passed to operation 305 and process 300 continues.
  • a searcher is selected.
  • An ambassador may select attributes of a searcher to provide a search result responsive to a query or request.
  • An ambassador may be presented with information regarding searchers available to respond to a request.
  • An ambassador may communicate with a user to modify criteria used to select a searcher. For example, an ambassador may determine time, rating, ranking, affiliation, demographic, geographic and/or other criteria for selection of a searcher.
  • an ambassador may determine whether an immediate response is required by a user in order to select a searcher.
  • an ambassador may select a searcher based on information such as a rating of search result which has been provided by a searcher.
  • a searcher may be selected automatically. For example, a searcher may be selected by the search system 130 (Fig. 1 ). Control is passed to operation 350 and process 300 continues.
  • a search is performed by a searcher.
  • a searcher may be a selected searcher.
  • a rating(s) of an ambassador may be determined.
  • a rating(s) of an ambassador may be based on various criteria as described herein above. For example, actions of a selected searcher may influence a rating of an ambassador that responded to a request.
  • a search result is provided by a searcher. Control is passed to operation 355 and process 300 continues.
  • a process 400 for reviewing and/or modifying a search result is provided.
  • the process 400 may be operative on any suitable elements of the system 100.
  • the process 400 is operative on a server associated with the search system 130.
  • a searcher may perform a review of a search result.
  • operation 405 a determination is made as to whether a search result is received. If in operation 405 it is determined that a search result is not received control remains at operation 405 and process 400 continues. If in operation 405 it is determined that a search result is received control is passed to operation 410 and process 400 continues.
  • information of a search result is presented for review by an ambassador who may also be a guide and/or searcher.
  • the guide is a first available ambassador selected by the search system 130 (Fig. 1 ).
  • a guide may be selected based on criteria such as a rating, a skill set of a guide, a capability(ies) of a searcher system such as the ambassador system 145 (Fig. 1 ), or other information indicated in the database 120.
  • Information of a search result may be presented to a guide using a GUI such as the GUI 1300 illustrated in Figure 13. Control is passed to operation 415 and process 400 continues.
  • a guide reviews information of a search result.
  • a review may be based on criteria such as relevance, credibility, completeness, reference type, media type, etc. Control is passed to operation 420 and process 400 continues.
  • a search result is modified and transmitted to a user. Modification of a search result may be performed according to various criteria.
  • a search result is restructured to a standard search result format.
  • An initial structured search result may be presented to an ambassador, based on an automated analysis of a search result.
  • a search result may be modified for various reasons. For example, an ambassador may elect to send only one search result. This may for example be due to a constraint of a user system. For example, if a user is utilizing a messaging system such as SMS or other messaging services such as multimedia messaging service (MMS), a search result may be modified to be suitable for the capabilities of a user device.
  • MMS multimedia messaging service
  • an ambassador may modify a search result to be suitable for voice communication such as VoIP, text-to-speech, or other voice based systems.
  • An ambassador may send a search result to more than one user device. For example, an SMS result may be sent to a text-based user device, and a search result may be posted to a web page of the search system 130 (Fig. 1 ) which may be accessed by a user at a later time, or a voice reply may be transmitted, and a text reply may be sent.
  • An ambassador may determine a format for replying to a user based on interaction with a user. An ambassador may select an advertisement to be sent to a user.
  • An exemplary GUI 1500 for reviewing, modifying and transmitting a response to a user is illustrated in Figure 15 Control is passed to operation 425 and process 400 continues.
  • a rating may be obtained.
  • a user rating of a search result, an ambassador, and/or an overall rating may be obtained.
  • a rating of an ambassador may be determined based on various factors such as time from receipt of a search result by an ambassador until a search result is sent to a user, number of search results sent to a user, number of search results accepted by a user, selection of an advertisement sent by an ambassador to a user, selection of a search result sent to a user by an ambassador, subsequent usage of the search system 130 by a user associated with an ambassador, number of search results requiring the use of a searcher other than the ambassador, etc. Control is passed to operation 430 and process 400 continues.
  • process information is recorded.
  • process information is recorded in the database 120 (Fig. 1 ).
  • content of the request may be recorded.
  • Information associated with processing of a request may be recorded for various purposes. While specific examples of information which may be recorded are used for the purposes of illustration, other information may be recorded, and equivalent ways of storing and accessing which are well known in the art may be used to implement the systems and methods described herein.
  • the raw query record 500 may include a query ID field 505, a keyword field 510, a category field 515, a structured query field 520, a user list field 525, a profile field 530, a search resource ID field 535, a search result ID field 540, and a guide ID field 545.
  • a raw query record may be created by various processes including a guided search session, an interactive training session, a non-interactive training session.
  • a raw query record may be imported from an external resource accessible to the search system, etc.
  • a raw query record is created if a request is received by the search system 130 (Fig. 1 ).
  • a raw query (or original query) is a query which has not been processed.
  • a raw query may include a fully-formed question/sentence, a keyword and/or a search phrase.
  • a raw query may include media, including images, audio, video, etc., and may include information such as a phone number, an IM credential, and/or other data which is associated with a user system and/or communication service.
  • the raw query ID field 505 preferably contains a unique identifier of a raw query, which is preferably used consistently.
  • the raw query ID field 505 can include a randomly generated numerical code, and/or a text string indicating the content of a raw query.
  • a query ID serves to distinguish a raw query record associated with a raw query from a raw query record associated with other raw queries.
  • a unique identifier is assigned to a raw query when it is entered into the database 120 (Fig. 1 ). If it is determined that a raw query is identical to an existing raw query in the search database, that query may be assigned the same query ID and may be added to the raw query record associated with the query ID.
  • a request may include other information than that explicitly provided by a user. For example, location information, profile information, etc., may be associated with a query which might differentiate the query from a raw query in the database 120 (Fig. 1 ).
  • a query may receive a new query ID and create an associated raw query record based on information indicated by a guide.
  • each request or raw query is assigned a unique ID. As illustrated in Fig. 5, the string 'Who is the #1 quarterback for the Bears?' is the query ID for the raw query record 500.
  • the raw query keyword field 510 may include information of a keyword associated with a raw query.
  • Content of the raw query keyword field 510 may be created by an automated processing or parsing of a raw query.
  • a human guide may select a keyword which is associated with a raw query. Processes for making and/or reviewing an association of a keyword with a raw query are further described herein.
  • the keyword 'bears' and the keyword 'quarterback' are associated with the request 'Who is the #1 quarterback for the Bears?' Keywords associated with a request may be ranked, and may be presented to an ambassador in an order based on the ranking. Keywords associated with a request may be used to rank information associated with the request.
  • the raw query category field 515 may include information of a category associated with a raw query.
  • Content of the raw query category field 515 may be created by an automated classification of a raw query.
  • a human guide may select a category which is associated with a raw query. Processes for making and/or reviewing an association of a category with a raw query is further described herein. For example, in the record illustrated in Fig. 5, the categories 'Sports>News>NFL' and 'Sports>NCAA>Football' are associated with the raw query 'Who is the #1 quarterback for the Bears?' A category associated with a raw query may be used to rank information which may be selected responsive to the query.
  • the raw query structured query field 520 may include information of a structured query associated with a raw query.
  • the raw query structured query field 520 may further include rating information of a structured query with respect to a raw query.
  • a ranking or rating of a structured query may affect an order in which a structured query is presented to a guide when constructing a structured query.
  • a structured query may be rated in various ways. For example, a more specific structured query such as 'Who is the current quarterback for the University of California Bears?' may be rated higher than a less specific structured query such as 'Who is the quarterback for the Bears?' Alternately, a structured query may be rated based on frequency of use in a given time interval.
  • a structured query might be rated based on keywords associated with a raw query. For example, if a user has submitted multiple queries which have been associated with a keyword, a structured query associated with that keyword might be rated higher. A structured query may be rated based on a category associated with a raw query. A structured query may be rated based on a combination of ratings. A method for ranking of structured queries associated with a request is further described herein.
  • the raw query user list field 525 may include information of a user that submitted a raw query.
  • the user list field 525 may be used to associate a user with a raw query.
  • a raw query record may be created or modified when a user is associated with a raw query.
  • the user 'BiIH 023' is associated with the raw query 'Who is the #1 quarterback for the Bears?'.
  • other information associated with a user may be presented to a guide.
  • the raw query profile field 530 may include information of a profile associated with a raw query.
  • the raw query profile field 530 may be used to associate a profile, which may contain additional information such as geographic data, demographic data, personality data, time data, location based data, user system data, etc., with a raw query.
  • a raw query record may be created or modified when a profile is associated with a raw query. Using the example illustrated in Fig.
  • the profile 'BiIH 023 profile' has been associated with the raw query 'Who is the #1 quarterback for the Bears?' This may indicate that profile information associated with the user 'BiIH 023' is associated with the raw query 'who is the #1 quarterback for the Bears?'
  • a profile associated with a raw query may be used to modify processing of a query, to rank an item which is to be associated with a raw query or request, to target advertisements, etc.
  • the raw query search resource ID field 535 may include information of a search resource associated with a raw query.
  • Information of a search resource associated with a raw query may be used to locate search resources which may be provided to a guide in order to obtain a search result responsive to a request.
  • a search resource is associated with a category.
  • the resources 'Sporting News', 'NFL Network' and 'Chicago SunTimes' may be associated with 'Sports>News>NFL', and NCAA News' is associated with 'Sports>NCAA>Football'.
  • a resource may have a rating associated with a category, which may be used to rank a resource.
  • a ranking of a resource associated with a category may affect a probability that a resource will be presented responsive to a request or raw query associated with the category.
  • the raw query search result ID field 540 may include information of a search result associated with a raw query.
  • Information of a search result associated with a raw query may be used to provide a search result to a user.
  • the result 'Kyle Orton is the QB - Go Bears' might indicate a text message to be provided to a user via SMS, and the result ⁇ www.nfl.com/kyleorton> might be provided using a web serving functionality to a browser of a user device.
  • Any number of search results may be associated with a request.
  • a search result may be associated with a request automatically and/or using the assistance of a guide.
  • a ranking of a search result associated with a structured query may affect the probability that a search result will be associated with a request associated with the structured query.
  • the raw query guide ID field 545 may include information of a guide associated with a raw query.
  • Information of a searcher associated with a raw query may be used to determine compensation for a guide. For example, a number of queries processed by an ambassador and/or a searcher may be used to determine compensation for the guide. Similarly, if a search result provided by a guide is reused, the guide may be compensated.
  • the guides 'Ambassadori OOO' and 'Searcher22' are associated with 'Who is the #1 quarterback for the Bears?'
  • a vetted query may be used to improve matching of a raw query.
  • a raw query may include errors, slang, anaphora, and/or other forms of natural language information which may be difficult to interpret using automated processing, but which may be readily understood by a person.
  • An ambassador and/or a guide may select a vetted query which is associated with an answer in order to provide a response to a request.
  • a vetted query may unambiguously map to a response, and/or may be used to assist a searcher to obtain a response.
  • a sample of a vetted query record 600 of which one or more may be associated with or resident in the database 120 (Fig. 1 ) is provided.
  • the vetted query record 600 may include information of vetted query record including a vetted query ID field 605, a keyword field 610, a category field 615, a raw query field 620, a user list field 625, a profile field 630, a search result ID field 635, and a searcher ID field 640
  • a vetted query record may be created by various processes, including a guided search session, an interactive training session, a non-interactive training session.
  • a vetted query record may be imported from an external resource accessible to the search system 130 (Fig. 1 ), etc.
  • a vetted query record is created by an ambassador.
  • a vetted query may be associated with a number of search results. If a vetted query is selected, a search result and/or other information associated with the vetted query may be provided responsive to the selection. For example, if a vetted query is selected by an ambassador responsive to a request, a search result associated with the vetted query may be provided to a user and/or the ambassador.
  • the vetted query ID field 605 preferably contains a unique identifier of the vetted query, which is preferably used consistently.
  • the vetted query ID field 605 can include a randomly generated numerical code, and/or a text string indicating the content of the vetted query.
  • a vetted query ID serves to distinguish a vetted query record associated with a vetted query from a vetted query record associated with other vetted queries.
  • a unique identifier may be assigned to a vetted query when it is entered into the database 120 (Fig. 1 ). If it is determined that a vetted query is identical to an existing vetted query in the search database, that vetted query may be assigned the same vetted query ID and information may be added to the vetted query record associated with the vetted query ID.
  • a vetted query may include other information than the explicit content of a request. For example, location information, profile information, etc. may be associated with a vetted query. For example, an identical structured query such as 'Where is a good place for dinner?' might be associated with multiple vetted queries depending on a profile (e.g. vegetarian, Muslim, Malawi) associated with the structured query. Information obtained by an ambassador may be used to differentiate one vetted query from another. As illustrated in Fig. 6, the string 'Who is the quarterback for the Chicago Bears?' is the query ID for the vetted query record 600 (Fig. 6).
  • the keyword field 610 for the vetted query may include information of a keyword associated with a vetted query. Content of the keyword field 610 may be created by an automated processing of a query. Keywords associated with a vetted query may be used to rank the vetted query. For example, if a raw query includes a keyword associated with a vetted query, the vetted query may be ranked higher and may be more likely to be provided to a user, an ambassador, a searcher, etc. For example, synonyms of nouns and/or noun phrases indicated in a vetted query might be indicated in the keyword field 610. Using the example in Fig. 6, the named entity 'Chicago Bears' and the noun 'quarterback' are associated with the vetted query record 600.
  • the category field 615 of the vetted query may include information of a category associated with a vetted query. If category associated with a vetted query is associated with a raw query, the vetted query may be ranked higher and may be more likely to be provided to a user, an ambassador, a searcher, etc. For example, if a category is associated with a raw query automatically, a ranking of a vetted query associated with the category might be used to determine an order in which the vetted query is presented to an ambassador. Using the example in Fig. 6, the categories 'Sports>News>NFL' and 'Sports>People>Chicago' are associated with the vetted query record 600.
  • the raw query field 620 of the vetted query may include information of a raw query which have been associated with a vetted query. It may further include rating information of a raw query with respect to a vetted query.
  • the rating of a raw query associated with a vetted query may determine an order in which a vetted query may be presented to an ambassador when constructing a structured query responsive to a user request for information. For example, if a raw query has been associated with a vetted query by ambassadors fifty percent of the time, the vetted query may be presented higher in a list of suggested queries than a vetted query which has been associated with a raw query by ambassadors twenty percent of the time.
  • a rating may be based on information such as ratings associated with a guide making an association of a raw query with a vetted query, voting by a guide associated with a category, keyword, profile and/or other information associated with a vetted query, etc.
  • a rating of a raw query associated with a vetted query may indicate a probability that a raw query is equivalent to the vetted query.
  • the user list field 625 for the vetted query may include information of a user that submitted a raw query which was associated with a vetted query to the search system 130 (Fig. 1 ).
  • the user list field 625 (Fig. 6) may be used to associate a user with a vetted query.
  • a vetted query record may be created and/or modified when a user is associated with a vetted query.
  • the user 'BiIH 023' may have submitted the raw query 'Who is the #1 quarterback for the Bears?' which is associated with the vetted query 'Who is the quarterback for the Chicago Bears?'.
  • Fig. 6 the user 'BiIH 023' may have submitted the raw query 'Who is the #1 quarterback for the Bears?' which is associated with the vetted query 'Who is the quarterback for the Chicago Bears?'.
  • a phrase such as 'Chicago Bears quarterback' might be a phrase which was not submitted by a user, but which might be used to rank a vetted query versus a raw query.
  • Association of a user with a previous query may be used to associate a raw query submitted by a user with a vetted query. For example, a user who previously submitted the raw query 'Who is the #1 quarterback for the Bears?' which was associated with the vetted query 'Who is the quarterback for the Chicago Bears?' might submit a query such as 'Who was the punter for the Bears in 1960?'. As the previous raw query was associated with the vetted query 'Who is the quarterback for the Chicago Bears?' the new raw query might be associated with the vetted query 'Who was the punter for the Chicago Bears in 1960?' Historical context information may be used automatically and/or may be presented to an ambassador and/or a searcher.
  • the profile field 630 of the vetted query may include information of a profile associated with a vetted query by the search system 130 (Fig. 1 ).
  • the profile field 630 (Fig. 6) may be used to associate a profile which may include information such as geographic data, demographic data, personality data, time data, location based data, user system data, etc. with a vetted query.
  • profile information may differentiate vetted queries which have similar query content. For example, if a raw query is associated with profile information, a ranking of a vetted query may be affected by a match between the profile information associated with the vetted query and the profile information associated with the raw query.
  • a search result which is provided automatically when a vetted query is selected may be customized to a user profile.
  • Profile information associated with a vetted query might also be used to determine information regarding a user.
  • the profile 'Bears Fans' is associated with the vetted query 'Who is the quarterback for the Chicago Bears?'
  • the association of the users 'BiIH 023' and 'Mark1000' with the vetted query record 600 may be used to infer that the users are 'Bears Fans' if multiple vetted queries associated with that profile are associated with the users.
  • a comparison of the profile 'Bears Fans' with profile information associated with 'BiIH 023' and 'Mark1000' may affect a ranking of the vetted query 'Who is the quarterback for the Chicago Bears?' for a query submitted by 'BiIH 023' and/or 'Marki 000'
  • the search result ID field 635 for the vetted query may include information of a search result associated with a vetted query. Search results may be rated in association with a vetted query. For example, a number of guides and/or users may be presented with a query associated with the vetted query and one or more search results and may indicate an opinion regarding the search result.
  • the search results 'Kyle Orton is the QB - Go Bears', ⁇ www.nfl.com/kyleorton>, 'I am in love with Kyle Orton!' and ⁇ www.kyleortonfans.com> are associated with the vetted query 'Who is the quarterback for the Chicago Bears?'
  • a highest rated search result associated with a vetted query may be provided to a user if the vetted query is selected.
  • search results associated with a vetted query may be presented to a guide in an order which may be based on a ranking of the search results.
  • the searcher ID field 640 for the vetted query includes information of a guide associated with a vetted query.
  • a searcher is associated with a search result produced by the searcher. This may for example allow the search system 130 (Fig. 1 ) to track use of search result provided by a searcher. For example, a searcher might provide multiple responses to a vetted query in order that reuse of a search result associated with the searcher may be increased. Likewise, a searcher might provide different results which might be delivered to users using various communication services. For example, a searcher might provide a brief answer as an SMS message, a link to a URL for a web browser, a mixed media message for email delivery, etc.
  • an identifier of an ambassador who has utilized a vetted query might be indicated in the search result ID field 640.
  • the searchers 'Searcher22' and 'Searcher2242' are associated with the vetted query record 600. This may indicate that 'Searcher22' and 'Searcher'2242' have provided a search result associated with the vetted query 'Who is the quarterback for the Chicago Bears?'
  • a raw query record such as the raw query record 500 may be used to track information of a query as it is processed by the search system 130 (Fig. 1 ). For example, when a request is received by the search system 130 a raw query record is created and the query ID field 505 and the raw query user list field 525 and the raw query profile field 530 are populated. A raw query may be processed automatically, and content of the raw query keyword field 510, the raw query category field 515, and the raw query structured query field 520 determined and rated. Automated categorization and keyword weighting may be performed. Matching to a database of structured queries may be performed using a full-text search facility such as Lucene which may perform stemming and fuzzy matching to determine a rating or probability that a raw query matches a structured query as further described herein.
  • a full-text search facility such as Lucene which may perform stemming and fuzzy matching to determine a rating or probability that a raw query matches a structured query as further described herein.
  • a search result may be associated with the raw query.
  • a search result may be associated with the raw query.
  • an ambassador elects to associate the vetted query 'Who is the quarterback for the Chicago Bears?' with the raw query 'Who is the #1 quarterback for the Bears?' that raw query is indicated in the raw query field 620 (Fig. 6).
  • Search results associated with the vetted query record 600 may be associated with the raw query record 500, as indicated in the raw query search result ID field 540.
  • An ambassador may select one or more of the search results associated with a vetted query to be provided to a user. In such an instance, the results selected may be indicated in the raw query search result ID field 540.
  • the searcher 'Searcher 22' is associated with the search results 'Kyle Orton is the QB - Go Bears' and ⁇ www.nfl.com/kyleorton>, as indicated in the vetted query searcher ID field 640 (Fig. 6), the ID of the searcher may be added to the Guide ID field 545 (Fig. 5).
  • An identifier of an ambassador may be added to the raw query record 500 in order that ratings and/or compensation of an ambassador may be determined, as indicated in the raw query guide ID field 545. If no search result is associated with a vetted query, or if a rating of a search result associated with a vetted query is below a predetermined threshold, a query may be assigned to a searcher, as described further herein.
  • a searcher may be provided with search resources associated with one or more categories selected by an ambassador.
  • a vetted query may allow the search system 130 (Fig. 1 ) to provide an automated response to a request. For example, if a vetted query is associated with a search result, if an ambassador selects and/or constructs a vetted query which is determined to match a vetted query in the database a highest ranking search result associated with the vetted query may be provided to a user.
  • a structured query may be used to obtain a dynamic search result automatically. For example, if a user asked 'What did the Dow do today?' an ambassador might provide the structured query 'What is the closing price for the Dow Jones Industrial Average?' which might produce a programmatic search result. Various forms of a natural language questions may be mapped by an ambassador to a vetted query, which may improve utilization of searchers and ambassadors.
  • a process 700 for responding to a user request is illustrated in Fig 7.
  • the process 700 may be operative on any or all elements of the system 100 (Fig. 1 ).
  • the process 700 is operative on a server associated with the search system 130.
  • operation 705 a determination is made as to whether a query (request) is received. If in operation 705 it is determined that a request is not received control remains at operation 705 and process 700 continues. If in operation 705 it is determined that a request is received control is passed to operation 710 and process 700 continues.
  • NLP Natural Language Processing
  • a raw query may have been converted from speech to text in order that NLP may be applied.
  • An algorithm for NLP is further described with respect to Fig. 8 herein below. Control is passed to operation 715 and process 700 continues.
  • operation 715 a determination is made as to whether a request may be answered automatically. If in operation 715 it is determined that a request may be answered automatically control is passed to operation 770 and process 700 continues. If in operation 715 it is determined that a request may not be answered automatically control is passed to operation 725 and process 700 continues. In at least one embodiment, it may be determined that a search result such as an advertisement, etc., may be provided automatically while a search result which might include an answer to a user query, a search resource, etc., might require further processing. In such an instance, control is passed to operation 770 and operation 725 simultaneously.
  • a search result such as an advertisement, etc.
  • an answer reuse function is calculated.
  • a process and functionality for determining a probability that a reusable answer responsive to a user request is available is further described herein with respect to Fig. 9. Control is passed to operation 730 and process 700 continues.
  • operation 730 a determination is made as to whether a reusable answer is available. If in operation 730 it is determined that a reusable answer is available control is passed to operation 770 and process 700 continues. If in operation 730 it is determined that a reusable answer is not available control is passed to operation 740 and process 700 continues.
  • operation 745 a determination is made as to whether an expeditor is required to process a request. If in operation 745 it is determined that an expeditor is required to process a request control is passed to operation 750 and process 700 continues. If in operation 745 it is determined that an expeditor is not required to process a request control is passed to operation 765 and process 700 continues.
  • an expeditor processes a request. Processing of a request by an expeditor is further described herein.
  • An expeditor may categorize a request, may form a structured query responsive to a request, may identify a type of automated processing which may respond to a request, may select an answer to a user request, may designate location, profile, and/or other information associated with a request, etc. Control is passed to operation 755 and process 700 continues.
  • operation 755 a determination is made as to whether an answer is available to a user request. If in operation 755 it is determined that an answer is available to a user request control is passed to operation 770 and process 700 continues. If in operation 755 it is determined that an answer is not available to a user request control is passed to operation 765 and process 700 continues.
  • a searcher performs a search for information and provides a search result.
  • a searcher may be selected based on any suitable criteria.
  • a highest ranking searcher associated with a category associated with a request is selected and is provided with search resources associated with information of the request such as a category, keyword, location, profile, etc. Control is passed to operation 770 and process 700 continues.
  • a search result is provided to a user.
  • a search result may be provided to a user using any communication service associated with a user.
  • a search result which has been provided by an automated resource, stored information, and/or by a human searcher is provided. Control is passed to operation 775 and process 700 continues.
  • operation 775 information of the process 700 is recorded.
  • information of a user, a guide, a request, a resource, a keyword, a category, a vetted query, a profile, etc. which may have been modified by the process 700 may be recorded and/or updated.
  • process information is recorded in the database 120 (Fig. 1 ). Control is passed to operation 705 and process 700 continues.
  • a process 800 for natural language based query formation and improvement is provided.
  • the process 800 may be referred to as the Re-Writing Process (RWP).
  • the process 800 may be operative on any or all elements of the system 100 (Fig. 1 ).
  • the process 800 is operative on a server associated with the search system 130 (Fig. 1 ).
  • a processing such as a signature stripping is performed.
  • Some queries such as SMS, MMS, email, or IM based queries may include template information which must be removed in order to effectively determine user intent. For example, a user may have a signature message which is added to an email or SMS message.
  • an expeditor identifies this content in a first message
  • subsequent messages may have the identified content removed without requiring the use of an expeditor.
  • standard email signatures which contain elements such as phone numbers, etc.
  • CNID calling number identification
  • a comparison of the trailing characters of a number of successive messages may be used to determine if a signature exists. For example if 3 or more successive messages include a common character string at the end of the message, it may be determined that the common characters represent a signature which is to be removed. Control is passed to operation 815 and process 800 continues.
  • language detection is performed.
  • Various types of language detection algorithms may be applied.
  • a technique described in Language detection in unknown, International Conference on Computational Linguistics, Proceedings of the 18th conference on Computational linguistics - Volume 2 Saarbr ⁇ cken, Germany SESSION: Project notes and demos Pages: 1021 - 1025, Year of Publication: 2000 relates to a typical algorithm, ora composite approach to language/encoding detection, S Li, K Momoi - Proc. 19th International Unicode Conference, 2001 - sfr-fresh.com.
  • a unique ID associated with a user may be used to recognize an individual user, such an algorithm might be applied over a query history, and/or may be influenced by factors such as profile information, area code, etc. which may be associated with a user. Control is passed to operation 820 and process 800 continues.
  • a spelling check is performed.
  • a query string may be 'tokenized' or separated into words based on white space indicators. Spelling check is performed using techniques such as word comparison and phrase comparison based on a language detected in operation 815. If language detection results were ambiguous, a spelling check versus more than one dictionary may be performed, and language detection may be repeated based on results of probable spelling corrections.
  • a spelling check may include variances based on a type of device associated with a request. For example, a different correction algorithm might be applied for a query submitted using a 12-digit keypad, than a QWERTY keyboard. Control is passed to operation 825 and process 800 continues. [00146] In operation 825 a grammar check is performed.
  • Grammar check is performed using well known techniques such as an annotation graph, which are well known.
  • an annotation graph For example ATLAS: A flexible and extensible architecture for linguistic annotation, Steven Bird, David Day, John Garofolo, John Henderson, Christophe Laprun, Mark Liberman, submitted on 13 JuI 2000, describes a software tool for producing and using annotation graphs.
  • An annotation graph such as a Penn Tree may be created wherein tokens are associated with parts of speech. Control is passed to operation 830 and process 800 continues.
  • Word substitution is the process of replacing common abbreviations and/or slang forms of speech with an alternate form of a word or words. For example 'b4' might be replaced with 'before', or 'u' might be replaced with 'you', etc. Such a process may include factors which may be tailored to a user based on factors such as profile information (e.g. regional information, demographic, etc.) and/or query history information, etc. Control is passed to operation 835 and process 800 continues.
  • profile information e.g. regional information, demographic, etc.
  • a query is formatted as a structured query.
  • Such a process may be performed by semantic processing of a query.
  • a query may be created by converting a statement into a question, (e.g. 'will it rain today?' might be constructed as 'what is the chance of rain today?')
  • a raw query might be compared to a database of raw queries which have been associated with a succinct query by a guide(s).
  • An annotation graph may be used to format a raw query. Control is passed to operation 840 and process 800 continues.
  • a taxonomy for categorization may be used which is customized in order to index guides who may be associated with keywords, topics, and/or categories. Training of a classification algorithm based on query classification information provided by guides may be used to improve classification performance.
  • a keyword of a query may be compared to keywords associated with an index of categories to determine a category which is associated with a query. For example, if a query includes a number of keywords which are associated with a category, the category may be more likely to be associated with the query. Likewise if queries associated with a category match a query, the category may be more likely to be associated with the query. Control is passed to operation 845 and process 800 continues.
  • automated and/or keyword based queries may be excluded from processing. For example, if a keyword is associated with a particular system response a query which begins with that keyword may be excluded from processing. This might include advertising keywords, and/or other words and/or phrases which are associated with an automated response. Likewise, a query which conforms to a query template after NLP may be selected for an automated response. For example, if a user submitted the query 'wether 46038', the corrected version 'weather 46038' would be automatically processed. Likewise a query such as 'stok nfs' which might be corrected to 'stock nfs' could be automatically processed. Control is passed to operation 850 and process 800 continues.
  • named entity processing is performed.
  • Named entity processing techniques are well known in the art.
  • Introduction to the CoNLL-2002 Shared Task: Language-Independent Named Entity Recognition, author Erik F. Tjong Kim Sang University of Antwerp, 2002 describes a typical process and method for performing named entity processing.
  • the search system 130 (Fig. 1 ) may access a database of named entities which may be included in the database 120, and/or may be an external search resource. Control is passed to operation 855 and process 800 continues.
  • operation 855 information associated with the NLP operations performed in conjunction with the process 800 is recorded.
  • information of ratings of a category associated with a raw query, an annotation graph, named entity tags, etc associated with NLP processing of a raw query may be recorded in the database 120 (Fig. 1 ).
  • results of the processes, the confidence and/or probability factors associated with various versions of a query, etc., which may be shared with other processes and/or may be used to evaluate training and efficacy of a particular parametric tuning, etc. may be recorded. Control is passed to operation 860 and process 800 continues.
  • a query record such as the raw query record 500 may be updated to reflect categorizations, structured queries,, keywords, and/or ratings information determined based on the process 800. Control is passed back to the parent process and process 800 terminates.
  • a search resource, an advertisement, a game, an answer, contact information for a supplier of items or services, etc. may have different quality and/or matching criteria.
  • Quality rating of a resource might be based for example on reliability of providing information.
  • Quality rating of an advertisement might be based on fit with a category and/or a payment by an advertiser.
  • Quality rating of a game might be based on a match to keyword and/or profile information associated with previous purchases.
  • matching to a query may have different criteria for different types of search results. For example a search result such as an advertisement might be provided if matching is less exact, while a search result such as an answer to a question might be provided if matching is more exact.
  • a function which describes a level of matching between a query and a reference query with which a search result is associated is needed.
  • software which implements an evaluation function is referred to as the "Answer Reuse Process" (ARP) and an ARP evaluation function is used to determine whether an answer is to be reused.
  • ARP Answer Reuse Process
  • a process which provides a set of candidate queries which match a reference query and answers associated with the candidate queries is implemented.
  • the "Query Vetting Process" (QVP) compares any query to a database of queries and returns any queries which match the reference query to a given threshold. For example, the QVP might return all queries which matched any word in a query or any queries that match 80% of the words in a query, etc. Techniques such as stemming, etc. may be utilized by the QVP and a matching algorithm may be adjusted to provide a manageable set of queries and associated answers to the ARP. An index of queries may be used to determine matching of queries to a reference query.
  • queries may be considered to be documents within a corpus, and a ranking of a query with respect to a reference query may be determined based on word frequency analysis.
  • Word frequency analysis may for example include a number of times a word appears in a reference query and a number of times the word appears in a query to be ranked.
  • Word frequency analysis may include a weighting based on frequency of a word within a corpus. For example, words such as 'the', 'of, etc. which may appear in a high percentage of queries in a corpus of queries may have a low weighting, while words which appear with low frequency such as nouns, noun phrases, named entities, etc. may have a high weighting.
  • the ARP evaluator function which determines the probability that a search result will be reused is defined as
  • f arp (a x ,q) f q ⁇ ial ⁇ a x )* f match ⁇ q,qj Eq. 1
  • L J J a ⁇ ⁇ ax ' c l is the ARP answer rating for a potential answer (a x ) with respect to the query (q).
  • the answer rating function f arp is normalized such that it as a value between zero and one inclusive. In other words ⁇ * arp ⁇ ⁇ ' ⁇ ' ⁇
  • An answer a max is selected which has the highest value of f arp for the set of answers which are associated with queries which are provided to the ARP by the QVP. If the value of f arp associated with the selected answer a max is below a threshold, the answer will not be reused.
  • the probability that an answer will be reused is a function of both the match between a query (e.g. a vetted query, a raw query, etc.) and the query associated with the answer and the quality rating associated with the answer and its related query.
  • the Answer Quality Function may be determined in various ways. For example, an answer which has been selected by a highly rated searcher may have a high value of fquai(a). An answer which has received a high percentage of positive ratings by users and/or guides may have a high value of f qua ⁇ (a). An answer which has been provided by an automated search may have a low value of f qua ⁇ (a).
  • the AQF may include other variables such as how often a result has been provided to users, time based modification, sponsorship factors, adjustment for characteristics of a searcher, etc.
  • a Query Matching Function is used to determine how closely related a test query is to a reference query.
  • the QMF may be based on any information associated with a query such as a category, a structured query, a raw query, a profile, a location, a language, etc.
  • a QMF acts to normalize a ranking of queries returned by a database query from an index such as that provided by the QVP.
  • the QMF is defined as:
  • the reference query, q is the query which is to be matched from Eq. 1.
  • q is a succinct query which has been produced by the RWP process 800 (Fig.
  • the category set q is the set of categories that have been selected to be associated with
  • q is the set of categories which have been determined by the RWP process 800.
  • the candidate answer a x is an answer which is associated with a set of vetted queries which has been selected for evaluation versus a reference query q.
  • the candidate category set q ° x is the set of categories associated with the query associated with the candidate answer a x .
  • the QMF includes the weighting factors w wb , w pb , and w ac .
  • the components of the match function f m atc h are the Wordbag Function (WBF), the Phrasebag Function (PBF) and the Autocat Function (ACF).
  • WBF is used to compare words and their grammatical functions between two queries.
  • the PBF is used to compare phrases and their grammatical functions between two queries.
  • the ACF is used to compare categories between two queries.
  • the WBF compares the tagged word content of the reference query q and the query associated with the answer a x , q a using the following formula:
  • Q q is the set of tagged tokens in the reference query q.
  • a "tagged token” is a word along with a tag which identifies the part of speech which has been determined from semantic analysis of context of the query. Tagged tokens are considered identical only if both the word content and the part of speech tag are identical.
  • Q q is the set of tagged tokens in the query associated with the answer a x .
  • the Phrasebag Function is similar to the Wordbag Function (WBF) but the PBF utilizes word phrases instead of words.
  • W ? is the set of tagged phrases in the reference query q.
  • a "tagged phrase” is a group of words along with a tag which identifies the part of speech associated with the group which has been determined from semantic analysis of context of the query (e.g. noun phrase, prepositional phrase, etc.). Tagged phrases are considered to be identical if both the phrase content and the associated tag are identical.
  • W ? is the set of tagged phrases in the query associated with the answer a x .
  • the PBF is defined as:
  • the WBF and the PBF may include consideration for synonyms and other word relations as well as the frequency of various tagged phrases.
  • a group of queries provided by the QVP might be indexed based on synonyms, and a WBF and PBF might be calculated based on the queries with the synonyms substituted.
  • a query such as 'what is the most common bike?' might cause the tokens 'most common bicycle', 'most common motorcycle' and 'most common cycle' to be used to retrieve candidate queries from an index.
  • a WBF and PBF calculation might weight a match to the 'most common bike' a higher weight than a match to 'most common bicycle', etc.
  • candidate queries which match the synonym tokens might be rated highly.
  • the Autocategorization Function may be defined as:
  • the value of the ACF is zero. If there are categories in the intersection, the highest rating value of any category in the intersection is selected.
  • the rating value associated with the selected category is modified by the smoothing function f smooth -
  • the WBF, PBF, and ACF may be ignored by setting the associated multiplier to zero. Weighting of the WBF, PBF, and ACF may be adjusted based on whether a person such as an expediter, a user, a searcher, etc., has reviewed a vetted query, category, rewritten query, etc.
  • Actions of guides and users responsive to results produced by NLP processing may be used to tune various elements of the performance of algorithms. For example, data of acceptance of a structured query by expeditors may be used to determine a weighting given to words and/or phrases when constructing a structured query. Similarly, language detection, named entity processing, spelling correction, etc., may be modified based on actions of guides. Feedback activities by guides may for example allow NLP, speech recognition, and/or other forms of automated conversion of requests to be personalized to a user without the need for a user to make such corrections.
  • a process 900 for determining whether a reusable search result as a response to a request is available is provided.
  • the process 900 may be operative on any or all elements of the system 100 (Fig. 1 ).
  • the process 900 is operative on a server associated with the search system 130 (Fig. 1 ).
  • parameters for the reuse evaluation function(s) such as those described herein above are determined. For example if the answer reuse function is to be evaluated (calculated) in operation 725 (Fig. 7) the parameters and/or the decision thresholds may be adjusted due to the lack of human intervention in the processing of the query. Likewise if a guide such as an expeditor, searcher, and/or transcriber has processed a query, parameters and thresholds for query matching and answer reuse and/or automation may be modified. Control is passed to operation 910 and process 900 continues.
  • a list (number) of structured or succinct queries which may be compared to a reference query and associated categories which are associated with a raw query are obtained.
  • the QVP may provide such information to the ARP.
  • candidate succinct queries may be selected based at least in part on content of search results or answers associated with the succinct queries. For example, an index such as a Lucene® index associated with a corpus of succinct queries may be queried and a number of the highest ranked queries associated with a raw query may be obtained. Control is passed to operation 915 and process 900 continues.
  • a ranked list (number) of structured or succinct queries and categories which are associated with a query produced by NLP operations such as those performed in the process 800 are obtained.
  • the QVP may provide such information to the ARP.
  • Succinct queries may be selected based at least in part on content of search results associated with the succinct queries. Control is passed to operation 920 and process 900 continues.
  • the ARP evaluation function is evaluated for the set of answers associated with the structured queries and categories obtained in operation 910 and 915.
  • the queries from operation 910 are weighted differently than the queries from operation 915.
  • answers and/or candidate queries associated with a raw query may be more or less likely to be selected.
  • Information of the ARP evaluator function associated with the candidate answer set is recorded. For example, the ARP evaluator function value for answers associated with the three highest ranked or rated vetted queries may be recorded. Control is passed back to the parent process and process 900 terminates.
  • a process 1000 for determining whether a request is to be directed to an expeditor or ambassador is provided.
  • the process 1000 may be operative on any or all elements of the system 100 (Fig. 1 ).
  • the process 1000 is operative on a server associated with the search system 130 (Fig. 1 ).
  • a function which measures a probability that a category selected to be associated with a request is valid is evaluated.
  • a function such as the ACF described herein above may be utilized.
  • a category confidence function may include factors such as user history, user profile, guides associated with a user, etc.
  • the ACF is applied to a raw query and a succinct query created using the process 800 (Fig. 8). Control is passed to operation 1015 and process 1000 continues.
  • the ACF value is below a predetermined value, it may be determined that an expediter is required to select a category associated with a request, or if an expeditor has previously categorized a query it may be determined that an expediter is not required to select a category.
  • a function (the Auto Guide Probability (AGP) function) which measures a probability that a programmatic (automatic) response from a search resource may be used to respond to a request is evaluated.
  • AGP Auto Guide Probability
  • a query may be analyzed to determine if it may be mapped to an API available to provide information such as weather, sports scores, etc.
  • keywords of a request may be examined in order to determine the probability that a response from an expeditor can be used.
  • a function such as the ARF evaluation function may be applied to a query structure or template associated with a resource and a query.
  • Items such as named entities detected in a query may be used to determine a probability that an expediter may respond to a request using an automated resource or 'finder' which utilizes stored information and/or responds to information provided using an API.
  • the AGP function may be applied to the process step 715 (Fig. 7). Control is passed to operation 1035 and process 1000 continues.
  • a threshold value associated with a probability function such as the AGP may be compared to a computed value of the probability function for a request to determine whether an expediter may provide a response to a request using a resource.
  • a threshold used in operation 1035 may be lower than a threshold used in operation 715, which may be used to vary quality and/or cost of answering a query.
  • a function which measures a matching range between a succinct query and a request is evaluated.
  • a function such as the WBF, PBF and/or ACF may be evaluated to determine if a candidate query, such as a query indicated in the database 120 (Fig. 1 ) matches a user request, which may include a succinct query.
  • a succinct query may exist which is not associated with an answer, and/or which is associated with an answer of low quality. Control is passed to operation 1045 and process 1000 continues.
  • a threshold applied to the value of the function computed in operation 1040 may be used to determine whether a matching succinct query is available.
  • a function which measures (calculates) the sensitivity of a request to location is evaluated. For example, if a user request includes a word which implies location (i.e. exophora) such as 'here', 'there', etc. and/or includes a named entity such as a street name, a city name, etc., a location sensitivity function may be affected by such content. For example, a query matching a user query may not be associated with an answer due to ambiguity which may be resolved based on location information. Likewise, a location sensitivity function may be affected by the presence and/or absence of location information associated with a user associated with a request. For example, if a query is associated with a reliable source of location information, and a query is for generic information, a value of a location sensitivity function may be reduced. Control is passed to operation 1055 and process 1000 continues.
  • operation 1055 a determination is made as to whether a query is location sensitive. If in operation 1055 it is determined that a query is location sensitive control is passed to operation 1020 and process 1000 continues. If in operation 1045 it is determined that a query is not location sensitive control is passed to operation 1060 and process 1000 continues.
  • operation 1060 information of a request is provided to a searcher.
  • a searcher may be selected based on the categorization, succinct query, and/or other information including data associated with the request. It should be noted that the cost of processing a query may be reduced as a searcher may be responding to well formed queries in the searcher's area of interest, rather than responding to queries which may require revision, correction, etc. Control is passed back to the parent process and process 1000 terminates.
  • an expeditor In operation 1020 information of a request is provided to an expeditor.
  • a query has been evaluated to determine the nature of the request which may improve the probability that an expeditor may be selected more effectively.
  • An expeditor may further be provided with a toolset which is adapted to the most likely response type for a query. As such an expeditor may be able to process a request more rapidly and with fewer errors. For example, if an expediter receives a request as a result of the determination in operation 1015, a categorization tool might be provided first, while if an expediter is provided with a query as a result of the determination in operation 1035, a tool for providing an answer using the most probable automated resource may be provided. Control is passed back to the parent process and process 1000 terminates.
  • a process 1100 for qualifying a query by an ambassador is provided.
  • the process 1100 may be performed automatically, or may be performed using the assistance of a guide in any or all operations.
  • the process 1100 may be operative on any or all elements of the system 100 (Fig. 1 ).
  • the process 1100 is operative on a server associated with the search system 130 (Fig. 1 ) and/or the ambassador system 145.
  • operation 1110 a determination is made as to whether a request is received. If in operation 1110 it is determined that a request is not received control remains at operation 1110 and process 1100 continues. If in operation 1110 it is determined that a request is received control is passed to operation 1115 and process 1100 continues. [00195] In operation 1115 information of a query including a category and a structured query is obtained. Any information associated with a query, such as a user history, a raw query, a processing history, etc., may be obtained. In at least one embodiment, information of a request is provided to an ambassador. Control is passed to operation 1120 and process 1100 continues.
  • information of a query is analyzed for compatibility with available resource(s) which may provide a response.
  • a function may be evaluated to determine if a query may be answered by an automated process. For example, if a query includes words, named entities, keywords, etc., such information may be evaluated.
  • information of a query is provided to an expeditor who may evaluate the ability to respond using a resource.
  • An expediter may modify a query, which may be reevaluated automatically and/or when triggered by an expeditor. For example, the AGP function associated with a modified query may be evaluated. Control is passed to operation 1125 and process 1100 continues.
  • a determination is made as to whether a resource may respond to a request (determine whether auto guide is applicable to the request received). If in operation 1125 it is determined that a resource may respond to a request, control is passed to operation 1130 and process 1100 continues. If in operation 1125 it is determined that a resource may not respond to a request, control is passed to operation 1135 and process 1100 continues. For example, if a guide has provided needed information for a resource to respond to a request, it may be determined that a resource may respond to a request.
  • a response to a request is provided by a resource (an answer is provided automatically).
  • a resource an answer is provided automatically.
  • a database, news feed, RSS feed, a translation service, a music recognition service, and/or other system which may provide a response to a user request may provide information to a user.
  • Multiple resources may be used to provide a response to a request.
  • a music lyrics feed and a translation service might be used to respond to the query 'what is the first line of 'Besame Mucho' in English?'
  • rating information of a result provided may be obtained. Control is passed to operation 1105 and process 1100 continues.
  • information of available answer(s) is evaluated with respect to information of a query including to determine suitability of an existing answer to be used to respond to a request.
  • a function such as the ARP function may be evaluated to determine if a query may be answered by a stored result.
  • information of a query is provided to an expeditor who may evaluate the ability to respond using an existing answer.
  • An expeditor may modify a query, which may cause a matching function to be reevaluated automatically and/or when triggered by an expeditor. Control is passed to operation 1140 and process 1100 continues.
  • operation 1140 a determination is made as to whether a stored answer may be used to respond to a request. If in operation 1140 it is determined that a stored answer may be used to respond to a request, control is passed to operation 1145 and process 1100 continues. If in operation 1140 it is determined that a stored answer may not be used to respond to a request, control is passed to operation 1150 and process 1100 continues. For example, if an expeditor selects an answer associated with a previous request, or if an expeditor indicates that an answer associated with a previous request is not suitable the probability that a stored answer may be used to respond to a user request may be increased or decreased.
  • a stored response is provided.
  • an answer stored in the database 120 (Fig. 1 ) may be provided to a user.
  • Rating information may be obtained based on actions of a user and/or a guide. For example if an expeditor has selected an answer to be provided to a user, a ranking and/or rating of the answer may be increased. Control is passed to operation 1105 and process 1100 continues.
  • operation 1150 information of a category associated with a request is confirmed.
  • an ambassador may actively confirm a category associated with a request.
  • a category may be not be confirmed when an expeditor does not modify the category. Control is passed to operation 1155 and process 1100 continues.
  • operation 1155 information of a succinct query associated with a request is confirmed.
  • an ambassador may be required to confirm a succinct query associated with a request. Control is passed to operation 1160 and process 1100 continues.
  • a request and information associated with the request is provided to a searcher.
  • a succinct query, a category, user information, search resources, etc. which may be associated with a request may be provided to a searcher at a searcher system (e.g. the searcher system 105 (Fig. 1 )) using a web server functionality of the search system 130 and a web browser functionality of a searcher system.
  • Control is passed to operation 1105 and process 1100 continues.
  • information associated with the query vetting process 1100 is recorded.
  • the database 120 Fig.
  • a category a succinct query, a response provided by a resource, a stored response provided, an ambassador, a guide, a query, etc., which have been modified by the process 1100 may be recorded.
  • usage information of a resource may be recorded to determine compensation for the resource.
  • use of a stored result may cause a rating of the stored result to be reduced.
  • an amount of time required by an expeditor may be recorded in order to evaluate efficiency of the expeditor. Control is passed to operation 1110 and process 1100 continues.
  • FIG. 12 to 15 An embodiment of tools for creation of a vetted query is illustrated in Figures 12 to 15. The embodiment may be used by an ambassador or expediter to process a request.
  • the GUI 1200 includes a customer information window 1202, an advertisement window 1208, a query building window 1210, a category selection window 1230, a location selection window 1238, a 'Send to Guide' button 1242, a status selection tool 1244, and action controls 1246.
  • the customer information window 1202 may include a last known location indicator 1204 and a recent activity indicator 1206.
  • the last known location indicator 1204 may display a user's most recent geographic location. This information may be used by the search system 130 and/or a guide to facilitate provision of an answer to a user that is relevant to a certain location, area, region, etc.
  • the recent activity indicator 1206 may display a user's most recent requests and/or other context information which may be relevant to a request. For example, the recent activity indicator 1206 may display a user's previous queries, system responses, any advertisements that may have been provided to the user, previous search results, profile information of a user, etc.
  • the advertisement window 1208 may display a selected advertisement based on content of the GUI 1200, which may be transmitted to a user based on actions of a guide. For example, one or more advertisements which have been selected based on content of a user query may be presented in an order determined by the search system 100 (Fig. 1 ). An expeditor may select an appropriate advertisement or other information based on information provided using the GUI 1200, which may be provided to a user as part of a clarification message, during a search by a guide, as an interstitial message, etc.
  • an initial guide or ambassador may use the GUI 1200 to develop a well-formed or structured query from a user submitted query, categorize the query, provide a database response, ask a user for clarification, report abuse to the search system, etc., before passing the well-formed query on to an additional guide, if needed.
  • the query building window 1210 may include query type controls 1212, a session time indicator 1214, a raw query indicator 1216, NLP processed query indicator 1248, a query building text box 1220, interrogative word selection controls 1222, a suggested questions window 1224, suggested question indicators 1226, and a scrolling control 1228.
  • the query building window 1210 may be used by a guide to develop a well-formed query based upon a user submitted query. For example, as illustrated in Figure 12, the query 'What did Buzz Nutter do?' has been submitted.
  • a guide may use the query type controls 1212 to select a GUI which may be used to provide a response to a user request.
  • the query type controls are provided as text links which may cause a GUI such as the GUI's 1400 (Fig. 14) and 1500 (Fig. 15) to be presented. If a raw query has been determined to match a classification of query with a pre-determined probability, a query processing GUI may be selected and presented first to an expeditor.
  • the content of a GUI may be pre-filled with information based on a request. For example, if it has been determined that a query is most likely to be a request for financial information, a GUI associated with the 'Stocks' query type indicator might be provided first to an expediter.
  • Selection of an 'Expedite' query type control 1212a may cause the GUI 1200 to be provided.
  • Selection of a 'Conversational' query type control 1212c may result in a GUI for creating a conversational response being presented.
  • Selection of a 'Weather' query type control 1212e may result in the GUI 1500 (Fig. 15) being presented.
  • a GUI associated a 'Stocks' query type control 1212d, and a 'Directions' query type control 1212f may be selected by activating the respective query type controls.
  • the session time indicator 1214 may display an amount of time that is spent by a guide performing a task using the GUI 1200. Selection of the query type control may be indicated by underlining, and/or any suitable indicator such as color, shading, etc., as is well known in the art.
  • the raw query indicator 1216 may allow a guide to select any element of a query to include in a well-formed query that may be assembled in the query building text box 1220. For example, a guide may choose to select the named entity 'Buzz Nutter' using the raw query indicator 1216.
  • the NLP processed query indicator 1248 may allow a guide to select any element of a succinct query which may have been produced by the search system 130 (Fig. 1 ) to include in a well-formed query that may be assembled in the query building text box 1220.
  • the interrogative word selection controls 1222 may include any number of interrogative words that a guide my select in assembling a well-formed query in the query building text box 1220. For example, if a query is submitted to the search system by a user in a declarative form, a guide may choose to start a well-formed query by activating the 'Who' word selection control 1222a, the 'What' word selection control 1222b, etc.
  • the query building window 1210 may include the suggested questions window 1224.
  • the suggested questions window 1224 may include any number of suggested question indicators 1226 indicating queries which have been determined to match an NLP processed query and/or a raw query.
  • the scrolling control 1228 may be used to view any additional suggested questions that may not be displayed.
  • a guide may select a suggested question indicator 1226 by clicking or otherwise selecting the question. If a guide selects the suggested question indicator 1226c, the text of the associated question (i.e. 'Who was Buzz Nutter') is transferred to the query building text box 1220.
  • the category selection window 1230 may contain the suggested categories window 1232, the categories selection control 1234, and the selected categories window 1236.
  • the suggested categories window 1232 may display any categories determined to be a potential category associated with the request. For example, as illustrated in Figure 12, the categories, 'Science>Astronauts', 'Sports>NFU and ⁇ ntertainment>Movies' are displayed in the suggested categories window 1232 as a potential 'match' to the user submitted query, 'What did Buzz Nutter do?'.
  • a guide may use the categories selection control 1234 to select any category that may not be contained in the suggested categories window 1232.
  • a guide selects a category from the suggested categories window 1232, such selection may be indicated by the addition of the suggested category to the selected category window 1234 as shown by the selected categories 'Sports>NFL' in Figure 12.
  • only one category may be associated with a query at a time.
  • the location selection window 1238 may include a location selection control 1240. If a guide determines that a user submitted query may be location related, a guide may select the location selection control 1240 in which case a subsequent guide may better provide an answer to a request. If the search system 130 (Fig. 1 ) determines that a user query may be location related the location selection control may be set to active by default.
  • the 'Send to a Guide' button 1242 may be selected if a guide has completed the assembly, formation, selection or otherwise of a well-formed query, and completed categorization of the query, or other operations as may be determined by the search system 130 (Fig. 1 ).
  • a guide may select the 'Send to Guide' button to pass the vetted query to another guide, and/or to the search system 130 which may in turn provide an answer to a user request based on a vetted query.
  • a guide may alternatively select from the user controls 1246.
  • the 'Customer Clarify' user control 1246a may be selected to obtain clarification of a user submitted query and/or obtain additional information from a user that may be advantageous in forming a succinct query.
  • the 'Abort' user control 1246b may be used to cancel the current session and close the GUI 1200.
  • the 'Abuse' user control 1246c may be used to report abuse or other prohibited behavior of a user.
  • the status selection tool 1244 may be used to indicate that a guide desires to change the status of the guide to 'Away' and not accept additional queries after completion of the current session.
  • a GUI for selecting an answer 1300 is illustrated in Figure 13.
  • the GUI 1300 may be provided as a 'pop-up' if a structured query from the suggested questions window 1220 (Fig. 12) is selected using the suggested question indicators 1226.
  • the GUI 1300 may be provided to a guide if it is determined that a structured query matches a user query, but the probability of a match is below a pre-determined threshold.
  • the search system 130 (Fig. 1 ) may determine that human judgment is required to confirm that a succinct query indeed matches a user request.
  • an expeditor may select an answer.
  • Such information may be used to modify a rating of the answers associated with the succinct query.
  • the GUI 1300 may be provided if an expeditor has selected the 'send to a guide' control 1242 and the vetted query is a match to a query indicated in a database.
  • the GUI 1300 includes a user query indicator 1302, a succinct query indicator 1312, an answer selection window 1304, answer selection controls 1306, an answer rejection control 1308, and a completion control 1310.
  • the user query indicator 1302 provides information of a user query.
  • the succinct query indicator 1312 provides information of a succinct query associated with a user request.
  • the answer selection controls 1306 may be used to select an answer to be provided responsive to a user request.
  • the answer selection control 1306a selects the answer 'Buzz Nutter was a football player. He was an All-Pro for the Colts in the 1950's.
  • the answer selection control 1306b selects the answer 'Buzz Nutter played center and linebacker.
  • the answer selection control 1306c selects the answer 'Buzz Nutter is a Colts All-Pro.
  • the answer selection controls 1306 may be mutually exclusive radio buttons, or other controls well known in the art.
  • the answer rejection control 1308 may be used to indicate that a guide is unsure if the answers are relevant, if none of the answers are suitable, that a vetted query may be incorrect, or that a query should be sent to a guide for any reason.
  • the completion control 1310 may be used to indicate completion of the selections in the GUI 1300.
  • the GUI 1400 includes a customer information window 1402, an advertisement window 1408, an answer building window 1410, query type controls 1412, a session time indicator 1414, a user query indicator 1416, a suggested query indicator 1430, response type selection tools 1418, a provided answer display window 1420, an answer building text box 1422, a 'Send Answer' button 1424, a status selection tool 1426, and user controls 1428.
  • the customer information window 1402 may include the last known location indicator 1404 and the recent activity indicator 1406.
  • the last known location indicator 1404 may display a user's most recent geographic location. This information may be used by the search system 130 (Fig. 1 ) and/or a guide to facilitate provision of an answer that is relevant to a certain location, area, region, etc.
  • the recent activity indicator 1406 may display a user's most recent use of the search system 130 (Fig. 1 ) and/or other context information which may be relevant to a request. For example, the recent activity indicator 1406 may display a user's previous queries, responses by the search system 130 (Fig. 1 ), any advertisements that may have been provided to the user, previous search results, profile information of a user, etc.
  • the advertisement window 1408 may display a selected advertisement based on content of the GUI 1400, which may be transmitted to a user based on actions of a guide. For example, one or more advertisements which have been selected based on content of a user query may be presented. An expeditor may select an appropriate advertisement and/or other information based on information provided using the GUI 1400, which may be provided to a user as part of a clarification message, during a search by a guide, as an interstitial message, etc.
  • a guide may select the 'Jokes' query type control 1412b to provide the GUI 1400 if the guide determines that a user request may be satisfied by a joke.
  • Selection of the 'Conversational' query type control 1412c may cause a 'Conversational' GUI to be provided.
  • Selection of the 'Weather' query type control 1412e may result in the GUI 4600 (Fig. 46) being presented.
  • a GUI associated with 'Stocks' 1412d, and 'Directions' 1412f may be selected.
  • Any number of query type controls 1412 may be provided in the GUI 1400.
  • a first GUI provided to an expeditor may be determined based on analysis of a query.
  • the answer building window 1410 may include the user query indicator 1416, and the suggested query indicator 1430.
  • the joke type selection tools 1418 may be selected by a guide based on a guide's interpretation of a query in the user query indicator 1416, and/or the suggested query indicator 1430.
  • the user submitted query, 'tell me a funny' may be interpreted by automated processing of a request to be a request for a joke as indicated by the suggested query 'What is a funny joke?'
  • the GUI 1400 may be provided initially to an expeditor instead of the GUI 1200.
  • a guide may select a category of response using the joke type selection tools 1418.
  • the joke category 'Lawyer' has been selected as indicated by the response type selection control 1418e.
  • a guide might select 'Blonde' jokes 1418a, 'Caveman' 1418b, 'One liners' 1418c and 'Religion' 1418d. Any number of joke type selection tools 1418 may be provided.
  • responses associated with the selected category in for example the search system database 120 may be displayed in the provided answer display window 1420.
  • a guide may select any response displayed in the provided answer display window 1420 to submit as a response to a query.
  • a guide may use any part of a provided answer and/or the guide's text as a response to a user submitted query by entering such text into the answer building text box 1422.
  • a guide may not be allowed to enter free form text in the answer building text box 1422.
  • the text remaining indicator 1423 may display the number of characters a guide has remaining to provide an answer in the answer building text box 1422 based on a total number of characters and/or words that a guide is allowed to use in building an answer.
  • the number of characters may be based on a user device constraint, or any system constraints.
  • the 'Send Answer' button 1424 may be selected if a guide has completed the assembly, formation, selection or otherwise an answer in response to a query.
  • a guide may alternatively select from the user controls 1428.
  • the 'Customer Clarify' user control 1428a may be selected to obtain clarification of a user submitted query and/or obtain additional information from a user that may be advantageous in forming an answer.
  • the 'Abort' user control 1428b may be used to cancel the current session and close the GUI 1400.
  • the 'Abuse' user control 1428c may be used to report abuse or other prohibited behavior of a user.
  • the status selection tool 1426 may be used to indicate that a guide desires to change the status of the guide to 'Away' and not accept additional queries after completion of the current session.14
  • the GUI 1500 includes a customer information window 1502, an advertisement window 1508, an answer building window 1510, query type controls 1512, a session time indicator 1514, a user submitted query indicator 1516, a suggested query indicator 1530, response type selection tools 1518, a provided answer display window 1520, an answer building text box 1522, a 'Send Answer' button 1524, a status selection tool 1526, and user controls 1528.
  • the customer information window 1502 may include the last known location indicator 1504 and the recent activity indicator 1506.
  • the last known location indicator 1504 may display a user's most recent geographic location. This information may be used by the search system and/or a guide to facilitate provision of a query answer to a user that is relevant to a certain location, area, region, etc.
  • the recent activity indicator 1506 may display a user's most recent use of the search system 130 (Fig. 1 ) and/or other context information which may be relevant to a request. For example, the recent activity indicator 1506 may display a user's previous queries, responses by the search system, any advertisements that may have been provided to the user, previous search results, profile information of a user, etc.
  • the advertisement window 1508 may display a selected advertisement based on content of the GUI 1500, which may be transmitted to a user based on actions of a guide. For example, one or more advertisements which have been selected based on content of a user query may be presented in an order determined by the search system 130 (Fig. 1 ). An expeditor may select an appropriate advertisement and/or other information based on information provided using the GUI 1500, which may be provided to a user as part of a clarification message, during a search by a guide, as an interstitial message, etc.
  • a guide may select the 'Jokes' query type control 1512b to provide the GUI 1400 if the guide determines that a user request may be satisfied by a joke.
  • Selection of a 'Conversational' query type control 1512c may result in a GUI for creating a conversational response being presented.15.
  • Selection of the 'Weather' query type control 1512e may result in the GUI 1500 (Fig. 15) being presented.
  • a GUI associated with 'Stocks' 1512d, and 'Directions' 1512f may be selected. Any number of query type controls 1512 may be provided in the GUI 1500.
  • a first GUI provided to an expeditor may be determined based on analysis of the user query.
  • the answer building window 1510 displays the user query indicator 1516, and the suggested query 1530.
  • the weather type selection tools 1518 may be selected by a guide based on a guide's interpretation of a query in the user query indicator 1516, and/or the suggested query indicator 1530.
  • the user submitted query 'will it be nice today?' may be interpreted to be a request for a weather information as indicated by the suggested query 'What is weather for Miami, FL?', which is at least in part determined by the location information indicated in the customer information window 1502.
  • the GUI 1500 may be provided to an expeditor first instead of the GUI 1200 (Fig. 12).
  • a guide may select a category of response using the weather type selection tools 1518.
  • the weather category 'Current' has been selected as indicated by the weather type selection control 1518a.
  • a guide might select the '1 -Day' 1518b, Tomorrow' 1518c, 'Week' 1518d and 'Extended' 1518e weather type selection tools. Any number of weather type selection tools 1518 may be provided.
  • responses associated with the selected type may be displayed in the provided answer display window 1520.
  • a guide may select any response displayed in the provided answer display window 1520 to submit as a response to a query.
  • a guide may use any part of a provided answer and/or the guide's text as a response to a user submitted query by entering such text into the answer building text box 1522.
  • a guide may not be allowed to enter free form text in the answer building text box 1522.
  • the text remaining indicator 1523 may display the number of characters a guide has remaining to provide in the answer building text box 1522 based on a total number of characters and/or words that a guide is allowed to use in building an answer. The number of characters may be based on a user device constraint, or any system constraints. In the case of a request which may be processed using an automated resource, an answer may be provided without requiring selections by a guide in order to optimize the time required to respond.
  • the 'Send Answer' button 1524 may be selected if a guide has completed the assembly, formation, selection or otherwise an answer in response to the user submitted query.
  • a guide may alternatively select from the user controls 1528.
  • the 'Customer Clarify' user control 1528a may be selected to obtain clarification of a user submitted query and/or obtain additional information from a user that may be advantageous in forming an answer.
  • the 'Abort' user control 1528b may be used to cancel the current session and close the GUI 1500.
  • the 'Abuse' user control 1528c may be used to report abuse or other prohibited behavior of a user.
  • the status selection tool 1526 may be used to indicate that a guide desires to change the status of the guide to 'Away' and not accept additional queries after completion of the current session..
  • Efficient utilization of human assistants or guides in a human assisted search system is an important element of success. Better utilization of the expeditor and searcher is desirable, but may be elusive.
  • the system and method disclosed herein combines the well proven expeditor concept with the use of Natural Language Processing NLP to further enhance system efficiency.
  • An NLP process is implemented to ensure that a request is processed using machine based processes which may improve the quality of the content of a request.
  • NLP is used to convert a user request into a suggested structured query which may be compared to a database of queries.
  • NLP is used to create estimation and matching functions which may be used to determine the probability that a response to a request is available. NLP is further used to automatically categorize a query to match with a system index. Results of NLP are used to determine whether a query may be answered automatically, may be routed to an expeditor, or may be routed to a searcher.
  • NLP processing is implemented in a multi-layered approach.
  • an unprocessed request or 'raw query' is passed through a series of automated processes which may improve the readability of a query, correct spelling, tokenize the query, detect named entities, determine locations, produce an annotation graph, etc., prior to the query being compared to a database of queries.
  • a target query is compared to a database of reference queries to determine whether a match to the query is found. If a reference query which is an exact match to a target query is found, a quality function associated with any answers associated with the matching query is evaluated, and is used to determine whether the answer should be automatically sent, and/or should be presented to a guide (e.g. and expediter) to determine whether the answer should be sent responsive to the target query. [00237] If a reference query is not found which exactly matches a target query, reference queries which are highest ranked based on an index match to the target query may be evaluated to determine if the queries are a sufficiently close match to a target query.
  • reference queries which have similar scores from the index may be compared to each other to determine whether any answers associated with the reference queries may be evaluated if a reference query is determined to be a suitable match to a target query. This may allow a higher quality answer associated with a lower ranked reference query to be provided responsive to a request including the target query.
  • Automated analysis of a query may be used to determine whether a request is to be presented to an expediter guide, and/or may be presented to a searcher guide directly.
  • Information such as query length, query category, keywords of a query, etc., may be used to determine whether a request is provided to an expediter, and/or may determine a toolset and/or an order of presentation of a toolset to a guide.
  • an expediter may modify a query associated with the request and the modified query may be used as a target query for NLP analysis of the request.
  • An expediter may be presented with results of comparisons to a revised target query, which may allow an expeditor to select and/or review a matching query and/or answer.
  • NLP processing may be adjusted based on whether a guide has reviewed a request.
  • a GUI presented to an expeditor may be based at least in part on evaluation of the type of response which an expeditor may need to provide to the query. This may significantly improve efficiency of the expeditor process resulting in better user experience.
  • An expeditor may select an advertisement to be provided to a user, which may significantly improve targeting of an advertisement, which may be highly valuable as an advertiser may obtain the exclusive attention of a user. Further a well targeted advertisement may be provided while a user is waiting for a search result.
  • the system described may increase fully automated responses as a user query may be standardized using algorithmic preprocessing which may increase matching to a database of reference queries.
  • An expediter is not required to correct common errors in punctuation, etc. which may be easily corrected automatically.
  • An expediter may modify a query to conform to a standardized query or otherwise, which may improve a match to a reference query. If a target query is matched by a guide to a reference query, information of the matching may cause the target query to be used as a reference query subsequently.
  • Use of NLP algorithms has been demonstrated to greatly improve automated and human-assisted matching of a user request to answers.
  • the median length of queries answered automatically using the new system may be equal to the median length of queries answered by human expediters given an equivalent size corpus of reference queries.
  • a percentage of queries requiring a search may be reduced by as much as eighty percent using the methods and systems described herein.
  • any or all of the operations described herein may be implemented via one or more hardware components. However, the present invention is not limited to any specific implementation of an operation. For example, one or more operations discussed herein may be implemented via software executed on a device while others may be executed via a specific hardware device.
  • the present invention may be implemented using a program stored, for example, in a computer-readable storage medium such as a CD-ROM, etc., or using one or more specialized terminals, devices or systems that is enabled to execute operation(s) described herein.
  • a computer-readable storage medium such as a CD-ROM, etc.
  • the storage or recording medium used in an embodiment can be selected from among various computer-readable media including, a disk, a DVD, an internal storage device (memory such as RAM or ROM) in a computer, etc.
  • the embodiments can be implemented in computing hardware (computing apparatus) and/or software, such as (in a non-limiting example) any computer that can store, retrieve, process and/or output data and/or communicate with other computers.
  • the results produced can be displayed on a display of the computing hardware.
  • a program/software implementing the embodiments may be recorded on computer-readable media comprising computer-readable recording media.
  • the program/software implementing the embodiments may also be transmitted over transmission communication media. Examples of the computer- readable recording media include a magnetic recording apparatus, an optical disk, a magneto- optical disk, and/or a semiconductor memory (for example, RAM, ROM, etc.).
  • Examples of the magnetic recording apparatus include a hard disk device (HDD), a flexible disk (FD), and a magnetic tape (MT).
  • Examples of the optical disk include a DVD (Digital Versatile Disc), a DVD- RAM, a CD-ROM (Compact Disc - Read Only Memory), and a CD-R (Recordable)/RW.
  • An example of communication media includes a carrier-wave signal.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

L’invention concerne un procédé et un système visant à améliorer l’utilisation d’un ou de plusieurs chercheurs humains dans un système de recherche, consistant à utiliser un traitement tel qu’un traitement automatique du langage naturel pour améliorer l’aptitude du système de recherche à reconnaître une intention d’une demande d’utilisateur, fournir des informations à un expéditeur pour répondre à des requêtes, et transmettre automatiquement une requête à un chercheur pertinent et qualifié. Une demande fait l’objet d’un traitement automatique du langage naturel pour améliorer l’usage de réponses automatisées, permettre une utilisation plus efficace de l’expéditeur et une utilisation plus efficace de spécialistes de recherche.
PCT/US2009/066406 2008-11-21 2009-12-02 Procédé et système visant à améliorer l’utilisation de chercheurs humains WO2010060117A1 (fr)

Applications Claiming Priority (8)

Application Number Priority Date Filing Date Title
US12/275,864 2008-11-21
US12/275,864 US8301651B2 (en) 2007-11-21 2008-11-21 Method and system for improving utilization of human searchers
US11920508P 2008-12-02 2008-12-02
US61/119,205 2008-12-02
US12/434,385 US8719256B2 (en) 2008-05-01 2009-05-01 Method and system for improvement of request processing
US12/434,385 2009-05-01
US12/629,500 US20100138402A1 (en) 2008-12-02 2009-12-02 Method and system for improving utilization of human searchers
US12/629,500 2009-12-02

Publications (1)

Publication Number Publication Date
WO2010060117A1 true WO2010060117A1 (fr) 2010-05-27

Family

ID=42198560

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2009/066406 WO2010060117A1 (fr) 2008-11-21 2009-12-02 Procédé et système visant à améliorer l’utilisation de chercheurs humains

Country Status (1)

Country Link
WO (1) WO2010060117A1 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2012154992A3 (fr) * 2011-05-10 2013-01-17 Decarta Systèmes et procédés d'exécution d'une recherche et d'une récupération de documents électroniques à l'aide d'un index de grande taille
US9165074B2 (en) 2011-05-10 2015-10-20 Uber Technologies, Inc. Systems and methods for performing geo-search and retrieval of electronic point-of-interest records using a big index
CN109446310A (zh) * 2018-10-30 2019-03-08 腾讯科技（武汉）有限公司 一种问句模板的质量评估方法、装置及存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040024752A1 (en) * 2002-08-05 2004-02-05 Yahoo! Inc. Method and apparatus for search ranking using human input and automated ranking
US20040243560A1 (en) * 2003-05-30 2004-12-02 International Business Machines Corporation System, method and computer program product for performing unstructured information management and automatic text analysis, including an annotation inverted file system facilitating indexing and searching
US20060117003A1 (en) * 1998-07-15 2006-06-01 Ortega Ruben E Search query processing to identify related search terms and to correct misspellings of search terms
US20060190436A1 (en) * 2005-02-23 2006-08-24 Microsoft Corporation Dynamic client interaction for search
US20070011154A1 (en) * 2005-04-11 2007-01-11 Textdigger, Inc. System and method for searching for a query
US20070174244A1 (en) * 2006-01-23 2007-07-26 Jones Scott A Scalable search system using human searchers

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060117003A1 (en) * 1998-07-15 2006-06-01 Ortega Ruben E Search query processing to identify related search terms and to correct misspellings of search terms
US20040024752A1 (en) * 2002-08-05 2004-02-05 Yahoo! Inc. Method and apparatus for search ranking using human input and automated ranking
US20040243560A1 (en) * 2003-05-30 2004-12-02 International Business Machines Corporation System, method and computer program product for performing unstructured information management and automatic text analysis, including an annotation inverted file system facilitating indexing and searching
US20060190436A1 (en) * 2005-02-23 2006-08-24 Microsoft Corporation Dynamic client interaction for search
US20070011154A1 (en) * 2005-04-11 2007-01-11 Textdigger, Inc. System and method for searching for a query
US20070174244A1 (en) * 2006-01-23 2007-07-26 Jones Scott A Scalable search system using human searchers

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2012154992A3 (fr) * 2011-05-10 2013-01-17 Decarta Systèmes et procédés d'exécution d'une recherche et d'une récupération de documents électroniques à l'aide d'un index de grande taille
US9165074B2 (en) 2011-05-10 2015-10-20 Uber Technologies, Inc. Systems and methods for performing geo-search and retrieval of electronic point-of-interest records using a big index
EP2947585A1 (fr) * 2011-05-10 2015-11-25 DeCarta Inc. Systèmes et procédés pour effectuer une recherche et une extraction de documents électroniques à l'aide d'un indice de grande taille
US9396276B2 (en) 2011-05-10 2016-07-19 Uber Technologies, Inc. Key-value database for geo-search and retrieval of point of interest records
EP3115913A1 (fr) * 2011-05-10 2017-01-11 Uber Technologies, Inc. Systèmes et procédés pour effectuer une recherche et une extraction de documents électroniques à l'aide d'un indice de grande taille
US9646108B2 (en) 2011-05-10 2017-05-09 Uber Technologies, Inc. Systems and methods for performing geo-search and retrieval of electronic documents using a big index
US10198530B2 (en) 2011-05-10 2019-02-05 Uber Technologies, Inc. Generating and providing spelling correction suggestions to search queries using a confusion set based on residual strings
US10210282B2 (en) 2011-05-10 2019-02-19 Uber Technologies, Inc. Search and retrieval of electronic documents using key-value based partition-by-query indices
CN109446310A (zh) * 2018-10-30 2019-03-08 腾讯科技（武汉）有限公司 一种问句模板的质量评估方法、装置及存储介质
CN109446310B (zh) * 2018-10-30 2020-11-03 腾讯科技（武汉）有限公司 一种问句模板的质量评估方法、装置及存储介质

Similar Documents

Publication Publication Date Title
US20100138402A1 (en) Method and system for improving utilization of human searchers
US11698908B2 (en) Content inversion for user searches and product recommendations systems and methods
AU2018383346B2 (en) Domain-specific natural language understanding of customer intent in self-help
US10567329B2 (en) Methods and apparatus for inserting content into conversations in on-line and digital environments
US20110106617A1 (en) Method and system of processing a query using human assistants
US8782069B2 (en) Method and system of providing a search tool
US11757807B2 (en) Interactive chatbot for multi-way communication
US8868590B1 (en) Method and system utilizing a personalized user model to develop a search request
US8112436B2 (en) Semantic and text matching techniques for network search
US20190220460A1 (en) Searchable index
US9444772B2 (en) Social search engine
US7403938B2 (en) Natural language query processing
US8935192B1 (en) Social search engine
US8788476B2 (en) Method and system of triggering a search request
US7099855B1 (en) System and method for electronic communication management
US20150178392A1 (en) Method and system of providing a search tool
US10586174B2 (en) Methods and systems for finding and ranking entities in a domain specific system
US11482223B2 (en) Systems and methods for automatically determining utterances, entities, and intents based on natural language inputs
US20160180237A1 (en) Managing a question and answer system
US20140344261A1 (en) Method and system for analyzing a request
US20150178390A1 (en) Natural language search engine using lexical functions and meaning-text criteria
US20140214820A1 (en) Method and system of creating a seach query
US20110072023A1 (en) Detect, Index, and Retrieve Term-Group Attributes for Network Search
US9940354B2 (en) Providing answers to questions having both rankable and probabilistic components
US20130166489A1 (en) System and method for using a statistical classifier to score contact entities

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 09828402

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 09828402

Country of ref document: EP

Kind code of ref document: A1

点击 这是indexloc提供的php浏览器服务，不要输入任何密码和下载