Computer program product and a method for using natural language for the description, search and retrieval of multi-media objects

Abstract

A method for using natural language for the description, search and retrieval of multi-media objects, this method comprises receiving a query for the database using natural language; recognizing the syntactic and semantic structure of the query; assigning a role representation of the query for permitting searching an archival database for a query response.

Claims

We claim: 1. A computer program product for retrieving a multi-media object, comprising: a computer readable storage medium having a computer program stored thereon for performing the steps of: (a) receiving a query composed in natural language for searching a database; (b) recognizing syntactic and semantic structure of the query; (c) based on step (b), assigning a role representation to the query, which said role representation further comprises assigning agent role that indicates an actor role who is performing an action, an action role that identifies an activity that is being performed, and a patient role that identifies an object affected in some way by the action, for permitting the search of the multimedia object database to generate a query response. 2. The computer program product as in claim 1, wherein step (b) includes recognizing word or phrases that do not contribute to the meaning of the query. 3. The computer program product as in claim 1 further comprising assigning either a location role that identifies any place name; a prepositional role; or an agent attribute role, patent attribute role, action attribute role or location attribute role. 4. The computer program product as in claim 1, wherein step (b) includes using prepositional phrases to enhance the query representation. 5. A computer program product for archiving both a multi-media object and an associated description composed in a natural language, comprising: a computer readable storage medium having a computer program stored thereon for performing the steps of: (a) receiving the description composed in natural language to be associated with the multi-media object; (b) recognizing the syntactic and semantic structure of the of the description; (c) based on step (b), assigning a role representation of the description, which said role representation further comprises assigning agent role that indicates an actor who is performing an action, an action role that identifies an activity that is being performed, and a patient role that identifies an object affected in some way by the action, for storage in an archival database. 6. The computer program product as in claim 5, wherein step (b) includes recognizing word or phrases that do not contribute to the meaning of the metadata. 7. The computer program product as in claim 5 further comprising assigning either a location role that identifies any place name; a prepositional role; or an agent attribute role, patent attribute role, action attribute role or location attribute role. 8. The computer program product as in claim 5, wherein step (b) includes using prepositional phrases to enhance the metadata representation. 9. A method for retrieving a multi-media object, comprising the steps of: (a) receiving a query composed in natural language for searching a database; (b) recognizing syntactic and semantic structure of the query; (c) based on step (b), assigning a role representation to the query, which said role representation further comprises assigning agent role that indicates an actor role who is performing an action, an action role that identifies an activity that is being performed, and a patient role that identifies an object affected in some way by the action, for permitting the search of the multimedia object database to generate a query response. 10. The method as in claim 9, wherein step (b) includes recognizing word or phrases that do not contribute to the meaning of the query. 11. The method as in claim further comprising assigning either a location role that identifies any place name; a prepositional role; or an agent attribute role, patent attribute role, action attribute role or location attribute role. 12. The method as in claim 9, wherein step (b) includes using prepositional phrases to enhance the query representation. 13. A method for archiving both a multi-media object and a associated description composed in a natural language, comprising the steps of: (a) receiving the description composed in natural language to be associated with the multi-media object; (b) recognizing the syntactic and semantic structure of the of the description; (c) based on step (b), assigning a role representation of the description, which said role representation further comprises assigning agent role that indicates an actor who is performing an action, an action role that identifies an activity that is being performed, and a patient role that identifies an object affected in some way by the action, for storage in an archival database. 14. The method as in claim 13, wherein step (b) includes recognizing word or phrases that do not contribute to the meaning of the metadata. 15. The method as in claim 13 further comprising assigning either a location role that identifies any place name; a prepositional role; or an agent attribute role, patent attribute role, action attribute role or location attribute role. 16. The method as in claim 13, wherein step (b) includes using prepositional phrases to enhance the metadata representation. 17. A computer program product for retrieving a multi-media object, comprising: a computer readable storage medium having a computer program stored thereon for performing the steps of: (a) receiving a query composed in natural language for searching a database; (b) recognizing syntactic and semantic structure of the query; (c) based on step (b), assigning a role representation to the query, which said role representation further comprises assigning agent that indicates an actor who is performing an action, an action that identifies an activity that is being performed, and a patient that identifies an object affected in some way by the action, for permitting the search of the multimedia object database to generate a query response; (d) comparing the query roles to an archived multimedia object description roles to identify matching roles; and (e) computing a quality of matched between the query and the archived multimedia object description based on the matching roles. 18. The computer program product as in claim 17, wherein step (b) includes recognizing word or phrases that do not contribute to the meaning of the query. 19. The computer program product as in claim 17 further comprising assigning either a location role that identifies any place name; a prepositional role; or an agent attribute role, patent attribute role, action attribute role or location attribute role. 20. The computer program product as in claim 17, wherein step (b) includes using prepositional phrases to enhance the query representation. 21. A method for retrieving a multi-media object, comprising the steps of: (a) receiving a query composed in natural language for searching a database; (b) recognizing syntactic and semantic structure of the query; (c) based on step (b), assigning a role representation to the query which said role representation further comprises assigning agent role that indicates an actor who is performing an action, an action role that identifies an activity that is being performed, and a patient role that identifies an object affected in some way by the action, for permitting the search of the multimedia object database to generate a query response; (d) comparing the query roles to an archived multimedia object description roles to identify matching roles; and (e) computing a quality of matched between the query and the archived multimedia object description based on the matching roles. 22. The computer program product as in claim 21, wherein step (b) includes recognizing word or phrases that do not contribute to the meaning of the query. 23. The computer program product as in claim 21 further comprising assigning either a location role that identifies any place name; a prepositional role; or an agent attribute role, patent attribute role, action attribute role or location attribute role. 24. The computer program product as in claim 21, wherein step (b) includes using prepositional phrases to enhance the query representation.
FIELD OF THE INVENTION The invention relates generally to the field of archival and retrieval of multi-media objects such as still images, audios, videos, graphics, computer generated graphics, drawings and the like and specifically, to archiving and retrieving multi-media objects using a natural language, such as English. Multi-media objects carry a great deal of information and as multi-media technology is growing, there has been an increasing demand for a system that allows user to easily describe, archive, search and retrieve these multi-media objects. Some conventional methods and their limitations are described as follows. In the past, people have used shoe boxes and albums to archive their images and then search and retrieval of these images is performed based on user's memory. Stock agencies have used index cards to keep track of stock images and search and retrieval is done using personnel experiences and preferences. Such methods of archiving and retrieving images are difficult, time-consuming and expensive. These methods can also produce imperfect results since these methods are subjective in nature. As computers became popular and more and more images were stored on-line, keyword based approach was developed. Keyword representations can be created either manually or automatically. In the manual approach, a set of keywords are assigned to each image in the database. The keywords describe the image content of interest (i.e. objects, events, concepts, place, activities, etc.) The KODAK PICTURE EXCHANGE (KPX) uses this approach. A shortcoming of this approach is that a multi-media object, in this instance images, can not always be described by a disjoint set of keywords, image retrieval depends on an exact match of a keyword used in the description and in the search, and the keywords used to describe/retrieve an image may change from user to user. Some incremental improvements can be made to this method by use of a thesaurus. In the automatic approach, keywords are selected from within the document itself based on statistics pertaining to the relative frequency of word occurrence. This approach is more suitable for document retrieval applications where the large amount of text is available to obtain accurate statistics, such as in the area of newspaper article retrieval. Many text retrieval engines have been developed using this approach. However, in the case of images, the caption will typically be a sentence or two, not enough to extract meaningful statistics from. Another limitation of the keyword based technique for image retrieval is that only the words, and not their meaning or context is taken into account. This makes this technique unsuitable for applications that contain sparse amount of text to describe an image. Images can also be searched and retrieved using Image Content Analysis techniques. Image content attributes are defined using color, texture, shape etc. Some of the existing systems that perform image content analysis are QBIC from IBM, Virage from Virage. The drawback of this approach is it only allows for image similarity type search and retrieval, that is responding to queries of the form "Find me images like this one . . . ". University of Buffalo has developed a system called PICTION which uses natural language captions to label human faces in an accompanying newspaper photograph. A key component of the system is the utilization of spatial and characteristic constraints (derived from caption) in labeling face candidates (generated by a face locator). The system is limited to only identifying faces based upon the spatial constraints defined in the caption, for example "John Doe is to the left of Jane Doe . . . ". Anil Chakravarthy at MIT has developed a program as part of his thesis "Information Access and Retrieval with Semantic Background Knowledge" for retrieving captions of pictures and video clips using natural language queries. He presents a limited framework for structured representation through the incorporation of semantic knowledge. However, the program only accepts images accompanied by well formed single sentence description. Queries also need to be well formed single sentence description. U.S. Pat. No. 5,493,677 discloses a natural language archival and retrieval system for images. This patent discloses inputting a search query in a natural language and then searching for archived images. It identifies name, location and noun phrases from the query. Other words are eliminated. For example, prepositions are not used for further processing. This eliminates the context of some sentences and may give inaccurate results during retrieval, for example, the difference between the two phrases, "A man on a horse." and "A man and a horse." In addition, when inputting information that is to be associated with an image into the database, it has to be specified in a standardized form. The user is involved for part-of-speech disambiguation and word-sense disambiguation. This is time consuming and labor intensive. Consequently, a need exist for a smart archival and retrieval system to eliminate the above-described drawbacks. SUMMARY OF THE INVENTION The present invention is directed to overcoming one or more of the problems set forth above. Briefly summarized, according to one aspect of the present invention, the invention is directed to a computer program product for retrieving a multi-media object, comprising computer readable storage medium having a computer program stored thereon for performing the steps of: (a) receiving a query for the database using natural language; (b) recognizing the syntactic and semantic structure of the query; (c) based on step (b), assigning a role representation of the query for permitting searching an archival database for a query response. It is an object of the present invention to provide a "smart" archival and retrieval system so that syntactic and semantic formalities in the search query are recognized and utilized. It is a further object of the present invention to provide an efficient and user-friendly means for inputting information into the database, requiring minimal interaction from the user. It is a feature of the present invention to assign a role representation of the query for permitting searching an archival database for a query response. It is an advantage of the present invention to provide means for archival and retrieval of images that is free of grammar restrictions. The above and other objects of the present invention will become more apparent when taken in conjunction with the following description and drawings wherein identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a diagram of a computer system on which the present invention may be utilized; FIG. 2 is a flowchart of the software program of the present invention for inputting data that is to be associated with an image using natural language; FIG. 3 is a detailed flowchart of a portion of FIG. 2; FIG. 4 is also a detailed flowchart of a portion of FIG. 2; FIG. 5 is a flowchart of the software program of the present invention for querying the database for searching and retrieving one or more desired images using natural language; and FIG. 6 is a detailed flowchart of a portion of FIG. 5. DETAILED DESCRIPTION OF THE INVENTION In the following description, the present invention will be described in the preferred embodiment as a software program. Those skilled in the art will readily recognize that the equivalent of such software may also be constructed in hardware. The preferred embodiment may be utilized for any multi-media object but the preferred embodiment is illustrated using images. The preferred embodiment uses text input for the natural language. It is instructive to note that speech input or input from a communication or a multi-media capture storage device could also be used, for example an image/video camera and video phone. Still further, as used herein, computer readable storage medium may comprise, for example; magnetic storage media such as a magnetic disk (such as a floppy disk) or magnetic tape; optical storage media such as an optical disc, optical tape, or machine readable bar code; solid state electronic storage devices such as random access memory (RAM), or read only memory (ROM); or any other physical device or medium employed to store a computer program. The below-described invention is illustrated using the English language as the preferred natural language. However, the present invention is not limited to the English, but may be used with any other natural language as will be obvious to those skilled in the art. Referring to FIG. 1, there is illustrated a computer system 10 for implementing the present invention. Although the computer system 10 is shown for the purpose of illustrating a preferred embodiment, the present invention is not limited to the computer system 10 shown, but could be a distributed system such as client-server or a web application. Images can be fed directly from a capture device or from the internet or some other live feed. The computer system 10 includes a microprocessor-based unit 20 for receiving and processing software programs and for performing other processing functions. A display 30 is electrically connected to the microprocessor-based unit 20 for displaying user-related information associated with the software. A keyboard 40 is also connected to the microprocessor-based unit 20 for permitting a user to input information to the software. As an alternative to using the keyboard 40 for input, a mouse 50 may be used for moving a selector 52 on the display 30 and for selecting an item on which the selector 52 overlays, as is well known in the art. A compact disc-read only memory (CD-ROM) 55 is connected to the microprocessor-based unit 20 for receiving software programs and other information, such as images, and for providing a means of inputting them into the microprocessor-based unit 20. A compact disc 57 stores the software and images, and is inserted into the CD-ROM 55 for permitting the CD-ROM 55 to retrieve the images and software. Storage medium is not limited to locally stored objects but could also consist of a file or multimedia object server. Images could also be fed directly to the system through an image capture device, such as a scanner, or a live feed as via the internet. In the case of images, they are displayed on the display 30. In addition, a floppy disk 61 may also include a software program, and is inserted into the microprocessor-based unit 20 for inputting the software program. Still further, the microprocessor-based unit 20 may be programmed, as is well know in the art, for storing the software program internally. A printer 56 is connected to the microprocessor-based unit 20 for printing a hardcopy of the output of the computer system 10. Images may also be displayed on the display 30 via a personal computer card (PC card) 62 or, as it was formerly known, a personal computer memory card international association card (PCMCIA card) which contains digitized images electronically embodied in the card 62. The PC card 62 is ultimately inserted into the microprocessor-based unit 20 for permitting visual display of the image on the display 30. Referring to FIG. 2, there is illustrated a flowchart of a software program of the present invention. The software is initiated S2 and images are input S4 to the microprocessor-based unit 20 from a storage device such as the CD-ROM 57. The user inputs description fields and caption to be associated with a particular image (metadata). The description fields may include but are not limited to, when image was taken, who the photographer was and any application specific information. These description fields will be stored as individual fields. Caption may include one or more natural language phrases or sentences without any grammar restrictions describing any characteristic of the image. The caption is then processed S6. It takes the caption as an input, performs the natural language processing and outputs the frame representation for the caption. The natural language processing S6 will be described in detail below. The next step is to catalog the caption frames in the frame database S8. Images are stored in the image data base. The image database may be physically the same or different from the frame database. Each frame in the frame database contains corresponding image id from the image database. The details of the processing step S6 are spell checking of the input caption and correcting any errors found using conventional methods S10. Referring now to FIGS. 2 and 3, next, the input caption is preprocessed S12 which includes comparing the input caption with a non-relevant phrase database S14. Comparison is made using standard string matching techniques. The non-relevant phrase database contains phrases that do not contribute to the meaning of a caption. Those skilled in the art will recognize that there are a plurality of phrases and words which are input for captioning that do not contribute to the meaning. If a match of a non-relevant phrase is found S16, the phrase is deleted S18; for example, an initial caption of "this is a picture of a black automobile." will have the phrase "This is a picture of" deleted from the caption. The remaining caption, if any non-relevant phrases have been deleted, or the original caption is then searched for any co-located words, that is compound words such as operating room. These are put together as one word, for example operating -- room. Then proper nouns and place names are identified and a tag identifying them as such is attached S20. Referring to FIG. 2 and FIG. 4, after preprocessing, the caption is sent through the part-of-speech tagging and Role Parser unit S22. There are several part-of-speech tagging tools available, for example Xerox PARC tagger and MIT media lab tagger. In the preferred embodiment, this invention uses ENGCG part-of-speech tagger from Lingsoft, Inc. ENGCG contains a set of rules related to part-of-speech tagging. It reduces words to their root form and assigns morphological and syntactic tags to each word of the caption, S24. Morphological tags identify nouns, verbs, adjectives, prepositions etc. Syntactic tags identify the function of the word in the sentence, such as subject, main verb, and the like. For example, in the caption "Little girl is standing near an orange car.", the word "orange" gets assigned the morphological tag of an adjective and the syntactic tag of pre-modifying adjective. ENGCG contains about 32 syntactic tags and a long list of morphological tags. The tagged caption is then sent to the Role Parser unit to assign the roles that different words play S26. The preferred roles are agent, action, patient, location, and prepositional roles, although other roles may be used especially for a different application. In the case of video or audio retrieval, a different set of roles may be more appropriate. The agent role indicates an actor who is performing an action, action role identifies an activity that is being performed, patient identifies an object affected in some way by an action. Some of these roles are similar to case grammar representation first suggested by Fillmore (reference: Fillmore, C. J. 1968. "The Case for Case," in Universals of Linguistic Theory, edited by E. Bach and R. T. Harms, Holt, New York.) Case grammar claims that there is a small fixed number of ways in which verbs can be used in a sentence. These describe the basic ways that a noun phrase or a prepositional phrase relates to the verb. All verbs must select their cases from this set. For example, possible cases for the verb "fly" could be agent, patient, source and destination. In the sentence, "Jack is flying a plane from New York to Boston", fly is the identified verb, Jack is an agent, plane is the patient, source is New York and the destination is Boston. However, the limitation of Case grammar approach is that it works only with well formed (enforced grammar rules) single sentences and the possible cases for each verb has to be predefined. In this preferred embodiment, these restrictions are removed and roles are generated dynamically based upon the caption description. The Role parser contains rules to map syntactic and morphological tags to the actual roles that the words play S26. Attributes for each role are also identified. For example, in the caption, "a black automobile", "black" becomes a pre-modifying adjective or attribute for the "automobile." In another example, "Skier skiing downhill.", "downhill" becomes a post modifying attribute for "ski." In this preferred embodiment, a matrix was created based upon syntactic and morphological tags available from the ENGCG part-of-speech tagger to identify possible role assignments, pre-modifying attribute tags and post-modifying attribute tags for the agent, action, patient, preposition, and location roles. Some of these tags can belong to multiple categories. For example, syntactic tag of generated by the ENGCG tagger "@NN>"could be an agent or an attribute of an agent based upon its context in the sentence. This matrix was translated into a set of rules for determining roles and attributes. It also included rules for ignoring tags of determiners like "a" "an" "the" and the main verb "be" which are not used for determining roles. Role parser, S26 analyzes each tagged word in the caption. If the combination of syntactic and morphological tags identifies that word to be ignored then it goes to the next word. Each word is analyzed based on the rules and identified either as an attribute tag or a role tag. It collects a list of attributes along with their values until a role is identified. Once a role is identified, an attribute list is assigned to that role and the remaining caption is searched for any post modifying attributes. If any post modifying attributes are present, they are also stored with the currently identified role. This process is repeated until each word of the caption is analyzed. For example, a caption "The flower garden is in full bloom." will result in "garden" as the agent role, "flower" as an attribute for the agent role, "in" as a preposition role containing "bloom" as its value and "full" as an attribute for "bloom". The location role is assigned to any place names that were identified in the preprocessing, S12. After the roles and attributes for all the words have been determined, a frame is created and populated. The roles are collectively referred to hereinafter as a frame. A frame, in general, is a cluster of facts that describe a typical situation, which in this application is the description of an image. The frame structure representation is created and populated S28 based on the roles identified. Each frame will have a unique identifier along with a list of roles. Each role will have a value and a list of attributes. In the case of plurality of roles, a list of roles will be included in the frame. For example, a caption "The flower garden is in full bloom." after role assignment, S26 will result into a frame depicted ______________________________________frame (pldl: (agent: (value: (garden), attribute: (flower)) (in: (value: (bloom), attribute: (full))) (imageid: pl))______________________________________ where p1d1 is the frame identifier for this caption representation, the roles are agent and in and p1 is the corresponding image id in the image database. In the case of multiple roles, a list of role values would be included here. The frame database populated with the frames described above S8. Now the frame database may be queried for finding an image having characteristics desired by the user. Referring now to FIG. 5, the user inputs a natural language query S29 which is processed S30 using the same method as described above in reference to step S6. Briefly described, these steps include spell checking, preprocessing, part-of-speech tagging, assigning roles and populating a frame. Next, the frame representation of the query is searched for matching frames in the frame database. If any matches are found, they are ranked in order of significance S32. Any fielded information which has been entered along with query, will be searched against the caption description fields using standard database search techniques. Referring to FIG. 6, step S32 is described in detail. The frame representation of the query is first matched role-to-role S34 with the caption frames in the frame database, that is the agent role of query frame is matched against the agent role of caption frames, action role of query frame is matched against the action role of caption frames and so on. A set of rules are created to govern the matching and every matching rule has an associated weight, as can be determined by those skilled in the art. The matching results are scored based on the number of role matches and their associated weights. In the preferred embodiment, the weights in descending order of weighting are the agent, the action, the location, the patient and the prepositional phrases. The weighting scheme may vary according to the application. The total weight is calculated by summing the individual weights contributed by each query role. The number and score of matched frames S36, is analyzed against a predefined threshold, as can be determined by those skilled in the art, which determines whether the set of matches is acceptable. If the retrieved matches are not acceptable S38, system performs the next level search S42. Optionally, the user may make the decision to continue to search further or not. If the next level search is not required, corresponding images are retrieved from the image database based on the image id's stored in the caption frames and displayed in the order of the ranking S40. If the next level search is required S42, an expanded query frame is generated. Query frame expansion is done by looking up the query role words in the semantic knowledge source and adding a set of appropriate words based on the relationships. In the preferred embodiment, WordNet, which was developed at Princeton University, is be used for semantic knowledge source. WordNet is an on-line dictionary of nouns, verbs, adjectives and adverbs which are organized into synonym sets. Different relationships link the synonym sets together. Synonym sets are also organized into different senses of words. In this application, the most common senses of words will be used. The most important relationship utilized in this preferred embodiment is the synonym. For the Agent role, query words are first augmented by adding synonyms. For example, if the Agent role word was "child", then the synonyms of "kid", "youngster" and so on, would also be used in the matching. Generalization or the a-kind-of (hypernym) relationship is also used. For the action role, in addition to synonyms, the cause and entail relationships is used. This expanded query is then searched against the caption frame database S44. Matching is still of type role-to-role matching, as described above S34. The results of the search are scored S46 and ranked in the same manner as described above S36. If retrieved matches are not acceptable S48 based on the predefined threshold, the next level of search is performed S50. Optionally, user may make the decision to continue further search or not. If the next level search is not required, corresponding images are retrieved from the image database in the order of the ranking S40. If the next level search is required , the frame database search using expanded keywords is performed S50. It consists of matching any expanded query role to any caption role in the caption frame database. This search is performed without regard to the role the words play in the sentence. This is analogous to a keyword match using the expanded query words. Referring back to FIG. 5, the images and associated captions are retrieved S52 in their order of ranking and are displayed on the display 30 in a suitable manner, preferably thumbnails, for permitting the user to view the images. If any one or all of the images are acceptable S54, the selected images may be printed, ordered, transmitted over network, stored in an album or further manipulated before exiting out S56. If the retrieved images are not acceptable to the user, the user may input another natural language query S29 or select one or more images from the retrieved set of images as new query S58. If an image from the retrieved set of images is selected as the new query, the corresponding frame representation will become the new query frame and frame database search and ranking step, S32 will proceed. In case of multiple selected images, either union or intersection of the corresponding retrieved images sets can be selected to satisfy the query. This can be chosen by the user or setup by the system. The invention has been described with reference to a preferred embodiment. However, it will be appreciated that variations and modifications can be effected by a person of ordinary skill in the art without departing from the scope of the invention. Parts List: 10 computer system 20 microprocessor-based unit 30 display 40 keyboard 50 mouse 52 selector 55 CD-ROM 56 printer 57 compact disc 61 floppy diskette 62 PC card

Description

Topics

Download Full PDF Version (Non-Commercial Use)

Patent Citations (10)

    Publication numberPublication dateAssigneeTitle
    US-4914590-AApril 03, 1990Emhart Industries, Inc.Natural language understanding system
    US-5197005-AMarch 23, 1993Intelligent Business SystemsDatabase retrieval system having a natural language interface
    US-5265065-ANovember 23, 1993West Publishing CompanyMethod and apparatus for information retrieval from a database by replacing domain specific stemmed phases in a natural language to create a search query
    US-5386556-AJanuary 31, 1995International Business Machines CorporationNatural language analyzing apparatus and method
    US-5418948-AMay 23, 1995West Publishing CompanyConcept matching of natural language queries with a database of document concepts
    US-5434777-AJuly 18, 1995Apple Computer, Inc.Method and apparatus for processing natural language
    US-5488725-AJanuary 30, 1996West Publishing CompanySystem of document representation retrieval by successive iterated probability sampling
    US-5493677-AFebruary 20, 1996Systems Research & Applications CorporationGeneration, archiving, and retrieval of digital images with evoked suggestion-set captions and natural language interface
    US-5528491-AJune 18, 1996Language Engineering CorporationApparatus and method for automated natural language translation
    US-5625814-AApril 29, 1997Apple Computer, Inc.Method and apparatus for processing natural language with a hierarchy of mapping routines

NO-Patent Citations (0)

    Title

Cited By (271)

    Publication numberPublication dateAssigneeTitle
    US-2005102251-A1May 12, 2005David GillespieMethod of document searching
    US-2006155684-A1July 13, 2006Microsoft CorporationSystems and methods to present web image search results for effective image browsing
    US-2004246255-A1December 09, 2004Tsang-Gang Lin, Chia-Chen Chen, Shi-Nine Yang, Shih-Pin Chao, Chih-Yi ChiuMethod for converting high level motion scripts to computer animations
    US-9300784-B2March 29, 2016Apple Inc.System and method for emergency calls initiated by voice command
    US-9430463-B2August 30, 2016Apple Inc.Exemplar-based natural language processing
    US-9785630-B2October 10, 2017Apple Inc.Text prediction using combined word N-gram and unigram language models
    US-2009234803-A1September 17, 2009Continental Electrical Construction Company, LlcKeyword search of business information system
    US-8073681-B2December 06, 2011Voicebox Technologies, Inc.System and method for a cooperative conversational voice user interface
    US-9691383-B2June 27, 2017Apple Inc.Multi-tiered voice feedback in an electronic device
    US-2008189110-A1August 07, 2008Tom Freeman, Mike KennewickSystem and method for selecting and presenting advertisements based on natural language processing of voice-based input
    US-2011161067-A1June 30, 2011Dynavox Systems, LlcSystem and method of using pos tagging for symbol assignment
    US-9668024-B2May 30, 2017Apple Inc.Intelligent automated assistant for TV user interactions
    US-2006071947-A1April 06, 2006Randy Ubillos, Laurent Perrodin, Dan WaylonisTechniques for displaying digital images on a display
    US-2009299745-A1December 03, 2009Kennewick Robert A, Chris WeiderSystem and method for an integrated, multi-modal, multi-device natural language voice services environment
    US-8265862-B1September 11, 2012Boadin Technology, LLCSystem, method, and computer program product for communicating location-related information
    US-9697822-B1July 04, 2017Apple Inc.System and method for updating an adaptive speech recognition model
    US-8713021-B2April 29, 2014Apple Inc.Unsupervised document clustering using latent semantic density analysis
    US-2010204994-A1August 12, 2010Voicebox Technologies, Inc.Systems and methods for responding to natural language speech utterance
    US-9858925-B2January 02, 2018Apple Inc.Using context information to facilitate processing of commands in a virtual assistant
    US-2003158723-A1August 21, 2003Fuji Xerox Co., Ltd.Syntactic information tagging support system and method
    US-6246977-B1June 12, 2001Microsoft CorporationInformation retrieval utilizing semantic representation of text and based on constrained expansion of query words
    US-9318108-B2April 19, 2016Apple Inc.Intelligent automated assistant
    US-6389181-B2May 14, 2002Eastman Kodak CompanyPhotocollage generation and modification using image recognition
    US-8326637-B2December 04, 2012Voicebox Technologies, Inc.System and method for processing multi-modal device interactions in a natural language voice services environment
    US-8719026-B2May 06, 2014Voicebox Technologies CorporationSystem and method for providing a natural language voice user interface in an integrated voice navigation services environment
    US-2010138392-A1June 03, 2010Caspedia CorporationContent manager
    US-2010299142-A1November 25, 2010Voicebox Technologies, Inc.System and method for selecting and presenting advertisements based on natural language processing of voice-based input
    US-8155962-B2April 10, 2012Voicebox Technologies, Inc.Method and system for asynchronously processing natural language utterances
    US-9697820-B2July 04, 2017Apple Inc.Unit-selection text-to-speech synthesis using concatenation-sensitive neural networks
    US-7962326-B2June 14, 2011Invention Machine CorporationSemantic answering system and method
    US-9626955-B2April 18, 2017Apple Inc.Intelligent text-to-speech conversion
    US-8930191-B2January 06, 2015Apple Inc.Paraphrasing of user requests and results by automated digital assistant
    US-8775442-B2July 08, 2014Apple Inc.Semantic search using a single-source semantic model
    US-8150694-B2April 03, 2012Voicebox Technologies, Inc.System and method for providing an acoustic grammar to dynamically sharpen speech interpretation
    US-7406415-B1July 29, 2008Georgia Tech Research CorporationPhonetic searching
    US-9305548-B2April 05, 2016Voicebox Technologies CorporationSystem and method for an integrated, multi-modal, multi-device natural language voice services environment
    US-8682649-B2March 25, 2014Apple Inc.Sentiment prediction from textual data
    US-8731942-B2May 20, 2014Apple Inc.Maintaining context information between user interactions with a voice assistant
    US-6665083-B1December 16, 2003Fuji Photo Film Co., Ltd.Computer-readable recording medium for recording photograph print ordering information
    US-6654749-B1November 25, 2003Choice Media, Inc.Method and system for searching indexed information databases with automatic user registration via a communication network
    US-9190062-B2November 17, 2015Apple Inc.User profiling for voice input processing
    US-7917497-B2March 29, 2011Iac Search & Media, Inc.Natural language query processing
    US-6517002-B1February 11, 2003Portable Data Technologies, Inc.System and method for intelligent data input from a machine-readable medium
    US-2011066630-A1March 17, 2011Marcello Balduccini, Sara GirottoMultimedia object retrieval from natural language queries
    US-6173279-B1January 09, 2001At&T Corp.Method of using a natural language interface to retrieve information from one or more data resources
    US-9842105-B2December 12, 2017Apple Inc.Parsimonious continuous-space phrase representations for natural language processing
    US-8190692-B1May 29, 2012Boadin Technology, LLCLocation-based messaging system, method, and computer program product
    WO-0120500-A2March 22, 2001Sri InternationalRecuperation d'informations au moyen d'interrogations formulees en langage naturel
    US-9424862-B2August 23, 2016Newvaluexchange LtdApparatuses, methods and systems for a digital conversation management platform
    US-2007070038-A1March 29, 2007Hoffberg Steven M, Hoffberg-Borghesani Linda IErgonomic man-machine interface incorporating adaptive pattern recognition based control system
    US-9105266-B2August 11, 2015Voicebox Technologies CorporationSystem and method for processing multi-modal device interactions in a natural language voice services environment
    US-2010325569-A1December 23, 2010Oracle International CorporationSecurity policy verification system
    US-8515765-B2August 20, 2013Voicebox Technologies, Inc.System and method for a cooperative conversational voice user interface
    US-9098489-B2August 04, 2015Abbyy Infopoisk LlcMethod and system for semantic searching
    US-7818176-B2October 19, 2010Voicebox Technologies, Inc.System and method for selecting and presenting advertisements based on natural language processing of voice-based input
    US-8069046-B2November 29, 2011Voicebox Technologies, Inc.Dynamic speech sharpening
    US-9495129-B2November 15, 2016Apple Inc.Device, method, and user interface for voice-activated navigation and browsing of a document
    US-2007050191-A1March 01, 2007Voicebox Technologies, Inc.Mobile systems and methods of supporting natural language human-machine interactions
    US-2010145700-A1June 10, 2010Voicebox Technologies, Inc.Mobile systems and methods for responding to natural language speech utterance
    US-8332224-B2December 11, 2012Voicebox Technologies, Inc.System and method of supporting adaptive misrecognition conversational speech
    US-9495957-B2November 15, 2016Nuance Communications, Inc.Mobile systems and methods of supporting natural language human-machine interactions
    US-8452598-B2May 28, 2013Voicebox Technologies, Inc.System and method for providing advertisements in an integrated voice navigation services environment
    US-8781836-B2July 15, 2014Apple Inc.Hearing assistance system for providing consistent human speech
    WO-02069197-A1September 06, 2002Motorola, Inc., A Corporation Of The State Of DelawareGénération automatique de sql d'achèvement des trames
    US-9361886-B2June 07, 2016Apple Inc.Providing text input using speech data and non-speech data
    US-8078397-B1December 13, 2011Boadin Technology, LLCSystem, method, and computer program product for social networking utilizing a vehicular assembly
    US-2003105622-A1June 05, 2003Netbytel, Inc.Retrieval of records using phrase chunking
    US-9483461-B2November 01, 2016Apple Inc.Handling speech synthesis of content for multiple languages
    US-8145489-B2March 27, 2012Voicebox Technologies, Inc.System and method for selecting and presenting advertisements based on natural language processing of voice-based input
    US-8738380-B2May 27, 2014Voicebox Technologies CorporationSystem and method for processing multi-modal device interactions in a natural language voice services environment
    US-9502031-B2November 22, 2016Apple Inc.Method for supporting dynamic grammars in WFST-based ASR
    US-2011301941-A1December 08, 2011Syl Research LimitedNatural language processing method and system
    US-2011258196-A1October 20, 2011Skjalg Lepsoy, Gianluca Francini, Fabrizio AntonelliMethod and system of content recommendation
    US-7769587-B2August 03, 2010Georgia Tech Research CorporationPhonetic searching
    US-9311043-B2April 12, 2016Apple Inc.Adaptive audio feedback system and method
    US-8670985-B2March 11, 2014Apple Inc.Devices and methods for identifying a prompt corresponding to a voice input in a sequence of prompts
    US-9406078-B2August 02, 2016Voicebox Technologies CorporationSystem and method for delivering targeted advertisements and/or providing natural language processing based on advertisements
    US-7181434-B2February 20, 2007Industrial Technology Research InstituteMethod for converting high level motion scripts to computer animations
    US-2008263019-A1October 23, 2008Iac Search & Media, Inc.Natural language query processing
    US-7974714-B2July 05, 2011Steven Mark HoffbergIntelligent electronic appliance system and method
    US-9620105-B2April 11, 2017Apple Inc.Analyzing audio input for efficient speech and music recognition
    US-6601026-B2July 29, 2003Discern Communications, Inc.Information retrieval by natural language querying
    US-9330720-B2May 03, 2016Apple Inc.Methods and apparatus for altering audio output signals
    US-8326634-B2December 04, 2012Voicebox Technologies, Inc.Systems and methods for responding to natural language speech utterance
    US-8768702-B2July 01, 2014Apple Inc.Multi-tiered voice feedback in an electronic device
    US-2010057443-A1March 04, 2010Voicebox Technologies, Inc.Systems and methods for responding to natural language speech utterance
    US-8942986-B2January 27, 2015Apple Inc.Determining user intent based on ontologies of domains
    US-9721563-B2August 01, 2017Apple Inc.Name recognition system
    US-7177798-B2February 13, 2007Rensselaer Polytechnic InstituteNatural language interface using constrained intermediate dictionary of results
    US-6910003-B1June 21, 2005Discern Communications, Inc.System, method and article of manufacture for concept based information searching
    US-9760566-B2September 12, 2017Microsoft Technology Licensing, LlcAugmented conversational understanding agent to identify conversation context between two humans and taking an agent action thereof
    US-6523029-B1February 18, 2003Bitmobile TechnologiesSystem and method for embedded information retrieval in a distributed free-text application environment
    US-6745161-B1June 01, 2004Discern Communications, Inc.System and method for incorporating concept-based retrieval within boolean search engines
    US-9075783-B2July 07, 2015Apple Inc.Electronic device with text error correction based on voice recognition data
    US-8670979-B2March 11, 2014Apple Inc.Active input elicitation by intelligent automated assistant
    WO-0120500-A3February 19, 2004Stanford Res Inst IntRecuperation d'informations au moyen d'interrogations formulees en langage naturel
    US-6965857-B1November 15, 2005Cogilex Recherches & Developpement Inc.Method and apparatus for deriving information from written text
    US-9733821-B2August 15, 2017Apple Inc.Voice control to diagnose inadvertent activation of accessibility features
    US-7324939-B1January 29, 2008Georgia Tech Research CorporationPhonetic searching
    US-7483892-B1January 27, 2009Kroll Ontrack, Inc.Method and system for optimally searching a document database using a representative semantic space
    US-9842101-B2December 12, 2017Apple Inc.Predictive conversion of language input
    US-9620104-B2April 11, 2017Apple Inc.System and method for user-specified pronunciation of words for speech synthesis and recognition
    US-9031845-B2May 12, 2015Nuance Communications, Inc.Mobile systems and methods for responding to natural language speech utterance
    US-RE46310-EFebruary 14, 2017Blanding Hovenweep, LlcErgonomic man-machine interface incorporating adaptive pattern recognition based control system
    US-9672591-B2June 06, 2017Apple Inc.Collage display of image projects
    US-8731929-B2May 20, 2014Voicebox Technologies CorporationAgent architecture for determining meanings of natural language utterances
    US-7475065-B1January 06, 2009Georgia Tech Research CorporationPhonetic searching
    US-2012117100-A1May 10, 2012Marcello Balduccini, Sara GirottoMultimedia object retrieval from natural language queries
    WO-03051033-A1June 19, 2003Koninklijke Philips Electronics N.V.Album de photos virtuel base sur l'humeur
    US-9626959-B2April 18, 2017Nuance Communications, Inc.System and method of supporting adaptive misrecognition in conversational speech
    US-7949529-B2May 24, 2011Voicebox Technologies, Inc.Mobile systems and methods of supporting natural language human-machine interactions
    US-2010204986-A1August 12, 2010Voicebox Technologies, Inc.Systems and methods for responding to natural language speech utterance
    US-9424861-B2August 23, 2016Newvaluexchange LtdApparatuses, methods and systems for a digital conversation management platform
    US-8849652-B2September 30, 2014Voicebox Technologies CorporationMobile systems and methods of supporting natural language human-machine interactions
    US-9548050-B2January 17, 2017Apple Inc.Intelligent automated assistant
    US-2009162818-A1June 25, 2009Martin KosakowskiMethod for the determination of supplementary content in an electronic device
    US-2003041058-A1February 27, 2003Fujitsu LimitedQueries-and-responses processing method, queries-and-responses processing program, queries-and-responses processing program recording medium, and queries-and-responses processing apparatus
    US-8892446-B2November 18, 2014Apple Inc.Service orchestration for intelligent automated assistant
    US-2013124551-A1May 16, 2013Koninklijke Philips Electronics N.V.Obtaining keywords for searching
    US-9633004-B2April 25, 2017Apple Inc.Better resolution when referencing to concepts
    US-7917367-B2March 29, 2011Voicebox Technologies, Inc.Systems and methods for responding to natural language speech utterance
    US-2010146447-A1June 10, 2010Randy Ubillos, Laurent Perrodin, Dan WaylonisTechniques For Displaying Digital Images On A Display
    US-2010198375-A1August 05, 2010Apple Inc.Audio user interface for displayless electronic device
    US-9547647-B2January 17, 2017Apple Inc.Voice-based media searching
    US-8718047-B2May 06, 2014Apple Inc.Text to speech conversion of text messages from mobile communication devices
    US-2008091406-A1April 17, 2008Voicebox Technologies, Inc.System and method for a cooperative conversational voice user interface
    US-8996376-B2March 31, 2015Apple Inc.Intelligent text-to-speech conversion
    US-7076503-B2July 11, 2006Microsoft CorporationManaging media objects in a database
    US-2009306962-A1December 10, 2009International Business Machines CorporationSystem and method to provide warnings associated with natural language searches to determine intended actions and accidental omissions
    US-2003157470-A1August 21, 2003Michael Altenhofen, Joachim Schaper, Wolfgang Gerteis, Krebs Andreas S., Torsten LeidigE-learning station and interface
    US-9646609-B2May 09, 2017Apple Inc.Caching apparatus for serving phonetic pronunciations
    US-2014032574-A1January 30, 2014Emdadur R. KhanNatural language understanding using brain-like approach: semantic engine using brain-like approach (sebla) derives semantics of words and sentences
    US-8589161-B2November 19, 2013Voicebox Technologies, Inc.System and method for an integrated, multi-modal, multi-device natural language voice services environment
    US-8140327-B2March 20, 2012Voicebox Technologies, Inc.System and method for filtering and eliminating noise from natural language utterances to improve speech recognition and parsing
    US-7548936-B2June 16, 2009Microsoft CorporationSystems and methods to present web image search results for effective image browsing
    US-8161063-B2April 17, 2012Eastman Kodak CompanyMultimedia object retrieval from natural language queries
    US-9886432-B2February 06, 2018Apple Inc.Parsimonious handling of word inflection via categorical stem + suffix N-gram language models
    US-2006111893-A1May 25, 2006International Business Machines CorporationDisplay of results of cross language search
    US-8706472-B2April 22, 2014Apple Inc.Method for disambiguating multiple readings in language conversion
    US-8983839-B2March 17, 2015Voicebox Technologies CorporationSystem and method for dynamically generating a recognition grammar in an integrated voice navigation services environment
    US-7403938-B2July 22, 2008Iac Search & Media, Inc.Natural language query processing
    US-7685175-B2March 23, 2010Michael Lee Carroll, Paul Thomas SchweizerContent manager
    US-9865248-B2January 09, 2018Apple Inc.Intelligent text-to-speech conversion
    US-8495703-B2July 23, 2013Oracle International CorporationSecurity policy verification system
    US-6233547-B1May 15, 2001Eastman Kodak CompanyComputer program product for retrieving multi-media objects using a natural language having a pronoun
    US-2008195396-A1August 14, 2008Mark Greene, Michael Hegarty, Dermot CantwellSystem, method and computer program product for adding voice activation and voice control to a media player
    US-9711141-B2July 18, 2017Apple Inc.Disambiguating heteronyms in speech synthesis
    US-8775953-B2July 08, 2014Apple Inc.Collage display of image projects
    US-6931147-B2August 16, 2005Koninklijke Philips Electronics N.V.Mood based virtual photo album
    US-8898568-B2November 25, 2014Apple Inc.Audio user interface
    US-6947589-B2September 20, 2005Canon Kabushiki KaishaDynamic gamut mapping selection
    US-9633674-B2April 25, 2017Apple Inc.System and method for detecting errors in interactions with a voice-based digital assistant
    US-2011145264-A1June 16, 2011Electronics And Telecommunications Research InstituteMethod and apparatus for automatically creating allomorphs
    US-2001044720-A1November 22, 2001Lee Victor Wai Leung, Chris Semotok, Otman Basir, Fakhri KarrayNatural English language search and retrieval system and method
    US-2004111404-A1June 10, 2004Hiroko Mano, Hideo ItohMethod and system for searching text portions based upon occurrence in a specific area
    US-9269097-B2February 23, 2016Voicebox Technologies CorporationSystem and method for delivering targeted advertisements and/or providing natural language processing based on advertisements
    US-9431028-B2August 30, 2016Newvaluexchange LtdApparatuses, methods and systems for a digital conversation management platform
    US-9734193-B2August 15, 2017Apple Inc.Determining domain salience ranking from ambiguous words in natural speech
    US-8719009-B2May 06, 2014Voicebox Technologies CorporationSystem and method for processing multi-modal device interactions in a natural language voice services environment
    US-9798393-B2October 24, 2017Apple Inc.Text correction processing
    US-6161084-ADecember 12, 2000Microsoft CorporationInformation retrieval utilizing semantic representation of text by identifying hypernyms and indexing multiple tokenized semantic structures to a same passage of text
    US-8370376-B2February 05, 2013Eastman Kodak CompanyMultimedia object retrieval from natural language queries
    US-2007038687-A1February 15, 2007Carroll Micheal L, Schweizer Paul TContent Manager
    US-8762469-B2June 24, 2014Apple Inc.Electronic devices with voice command and contextual data processing capabilities
    US-9015049-B2April 21, 2015Voicebox Technologies CorporationSystem and method for a cooperative conversational voice user interface
    US-9053089-B2June 09, 2015Apple Inc.Part-of-speech tagging using latent analogy
    US-2002081024-A1June 27, 2002Sung Hee Park, Hyun Jin Kim, Soo Jun Park, Myung Gil JangApparatus and method for retrieving color and shape of image based on natural language
    US-2002143524-A1October 03, 2002Lingomotors, Inc.Method and resulting system for integrating a query reformation module onto an information retrieval system
    US-7313521-B1December 25, 2007Georgia Tech Research CorporationPhonetic searching
    US-7904296-B2March 08, 2011Nexidia Inc.Spoken word spotting queries
    US-8903716-B2December 02, 2014Apple Inc.Personalized vocabulary for digital assistant
    US-8074171-B2December 06, 2011International Business Machines CorporationSystem and method to provide warnings associated with natural language searches to determine intended actions and accidental omissions
    US-9117447-B2August 25, 2015Apple Inc.Using event alert text as input to an automated assistant
    US-8886536-B2November 11, 2014Voicebox Technologies CorporationSystem and method for delivering targeted advertisements and tracking advertisement interactions in voice recognition contexts
    US-9711143-B2July 18, 2017Voicebox Technologies CorporationSystem and method for an integrated, multi-modal, multi-device natural language voice services environment
    US-9262612-B2February 16, 2016Apple Inc.Device access using voice authentication
    US-8073590-B1December 06, 2011Boadin Technology, LLCSystem, method, and computer program product for utilizing a communication channel of a mobile device by a vehicular assembly
    US-8677377-B2March 18, 2014Apple Inc.Method and apparatus for building an intelligent automated assistant
    US-2002022955-A1February 21, 2002Galina Troyanova, Andrei Bondarenko, Yaraslau Bialiauski, Igar ZhygalkaSynonym extension of search queries with validation
    US-6430531-B1August 06, 2002Soliloquy, Inc.Bilateral speech system
    US-2003101182-A1May 29, 2003Omri Govrin, Govrin Eri MosheMethod and system for smart search engine and other applications
    US-9280610-B2March 08, 2016Apple Inc.Crowd sourcing information to fulfill user requests
    US-8326627-B2December 04, 2012Voicebox Technologies, Inc.System and method for dynamically generating a recognition grammar in an integrated voice navigation services environment
    US-8849670-B2September 30, 2014Voicebox Technologies CorporationSystems and methods for responding to natural language speech utterance
    US-8180737-B2May 15, 2012Panstoria, Inc.Content manager
    US-9858343-B2January 02, 2018Microsoft Technology Licensing LlcPersonalization of queries, conversations, and searches
    US-2009083033-A1March 26, 2009Georgia Tech Research CorporationPhonetic Searching
    US-2012246135-A1September 27, 2012Enpulz L.L.C.Image search engine augmenting search text based upon category selection
    US-8719014-B2May 06, 2014Apple Inc.Electronic device with text error correction based on voice recognition data
    US-2011257963-A1October 20, 2011Konstantin Zuev, Alexander RylovMethod and system for semantic searching
    US-9668121-B2May 30, 2017Apple Inc.Social reminders
    US-9886953-B2February 06, 2018Apple Inc.Virtual assistant activation
    US-9721566-B2August 01, 2017Apple Inc.Competing devices responding to voice triggers
    US-9171541-B2October 27, 2015Voicebox Technologies CorporationSystem and method for hybrid processing in a natural language voice services environment
    US-9760559-B2September 12, 2017Apple Inc.Predictive text input
    US-2010049514-A1February 25, 2010Voicebox Technologies, Inc.Dynamic speech sharpening
    US-2003126126-A1July 03, 2003Lee Jin Soo, Kim Heon JunApparatus and method for searching multimedia object
    US-9633660-B2April 25, 2017Apple Inc.User profiling for voice input processing
    US-9626703-B2April 18, 2017Voicebox Technologies CorporationVoice commerce
    US-8762156-B2June 24, 2014Apple Inc.Speech recognition repair using contextual information
    US-8935167-B2January 13, 2015Apple Inc.Exemplar-based latent perceptual modeling for automatic speech recognition
    US-8112275-B2February 07, 2012Voicebox Technologies, Inc.System and method for user-specific speech recognition
    US-7705858-B2April 27, 2010Apple Inc.Techniques for displaying digital images on a display
    US-8688446-B2April 01, 2014Apple Inc.Providing text input using speech data and non-speech data
    US-8676904-B2March 18, 2014Apple Inc.Electronic devices with voice command and contextual data processing capabilities
    US-9620113-B2April 11, 2017Voicebox Technologies CorporationSystem and method for providing a natural language voice user interface
    US-9818400-B2November 14, 2017Apple Inc.Method and apparatus for discovering trending terms in speech requests
    US-2006074973-A1April 06, 2006Microsoft CorporationManaging media objects in a database
    US-7870145-B2January 11, 2011International Business Machines CorporationUtilization of logical fields with conditional constraints in abstract queries
    US-8682667-B2March 25, 2014Apple Inc.User profiling for selecting user specific voice input processing information
    US-8799000-B2August 05, 2014Apple Inc.Disambiguation based on active input elicitation by intelligent automated assistant
    US-8812294-B2August 19, 2014Apple Inc.Translating phrases from one language into another using an order-based set of declarative rules
    US-8140335-B2March 20, 2012Voicebox Technologies, Inc.System and method for providing a natural language voice user interface in an integrated voice navigation services environment
    US-2009148064-A1June 11, 2009Egan SchulzCollage display of image projects
    US-2003053683-A1March 20, 2003Newman Todd D., Kohler Timothy L., Haikin John S.Dynamic gamut mapping selection
    US-8660849-B2February 25, 2014Apple Inc.Prioritizing selection criteria by automated assistant
    US-9338493-B2May 10, 2016Apple Inc.Intelligent automated assistant for TV user interactions
    US-9570070-B2February 14, 2017Voicebox Technologies CorporationSystem and method for processing multi-modal device interactions in a natural language voice services environment
    US-2002035571-A1March 21, 2002Coult John HDigital patent marking method
    US-8862252-B2October 14, 2014Apple Inc.Audio user interface for displayless electronic device
    US-8117225-B1February 14, 2012Boadin Technology, LLCDrill-down system, method, and computer program product for focusing a search
    US-8447607-B2May 21, 2013Voicebox Technologies, Inc.Mobile systems and methods of supporting natural language human-machine interactions
    US-8977584-B2March 10, 2015Newvaluexchange Global Ai LlpApparatuses, methods and systems for a digital conversation management platform
    US-9502025-B2November 22, 2016Voicebox Technologies CorporationSystem and method for providing a natural language content dedication service
    US-9747896-B2August 29, 2017Voicebox Technologies CorporationSystem and method for providing follow-up responses to prior natural language inputs of a user
    US-2011161068-A1June 30, 2011Dynavox Systems, LlcSystem and method of using a sense model for symbol assignment
    US-2008319751-A1December 25, 2008Kennewick Robert A, David Locke, Kennewick Sr Michael R, Kennewick Jr Michael R, Richard Kennewick, Tom FreemanSystems and methods for responding to natural language speech utterance
    US-8719006-B2May 06, 2014Apple Inc.Combined statistical and rule-based part-of-speech tagging for text-to-speech synthesis
    US-9715875-B2July 25, 2017Apple Inc.Reducing the need for manual start/end-pointing and trigger phrases
    US-2008228737-A1September 18, 2008Dettinger Richard D, Tenner Jeffrey WUtilization of logical fields with conditional constraints in abstract queries
    US-2003069880-A1April 10, 2003Ask Jeeves, Inc.Natural language query processing
    US-8527274-B2September 03, 2013Voicebox Technologies, Inc.System and method for delivering targeted advertisements and tracking advertisement interactions in voice recognition contexts
    US-9431006-B2August 30, 2016Apple Inc.Methods and apparatuses for automatic speech recognition
    US-9576574-B2February 21, 2017Apple Inc.Context-sensitive handling of interruptions by intelligent digital assistant
    US-8370147-B2February 05, 2013Voicebox Technologies, Inc.System and method for providing a natural language voice user interface in an integrated voice navigation services environment
    US-8195468-B2June 05, 2012Voicebox Technologies, Inc.Mobile systems and methods of supporting natural language human-machine interactions
    US-2002120651-A1August 29, 2002Lingomotors, Inc.Natural language search method and system for electronic books
    US-7263484-B1August 28, 2007Georgia Tech Research CorporationPhonetic searching
    US-8620659-B2December 31, 2013Voicebox Technologies, Inc.System and method of supporting adaptive misrecognition in conversational speech
    US-8713119-B2April 29, 2014Apple Inc.Electronic devices with voice command and contextual data processing capabilities
    US-9842168-B2December 12, 2017Microsoft Technology Licensing, LlcTask driven user intents
    US-8712776-B2April 29, 2014Apple Inc.Systems and methods for selective text to speech synthesis
    US-6847966-B1January 25, 2005Engenium CorporationMethod and system for optimally searching a document database using a representative semantic space
    US-8131458-B1March 06, 2012Boadin Technology, LLCSystem, method, and computer program product for instant messaging utilizing a vehicular assembly
    US-8751238-B2June 10, 2014Apple Inc.Systems and methods for determining the language to use for speech generated by a text to speech engine
    US-9263039-B2February 16, 2016Nuance Communications, Inc.Systems and methods for responding to natural language speech utterance
    US-8977255-B2March 10, 2015Apple Inc.Method and system for operating a multi-function portable electronic device using voice-activation
    US-7120574-B2October 10, 2006Invention Machine CorporationSynonym extension of search queries with validation
    US-7983917-B2July 19, 2011Voicebox Technologies, Inc.Dynamic speech sharpening
    US-9535906-B2January 03, 2017Apple Inc.Mobile device having human language translation capability with positional feedback
    US-2003108241-A1June 12, 2003Koninklijke Philips Electronics N.V.Mood based virtual photo album
    US-7809570-B2October 05, 2010Voicebox Technologies, Inc.Systems and methods for responding to natural language speech utterance
    US-8015006-B2September 06, 2011Voicebox Technologies, Inc.Systems and methods for processing natural language speech utterances with context-specific domain agents
    US-8117242-B1February 14, 2012Boadin Technology, LLCSystem, method, and computer program product for performing a search in conjunction with use of an online application
    US-9412392-B2August 09, 2016Apple Inc.Electronic devices with voice command and contextual data processing capabilities
    US-2002059069-A1May 16, 2002Cheng Hsu, Veera BoonjingNatural language interface
    US-8046313-B2October 25, 2011Hoffberg Steven M, Linda Irene Hoffberg-BorghesaniErgonomic man-machine interface incorporating adaptive pattern recognition based control system
    US-9311391-B2April 12, 2016Telecom Italia S.P.A.Method and system of content recommendation
    US-9646614-B2May 09, 2017Apple Inc.Fast, language-independent method for user authentication by voice
    US-7783633-B2August 24, 2010International Business Machines CorporationDisplay of results of cross language search
    US-9368114-B2June 14, 2016Apple Inc.Context-sensitive handling of interruptions
    US-9501741-B2November 22, 2016Apple Inc.Method and apparatus for building an intelligent automated assistant
    US-2013346068-A1December 26, 2013Apple Inc.Voice-Based Image Tagging and Searching
    US-6823308-B2November 23, 2004Canon Kabushiki KaishaSpeech recognition accuracy in a multimodal input system
    US-9582608-B2February 28, 2017Apple Inc.Unified ranking with entropy-weighted information for phrase-based semantic auto-completion
    US-2002116176-A1August 22, 2002Valery Tsourikov, Leonid Batchilo, Igor Sovpel, Alexander KorzunSemantic answering system and method
    US-8194099-B2June 05, 2012Apple Inc.Techniques for displaying digital images on a display
    WO-0016243-A1March 23, 2000Mate - Media Access Technologies Ltd.Procede d'indexation de visages servant dans la navigation et recherche efficaces de personnes dans des images video
    US-2003028564-A1February 06, 2003Lingomotors, Inc.Natural language method and system for matching and ranking documents in terms of semantic relatedness
    US-9865280-B2January 09, 2018Apple Inc.Structured dictation using intelligent automated assistants
    US-8706503-B2April 22, 2014Apple Inc.Intent deduction based on previous user interactions with voice assistant