Multimedia information retrieval

This article has multiple issues. Please help improve it or discuss these issues on the talk page. (Learn how and when to remove these messages) A major contributor to this article appears to have a close connection with its subject. It may require cleanup to comply with Wikipedia's content policies, particularly neutral point of view. Please discuss further on the talk page. (July 2014) (Learn how and when to remove this message) This article possibly contains original research. Please improve it by verifying the claims made and adding inline citations. Statements consisting only of original research should be removed. (July 2014) (Learn how and when to remove this message) (Learn how and when to remove this message)

Multimedia information retrieval (MMIR or MIR) is a research discipline of computer science that aims at extracting semantic information from multimedia data sources.^{[1][failed verification]} Data sources include directly perceivable media such as audio, image and video, indirectly perceivable sources such as text, semantic descriptions,^[2] biosignals as well as not perceivable sources such as bioinformation, stock prices, etc. The methodology of MMIR can be organized in three groups:

1. Methods for the summarization of media content (feature extraction). The result of feature extraction is a description.
2. Methods for the filtering of media descriptions (for example, elimination of redundancy)
3. Methods for the categorization of media descriptions into classes.

Feature extraction is motivated by the sheer size of multimedia objects as well as their redundancy and, possibly, noisiness.^{[1]: 2 [failed verification]} Generally, two possible goals can be achieved by feature extraction:

• Summarization of media content. Methods for summarization include in the audio domain, for example, mel-frequency cepstral coefficients, Zero Crossings Rate, Short-Time Energy. In the visual domain, color histograms^[3] such as the MPEG-7 Scalable Color Descriptor can be used for summarization.
• Detection of patterns by auto-correlation and/or cross-correlation. Patterns are recurring media chunks that can either be detected by comparing chunks over the media dimensions (time, space, etc.) or comparing media chunks to templates (e.g. face templates, phrases). Typical methods include Linear Predictive Coding in the audio/biosignal domain,^[4] texture description in the visual domain and n-grams in text information retrieval.

Multimedia Information Retrieval implies that multiple channels are employed for the understanding of media content.^[5] Each of this channels is described by media-specific feature transformations. The resulting descriptions have to be merged to one description per media object. Merging can be performed by simple concatenation if the descriptions are of fixed size. Variable-sized descriptions – as they frequently occur in motion description – have to be normalized to a fixed length first.

Frequently used methods for description filtering include factor analysis (e.g. by PCA), singular value decomposition (e.g. as latent semantic indexing in text retrieval) and the extraction and testing of statistical moments. Advanced concepts such as the Kalman filter are used for merging of descriptions.

Generally, all forms of machine learning can be employed for the categorization of multimedia descriptions^{[1]: 125 [failed verification]} though some methods are more frequently used in one area than another. For example, hidden Markov models are state-of-the-art in speech recognition, while dynamic time warping – a semantically related method – is state-of-the-art in gene sequence alignment. The list of applicable classifiers includes the following:

• Metric approaches (Cluster analysis, vector space model, Minkowski distances, dynamic alignment)
• Nearest Neighbor methods (K-nearest neighbors algorithm, K-means, self-organizing map)
• Risk Minimization (Support vector regression, support vector machine, linear discriminant analysis)
• Density-based Methods (Bayes nets, Markov processes, mixture models)
• Neural Networks (Perceptron, associative memories, spiking nets)
• Heuristics (Decision trees, random forests, etc.)

The selection of the best classifier for a given problem (test set with descriptions and class labels, so-called ground truth) can be performed automatically, for example, using the Weka Data Miner.

The quality of MMIR Systems^[6] depends heavily on the quality of the training data. Discriminative descriptions can be extracted from media sources in various forms. Machine learning provides categorization methods for all types of data. However, the classifier can only be as good as the given training data. On the other hand, it requires considerable effort to provide class labels for large databases. The future success of MMIR will depend on the provision of such data.^[7] The annual TRECVID competition is currently one of the most relevant sources of high-quality ground truth.

MMIR provides an overview over methods employed in the areas of information retrieval.^[8][9] Methods of one area are adapted and employed on other types of media. Multimedia content is merged before the classification is performed. MMIR methods are, therefore, usually reused from other areas such as:

• Bioinformation analysis
• Biosignal processing
• Content-based image and video retrieval
• Face recognition
• Audio and music classification (Music information retrieval)
• Automatic content recognition
• Speech recognition
• Technical chart analysis
• Video browsing
• Text information retrieval
• Image retrieval
• Learning to rank

The International Journal of Multimedia Information Retrieval^[10] documents the development of MMIR as a research discipline that is independent of these areas. See also Handbook of Multimedia Information Retrieval^[11] for a complete overview over this research discipline.