Sentence boundary disambiguation

Sentence boundary disambiguation (SBD), also known as sentence breaking, sentence boundary detection, and sentence segmentation, is the problem in natural language processing of deciding where sentences begin and end. Natural language processing tools often require their input to be divided into sentences; however, sentence boundary identification can be challenging due to the potential ambiguity of punctuation marks. In written English, a period may indicate the end of a sentence, or may denote an abbreviation, a decimal point, an ellipsis, or an email address, among other possibilities. About 47% of the periods in The Wall Street Journal corpus denote abbreviations.[1] Question marks and exclamation marks can be similarly ambiguous due to use in emoticons, source code, and slang.

Some languages including Japanese and Chinese have unambiguous sentence-ending markers.

## Strategies

[edit]

The standard 'vanilla' approach to locate the end of a sentence:[clarification needed]

(a) If it is a period, it ends a sentence.
(b) If the preceding token is in the hand-compiled list of abbreviations, then it does not end a sentence.
(c) If the next token is capitalized, then it ends a sentence.

This strategy gets about 95% of sentences correct.[2] Things such as shortened names, e.g. "D. H. Lawrence" (with whitespaces between the individual words that form the full name), idiosyncratic orthographical spellings used for stylistic purposes (often referring to a single concept, e.g. an entertainment product title like ".hack//SIGN") and usage of non-standard punctuation (or non-standard usage of punctuation) in a text often fall under the remaining 5%.

Another approach is to automatically learn a set of rules from a set of documents where the sentence breaks are pre-marked. Solutions have been based on a maximum entropy model.[3] The SATZ[4] architecture uses a neural network to disambiguate sentence boundaries and achieves 98.5% accuracy.

## Software

[edit]

Examples of use of Perl compatible regular expressions ("PCRE")

* `((?<=[a-z0-9][.?!])|(?<=[a-z0-9][.?!]\"))(\s|\r\n)(?=\"?[A-Z])`
* `$sentences = preg_split("/(?<!\..)([\?\!\.]+)\s(?!.\.)/", $text, -1, PREG_SPLIT_DELIM_CAPTURE);` (for PHP)

Online use, libraries, and APIs

* sent_detector - Java[5]
* Lingua-EN-Sentence - perl[6]
* Sentence.pm - perl[7]
* SATZ - An Adaptive Sentence Segmentation System - by David D. Palmer - C[4]

Toolkits that include sentence detection

* Apache OpenNLP[8]
* Freeling (software)[9]
* Natural Language Toolkit[10]
* Stanford NLP[11]
* GExp[12]
* CogComp-NLP[13]

## See also

[edit]

* Multiword expression
* Punctuation
* Sentence extraction
* Sentence spacing
* Speech segmentation
* Syllabification
* Text segmentation
* Translation memory
* Word divider

## References

[edit]

1. ^ E. Stamatatos; N. Fakotakis & G. Kokkinakis. "Automatic extraction of rules for sentence boundary disambiguation". Proceedings of the Workshop on Machine Learning in Human Language Technology. University of Patras. pp. 88–92.
2. ^ O'Neil, John. "Doing Things with Words, Part Two: Sentence Boundary Detection". Archived from the original on 2009-02-21. Retrieved 2009-01-03.
3. ^ Reynar, JC; Ratnaparkhi, A. "A Maximum Entropy Approach to Identifying Sentence Boundaries" (PDF). Retrieved 2009-01-03.
4. ^ a b "SATZ: An Adaptive Sentence Boundary Detector". Archived from the original on 2007-09-22.
5. ^ "SentParBreaker Web page". Archived from the original on 2007-11-12.
6. ^ "Lingua-EN-Sentence-0.25 - Module for splitting text into sentences. - metacpan.org". metacpan.org.
7. ^ "Text::Sentence - module for splitting text into sentences - metacpan.org". metacpan.org.
8. ^ "Apache OpenNLP". opennlp.apache.org.
9. ^ "Welcome | FreeLing Home Page".
10. ^ "NLTK :: Natural Language Toolkit". www.nltk.org.
11. ^ "Software - The Stanford Natural Language Processing Group". nlp.stanford.edu.
12. ^ "Google Code Archive - Long-term storage for Google Code Project Hosting". code.google.com.
13. ^ "CogCompNLP". January 2, 2024 - via GitHub.

## External links

[edit]

* pySBD - python Sentence Boundary Disambiguation

* v
* t
* e

Natural language processing

General terms|

* AI-complete
* Bag-of-words
* n-gram
* Bigram
* Trigram
* Computational linguistics
* Natural language understanding
* Stop words
* Text processing

Text analysis|

* Argument mining
* Collocation extraction
* Concept mining
* Coreference resolution
* Deep linguistic processing
* Distant reading
* Information extraction
* Named-entity recognition
* Ontology learning
* Parsing
* Semantic parsing
* Syntactic parsing
* Part-of-speech tagging
* Semantic analysis
* Semantic role labeling
* Semantic decomposition
* Semantic similarity
* Sentiment analysis

* Terminology extraction
* Text mining
* Textual entailment
* Truecasing
* Word-sense disambiguation
* Word-sense induction

| Text segmentation|

* Compound-term processing
* Lemmatisation
* Lexical analysis
* Text chunking
* Stemming
* Sentence segmentation
* Word segmentation

|

Automatic summarization|

* Multi-document summarization
* Sentence extraction
* Text simplification

Machine translation|

* Computer-assisted
* Example-based
* Rule-based
* Statistical
* Transfer-based
* Neural

Distributional semantics models|

* BERT
* Document-term matrix
* Explicit semantic analysis
* fastText
* GloVe
* Language model (large)
* Latent semantic analysis
* Seq2seq
* Word embedding
* Word2vec

Language resources,
datasets and corpora|

| Types and
standards|

* Corpus linguistics
* Lexical resource
* Linguistic Linked Open Data
* Machine-readable dictionary
* Parallel text
* PropBank
* Semantic network
* Simple Knowledge Organization System
* Speech corpus
* Text corpus
* Thesaurus (information retrieval)
* Treebank
* Universal Dependencies

|

Data|

* BabelNet
* Bank of English
* DBpedia
* FrameNet
* Google Ngram Viewer
* UBY
* WordNet
* Wikidata

Automatic identification
and data capture|

* Speech recognition
* Speech segmentation
* Speech synthesis
* Natural language generation
* Optical character recognition

Topic model|

* Document classification
* Latent Dirichlet allocation
* Pachinko allocation

Computer-assisted
reviewing|

* Automated essay scoring
* Concordancer
* Grammar checker
* Predictive text
* Pronunciation assessment
* Spell checker

Natural language
user interface|

* Chatbot
* Interactive fiction
* Question answering
* Virtual assistant
* Voice user interface

Related|

* Formal semantics
* Hallucination
* Natural Language Toolkit
* spaCy

*[v]: View this template
*[t]: Discuss this template
*[e]: Edit this template