What are stemming and lemmatization in NLP?

Stemming and lemmatization, are two methods used in Natural Language Processing (NLP) to standardize text and get words or documents ready for some more machine learning processing. 

Though these are two of the most popular canonicalization techniques, they happen to have certain limitations, our task is to reduce the words to their root form for pre-processing and this technique is known as canonicalization. Canonicalization is a method for transforming data with many representations into a standard or normal form. In this article, we’ll try to understand these limitations and how phonetic hashing comes to the rescue.

What is the difference between stemming and lemmatization?

Stemming is a canonicalization technique that attempts to reduce a word down to its root form by dropping its affixes. Lemmatization is another canonicalization technique, that tries to map a word to its lemma – thus, for correct results using lemmatization, the words must be spelled correctly in the corpus.

Limitations of Stemming and Lemmatization

A significant issue occurs in both methods when dealing with words with multiple variants of spellings due to different pronunciations. For example, the words ‘Colour’ and ‘Color’ might be treated differently by a stemmer, although they both mean the same thing. Likewise, ‘traveling’ and ‘traveling’ would give rise to two stems/lemmas despite being the variations of the same word. 

In this article, we will cover word similarity matching using the Soundex algorithm in Python.