Understanding Stemmers (Natural Language Processing)

Post image

I am interested in NLP and have already some experience with Apache Solr. It’s time to dig a little in-deep regarding stemmers. First of all, I was looking for a general definition of what a stemmer is, and I found this one, which IMHO is quite good:

stemmer — an algorithm for removing inflectional and derivational endings in order to reduce word forms to a common stem

So what a stemmer does is nothing more, than converting words to their word stem. For example, the three words developing, developer, development will be converted to develop. Therefore, stemmers are often used as filters.

This can be very handy in different situaltions e.g. when writing to a fulltext search index like SOLR or Elasticsearch. There are plenty of different stemming algorithms out there. To use them in a project, some different things have to be concerned:


Stemming is based on common patterns, so it works a bit different in English than e.g. in German. Some stemmers support a variety of languages, some are only available in English. I would not recommend to use a stemmer in another language than your content is, because it maybe works, but only maybe.

Here is a list of different stemmers which can be used in Apache SOLR, including supported languages.

Time consumed

As with every algorithm, there are faster and slower ones. Depending on where they are used, performance could have a huge impact on success. Generally said: Slower does not necessarily mean more precise.


Because a stemmer does not really “understand” the language of the content which it’s processing, it all depends on the patterns which are applied. The PorterStemmer for example removes word endings like “e”, “er” and “ing”. Therefore, the following conversion will be made:

horse, horses > hors

develoment, developer, developing > develop

Depending on the task to be done, this can be great or totally disfunctional. I would recommend to try out different stemming algorithms whith some edge-cases of specific domain and evaluating the result. For many stemmers, there is good documentation describing the agressiveness of the algorithm.

Different stemmers

Usage in Java

For a recent project, I used some of the SOLR / Lucene filters in Java. Just a short example how they can be used:

public class EnglishTokenizer implements TokenizerInterface {

    public Collection<String> tokenize(String content) throws TokenizerException {
        try {
            // read content
            StringReader inputText = new StringReader(content);
            Map<String, String> tkargs = new HashMap<String, String>();
            tkargs.put("luceneMatchVersion", "LUCENE_51");

            // char filter (html)
            CharFilterFactory hcff = new HTMLStripCharFilterFactory(tkargs);
            Reader strippedInput = hcff.create(inputText);

            // tokenizer
            TokenizerFactory tkf = new StandardTokenizerFactory(tkargs);
            Tokenizer tkz = tkf.create();

            // stopwords filter
            Map<String, String> stfargs = new HashMap<String, String>();
            stfargs.put("luceneMatchVersion", "LUCENE_51");
            stfargs.put("words", "lucene/en/stopwords.txt");
            stfargs.put("ignoreCase", "true");
            StopFilterFactory stff = new StopFilterFactory(stfargs);
            stff.inform(new ClasspathResourceLoader());
            TokenStream stfts = stff.create(tkz);

            // K stem filter
            Map<String, String> ksffparam = new HashMap<String, String>();
            KStemFilterFactory ksff = new KStemFilterFactory(ksffparam);
            TokenStream ksts = ksff.create(stfts);

            // synonyms filter
            Map<String, String> syffargs = new HashMap<String, String>();
            syffargs.put("luceneMatchVersion", "LUCENE_51");
            syffargs.put("synonyms", "lucene/en/synonyms.txt");
            syffargs.put("ignoreCase", "true");
            syffargs.put("expand", "false");
            SynonymFilterFactory syff = new SynonymFilterFactory(syffargs);
            syff.inform(new ClasspathResourceLoader());
            TokenStream syfts = syff.create(ksts);

            // lower case filter
            LowerCaseFilterFactory lcf = new LowerCaseFilterFactory(tkargs);
            TokenStream ts = lcf.create(syfts);

            // process token stream

            CharTermAttribute termAttrib = (CharTermAttribute) ts.getAttribute(CharTermAttribute.class);

            Collection<String> tokens = new ArrayList<String>();
            while (ts.incrementToken()) {


            return tokens;
        } catch (IOException e) {
            throw new TokenizerException(e);

In the above example, a set of filters is applied to a string to tokenize, filter (and stem) words. For stemming, the KStem filter is used. The filters are all from the package org.apache.solr:solr-core:5.2.1 which is available on the maven repository.

Tokeinzer Example


Java Developer - Zurich - financial markets - Salary negotiable depending on experience. A number of exciting Java opportunities exist to join a highly successful and growing provider…

Result (Tokens)

java, development, zurich, financial, market, salary, negotiable, depend, experience, number, exciting, java, opportunity, exist, join, highly, successful, grow, provider

Sources / More Information

You May Also Like

How a Strong Type System Saves You Documentation, Tests, and Nerves

How a Strong Type System Saves You Documentation, Tests, and Nerves

I was recently inspired to finally write this post. Especially in weakly- or untyped languages, such as the JavaScript or PHP world, the added value of strict type systems is often not recognized. Instead, many discussions and comments revolve around the need for tests or code comments. Contrary to that, in the functional programming world, we leave such checks to the compiler. In this post I would like to give a short overview and explain how to use a strict type system for everyday checks instead of writing type checks, tests and documentation for it.

How to effectively visualize an Application Landscape in Enterprise Architecture

How to effectively visualize an Application Landscape in Enterprise Architecture

In enterprise & solution architecture, connecting boxes with arrows is an often used and overrated visualization from high-level, thru component architecture, down to data and class diagrams. However, to create a holistic view on systems, component diagrams are not enough! When it comes to analysis or transformation of a high-level application- or service-architecture, I prefer to draw an Application Landscape Diagram, which I would like to show and elaborate on in this post.