Lingua-Go, the most accurate language detection for Go

Lingua-Go, the most accurate language detection for Go


Build Status
supported languages
Go Reference
Go Report Card

Table of Contents

  1. What does this library do?
  2. Why does this library exist?
  3. Which languages are supported?
  4. How good is it?
  5. Why is it better than other libraries?
  6. Test report generation
  7. How to add it to your project?
  8. How to build?
  9. How to use?
  10. What’s next for version 1.1.0?
  11. Contributions

1. What does this library do? Top ▲

Its task is simple: It tells you which language some provided textual data is written in.
This is very useful as a preprocessing step for linguistic data in natural language
processing applications such as text classification and spell checking.
Other use cases, for instance, might include routing e-mails to the right geographically
located customer service department, based on the e-mails’ languages.

2. Why does this library exist? Top ▲

Language detection is often done as part of large machine learning frameworks or natural
language processing applications. In cases where you don’t need the full-fledged
functionality of those systems or don’t want to learn the ropes of those,
a small flexible library comes in handy.

So far, the only other comprehensive open source library in the Go ecosystem for
this task is Whatlanggo.
Unfortunately, it has two major drawbacks:

  1. Detection only works with quite lengthy text fragments. For very short text snippets
    such as Twitter messages, it does not provide adequate results.
  2. The more languages take part in the decision process, the less accurate are the
    detection results.

Lingua aims at eliminating these problems. It nearly does not need any configuration and
yields pretty accurate results on both long and short text, even on single words and phrases.
It draws on both rule-based and statistical methods but does not use any dictionaries of words.
It does not need a connection to any external API or service either.
Once the library has been downloaded, it can be used completely offline.

3. Which languages are supported? Top ▲

Compared to other language detection libraries, Lingua’s focus is on quality over quantity, that is,
getting detection right for a small set of languages first before adding new ones.
Currently, the following 75 languages are supported:

  • A
    • Afrikaans
    • Albanian
    • Arabic
    • Armenian
    • Azerbaijani
  • B
    • Basque
    • Belarusian
    • Bengali
    • Norwegian Bokmal
    • Bosnian
    • Bulgarian
  • C
    • Catalan
    • Chinese
    • Croatian
    • Czech
  • D
    • Danish
    • Dutch
  • E
    • English
    • Esperanto
    • Estonian
  • F
    • Finnish
    • French
  • G
    • Ganda
    • Georgian
    • German
    • Greek
    • Gujarati
  • H
    • Hebrew
    • Hindi
    • Hungarian
  • I
    • Icelandic
    • Indonesian
    • Irish
    • Italian
  • J
    • Japanese
  • K
    • Kazakh
    • Korean
  • L
    • Latin
    • Latvian
    • Lithuanian
  • M
    • Macedonian
    • Malay
    • Maori
    • Marathi
    • Mongolian
  • N
    • Norwegian Nynorsk
  • P
    • Persian
    • Polish
    • Portuguese
    • Punjabi
  • R
    • Romanian
    • Russian
  • S
    • Serbian
    • Shona
    • Slovak
    • Slovene
    • Somali
    • Sotho
    • Spanish
    • Swahili
    • Swedish
  • T
    • Tagalog
    • Tamil
    • Telugu
    • Thai
    • Tsonga
    • Tswana
    • Turkish
  • U
    • Ukrainian
    • Urdu
  • V
    • Vietnamese
  • W
    • Welsh
  • X
    • Xhosa
  • Y
    • Yoruba
  • Z
    • Zulu

4. How good is it? Top ▲

Lingua is able to report accuracy statistics for some bundled test data available for each
supported language. The test data for each language is split into three parts:

  1. a list of single words with a minimum length of 5 characters
  2. a list of word pairs with a minimum length of 10 characters
  3. a list of complete grammatical sentences of various lengths

Both the language models and the test data have been created from separate documents of the
Wortschatz corpora offered by Leipzig University, Germany.
Data crawled from various news websites have been used for training, each corpus comprising one
million sentences. For testing, corpora made of arbitrarily chosen websites have been used,
each comprising ten thousand sentences. From each test corpus, a random unsorted subset of
1000 single words, 1000 word pairs and 1000 sentences has been extracted, respectively.

Given the generated test data, I have compared the detection results of Lingua and Whatlanggo
running over the data of Lingua’s supported 75 languages. Additionally, I have added Google‘s
CLD3 to the comparison with the help of the
gocld3 bindings. Languages that are not supported
by CLD3 or Whatlanggo are simply ignored during the detection process.

The box plot below shows the distribution of the averaged accuracy values for all three performed tasks:
Single word detection, word pair detection and sentence detection. Lingua clearly outperforms its contenders.
Bar plots for each language and further box plots for the separate detection tasks can be found in the file
Detailed statistics including mean, median and standard deviation values for each language and classifier are
available in the file

Average Detection Performance

5. Why is it better than other libraries? Top ▲

Every language detector uses a probabilistic n-gram model trained on the
character distribution in some training corpus. Most libraries only use n-grams of size 3 (trigrams) which is
satisfactory for detecting the language of longer text fragments consisting of multiple sentences. For short
phrases or single words, however, trigrams are not enough. The shorter the input text is, the less n-grams are
available. The probabilities estimated from such few n-grams are not reliable. This is why Lingua makes use
of n-grams of sizes 1 up to 5 which results in much more accurate prediction of the correct language.

A second important difference is that Lingua does not only use such a statistical model, but also a rule-based
engine. This engine first determines the alphabet of the input text and searches for characters which are unique
in one or more languages. If exactly one language can be reliably chosen this way, the statistical model is not
necessary anymore. In any case, the rule-based engine filters out languages that do not satisfy the conditions
of the input text. Only then, in a second step, the probabilistic n-gram model is taken into consideration.
This makes sense because loading less language models means less memory consumption and better runtime performance.

In general, it is always a good idea to restrict the set of languages to be considered in the classification process
using the respective api methods. If you know beforehand that certain languages are
never to occur in an input text, do not let those take part in the classifcation process. The filtering mechanism
of the rule-based engine is quite good, however, filtering based on your own knowledge of the input text is always preferable.

6. Test report generation Top ▲

If you want to reproduce the accuracy results above, you can generate the test reports yourself for both classifiers
and all languages by doing:

cd cmd
go run accuracy_reporter.go

For gocld3 to run successfully, you need to install the exact
version 3.17.3 of Google‘s protocol buffers which is a bit
unfortunate. For each detector and language, a test report file is then written into
As an example, here is the current output of the Lingua German report:

7. How to add it to your project? Top ▲
go get

8. How to build? Top ▲

Lingua requires at least Go version 1.16.

git clone
cd lingua-go
go build

The source code is accompanied by an extensive unit test suite. To run them, simply say:

9. How to use? Top ▲

9.1 Basic usage

package main

import (

func main() {
    languages := []lingua.Language{

    detector := lingua.NewLanguageDetectorBuilder().

    if language, exists := detector.DetectLanguageOf("languages are awesome"); exists {

    // Output: English

9.2 Minimum relative distance

By default, Lingua returns the most likely language for a given input text. However, there are
certain words that are spelled the same in more than one language. The word prologue, for
instance, is both a valid English and French word. Lingua would output either English or
French which might be wrong in the given context. For cases like that, it is possible to
specify a minimum relative distance that the logarithmized and summed up probabilities for
each possible language have to satisfy. It can be stated in the following way:

package main

import (

func main() {
    languages := []lingua.Language{

    detector := lingua.NewLanguageDetectorBuilder().

    language, exists := detector.DetectLanguageOf("languages are awesome")


    // Output:
    // Unknown
    // false

Be aware that the distance between the language probabilities is dependent on the length of the
input text. The longer the input text, the larger the distance between the languages. So if you
want to classify very short text phrases, do not set the minimum relative distance too high.
Otherwise Unknown will be
returned most of the time as in the example above. This is the return value for cases where
language detection is not reliably possible. This value is not meant to be included in the set
of input languages when building the language detector. If you include it, it will be
automatically removed from the set of input languages.

9.3 Confidence values

Knowing about the most likely language is nice but how reliable is the computed likelihood?
And how less likely are the other examined languages in comparison to the most likely one?
These questions can be answered as well:

package main

import (

func main() {
    languages := []lingua.Language{

    detector := lingua.NewLanguageDetectorBuilder().

    confidenceValues := detector.ComputeLanguageConfidenceValues("languages are awesome")

    for _, elem := range confidenceValues {
        fmt.Printf("%s: %.2fn", elem.Language(), elem.Value())

    // Output:
    // English: 1.00
    // French: 0.79
    // German: 0.75
    // Spanish: 0.72

In the example above, a slice of
is returned containing all possible languages sorted by their confidence value in descending
order. The values that this method computes are part of a relative confidence metric, not of
an absolute one. Each value is a number between 0.0 and 1.0. The most likely language is always
returned with value 1.0. All other languages get values assigned which are lower than 1.0,
denoting how less likely those languages are in comparison to the most likely language.

The slice returned by this method does not necessarily contain all languages which the calling
instance of LanguageDetector was built from. If the rule-based engine decides that a specific
language is truly impossible, then it will not be part of the returned slice. Likewise, if no
ngram probabilities can be found within the detector’s languages for the given input text, the
returned slice will be empty. The confidence value for each language not being part of the returned
slice is assumed to be 0.0.

9.4 Eager loading versus lazy loading

By default, Lingua uses lazy-loading to load only those language models on demand which are
considered relevant by the rule-based filter engine. For web services, for instance, it is
rather beneficial to preload all language models into memory to avoid unexpected latency while
waiting for the service response. If you want to enable the eager-loading mode, you can do it
like this:


Multiple instances

Read More



“Simplicity, patience, compassion.
These three are your greatest treasures.
Simple in actions and thoughts, you return to the source of being.
Patient with both friends and enemies,
you accord with the way things are.
Compassionate toward yourself,
you reconcile all beings in the world.”
― Lao Tzu, Tao Te Ching