stringdist_inner_join: Correcting misspellings against a dictionaryOften you find yourself with a set of words that you want to combine with a “dictionary”- it could be a literal dictionary (as in this case) or a domain-specific category system. But you want to allow for small differences in spelling or punctuation.
The fuzzyjoin package comes with a set of common misspellings (from Wikipedia):
library(dplyr)## Warning: package 'dplyr' was built under R version 3.5.2library(fuzzyjoin)
data(misspellings)
misspellings## # A tibble: 4,505 x 2
## misspelling correct
## <chr> <chr>
## 1 abandonned abandoned
## 2 aberation aberration
## 3 abilties abilities
## 4 abilty ability
## 5 abondon abandon
## 6 abbout about
## 7 abotu about
## 8 abouta about a
## 9 aboutit about it
## 10 aboutthe about the
## # … with 4,495 more rows# use the dictionary of words from the qdapDictionaries package,
# which is based on the Nettalk corpus.
library(qdapDictionaries)
words <- tbl_df(DICTIONARY)
words## # A tibble: 20,137 x 2
## word syllables
## <chr> <dbl>
## 1 hm 1
## 2 hmm 1
## 3 hmmm 1
## 4 hmph 1
## 5 mmhmm 2
## 6 mmhm 2
## 7 mm 1
## 8 mmm 1
## 9 mmmm 1
## 10 pff 1
## # … with 20,127 more rowsAs an example, we’ll pick 1000 of these words (you could try it on all of them though), and use stringdist_inner_join to join them against our dictionary.
set.seed(2016)
sub_misspellings <- misspellings %>%
sample_n(1000)joined <- sub_misspellings %>%
stringdist_inner_join(words, by = c(misspelling = "word"), max_dist = 1)By default, stringdist_inner_join uses optimal string alignment (Damerau–Levenshtein distance), and we’re setting a maximum distance of 1 for a join. Notice that they’ve been joined in cases where misspelling is close to (but not equal to) word:
joined## # A tibble: 728 x 4
## misspelling correct word syllables
## <chr> <chr> <chr> <dbl>
## 1 sould should could 1
## 2 sould should should 1
## 3 sould should sold 1
## 4 sould should soul 1
## 5 sould should sound 1
## 6 sould should would 1
## 7 fiels feels field 1
## 8 fiels feels fils 1
## 9 conscent consent consent 2
## 10 fleed freed bleed 1
## # … with 718 more rowsNote that there are some redundancies; words that could be multiple items in the dictionary. These end up with one row per “guess” in the output. How many words did we classify?
joined %>%
count(misspelling, correct)## # A tibble: 455 x 3
## misspelling correct n
## <chr> <chr> <int>
## 1 abritrary arbitrary 1
## 2 accademic academic 1
## 3 accension ascension 2
## 4 accessable accessible 1
## 5 accidant accident 1
## 6 accidentaly accidentally 1
## 7 accordeon accordion 1
## 8 addopt adopt 1
## 9 addtional additional 1
## 10 admendment amendment 1
## # … with 445 more rowsSo we found a match in the dictionary for about half of the misspellings. In how many of the ones we classified did we get at least one of our guesses right?
which_correct <- joined %>%
group_by(misspelling, correct) %>%
summarize(guesses = n(), one_correct = any(correct == word))
which_correct## # A tibble: 455 x 4
## # Groups: misspelling [444]
## misspelling correct guesses one_correct
## <chr> <chr> <int> <lgl>
## 1 abritrary arbitrary 1 TRUE
## 2 accademic academic 1 TRUE
## 3 accension ascension 2 TRUE
## 4 accessable accessible 1 TRUE
## 5 accidant accident 1 TRUE
## 6 accidentaly accidentally 1 FALSE
## 7 accordeon accordion 1 TRUE
## 8 addopt adopt 1 TRUE
## 9 addtional additional 1 TRUE
## 10 admendment amendment 1 TRUE
## # … with 445 more rows# percentage of guesses getting at least one right
mean(which_correct$one_correct)## [1] 0.8527473# number uniquely correct (out of the original 1000)
sum(which_correct$guesses == 1 & which_correct$one_correct)## [1] 294Not bad.
Note that stringdist_inner_join is not the only function we can use. If we’re interested in including the words that we couldn’t classify, we could have used stringdist_left_join:
left_joined <- sub_misspellings %>%
stringdist_left_join(words, by = c(misspelling = "word"), max_dist = 1)
left_joined## # A tibble: 1,273 x 4
## misspelling correct word syllables
## <chr> <chr> <chr> <dbl>
## 1 charactors characters <NA> NA
## 2 Brasillian Brazilian <NA> NA
## 3 sould should could 1
## 4 sould should should 1
## 5 sould should sold 1
## 6 sould should soul 1
## 7 sould should sound 1
## 8 sould should would 1
## 9 belligerant belligerent <NA> NA
## 10 incorportaed incorporated <NA> NA
## # … with 1,263 more rowsleft_joined %>%
filter(is.na(word))## # A tibble: 545 x 4
## misspelling correct word syllables
## <chr> <chr> <chr> <dbl>
## 1 charactors characters <NA> NA
## 2 Brasillian Brazilian <NA> NA
## 3 belligerant belligerent <NA> NA
## 4 incorportaed incorporated <NA> NA
## 5 awya away <NA> NA
## 6 occuring occurring <NA> NA
## 7 surveilence surveillance <NA> NA
## 8 abondoned abandoned <NA> NA
## 9 alledges alleges <NA> NA
## 10 deliberatly deliberately <NA> NA
## # … with 535 more rows(To get just the ones without matches immediately, we could have used stringdist_anti_join). If we increase our distance threshold, we’ll increase the fraction with a correct guess, but also get more false positive guesses:
left_joined2 <- sub_misspellings %>%
stringdist_left_join(words, by = c(misspelling = "word"), max_dist = 2)
left_joined2## # A tibble: 7,691 x 4
## misspelling correct word syllables
## <chr> <chr> <chr> <dbl>
## 1 charactors characters character 3
## 2 charactors characters charactery 4
## 3 Brasillian Brazilian <NA> NA
## 4 sould should auld 1
## 5 sould should bold 1
## 6 sould should bound 1
## 7 sould should cold 1
## 8 sould should could 1
## 9 sould should fold 1
## 10 sould should foul 1
## # … with 7,681 more rowsleft_joined2 %>%
filter(is.na(word))## # A tibble: 264 x 4
## misspelling correct word syllables
## <chr> <chr> <chr> <dbl>
## 1 Brasillian Brazilian <NA> NA
## 2 belligerant belligerent <NA> NA
## 3 occuring occurring <NA> NA
## 4 abondoned abandoned <NA> NA
## 5 correponding corresponding <NA> NA
## 6 archeaologist archaeologist <NA> NA
## 7 emmediately immediately <NA> NA
## 8 possessess possesses <NA> NA
## 9 unahppy unhappy <NA> NA
## 10 Guilio Giulio <NA> NA
## # … with 254 more rowsMost of the missing words here simply aren’t in our dictionary.
You can try other distance thresholds, other dictionaries, and other distance metrics (see stringdist-metrics for more). This function is especially useful on a domain-specific dataset, such as free-form survey input that is likely to be close to one of a handful of responses.