Improvements in Spell correction model

manish · May 11, 2020, 6:57am

Some of the improvements in error model,

In the below code snippet, we are iterating over all prefixes. Can we use LevenshteinSearcher style to lookup candidates up to certain edit distance with combination of error model probabilities.

deepmipt/DeepPavlov/blob/c10b079b972493220c82a643d47d718d5358c7f4/deeppavlov/models/spelling_correction/brillmoore/error_model.py#L110


threshold = log(prop_threshold)
word = '⟬{}⟭'.format(word.lower().replace('ё', 'е'))
word_len = len(word) + 1
inf = float('-inf')
d = defaultdict(list)
d[''] = [0.] + [inf] * (word_len - 1)
prefixes_heap = [(0, self.dictionary.words_trie[''])]
candidates = [(inf, '')] * self.candidates_count
while prefixes_heap and -prefixes_heap[0][0] > candidates[0][0]:
    _, prefixes = heappop(prefixes_heap)
    for prefix in prefixes:
        prefix_len = len(prefix)
        d[prefix] = res = [inf]
        for i in range(1, word_len):
            c_res = [inf]
            for li in range(1, min(prefix_len + 1, self.window + 2)):
                for ri in range(1, min(i + 1, self.window + 2)):
                    prev = d[prefix[:-li]][i - ri]
                    if prev > threshold:
                        edit = (prefix[-li:], word[i - ri:i])
                        if edit in self.costs:

Changes required to support “Phrase based” error probabilities. I have come across this but not sure how this can be fit in current settings.

Thanks a lot !!

Topic		Replies	Views
Predicting user input in search field (autocomplete) DeepPavlov Library	1	38	May 20, 2024
Как добавить дополнительный словарь в файл levenshtein_corrector_ru.json? DeepPavlov Library	0	222	November 3, 2021
"Оценка качества "Automatic spelling correction component"" was flagged as spam Models	1	359	June 25, 2019
Lemmatization using DeepPavlov pre-trained models Models	3	573	February 16, 2021
Paraphrase detection model Models	4	1156	May 25, 2020

Improvements in Spell correction model

Related topics