Reproog/novelWriter
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Remove the difflib spellchecker (#875)

Browse Source 
* Delete simple spell checker
* Clean up the spell checker code
* Update tests
* Delete the bundled dictionaries
* Update documentation
* Use set() for user word list
* Fix tests so they pass also without dictionaries installed
This commit is contained in:
Veronica Berglyd Olsen
2021-08-30 22:39:27 +02:00
committed by GitHub
parent 4b52f343c1
commit f6845a9820
24 changed files with 151 additions and 374876 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source/int_started.rst
+4 -11
View File
@@ -39,24 +39,17 @@ format of the main project file. Everything else is handled with standard Python
The following Python packages are needed to run novelWriter:
* ``pyqt5`` – needed for connecting with the Qt5 libraries.
* ``PyQt5`` – needed for connecting with the Qt5 libraries.
* ``lxml`` – needed for full XML support.
* ``pyenchant`` – needed for efficient spell checking (optional).
* ``PyEnchant`` – needed for efficient spell checking (optional).
PyQt/Qt should be at least 5.3, but ideally 5.10 or higher for nearly all features to work. For
instance, searching using regular expressions with full Unicode support requires 5.13. There is no
known minimum version requirement for package ``lxml``, but the code was originally written with
4.2, which is therefore set as the minimum. It may work on lower versions. You have to test it.
Optionally, a package can be installed to interface with the Enchant spell checking libaries, but
this isn't strictly required. If no external spell checking library is available, novelWriter falls
back to using the internal ``difflib`` of Python to check spelling. This is a slower and less
sophisticated spell checker than the full spell checking libaries. The spell check library must be
at least 3.0 to work with Windows. On Linux, 2.0 also works fine.
If you do use the internal spell checker, you also have to provide a text file with a list of all
words correctly spelled. One word on each line. Dictionary files for English is included with
novelWriter. For other languages, you will need to add more files yourself.
If you want spell checking, you must install the ``PyEnchant`` package. The spell check library
must be at least 3.0 to work with Windows. On Linux, 2.0 also works fine.
If you install from PyPi, these dependencies should be installed automatically. If you install from
source, dependencies can still be installed from PyPi with:
docs/source/setup_windows.rst
+1 -1
View File
@@ -101,7 +101,7 @@ containing either the minimal package, or the extracted full source package.
The first command will install the three main dependencies of novelWriter using the
`Python Package Index`_ install tool. The packages are ``PyQt5`` for the GUI, ``lxml`` for handling
XML data files, and ``PyEncant`` for spell checking.
XML data files, and ``PyEnchant`` for spell checking.
The second command installs a Python tool for Windows that assists the setup script in installing
icons and setting a few registry keys.
novelwriter/assets/dict/README.md
-12
View File
@@ -1,12 +0,0 @@
## Dictionary Conversion
Source: https://ftp.gnu.org/gnu/aspell/dict/0index.html
```
./configure
make
aspell --dict-dir=. -d en dump master > en.dict
aspell --dict-dir=. -d en_GB-ise-w_accents dump master > en_GB.dict
aspell --dict-dir=. -d en_US-w_accents dump master > en_US.dict
```
novelwriter/assets/dict/copyright.txt
-254
View File
@@ -1,254 +0,0 @@
This English word list is comes directly from SCOWL 2019.10.06 (up to level 60,
using the speller/make-aspell-dict script, http://wordlist.sourceforge.net/)
and is thus under the same copyright of SCOWL. The affix file (only
included in the aspell6 package) is based on the Ispell one which is
under the same copyright of Ispell. Part of SCOWL is also based on
Ispell thus the Ispell copyright is included with the SCOWL copyright.
The collective work is Copyright 2000-2018 by Kevin Atkinson as well
as any of the copyrights mentioned below:
Copyright 2000-2018 by Kevin Atkinson
Permission to use, copy, modify, distribute and sell these word
lists, the associated scripts, the output created from the scripts,
and its documentation for any purpose is hereby granted without fee,
provided that the above copyright notice appears in all copies and
that both that copyright notice and this permission notice appear in
supporting documentation. Kevin Atkinson makes no representations
about the suitability of this array for any purpose. It is provided
"as is" without express or implied warranty.
Alan Beale <biljir@pobox.com> also deserves special credit as he has,
in addition to providing the 12Dicts package and being a major
contributor to the ENABLE word list, given me an incredible amount of
feedback and created a number of special lists (those found in the
Supplement) in order to help improve the overall quality of SCOWL.
The 10 level includes the 1000 most common English words (according to
the Moby (TM) Words II [MWords] package), a subset of the 1000 most
common words on the Internet (again, according to Moby Words II), and
frequently class 16 from Brian Kelk's "UK English Wordlist
with Frequency Classification".
The MWords package was explicitly placed in the public domain:
The Moby lexicon project is complete and has
been place into the public domain. Use, sell,
rework, excerpt and use in any way on any platform.
Placing this material on internal or public servers is
also encouraged. The compiler is not aware of any
export restrictions so freely distribute world-wide.
You can verify the public domain status by contacting
Grady Ward
3449 Martha Ct.
Arcata, CA 95521-4884
grady@netcom.com
grady@northcoast.com
The "UK English Wordlist With Frequency Classification" is also in the
Public Domain:
Date: Sat, 08 Jul 2000 20:27:21 +0100
From: Brian Kelk <Brian.Kelk@cl.cam.ac.uk>
> I was wondering what the copyright status of your "UK English
> Wordlist With Frequency Classification" word list as it seems to
> be lacking any copyright notice.
There were many many sources in total, but any text marked
"copyright" was avoided. Locally-written documentation was one
source. An earlier version of the list resided in a filespace called
PUBLIC on the University mainframe, because it was considered public
domain.
Date: Tue, 11 Jul 2000 19:31:34 +0100
> So are you saying your word list is also in the public domain?
That is the intention.
The 20 level includes frequency classes 7-15 from Brian's word list.
The 35 level includes frequency classes 2-6 and words appearing in at
least 11 of 12 dictionaries as indicated in the 12Dicts package. All
words from the 12Dicts package have had likely inflections added via
my inflection database.
The 12Dicts package and Supplement is in the Public Domain.
The WordNet database, which was used in the creation of the
Inflections database, is under the following copyright:
This software and database is being provided to you, the LICENSEE,
by Princeton University under the following license. By obtaining,
using and/or copying this software and database, you agree that you
have read, understood, and will comply with these terms and
conditions.:
Permission to use, copy, modify and distribute this software and
database and its documentation for any purpose and without fee or
royalty is hereby granted, provided that you agree to comply with
the following copyright notice and statements, including the
disclaimer, and that the same appear on ALL copies of the software,
database and documentation, including modifications that you make
for internal use or for distribution.
WordNet 1.6 Copyright 1997 by Princeton University. All rights
reserved.
THIS SOFTWARE AND DATABASE IS PROVIDED "AS IS" AND PRINCETON
UNIVERSITY MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR
IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PRINCETON
UNIVERSITY MAKES NO REPRESENTATIONS OR WARRANTIES OF MERCHANT-
ABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF THE
LICENSED SOFTWARE, DATABASE OR DOCUMENTATION WILL NOT INFRINGE ANY
THIRD PARTY PATENTS, COPYRIGHTS, TRADEMARKS OR OTHER RIGHTS.
The name of Princeton University or Princeton may not be used in
advertising or publicity pertaining to distribution of the software
and/or database. Title to copyright in this software, database and
any associated documentation shall at all times remain with
Princeton University and LICENSEE agrees to preserve same.
The 40 level includes words from Alan's 3esl list found in version 4.0
of his 12dicts package. Like his other stuff the 3esl list is also in the
public domain.
The 50 level includes Brian's frequency class 1, words appearing
in at least 5 of 12 of the dictionaries as indicated in the 12Dicts
package, and uppercase words in at least 4 of the previous 12
dictionaries. A decent number of proper names is also included: The
top 1000 male, female, and Last names from the 1990 Census report; a
list of names sent to me by Alan Beale; and a few names that I added
myself. Finally a small list of abbreviations not commonly found in
other word lists is included.
The name files form the Census report is a government document which I
don't think can be copyrighted.
The file special-jargon.50 uses common.lst and word.lst from the
"Unofficial Jargon File Word Lists" which is derived from "The Jargon
File". All of which is in the Public Domain. This file also contain
a few extra UNIX terms which are found in the file "unix-terms" in the
special/ directory.
The 55 level includes words from Alan's 2of4brif list found in version
4.0 of his 12dicts package. Like his other stuff the 2of4brif is also
in the public domain.
The 60 level includes all words appearing in at least 2 of the 12
dictionaries as indicated by the 12Dicts package.
The 70 level includes Brian's frequency class 0 and the 74,550 common
dictionary words from the MWords package. The common dictionary words,
like those from the 12Dicts package, have had all likely inflections
added. The 70 level also included the 5desk list from version 4.0 of
the 12Dics package which is in the public domain.
The 80 level includes the ENABLE word list, all the lists in the
ENABLE supplement package (except for ABLE), the "UK Advanced Cryptics
Dictionary" (UKACD), the list of signature words from the YAWL package,
and the 10,196 places list from the MWords package.
The ENABLE package, mainted by M\Cooper <thegrendel@theriver.com>,
is in the Public Domain:
The ENABLE master word list, WORD.LST, is herewith formally released
into the Public Domain. Anyone is free to use it or distribute it in
any manner they see fit. No fee or registration is required for its
use nor are "contributions" solicited (if you feel you absolutely
must contribute something for your own peace of mind, the authors of
the ENABLE list ask that you make a donation on their behalf to your
favorite charity). This word list is our gift to the Scrabble
community, as an alternate to "official" word lists. Game designers
may feel free to incorporate the WORD.LST into their games. Please
mention the source and credit us as originators of the list. Note
that if you, as a game designer, use the WORD.LST in your product,
you may still copyright and protect your product, but you may *not*
legally copyright or in any way restrict redistribution of the
WORD.LST portion of your product. This *may* under law restrict your
rights to restrict your users' rights, but that is only fair.
UKACD, by J Ross Beresford <ross@bryson.demon.co.uk>, is under the
following copyright:
Copyright (c) J Ross Beresford 1993-1999. All Rights Reserved.
The following restriction is placed on the use of this publication:
if The UK Advanced Cryptics Dictionary is used in a software package
or redistributed in any form, the copyright notice must be
prominently displayed and the text of this document must be included
verbatim.
There are no other restrictions: I would like to see the list
distributed as widely as possible.
The 95 level includes the 354,984 single words, 256,772 compound
words, 4,946 female names and the 3,897 male names, and 21,986 names
from the MWords package, ABLE.LST from the ENABLE Supplement, and some
additional words found in my part-of-speech database that were not
found anywhere else.
Accent information was taken from UKACD.
The VarCon package was used to create the American, British, Canadian,
and Australian word list. It is under the following copyright:
Copyright 2000-2016 by Kevin Atkinson
Permission to use, copy, modify, distribute and sell this array, the
associated software, and its documentation for any purpose is hereby
granted without fee, provided that the above copyright notice appears
in all copies and that both that copyright notice and this permission
notice appear in supporting documentation. Kevin Atkinson makes no
representations about the suitability of this array for any
purpose. It is provided "as is" without express or implied warranty.
Copyright 2016 by Benjamin Titze
Permission to use, copy, modify, distribute and sell this array, the
associated software, and its documentation for any purpose is hereby
granted without fee, provided that the above copyright notice appears
in all copies and that both that copyright notice and this permission
notice appear in supporting documentation. Benjamin Titze makes no
representations about the suitability of this array for any
purpose. It is provided "as is" without express or implied warranty.
Since the original words lists come from the Ispell distribution:
Copyright 1993, Geoff Kuenning, Granada Hills, CA
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
3. All modifications to the source code must be clearly marked as
such. Binary redistributions based on modified source code
must be clearly marked as modified versions in the documentation
and/or other materials provided with the distribution.
(clause 4 removed with permission from Geoff Kuenning)
5. The name of Geoff Kuenning may not be used to endorse or promote
products derived from this software without specific prior
written permission.
THIS SOFTWARE IS PROVIDED BY GEOFF KUENNING AND CONTRIBUTORS ``AS IS'' AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL GEOFF KUENNING OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.
novelwriter/assets/dict/en.dict
-127250
View File
File diff suppressed because it is too large Load Diff
novelwriter/assets/dict/en_GB.dict
-123295
View File
File diff suppressed because it is too large Load Diff
novelwriter/assets/dict/en_US.dict
-123592
View File
File diff suppressed because it is too large Load Diff
novelwriter/config.py
+1 -6
View File
@@ -38,7 +38,7 @@ from PyQt5.QtCore import (
from novelwriter.error import logException
from novelwriter.common import splitVersionNumber, formatTimeStamp, NWConfigParser
from novelwriter.constants import nwConst, nwFiles, nwUnicode
from novelwriter.constants import nwFiles, nwUnicode
logger = logging.getLogger(__name__)
@@ -163,7 +163,6 @@ class Config:
self.fmtPadThin = False
# Spell Checking
self.spellTool = None
self.spellLanguage = None
# Search Bar Switches
@@ -353,8 +352,6 @@ class Config:
# Check the availability of optional packages
self._checkOptionalPackages()
if self.spellTool is None:
self.spellTool = nwConst.SP_INTERNAL
if self.spellLanguage is None:
self.spellLanguage = "en"
@@ -497,7 +494,6 @@ class Config:
self.fmtPadBefore = theConf.rdStr(cnfSec, "fmtpadbefore", self.fmtPadBefore)
self.fmtPadAfter = theConf.rdStr(cnfSec, "fmtpadafter", self.fmtPadAfter)
self.fmtPadThin = theConf.rdBool(cnfSec, "fmtpadthin", self.fmtPadThin)
self.spellTool = theConf.rdStr(cnfSec, "spelltool", self.spellTool)
self.spellLanguage = theConf.rdStr(cnfSec, "spellcheck", self.spellLanguage)
self.showTabsNSpaces = theConf.rdBool(cnfSec, "showtabsnspaces", self.showTabsNSpaces)
self.showLineEndings = theConf.rdBool(cnfSec, "showlineendings", self.showLineEndings)
@@ -620,7 +616,6 @@ class Config:
"fmtpadbefore": str(self.fmtPadBefore),
"fmtpadafter": str(self.fmtPadAfter),
"fmtpadthin": str(self.fmtPadThin),
"spelltool": str(self.spellTool),
"spellcheck": str(self.spellLanguage),
"showtabsnspaces": str(self.showTabsNSpaces),
"showlineendings": str(self.showLineEndings),
novelwriter/constants.py
-4
View File
@@ -46,10 +46,6 @@ class nwConst():
MAX_DOCSIZE = 5000000 # Maxium size of a single document
MAX_BUILDSIZE = 10000000 # Maxium size of a project build
# Spell Check Providers
SP_INTERNAL = "internal"
SP_ENCHANT = "enchant"
# END Class nwConst
novelwriter/core/__init__.py
+1 -3
View File
@@ -22,7 +22,7 @@ along with this program. If not, see <https://www.gnu.org/licenses/>.
from novelwriter.core.document import NWDoc
from novelwriter.core.index import NWIndex, countWords
from novelwriter.core.project import NWProject
from novelwriter.core.spellcheck import NWSpellCheck, NWSpellEnchant, NWSpellSimple
from novelwriter.core.spellcheck import NWSpellEnchant
from novelwriter.core.tohtml import ToHtml
from novelwriter.core.toodt import ToOdt
from novelwriter.core.tomd import ToMarkdown
@@ -32,9 +32,7 @@ __all__ = [
"NWDoc",
"NWIndex",
"NWProject",
"NWSpellCheck",
"NWSpellEnchant",
"NWSpellSimple",
"ToHtml",
"ToOdt",
"ToMarkdown",
novelwriter/core/spellcheck.py
+55 -208
View File
@@ -24,115 +24,25 @@ along with this program. If not, see <https://www.gnu.org/licenses/>.
"""
import os
import difflib
import logging
import novelwriter
logger = logging.getLogger(__name__)
# =============================================================================================== #
# SpellChecking SuperClass
# =============================================================================================== #
class NWSpellCheck():
theDict = None
projDict = []
class NWSpellEnchant():
def __init__(self):
self.mainConf = novelwriter.CONFIG
self.theDict = None
self.projDict = set()
self.projectDict = None
self.spellLanguage = None
return
def setLanguage(self, theLang, projectDict=None):
"""Default function.
"""
return
def checkWord(self, theWord):
"""Default function.
"""
return True
def suggestWords(self, theWord):
"""Default function.
"""
return []
def addWord(self, newWord):
"""Add a word to the project dictionary.
"""
if self.projectDict is not None and newWord not in self.projDict:
newWord = newWord.strip()
try:
with open(self.projectDict, mode="a+", encoding="utf-8") as outFile:
outFile.write("%s\n" % newWord)
self.projDict.append(newWord)
except Exception:
logger.error("Failed to add word to project word list %s", str(self.projectDict))
novelwriter.logException()
return False
return True
return False
def listDictionaries(self):
"""Default function.
"""
return []
def describeDict(self):
"""Default function.
"""
return "", ""
##
# Internal Functions
##
def _readProjectDictionary(self, projectDict):
"""Read the content of the project dictionary, and add it to the
lookup lists.
"""
self.projDict = []
self.projectDict = projectDict
if projectDict is None:
return False
if not os.path.isfile(projectDict):
return False
try:
logger.debug("Loading project word list")
with open(projectDict, mode="r", encoding="utf-8") as wordsFile:
for theLine in wordsFile:
theLine = theLine.strip()
if len(theLine) > 0 and theLine not in self.projDict:
self.projDict.append(theLine)
logger.debug("Project word list contains %d words", len(self.projDict))
except Exception:
logger.error("Failed to load project word list")
novelwriter.logException()
return False
return True
# END Class NWSpellCheck
# =============================================================================================== #
# Enchant Based SpellChecking
# =============================================================================================== #
class NWSpellEnchant(NWSpellCheck):
def __init__(self):
NWSpellCheck.__init__(self)
logger.debug("Enchant spell checking activated")
self.theBroker = None
logger.debug("Enchant spell checking activated")
return
def setLanguage(self, theLang, projectDict=None):
@@ -173,11 +83,23 @@ class NWSpellEnchant(NWSpellCheck):
return self.theDict.suggest(theWord)
def addWord(self, newWord):
"""Wrapper function for pyenchant.
"""Add a word to the project dictionary.
"""
self.theDict.add_to_session(newWord)
NWSpellCheck.addWord(self, newWord)
return
if self.projectDict is not None and newWord not in self.projDict:
newWord = newWord.strip()
try:
with open(self.projectDict, mode="a+", encoding="utf-8") as outFile:
outFile.write("%s\n" % newWord)
self.projDict.add(newWord)
except Exception:
logger.error("Failed to add word to project word list %s", str(self.projectDict))
novelwriter.logException()
return False
return True
return False
def listDictionaries(self):
"""Wrapper function for pyenchant.
@@ -207,6 +129,39 @@ class NWSpellEnchant(NWSpellCheck):
return spTag, spName
##
# Internal Functions
##
def _readProjectDictionary(self, projectDict):
"""Read the content of the project dictionary, and add it to the
lookup lists.
"""
self.projDict = set()
self.projectDict = projectDict
if projectDict is None:
return False
if not os.path.isfile(projectDict):
return False
try:
logger.debug("Loading project word list")
with open(projectDict, mode="r", encoding="utf-8") as wordsFile:
for theLine in wordsFile:
theLine = theLine.strip()
if len(theLine) > 0 and theLine not in self.projDict:
self.projDict.add(theLine)
logger.debug("Project word list contains %d words", len(self.projDict))
except Exception:
logger.error("Failed to load project word list")
novelwriter.logException()
return False
return True
# END Class NWSpellEnchant
@@ -226,111 +181,3 @@ class FakeEnchant:
return
# END Class FakeEnchant
# =============================================================================================== #
# Fallback SpellChecking Using difflib
# =============================================================================================== #
class NWSpellSimple(NWSpellCheck):
"""Internal spell check tool that uses standard Python packages with
no other external dependencies. This is the fallback spell checker
when no other is available. This method is slower than enchant.
"""
theWords = set()
def __init__(self):
NWSpellCheck.__init__(self)
self.theLang = ""
logger.debug("Simple spell checking activated")
return
def setLanguage(self, theLang, projectDict=None):
"""Load a dictionary as a list from the app assets folder.
"""
self.theLang = theLang
self.theWords = set()
dictFile = os.path.join(self.mainConf.dictPath, theLang+".dict")
try:
with open(dictFile, mode="r", encoding="utf-8") as wordsFile:
for theLine in wordsFile:
if len(theLine) == 0 or theLine.startswith("#"):
continue
self.theWords.add(theLine.strip().lower())
logger.debug("Spell check dictionary for language '%s' loaded", theLang)
logger.debug("Dictionary contains %d words", len(self.theWords))
self.spellLanguage = theLang
except Exception:
logger.error("Failed to load spell check word list for language '%s'", theLang)
novelwriter.logException()
self.spellLanguage = None
self._readProjectDictionary(projectDict)
for pWord in self.projDict:
self.theWords.add(pWord)
return
def checkWord(self, theWord):
"""Check if a word exists in the word list. Make sure to keep
this function as fast as possible as it is called for every
word by the syntax highlighter.
"""
theWord = theWord.replace(self.mainConf.fmtApostrophe, "'").lower()
return theWord in self.theWords
def suggestWords(self, theWord):
"""Get suggestions for correct word from difflib, and make sure
the first character is upper case if that was also the case for
the word be3ing checked. Also make sure the apostrophe is
changed to the one in the dictionary, and then put back in the
results.
"""
theWord = theWord.strip()
if len(theWord) == 0:
return []
theMatches = difflib.get_close_matches(theWord.lower(), self.theWords, n=10, cutoff=0.75)
theOptions = []
for aWord in theMatches:
if len(aWord) == 0:
continue
if theWord[0].isupper():
aWord = aWord[0].upper() + aWord[1:]
aWord = aWord.replace("'", self.mainConf.fmtApostrophe)
theOptions.append(aWord)
return theOptions
def addWord(self, newWord):
"""Wrapper for the internal project dictionary feature.
"""
newWord = newWord.strip().lower()
if newWord not in self.theWords:
self.theWords.add(newWord)
NWSpellCheck.addWord(self, newWord)
return
def listDictionaries(self):
"""Lists the dictionary files in the app assets folder.
"""
retList = []
for dictFile in os.listdir(self.mainConf.dictPath):
fRoot, fExt = os.path.splitext(dictFile)
if fExt != ".dict":
continue
retList.append((fRoot, "difflib"))
return retList
def describeDict(self):
"""Return the tag and provider of the currently loaded
dictionary.
"""
return self.theLang, ""
# END Class NWSpellSimple
novelwriter/dialogs/preferences.py
+21 -60
View File
@@ -34,9 +34,7 @@ from PyQt5.QtWidgets import (
QLineEdit, QFileDialog, QFontDialog, QDoubleSpinBox
)
from novelwriter.core import NWSpellSimple, NWSpellEnchant
from novelwriter.enum import nwAlert
from novelwriter.constants import nwConst
from novelwriter.gui.custom import QSwitch, QConfigLayout, PagedDialog
from novelwriter.dialogs.quotes import GuiQuoteSelect
@@ -702,32 +700,30 @@ class GuiPreferencesEditor(QWidget):
self.mainForm.addGroupLabel(self.tr("Spell Checking"))
# Spell Check Provider and Language
self.spellLangList = QComboBox(self)
self.spellLangList.setMaximumWidth(mW)
self.spellLanguage = QComboBox(self)
self.spellLanguage.setMaximumWidth(mW)
self.spellToolList = QComboBox(self)
self.spellToolList.setMaximumWidth(mW)
self.spellToolList.addItem("%s (difflib)" % self.tr("Internal"), nwConst.SP_INTERNAL)
self.spellToolList.addItem("Spell Enchant (pyenchant)", nwConst.SP_ENCHANT)
langAvail = self.theParent.docEditor.spEnchant.listDictionaries()
if self.mainConf.hasEnchant:
if langAvail:
for spTag, spProv in langAvail:
qLocal = QLocale(spTag)
spLang = qLocal.nativeLanguageName().title()
self.spellLanguage.addItem("%s [%s]" % (spLang, spProv), spTag)
else:
self.spellLanguage.addItem(self.tr("None"), "")
self.spellLanguage.setEnabled(False)
else:
self.spellLanguage.addItem(self.tr("Not installed"), "")
self.spellLanguage.setEnabled(False)
theModel = self.spellToolList.model()
idEnchant = self.spellToolList.findData(nwConst.SP_ENCHANT)
theModel.item(idEnchant).setEnabled(self.mainConf.hasEnchant)
self.spellToolList.currentIndexChanged.connect(self._doUpdateSpellTool)
toolIdx = self.spellToolList.findData(self.mainConf.spellTool)
if toolIdx != -1:
self.spellToolList.setCurrentIndex(toolIdx)
self._doUpdateSpellTool(0)
spellIdx = self.spellLanguage.findData(self.mainConf.spellLanguage)
if spellIdx != -1:
self.spellLanguage.setCurrentIndex(spellIdx)
self.mainForm.addRow(
self.tr("Spell check provider"),
self.spellToolList,
self.tr("Note that the internal spell check tool is quite slow.")
)
self.mainForm.addRow(
self.tr("Spell check language"),
self.spellLangList,
self.tr("Spell check language ({0})").format("PyEnchant"),
self.spellLanguage,
self.tr("Available languages are determined by your system.")
)
@@ -825,8 +821,7 @@ class GuiPreferencesEditor(QWidget):
"""Save the values set for this tab.
"""
# Spell Checking
self.mainConf.spellTool = self.spellToolList.currentData()
self.mainConf.spellLanguage = self.spellLangList.currentData()
self.mainConf.spellLanguage = self.spellLanguage.currentData()
self.mainConf.bigDocLimit = self.bigDocLimit.value()
# Word Count
@@ -845,40 +840,6 @@ class GuiPreferencesEditor(QWidget):
return
##
# Internal Functions
##
def _doUpdateSpellTool(self, currIdx):
"""Update the list of dictionaries based on spell tool selected.
"""
spellTool = self.spellToolList.currentData()
self._updateLanguageList(spellTool)
return
def _updateLanguageList(self, spellTool):
"""Updates the list of available spell checking dictionaries
available for the selected spell check tool. It will try to
preserve the language choice, if the language exists in the
updated list.
"""
if spellTool == nwConst.SP_ENCHANT:
theDict = NWSpellEnchant()
else:
theDict = NWSpellSimple()
self.spellLangList.clear()
for spTag, spProv in theDict.listDictionaries():
qLocal = QLocale(spTag)
spLang = qLocal.nativeLanguageName().title()
self.spellLangList.addItem("%s [%s]" % (spLang, spProv), spTag)
spellIdx = self.spellLangList.findData(self.mainConf.spellLanguage)
if spellIdx != -1:
self.spellLangList.setCurrentIndex(spellIdx)
return
# END Class GuiPreferencesEditor
novelwriter/dialogs/projsettings.py
+5 -6
View File
@@ -208,13 +208,12 @@ class GuiProjectEditMain(QWidget):
self.spellLang = QComboBox(self)
self.spellLang.setMaximumWidth(xW)
theDict = self.theParent.docEditor.currentDictionary()
self.spellLang.addItem(self.tr("Default"), "None")
if theDict is not None:
for spTag, spProv in theDict.listDictionaries():
qLocal = QLocale(spTag)
spLang = qLocal.nativeLanguageName().title()
self.spellLang.addItem("%s [%s]" % (spLang, spProv), spTag)
langAvail = self.theParent.docEditor.spEnchant.listDictionaries()
for spTag, spProv in langAvail:
qLocal = QLocale(spTag)
spLang = qLocal.nativeLanguageName().title()
self.spellLang.addItem("%s [%s]" % (spLang, spProv), spTag)
self.mainForm.addRow(
self.tr("Spell check language"),
novelwriter/gui/doceditor.py
+27 -40
View File
@@ -48,7 +48,7 @@ from PyQt5.QtWidgets import (
QFrame
)
from novelwriter.core import NWDoc, NWSpellSimple, countWords
from novelwriter.core import NWDoc, NWSpellEnchant, countWords
from novelwriter.enum import nwAlert, nwDocAction, nwDocInsert, nwItemClass
from novelwriter.common import transferCase
from novelwriter.constants import nwConst, nwKeyWords, nwUnicode
@@ -89,7 +89,6 @@ class GuiDocEditor(QTextEdit):
self._docHeaders = [] # Record of headers in the file
self._spellCheck = False # Flag for spell checking enabled
self._theDict = None # The current spell check dictionary
self._nonWord = "\"'" # Characters to not include in spell checking
# Document Variables
@@ -121,7 +120,8 @@ class GuiDocEditor(QTextEdit):
self.docSearch = GuiDocEditSearch(self)
# Syntax
self.hLight = GuiDocHighlighter(qDoc, self.theParent)
self.spEnchant = NWSpellEnchant()
self.highLight = GuiDocHighlighter(qDoc, self.theParent, self.spEnchant)
# Context Menu
self.setContextMenuPolicy(Qt.CustomContextMenu)
@@ -216,7 +216,6 @@ class GuiDocEditor(QTextEdit):
self._typSQClose = self.mainConf.fmtSingleQuotes[1]
# Reload spell check and dictionaries
self._setupSpellChecking()
self.setDictionaries()
# Set font
@@ -280,7 +279,7 @@ class GuiDocEditor(QTextEdit):
self.setTabStopWidth(self.mainConf.getTabWidth())
# Initialise the syntax highlighter
self.hLight.initHighlighter()
self.highLight.initHighlighter()
# Configure word count timer
self.wcInterval = self.mainConf.wordCountTimer
@@ -328,14 +327,14 @@ class GuiDocEditor(QTextEdit):
return False
qApp.setOverrideCursor(QCursor(Qt.WaitCursor))
self.hLight.setHandle(tHandle)
self.highLight.setHandle(tHandle)
# Check that the document is not too big for full, initial spell
# checking. If it is too big, we switch to only check as we type
self._checkDocSize(docSize)
spTemp = self.hLight.spellCheck
spTemp = self.highLight.spellCheck
if self._bigDoc:
self.hLight.spellCheck = False
self.highLight.spellCheck = False
bfTime = time()
self._allowAutoReplace(False)
@@ -356,7 +355,7 @@ class GuiDocEditor(QTextEdit):
self.docHeader.setTitleFromHandle(self._docHandle)
self.docFooter.setHandle(self._docHandle)
self.updateDocMargins()
self.hLight.spellCheck = spTemp
self.highLight.spellCheck = spTemp
if tLine is None and self._nwItem is not None:
# For large documents we queue the repositioning until the
@@ -394,7 +393,7 @@ class GuiDocEditor(QTextEdit):
def updateTagHighLighting(self):
"""Rerun the syntax highlighter on all meta data lines.
"""
self.hLight.rehighlightByType(GuiDocHighlighter.BLOCK_META)
self.highLight.rehighlightByType(GuiDocHighlighter.BLOCK_META)
return
def redrawText(self):
@@ -576,11 +575,6 @@ class GuiDocEditor(QTextEdit):
"""
return self.document().isEmpty()
def currentDictionary(self):
"""Return the current dictionary object.
"""
return self._theDict
##
# Getters
##
@@ -672,8 +666,8 @@ class GuiDocEditor(QTextEdit):
else:
theLang = self.theProject.projSpell
self._theDict.setLanguage(theLang, self.theProject.projDict)
_, theProvider = self._theDict.describeDict()
self.spEnchant.setLanguage(theLang, self.theProject.projDict)
_, theProvider = self.spEnchant.describeDict()
self.spellDictionaryChanged.emit(str(theLang), str(theProvider))
@@ -682,7 +676,7 @@ class GuiDocEditor(QTextEdit):
return True
def setSpellCheck(self, theMode):
def toggleSpellCheck(self, theMode):
"""This is the master spell check setting function, and this one
should call all other setSpellCheck functions in other classes.
If the spell check mode (theMode) is not defined (None), then
@@ -691,13 +685,21 @@ class GuiDocEditor(QTextEdit):
if theMode is None:
theMode = not self._spellCheck
if self._theDict.spellLanguage is None:
if not self.mainConf.hasEnchant:
if theMode:
self.theParent.makeAlert(self.tr(
"Spell checking requires the package PyEncant. "
"It does not appear to be installed."
), nwAlert.INFO)
theMode = False
if self.spEnchant.spellLanguage is None:
theMode = False
self._spellCheck = theMode
self.theParent.mainMenu.setSpellCheck(theMode)
self.theProject.setSpellCheck(theMode)
self.hLight.setSpellCheck(theMode)
self.highLight.setSpellCheck(theMode)
if not self._bigDoc:
self.spellCheckDocument()
@@ -719,7 +721,7 @@ class GuiDocEditor(QTextEdit):
theText = self.getText()
self.setPlainText(theText)
else:
self.hLight.rehighlight()
self.highLight.rehighlight()
qApp.restoreOverrideCursor()
afTime = time()
logger.debug("Document highlighted in %.3f ms", 1000*(afTime-bfTime))
@@ -1140,14 +1142,14 @@ class GuiDocEditor(QTextEdit):
if spellCheck:
logger.verbose("Looking up '%s' in the dictionary", theWord)
spellCheck &= not self._theDict.checkWord(theWord)
spellCheck &= not self.spEnchant.checkWord(theWord)
if spellCheck:
mnuContext.addSeparator()
mnuHead = QAction(self.tr("Spelling Suggestion(s)"), mnuContext)
mnuContext.addAction(mnuHead)
theSuggest = self._theDict.suggestWords(theWord)[:15]
theSuggest = self.spEnchant.suggestWords(theWord)[:15]
if len(theSuggest) > 0:
for aWord in theSuggest:
mnuWord = QAction("%s %s" % (nwUnicode.U_ENDASH, aWord), mnuContext)
@@ -1192,9 +1194,8 @@ class GuiDocEditor(QTextEdit):
"""
theWord = theCursor.selectedText().strip().strip(self._nonWord)
logger.debug("Added '%s' to project dictionary", theWord)
self._theDict.addWord(theWord)
self.hLight.setDict(self._theDict)
self.hLight.rehighlightBlock(theCursor.block())
self.spEnchant.addWord(theWord)
self.highLight.rehighlightBlock(theCursor.block())
return
@pyqtSlot()
@@ -1968,20 +1969,6 @@ class GuiDocEditor(QTextEdit):
self._makeSelection(selMode)
return
def _setupSpellChecking(self):
"""Create the spell checking object based on the spellTool
setting in config.
"""
if self.mainConf.spellTool == nwConst.SP_ENCHANT:
from novelwriter.core.spellcheck import NWSpellEnchant
self._theDict = NWSpellEnchant()
else:
self._theDict = NWSpellSimple()
self.hLight.setDict(self._theDict)
return
def _allowAutoReplace(self, theState):
"""used to enable/disable the auto-replace feature temporarily.
"""
novelwriter/gui/dochighlight.py
+4 -10
View File
@@ -46,16 +46,16 @@ class GuiDocHighlighter(QSyntaxHighlighter):
BLOCK_META = 2
BLOCK_TITLE = 4
def __init__(self, theDoc, theParent):
def __init__(self, theDoc, theParent, spEnchant):
QSyntaxHighlighter.__init__(self, theDoc)
logger.debug("Initialising GuiDocHighlighter ...")
self.mainConf = novelwriter.CONFIG
self.theDoc = theDoc
self.spEnchant = spEnchant
self.theParent = theParent
self.theTheme = theParent.theTheme
self.theIndex = theParent.theIndex
self.theDict = None
self.theHandle = None
self.spellCheck = False
self.spellRx = None
@@ -237,12 +237,6 @@ class GuiDocHighlighter(QSyntaxHighlighter):
# Setters
##
def setDict(self, theDict):
"""Set the dictionary object for spell check underlines lookup.
"""
self.theDict = theDict
return True
def setSpellCheck(self, theMode):
"""Enable/disable the real time spell checker.
"""
@@ -389,13 +383,13 @@ class GuiDocHighlighter(QSyntaxHighlighter):
spFmt.merge(xFmt[xM])
self.setFormat(x, 1, spFmt)
if self.theDict is None or not self.spellCheck:
if not self.spellCheck:
return
rxSpell = self.spellRx.globalMatch(theText, 0)
while rxSpell.hasNext():
rxMatch = rxSpell.next()
if not self.theDict.checkWord(rxMatch.captured(0)):
if not self.spEnchant.checkWord(rxMatch.captured(0)):
if rxMatch.captured(0).isupper() or rxMatch.captured(0).isnumeric():
continue
xPos = rxMatch.capturedStart(0)
novelwriter/gui/mainmenu.py
+1 -1
View File
@@ -120,7 +120,7 @@ class GuiMainMenu(QMenuBar):
flag is handled by the document editor class, so we make no
decision, just pass a None to the function and let it decide.
"""
self.theParent.docEditor.setSpellCheck(None)
self.theParent.docEditor.toggleSpellCheck(None)
return True
def _toggleAutoOutline(self, theMode):
novelwriter/guimain.py
+1 -1
View File
@@ -526,7 +526,7 @@ class GuiMain(QMainWindow):
self._updateWindowTitle(self.theProject.projName)
self.rebuildTrees()
self.docEditor.setDictionaries()
self.docEditor.setSpellCheck(self.theProject.spellCheck)
self.docEditor.toggleSpellCheck(self.theProject.spellCheck)
self.mainMenu.setAutoOutline(self.theProject.autoOutline)
self.statusBar.setRefTime(self.theProject.projOpened)
self.statusBar.doUpdateProjectStats(self.theProject.currWCount, 0)
tests/reference/baseConfig_novelwriter.conf
+1 -2
View File
@@ -1,5 +1,5 @@
[Main]
timestamp = 2021-08-26 17:48:02
timestamp = 2021-08-30 21:27:27
theme = default
syntax = default_light
icons = typicons_light
@@ -53,7 +53,6 @@ fmtdoublequote = “, ”
fmtpadbefore =
fmtpadafter =
fmtpadthin = False
spelltool = internal
spellcheck = en
showtabsnspaces = False
showlineendings = False
tests/reference/guiPreferences_novelwriter.conf
+1 -2
View File
@@ -1,5 +1,5 @@
[Main]
timestamp = 2021-08-26 17:48:04
timestamp = 2021-08-30 21:45:17
theme = default
syntax = default_light
icons = typicons_light
@@ -53,7 +53,6 @@ fmtdoublequote = “, ”
fmtpadbefore =
fmtpadafter =
fmtpadthin = False
spelltool = internal
spellcheck = en
showtabsnspaces = True
showlineendings = True
tests/test_base/test_base_config.py
+1 -2
View File
@@ -29,7 +29,7 @@ from mock import causeOSError, MockApp
from tools import cmpFiles, writeFile
from novelwriter.config import Config
from novelwriter.constants import nwConst, nwFiles
from novelwriter.constants import nwFiles
@pytest.mark.base
@@ -133,7 +133,6 @@ def testBaseConfig_Init(monkeypatch, tmpDir, fncDir, outDir, refDir, filesDir):
# This should create a config file as well
with monkeypatch.context() as mp:
mp.setattr("os.path.expanduser", lambda *a: "")
tstConf.spellTool = nwConst.SP_INTERNAL
tstConf.initConfig(confPath=tmpDir, dataPath=tmpDir)
assert tstConf.confPath == tmpDir
assert tstConf.dataPath == tmpDir
tests/test_core/test_core_spell.py
+23 -113
View File
@@ -26,53 +26,7 @@ import pytest
from mock import causeOSError
from tools import readFile, writeFile
from novelwriter.core.spellcheck import (
NWSpellCheck, NWSpellEnchant, NWSpellSimple
)
@pytest.mark.core
def testCoreSpell_Super(monkeypatch, tmpDir):
"""Test the spell checker super class
"""
wList = os.path.join(tmpDir, "wordlist.txt")
writeFile(wList, "a_word\nb_word\nc_word\n")
spChk = NWSpellCheck()
# Check that default functions return results that reflects that spell
# checking is effectively disabled
assert spChk.setLanguage("", "") is None
assert spChk.checkWord("")
assert spChk.suggestWords("") == []
assert spChk.listDictionaries() == []
assert spChk.describeDict() == ("", "")
# Add a word to the user's dictionary
assert spChk._readProjectDictionary("stuff") is False
with monkeypatch.context() as mp:
mp.setattr("builtins.open", causeOSError)
assert spChk._readProjectDictionary(wList) is False
assert spChk._readProjectDictionary(None) is False
assert spChk._readProjectDictionary(wList) is True
assert spChk.projectDict == wList
# Cannot write to file
with monkeypatch.context() as mp:
mp.setattr("builtins.open", causeOSError)
assert spChk.addWord("d_word") is False
assert readFile(wList) == "a_word\nb_word\nc_word\n"
# First time, OK
assert spChk.addWord("d_word") is True
assert readFile(wList) == "a_word\nb_word\nc_word\nd_word\n"
# But not added twice
assert spChk.addWord("d_word") is False
assert readFile(wList) == "a_word\nb_word\nc_word\nd_word\n"
# END Test testCoreSpell_Super
from novelwriter.core.spellcheck import NWSpellEnchant
@pytest.mark.core
@@ -99,11 +53,32 @@ def testCoreSpell_Enchant(monkeypatch, tmpDir):
spChk.setLanguage("en", wList)
spChk.setLanguage("en", wList)
# Add a word to the user's dictionary
assert spChk._readProjectDictionary("stuff") is False
with monkeypatch.context() as mp:
mp.setattr("builtins.open", causeOSError)
assert spChk._readProjectDictionary(wList) is False
assert spChk._readProjectDictionary(None) is False
assert spChk._readProjectDictionary(wList) is True
assert spChk.projectDict == wList
# Cannot write to file
with monkeypatch.context() as mp:
mp.setattr("builtins.open", causeOSError)
assert spChk.addWord("d_word") is False
assert readFile(wList) == "a_word\nb_word\nc_word\n"
assert spChk.addWord("d_word") is True
assert readFile(wList) == "a_word\nb_word\nc_word\nd_word\n"
assert spChk.addWord("d_word") is False
# Check words
assert spChk.checkWord("a_word") is True
assert spChk.checkWord("b_word") is True
assert spChk.checkWord("c_word") is True
assert spChk.checkWord("d_word") is False
assert spChk.checkWord("d_word") is True
assert spChk.checkWord("e_word") is False
spChk.addWord("d_word")
assert spChk.checkWord("d_word") is True
@@ -120,68 +95,3 @@ def testCoreSpell_Enchant(monkeypatch, tmpDir):
assert aName != ""
# END Test testCoreSpell_Enchant
@pytest.mark.core
def testCoreSpell_Simple(monkeypatch, tmpDir):
"""Test the fallback simple spell checker
"""
wList = os.path.join(tmpDir, "wordlist.txt")
wDict = os.path.join(tmpDir, "en.dict")
writeFile(wList, "a_word\nb_word\nc_word\n")
writeFile(wDict, "# Comment\ne_word\nf_word\ng_word\n")
spChk = NWSpellSimple()
spChk.mainConf.dictPath = tmpDir
# Load dictionary, but fail
with monkeypatch.context() as mp:
mp.setattr("builtins.open", causeOSError)
spChk.setLanguage("en", wList)
assert spChk.spellLanguage is None
assert spChk.theWords == set(spChk.projDict)
# Load dictionary properly
spChk.setLanguage("en", wList)
assert spChk.projDict == ["a_word", "b_word", "c_word"]
assert spChk.theWords == {"e_word", "f_word", "g_word", "a_word", "b_word", "c_word"}
# Check words
assert spChk.checkWord("a_word")
assert spChk.checkWord("b_word")
assert spChk.checkWord("c_word")
assert not spChk.checkWord("d_word")
assert spChk.checkWord("e_word")
assert spChk.checkWord("f_word")
assert spChk.checkWord("g_word")
# Add word
spChk.addWord("d_word")
assert spChk.checkWord("d_word")
# Check spelling
assert spChk.suggestWords(" \t") == []
wSuggest = spChk.suggestWords("d_wrod")
assert len(wSuggest) > 0
assert "d_word" in wSuggest
# Break the matching
with monkeypatch.context() as mp:
mp.setattr("difflib.get_close_matches", lambda *args, **kwargs: [""])
assert spChk.suggestWords("word") == []
# Capitalisation
wSuggest = spChk.suggestWords("D_wrod")
assert len(wSuggest) > 0
assert "D_word" in wSuggest
# List dictionaries
assert spChk.listDictionaries() == [("en", "difflib")]
# Description
aTag, aName = spChk.describeDict()
assert aTag == "en"
assert aName == ""
# END Test testCoreSpell_Simple
tests/test_dialogs/test_dlg_preferences.py
+2 -2
View File
@@ -33,7 +33,6 @@ from PyQt5.QtWidgets import (
from novelwriter.config import Config
from novelwriter.dialogs import GuiPreferences, GuiQuoteSelect
from novelwriter.constants import nwConst
keyDelay = 2
typeDelay = 1
@@ -67,10 +66,11 @@ def testDlgPreferences_Main(qtbot, monkeypatch, fncDir, outDir, refDir):
theConf = nwGUI.mainConf
assert theConf.confPath == fncDir
theConf.spellTool = nwConst.SP_INTERNAL
monkeypatch.setattr(GuiPreferences, "exec_", lambda *a: None)
monkeypatch.setattr(GuiPreferences, "result", lambda *a: QDialog.Accepted)
monkeypatch.setattr(nwGUI.docEditor.spEnchant, "listDictionaries", lambda: [("en", "none")])
nwGUI.mainMenu.aPreferences.activate(QAction.Trigger)
qtbot.waitUntil(lambda: getGuiItem("GuiPreferences") is not None, timeout=1000)
tests/test_dialogs/test_dlg_projsettings.py
+2
View File
@@ -66,6 +66,8 @@ def testDlgProjSettings_Dialog(qtbot, monkeypatch, nwGUI, fncDir, fncProj, outDi
# Get the dialog object
monkeypatch.setattr(GuiProjectSettings, "exec_", lambda *a: None)
monkeypatch.setattr(GuiProjectSettings, "result", lambda *a: QDialog.Accepted)
monkeypatch.setattr(nwGUI.docEditor.spEnchant, "listDictionaries", lambda: [("en", "none")])
nwGUI.mainMenu.aProjectSettings.activate(QAction.Trigger)
qtbot.waitUntil(lambda: getGuiItem("GuiProjectSettings") is not None, timeout=1000)
tests/test_gui/test_gui_doceditor.py
-1
View File
@@ -229,7 +229,6 @@ def testGuiEditor_MetaData(qtbot, monkeypatch, nwGUI, nwMinimal):
assert nwGUI.docEditor.docHandle() == sHandle
assert nwGUI.docEditor.lastActive() > 0.0
assert nwGUI.docEditor.isEmpty() is False
assert nwGUI.docEditor.currentDictionary() is not None
# Cursor Position
assert nwGUI.docEditor.setCursorPosition(None) is False