Reproog/novelWriter
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Improve tree class and auto-rebuild index (#1236)

Browse Source
This commit is contained in:
Veronica Berglyd Olsen
2022-11-12 18:29:49 +01:00
committed by GitHub
parent 116ebd8243 e7ce13f647
commit 8a9b79abd0
15 changed files with 3009 additions and 2939 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+86
View File
@@ -1,5 +1,91 @@
# novelWriter Changelog
## Version 2.0 RC 2 [2022-11-13]
### Release Notes
This is a release candidate of the next release version, and is intended for testing purposes.
Please be careful when using this version on live writing projects, and make sure you take frequent
backups.
Please check the changelog for an overview of changes. The full release notes will be added to the
final release.
### Detailed Changelog
Note: This release introduces a new Project XML format with version number 1.5. When the project is
opened, a request to update the file format will show up.
**Bugfixes**
* The custom folders for user defined themes and syntax were not created properly when the app was
first launched on a new computer. The folders were successfully created on a second launch. The
error was handled, but reported. This was caused by the folder creation process being in the
wrong order. Issue #1180. PR #1184.
* Fixed context menu entries for split and merge having inconsistent labels. Issue #1199. PR #1197.
**User Interface**
* The exported status for document items have been renamed to active/inactive. Their icons have
also been updated. Issues #1196 and #1198. PRs #1200 and #1216.
* The status/importance context menu now shows which label is the current. Issue #1202. PR #1207.
* The status/importance context menu now has a "Manage Labels" action that opens the Project
Settings dialog at the correct place. Issue #1203. PR #1207.
* Both GUI theme and syntax theme can now be updated without restarting the app. Issue #1171.
PR #1212.
* The GUI theme now determines which icon theme is to be loaded. It is no longer a separate
setting. The icon theme can also be reloaded without restart. Issue #1172. PR #1212.
* The block formating features in the Format menu now also works on empty lines. Issue #1178.
PR #1214.
* There is now a Format menu entry and shortcut code for synopsis comments. Issue #1177. PR #1214.
* The split document dialog now has the option to move teh source document to trash. Issue #1179.
PR #1217.
**Other Changes**
* Archived documents are now partially indexed. This mainly means that the item will have the
correct document icon in the project tree corresponding to its main heading. Issue #1176.
PR #1183.
* The option to add notes files in the Project Wizard now automatically switches off if there are
no notes categories enabled. Issue #1192. PR #1201.
* When a project is opened for the first time, the first document in the project is also opened.
Issue #1219. PR #1223.
* When there is no project open, the toolbars on the Project Tree, Novel View and Outline View are
disabled. They are enabled only when a project is loaded. Issue #1220. PR #1230.
**Installation and Packaging**
* The AppImage release now has version information in the package name. Issue #1182. PR #1218.
**Code Improvements**
* The main heading of a document is now stored in the item class instead of the index. PR #1183.
* The common module checker functions no longer allow None values. The only one needing it was the
string checker. A new string checker that allows None has been added for those cases. This makes
type discovery in the code editor easier. Issue #1185. PR #1188.
* Verbose logging has been removed. The lowest severity level is now DEBUG. Issue #1186. PR #1191.
* Added a number of None checks in the code where especially Qt calls could potentially return
None, even if they were unlikely to do so. PR #1197.
* Renamed the status bar attribute in the main GUI class as it conflicts with a Qt method.
Issue #1190. PR #1197.
* The data access methods for the custom config file parser have been improved to better report
correct type information. PR #1197.
* Saving and loading of XML data is now handled by a separate set of reader and writer classes. The
reader class is capable of reading all file formats that have been used thus far. The various
data classes have been improved, and a new XML file formart version 1.5 added. Issue #1189.
PRs #1221 and #1232.
* The project folder on disk is now wrapped in a storage class that the project accesses files
through. It also handles lock files and archiving used for backup. The change is in preparation
for adding a potential single file format. Issue #1222. PR #1225.
* The Project Wizard now creates the project on disk, and then opens it. This replaces the old
method where the new project was built directly into the current session. This caused a few
inconsistencies from time to time, and was a duplicate way of getting a project into the session.
Issue #1152. PR #1225.
* The Config class has been refactored extensively and now also uses pathlib for all paths. Tests
are also switched to using pathlib. Issue #1224. PRs #1228 and #1229.
----
## Version 2.0 RC 1 [2022-10-17]
### Release Notes
i18n/nw_base.ts
+922 -952
View File
File diff suppressed because it is too large Load Diff
i18n/nw_en_US.ts
+935 -965
View File
File diff suppressed because it is too large Load Diff
i18n/nw_nb_NO.ts
+934 -964
View File
File diff suppressed because it is too large Load Diff
novelwriter/__init__.py
+3 -3
View File
@@ -59,9 +59,9 @@ __license__ = "GPLv3"
__author__ = "Veronica Berglyd Olsen"
__maintainer__ = "Veronica Berglyd Olsen"
__email__ = "code@vkbo.net"
__version__ = "2.0-rc1"
__hexversion__ = "0x020000c1"
__date__ = "2022-10-17"
__version__ = "2.0-rc2"
__hexversion__ = "0x020000c2"
__date__ = "2022-11-13"
__status__ = "Stable"
__domain__ = "novelwriter.io"
__url__ = "https://novelwriter.io"
novelwriter/assets/text/release_notes.htm
+2 -2
View File
@@ -2,8 +2,8 @@
<html>
<body>
<h2>Release Notes for 2.0 RC 1</h2>
<p><i>Released on 17 October 2022</i></p>
<h2>Release Notes for 2.0 RC 2</h2>
<p><i>Released on 13 November 2022</i></p>
<p>This is a release candidate of the next release version, and is intended for testing purposes.
Please be careful when using this version on live writing projects, and make sure you take frequent
novelwriter/core/index.py
+32 -20
View File
@@ -58,23 +58,23 @@ class NWIndex:
The index data is cached in a JSON file between writing sessions.
"""
def __init__(self, theProject):
def __init__(self, project):
self.theProject = theProject
self._project = project
# Storage and State
self._tagsIndex = TagsIndex()
self._itemIndex = ItemIndex(theProject)
self._itemIndex = ItemIndex(project)
self._indexBroken = False
# TimeStamps
self._indexChange = 0
self._indexChange = 0.0
self._rootChange = {}
return
def __repr__(self):
return f"<NWIndex project='{self.theProject.data.name}'>"
return f"<NWIndex project='{self._project.data.name}'>"
##
# Properties
@@ -93,10 +93,22 @@ class NWIndex:
"""
self._tagsIndex.clear()
self._itemIndex.clear()
self._indexChange = 0
self._indexChange = 0.0
self._rootChange = {}
return
def rebuildIndex(self):
"""Rebuild the entire index from scratch.
"""
self.clearIndex()
for nwItem in self._project.tree:
if nwItem is not None and nwItem.isFileType():
tHandle = nwItem.itemHandle
theDoc = self._project.storage.getDocument(tHandle)
self.scanText(tHandle, theDoc.readDocument() or "")
self._indexBroken = False
return
def deleteHandle(self, tHandle):
"""Delete all entries of a given document handle.
"""
@@ -113,11 +125,11 @@ class NWIndex:
moved from the archive or trash folders back into the active
project.
"""
if not self.theProject.tree.checkType(tHandle, nwItemType.FILE):
if not self._project.tree.checkType(tHandle, nwItemType.FILE):
return False
logger.debug("Re-indexing item '%s'", tHandle)
theDoc = self.theProject.storage.getDocument(tHandle)
theDoc = self._project.storage.getDocument(tHandle)
self.scanText(tHandle, theDoc.readDocument() or "")
return True
@@ -125,13 +137,13 @@ class NWIndex:
def indexChangedSince(self, checkTime):
"""Check if the index has changed since a given time.
"""
return self._indexChange > checkTime
return self._indexChange > float(checkTime)
def rootChangedSince(self, rootHandle, checkTime):
"""Check if the index has changed since a given time for a
given root item.
"""
return self._rootChange.get(rootHandle, self._indexChange) > checkTime
return self._rootChange.get(rootHandle, self._indexChange) > float(checkTime)
##
# Load and Save Index to/from File
@@ -140,7 +152,7 @@ class NWIndex:
def loadIndex(self):
"""Load index from last session from the project meta folder.
"""
indexFile = self.theProject.storage.getMetaFile(nwFiles.INDEX_FILE)
indexFile = self._project.storage.getMetaFile(nwFiles.INDEX_FILE)
if not isinstance(indexFile, Path):
return False
@@ -171,12 +183,12 @@ class NWIndex:
logger.debug("Checking index")
# Check that all files are indexed
for fHandle in self.theProject.projFiles:
for fHandle in self._project.projFiles:
if fHandle not in self._itemIndex:
logger.warning("Item '%s' is not in the index", fHandle)
self.reIndexHandle(fHandle)
self._indexChange = round(time())
self._indexChange = time()
logger.debug("Index loaded in %.3f ms", (time() - tStart)*1000)
@@ -186,7 +198,7 @@ class NWIndex:
"""Save the current index as a json file in the project meta
data folder.
"""
indexFile = self.theProject.storage.getMetaFile(nwFiles.INDEX_FILE)
indexFile = self._project.storage.getMetaFile(nwFiles.INDEX_FILE)
if not isinstance(indexFile, Path):
return False
@@ -222,7 +234,7 @@ class NWIndex:
files before we save them, in which case we already have the
text.
"""
theItem = self.theProject.tree[tHandle]
theItem = self._project.tree[tHandle]
if theItem is None:
logger.info("Not indexing unknown item '%s'", tHandle)
return False
@@ -256,7 +268,7 @@ class NWIndex:
self._scanActive(tHandle, theItem, theText, itemTags)
# Update timestamps for index changes
nowTime = round(time())
nowTime = time()
self._indexChange = nowTime
self._rootChange[theItem.itemRoot] = nowTime
@@ -737,8 +749,8 @@ class ItemIndex:
IndexHeading object for each header of the text.
"""
def __init__(self, theProject):
self.theProject = theProject
def __init__(self, project):
self._project = project
self._items = {}
return
@@ -802,7 +814,7 @@ class ItemIndex:
"""Iterate over all items and headers in the novel structure for
a given root handle, or for all if root handle is None.
"""
for tItem in self.theProject.tree:
for tItem in self._project.tree:
if tItem is None:
continue
if tItem.isNoteLayout():
@@ -885,7 +897,7 @@ class ItemIndex:
if not isHandle(tHandle):
raise ValueError("itemIndex keys must be handles")
nwItem = self.theProject.tree[tHandle]
nwItem = self._project.tree[tHandle]
if nwItem is not None:
tItem = IndexItem(tHandle, nwItem)
tItem.unpackData(tData)
novelwriter/core/project.py
+6 -3
View File
@@ -369,8 +369,11 @@ class NWProject(QObject):
self._scanProjectFolder()
self._index.loadIndex()
self.updateWordCounts()
if xmlReader.state == XMLReadState.WAS_LEGACY:
# Often, the index needs to be rebuilt when updating format
self._index.rebuildIndex()
self.updateWordCounts()
self._projOpened = time()
self._projAltered = False
@@ -482,8 +485,8 @@ class NWProject(QObject):
return False
archName = baseDir / self.tr(
"Backup from {0}.zip"
).format(formatTimeStamp(time(), fileSafe=True))
"Backup from {0}"
).format(formatTimeStamp(time(), fileSafe=True) + ".zip")
if self._storage.zipIt(archName, compression=2):
if doNotify:
self.mainGui.makeAlert(self.tr(
novelwriter/core/tree.py
+9 -14
View File
@@ -331,20 +331,15 @@ class NWTree:
def setOrder(self, newOrder):
"""Reorders the tree based on a list of items.
"""
tmpOrder = []
# Add all known elements to a new temp list
for tHandle in newOrder:
if tHandle in self._projTree:
tmpOrder.append(tHandle)
else:
logger.error("Handle '%s' in new tree order is not in project tree", tHandle)
# Do a reverse lookup to check for items that will be lost
# This is mainly for debugging purposes
for tHandle in self._treeOrder:
if tHandle not in tmpOrder:
logger.warning("Handle '%s' in old tree order is not in new tree order", tHandle)
tmpOrder = [tHandle for tHandle in newOrder if tHandle in self._projTree]
if not (len(tmpOrder) == len(newOrder) == len(self._treeOrder)):
# Something is wrong, so let's debug it
for tHandle in newOrder:
if tHandle not in self._projTree:
logger.error("Handle '%s' in new tree order is not in old order", tHandle)
for tHandle in self._treeOrder:
if tHandle not in tmpOrder:
logger.warning("Handle '%s' in old tree order is not in new order", tHandle)
# Save the temp list
self._treeOrder = tmpOrder
novelwriter/guimain.py
+2 -11
View File
@@ -833,17 +833,8 @@ class GuiMain(QMainWindow):
tStart = time()
self.projView.saveProjectTasks()
self.theProject.index.clearIndex()
for tItem in self.theProject.tree:
if tItem is None: # pragma: no cover
continue # This is a bug trap
logger.debug("Indexing '%s'", tItem.itemName)
if self.theProject.index.reIndexHandle(tItem.itemHandle):
# Update Word Counts
self.projView.propagateCount(tItem.itemHandle, tItem.wordCount, countChildren=True)
self.projView.setTreeItemValues(tItem.itemHandle)
self.theProject.index.rebuildIndex()
self.projView.populateTree()
tEnd = time()
self.setStatus(
sample/nwProject.nwx
+2 -2
View File
@@ -1,6 +1,6 @@
<?xml version='1.0' encoding='utf-8'?>
<novelWriterXML appVersion="2.0-rc1" hexVersion="0x020000c1" fileVersion="1.5" timeStamp="2022-11-07 13:00:48">
<project id="e2be99af-f9bf-4403-857a-c3d1ac25abea" saveCount="1448" autoCount="237" editTime="69737">
<novelWriterXML appVersion="2.0-rc2" hexVersion="0x020000c2" fileVersion="1.5" timeStamp="2022-11-10 17:56:30">
<project id="e2be99af-f9bf-4403-857a-c3d1ac25abea" saveCount="1450" autoCount="237" editTime="69742">
<name>Sample Project</name>
<title>Sample Project</title>
<author>Jane Smith</author>
tests/test_core/test_core_index.py
+8 -1
View File
@@ -90,7 +90,7 @@ def testCoreIndex_LoadSave(monkeypatch, prjLipsum, mockGUI, tstPaths):
assert theIndex._tagsIndex._tags == {}
assert theIndex._itemIndex._items == {}
# No folder for sloading
# No folder for loading
with monkeypatch.context() as mp:
mp.setattr("novelwriter.core.storage.NWStorage.getMetaFile", lambda *a: None)
assert theIndex.loadIndex() is False
@@ -108,6 +108,13 @@ def testCoreIndex_LoadSave(monkeypatch, prjLipsum, mockGUI, tstPaths):
assert str(theIndex._tagsIndex.packData()) == tagIndex
assert str(theIndex._itemIndex.packData()) == itemsIndex
# Rebuild index
theIndex.clearIndex()
theIndex.rebuildIndex()
assert str(theIndex._tagsIndex.packData()) == tagIndex
assert str(theIndex._itemIndex.packData()) == itemsIndex
# Check File
copyfile(projFile, testFile)
assert cmpFiles(testFile, compFile)
tests/test_core/test_core_project.py
+13 -1
View File
@@ -230,7 +230,7 @@ def testCoreProject_Open(monkeypatch, caplog, mockGUI, fncPath, mockRnd):
assert "The file format of your project is about to be" in mockGUI.lastQuestion[1]
mockGUI.askResponse = True
# Won't convert legacy file
# Won't open project from newer version
with monkeypatch.context() as mp:
mp.setattr(ProjectXMLReader, "hexVersion", property(lambda *a: 0x99999999))
mockGUI.askResponse = False
@@ -245,6 +245,18 @@ def testCoreProject_Open(monkeypatch, caplog, mockGUI, fncPath, mockRnd):
assert theProject.closeProject()
# Trigger an index rebuild
with monkeypatch.context() as mp:
mp.setattr(ProjectXMLReader, "state", property(lambda *a: XMLReadState.WAS_LEGACY))
mp.setattr("novelwriter.core.index.NWIndex.loadIndex", lambda *a: True)
mockGUI.askResponse = True
theProject.index._indexBroken = True
assert theProject.openProject(fncPath) is True
assert "The file format of your project is about to be" in mockGUI.lastQuestion[1]
assert theProject.index._indexBroken is False
assert theProject.closeProject()
# END Test testCoreProject_Open
tests/test_core/test_core_projectxml.py
+7
View File
@@ -38,6 +38,13 @@ class MockProject:
pass
@pytest.fixture(scope="function", autouse=True)
def mockVersion(monkeypatch):
monkeypatch.setattr("novelwriter.__version__", "2.0-rc1")
monkeypatch.setattr("novelwriter.__hexversion__", "0x020000c1")
return
@pytest.mark.core
def testCoreProjectXML_ReadCurrent(monkeypatch, tstPaths, fncPath):
"""Test reading the current XML file format.
tests/test_core/test_core_tree.py
+48 -1
View File
@@ -24,6 +24,7 @@ import random
from pathlib import Path
from mock import causeOSError
from tools import readFile
from novelwriter.enum import nwItemClass, nwItemType, nwItemLayout
@@ -230,6 +231,36 @@ def testCoreTree_BuildTree(mockGUI, mockItems):
# END Test testCoreTree_BuildTree
@pytest.mark.core
def testCoreTree_PackUnpack(mockGUI, mockItems):
"""Test packing and unpacking data.
"""
theProject = NWProject(mockGUI)
theTree = NWTree(theProject)
aHandles = []
for tHandle, pHandle, nwItem in mockItems:
aHandles.append(tHandle)
theTree.append(tHandle, pHandle, nwItem)
theTree.updateItemData(tHandle)
assert len(theTree) == len(mockItems)
# Pack
tree = theTree.pack()
for i, (tHandle, pHandle, nwItem) in enumerate(mockItems):
assert tree[i]["itemAttr"]["handle"] == tHandle
# Unpack
theTree.clear()
assert len(theTree) == 0
assert theTree.handles() == []
assert theTree.unpack(tree) is True
assert theTree.handles() == aHandles
# END Test testCoreTree_PackUnpack
@pytest.mark.core
def testCoreTree_Methods(mockGUI, mockItems):
"""Test various class methods.
@@ -272,6 +303,13 @@ def testCoreTree_Methods(mockGUI, mockItems):
assert theTree.findRoot(nwItemClass.NOVEL) == "a000000000001"
assert theTree.findRoot(nwItemClass.CHARACTER) == "a000000000004"
# Iter roots
roots = list(theTree.iterRoots(None))
assert roots[0][0] == "a000000000001"
assert roots[1][0] == "a000000000002"
assert roots[2][0] == "a000000000003"
assert roots[3][0] == "a000000000004"
# Add a fake item to root and check that it can handle it
theTree._treeRoots["0000000000000"] = NWItem(theProject)
assert theTree.findRoot(nwItemClass.WORLD) is None
@@ -363,7 +401,7 @@ def testCoreTree_Stats(mockGUI, mockItems):
@pytest.mark.core
def testCoreTree_Reorder(mockGUI, mockItems):
def testCoreTree_Reorder(caplog, mockGUI, mockItems):
"""Test changing tree order.
"""
theProject = NWProject(mockGUI)
@@ -384,12 +422,16 @@ def testCoreTree_Reorder(mockGUI, mockItems):
theTree.setOrder(bHandle)
assert theTree.handles() == bHandle
caplog.clear()
theTree.setOrder(bHandle + ["stuff"])
assert theTree.handles() == bHandle
assert "Handle 'stuff' in new tree order is not in old order" in caplog.text
caplog.clear()
theTree._treeOrder.append("stuff")
theTree.setOrder(bHandle)
assert theTree.handles() == bHandle
assert "Handle 'stuff' in old tree order is not in new order" in caplog.text
# END Test testCoreTree_Reorder
@@ -421,6 +463,11 @@ def testCoreTree_ToCFile(monkeypatch, mockGUI, mockItems, tmpPath):
theProject._storage._runtimePath = None
assert theTree.writeToCFile() is False
theProject._storage._runtimePath = tmpPath
with monkeypatch.context() as mp:
mp.setattr("builtins.open", causeOSError)
assert theTree.writeToCFile() is False
theProject._storage._runtimePath = tmpPath
assert theTree.writeToCFile() is True