Improve tree class and auto-rebuild index (#1236)

CHANGELOG.md

@@ -1,5 +1,91 @@
 # novelWriter Changelog
 
+## Version 2.0 RC 2 [2022-11-13]
+
+### Release Notes
+
+This is a release candidate of the next release version, and is intended for testing purposes.
+Please be careful when using this version on live writing projects, and make sure you take frequent
+backups.
+
+Please check the changelog for an overview of changes. The full release notes will be added to the
+final release.
+
+### Detailed Changelog
+
+Note: This release introduces a new Project XML format with version number 1.5. When the project is
+opened, a request to update the file format will show up.
+
+**Bugfixes**
+
+* The custom folders for user defined themes and syntax were not created properly when the app was
+  first launched on a new computer. The folders were successfully created on a second launch. The
+  error was handled, but reported. This was caused by the folder creation process being in the
+  wrong order. Issue #1180. PR #1184.
+* Fixed context menu entries for split and merge having inconsistent labels. Issue #1199. PR #1197.
+
+**User Interface**
+
+* The exported status for document items have been renamed to active/inactive. Their icons have
+  also been updated. Issues #1196 and #1198. PRs #1200 and #1216.
+* The status/importance context menu now shows which label is the current. Issue #1202. PR #1207.
+* The status/importance context menu now has a "Manage Labels" action that opens the Project
+  Settings dialog at the correct place. Issue #1203. PR #1207.
+* Both GUI theme and syntax theme can now be updated without restarting the app. Issue #1171.
+  PR #1212.
+* The GUI theme now determines which icon theme is to be loaded. It is no longer a separate
+  setting. The icon theme can also be reloaded without restart. Issue #1172. PR #1212.
+* The block formating features in the Format menu now also works on empty lines. Issue #1178.
+  PR #1214.
+* There is now a Format menu entry and shortcut code for synopsis comments. Issue #1177. PR #1214.
+* The split document dialog now has the option to move teh source document to trash. Issue #1179.
+  PR #1217.
+
+**Other Changes**
+
+* Archived documents are now partially indexed. This mainly means that the item will have the
+  correct document icon in the project tree corresponding to its main heading. Issue #1176.
+  PR #1183.
+* The option to add notes files in the Project Wizard now automatically switches off if there are
+  no notes categories enabled. Issue #1192. PR #1201.
+* When a project is opened for the first time, the first document in the project is also opened.
+  Issue #1219. PR #1223.
+* When there is no project open, the toolbars on the Project Tree, Novel View and Outline View are
+  disabled. They are enabled only when a project is loaded. Issue #1220. PR #1230.
+
+**Installation and Packaging**
+
+* The AppImage release now has version information in the package name. Issue #1182. PR #1218.
+
+**Code Improvements**
+
+* The main heading of a document is now stored in the item class instead of the index. PR #1183.
+* The common module checker functions no longer allow None values. The only one needing it was the
+  string checker. A new string checker that allows None has been added for those cases. This makes
+  type discovery in the code editor easier. Issue #1185. PR #1188.
+* Verbose logging has been removed. The lowest severity level is now DEBUG. Issue #1186. PR #1191.
+* Added a number of None checks in the code where especially Qt calls could potentially return
+  None, even if they were unlikely to do so. PR #1197.
+* Renamed the status bar attribute in the main GUI class as it conflicts with a Qt method.
+  Issue #1190. PR #1197.
+* The data access methods for the custom config file parser have been improved to better report
+  correct type information. PR #1197.
+* Saving and loading of XML data is now handled by a separate set of reader and writer classes. The
+  reader class is capable of reading all file formats that have been used thus far. The various
+  data classes have been improved, and a new XML file formart version 1.5 added. Issue #1189.
+  PRs #1221 and #1232.
+* The project folder on disk is now wrapped in a storage class that the project accesses files
+  through. It also handles lock files and archiving used for backup. The change is in preparation
+  for adding a potential single file format. Issue #1222. PR #1225.
+* The Project Wizard now creates the project on disk, and then opens it. This replaces the old
+  method where the new project was built directly into the current session. This caused a few
+  inconsistencies from time to time, and was a duplicate way of getting a project into the session.
+  Issue #1152. PR #1225.
+* The Config class has been refactored extensively and now also uses pathlib for all paths. Tests
+  are also switched to using pathlib. Issue #1224. PRs #1228 and #1229.
+
+----
+
 ## Version 2.0 RC 1 [2022-10-17]
 
 ### Release Notes

novelwriter/__init__.py

@@ -59,9 +59,9 @@ __license__    = "GPLv3"
 __author__     = "Veronica Berglyd Olsen"
 __maintainer__ = "Veronica Berglyd Olsen"
 __email__      = "code@vkbo.net"
-__version__    = "2.0-rc1"
+__version__    = "2.0-rc2"
-__hexversion__ = "0x020000c1"
+__hexversion__ = "0x020000c2"
-__date__       = "2022-10-17"
+__date__       = "2022-11-13"
 __status__     = "Stable"
 __domain__     = "novelwriter.io"
 __url__        = "https://novelwriter.io"

novelwriter/assets/text/release_notes.htm

@@ -2,8 +2,8 @@
 <html>
 <body>
 
-<h2>Release Notes for 2.0 RC 1</h2>
+<h2>Release Notes for 2.0 RC 2</h2>
-<p><i>Released on 17 October 2022</i></p>
+<p><i>Released on 13 November 2022</i></p>
 
 <p>This is a release candidate of the next release version, and is intended for testing purposes.
 Please be careful when using this version on live writing projects, and make sure you take frequent

novelwriter/core/index.py

@@ -58,23 +58,23 @@ class NWIndex:
     The index data is cached in a JSON file between writing sessions.
     """
 
-    def __init__(self, theProject):
+    def __init__(self, project):
 
-        self.theProject = theProject
+        self._project = project
 
         # Storage and State
         self._tagsIndex = TagsIndex()
-        self._itemIndex = ItemIndex(theProject)
+        self._itemIndex = ItemIndex(project)
         self._indexBroken = False
 
         # TimeStamps
-        self._indexChange = 0
+        self._indexChange = 0.0
         self._rootChange = {}
 
         return
 
     def __repr__(self):
-        return f"<NWIndex project='{self.theProject.data.name}'>"
+        return f"<NWIndex project='{self._project.data.name}'>"
 
     ##
     #  Properties
@@ -93,10 +93,22 @@ class NWIndex:
         """
         self._tagsIndex.clear()
         self._itemIndex.clear()
-        self._indexChange = 0
+        self._indexChange = 0.0
         self._rootChange = {}
         return
 
+    def rebuildIndex(self):
+        """Rebuild the entire index from scratch.
+        """
+        self.clearIndex()
+        for nwItem in self._project.tree:
+            if nwItem is not None and nwItem.isFileType():
+                tHandle = nwItem.itemHandle
+                theDoc = self._project.storage.getDocument(tHandle)
+                self.scanText(tHandle, theDoc.readDocument() or "")
+        self._indexBroken = False
+        return
+
     def deleteHandle(self, tHandle):
         """Delete all entries of a given document handle.
         """
@@ -113,11 +125,11 @@ class NWIndex:
         moved from the archive or trash folders back into the active
         project.
         """
-        if not self.theProject.tree.checkType(tHandle, nwItemType.FILE):
+        if not self._project.tree.checkType(tHandle, nwItemType.FILE):
             return False
 
         logger.debug("Re-indexing item '%s'", tHandle)
-        theDoc = self.theProject.storage.getDocument(tHandle)
+        theDoc = self._project.storage.getDocument(tHandle)
         self.scanText(tHandle, theDoc.readDocument() or "")
 
         return True
@@ -125,13 +137,13 @@ class NWIndex:
     def indexChangedSince(self, checkTime):
         """Check if the index has changed since a given time.
         """
-        return self._indexChange > checkTime
+        return self._indexChange > float(checkTime)
 
     def rootChangedSince(self, rootHandle, checkTime):
         """Check if the index has changed since a given time for a
         given root item.
         """
-        return self._rootChange.get(rootHandle, self._indexChange) > checkTime
+        return self._rootChange.get(rootHandle, self._indexChange) > float(checkTime)
 
     ##
     #  Load and Save Index to/from File
@@ -140,7 +152,7 @@ class NWIndex:
     def loadIndex(self):
         """Load index from last session from the project meta folder.
         """
-        indexFile = self.theProject.storage.getMetaFile(nwFiles.INDEX_FILE)
+        indexFile = self._project.storage.getMetaFile(nwFiles.INDEX_FILE)
         if not isinstance(indexFile, Path):
             return False
 
@@ -171,12 +183,12 @@ class NWIndex:
         logger.debug("Checking index")
 
         # Check that all files are indexed
-        for fHandle in self.theProject.projFiles:
+        for fHandle in self._project.projFiles:
             if fHandle not in self._itemIndex:
                 logger.warning("Item '%s' is not in the index", fHandle)
                 self.reIndexHandle(fHandle)
 
-        self._indexChange = round(time())
+        self._indexChange = time()
 
         logger.debug("Index loaded in %.3f ms", (time() - tStart)*1000)
 
@@ -186,7 +198,7 @@ class NWIndex:
         """Save the current index as a json file in the project meta
         data folder.
         """
-        indexFile = self.theProject.storage.getMetaFile(nwFiles.INDEX_FILE)
+        indexFile = self._project.storage.getMetaFile(nwFiles.INDEX_FILE)
         if not isinstance(indexFile, Path):
             return False
 
@@ -222,7 +234,7 @@ class NWIndex:
         files before we save them, in which case we already have the
         text.
         """
-        theItem = self.theProject.tree[tHandle]
+        theItem = self._project.tree[tHandle]
         if theItem is None:
             logger.info("Not indexing unknown item '%s'", tHandle)
             return False
@@ -256,7 +268,7 @@ class NWIndex:
             self._scanActive(tHandle, theItem, theText, itemTags)
 
         # Update timestamps for index changes
-        nowTime = round(time())
+        nowTime = time()
         self._indexChange = nowTime
         self._rootChange[theItem.itemRoot] = nowTime
 
@@ -737,8 +749,8 @@ class ItemIndex:
     IndexHeading object for each header of the text.
     """
 
-    def __init__(self, theProject):
+    def __init__(self, project):
-        self.theProject = theProject
+        self._project = project
         self._items = {}
         return
 
@@ -802,7 +814,7 @@ class ItemIndex:
         """Iterate over all items and headers in the novel structure for
         a given root handle, or for all if root handle is None.
         """
-        for tItem in self.theProject.tree:
+        for tItem in self._project.tree:
             if tItem is None:
                 continue
             if tItem.isNoteLayout():
@@ -885,7 +897,7 @@ class ItemIndex:
             if not isHandle(tHandle):
                 raise ValueError("itemIndex keys must be handles")
 
-            nwItem = self.theProject.tree[tHandle]
+            nwItem = self._project.tree[tHandle]
             if nwItem is not None:
                 tItem = IndexItem(tHandle, nwItem)
                 tItem.unpackData(tData)

novelwriter/core/project.py

@@ -369,8 +369,11 @@ class NWProject(QObject):
 
         self._scanProjectFolder()
         self._index.loadIndex()
-        self.updateWordCounts()
+        if xmlReader.state == XMLReadState.WAS_LEGACY:
+            # Often, the index needs to be rebuilt when updating format
+            self._index.rebuildIndex()
 
+        self.updateWordCounts()
         self._projOpened = time()
         self._projAltered = False
 
@@ -482,8 +485,8 @@ class NWProject(QObject):
             return False
 
         archName = baseDir / self.tr(
-            "Backup from {0}.zip"
+            "Backup from {0}"
-        ).format(formatTimeStamp(time(), fileSafe=True))
+        ).format(formatTimeStamp(time(), fileSafe=True) + ".zip")
         if self._storage.zipIt(archName, compression=2):
             if doNotify:
                 self.mainGui.makeAlert(self.tr(

novelwriter/core/tree.py

@@ -331,20 +331,15 @@ class NWTree:
     def setOrder(self, newOrder):
         """Reorders the tree based on a list of items.
         """
-        tmpOrder = []
+        tmpOrder = [tHandle for tHandle in newOrder if tHandle in self._projTree]
-
+        if not (len(tmpOrder) == len(newOrder) == len(self._treeOrder)):
-        # Add all known elements to a new temp list
+            # Something is wrong, so let's debug it
-        for tHandle in newOrder:
+            for tHandle in newOrder:
-            if tHandle in self._projTree:
+                if tHandle not in self._projTree:
-                tmpOrder.append(tHandle)
+                    logger.error("Handle '%s' in new tree order is not in old order", tHandle)
-            else:
+            for tHandle in self._treeOrder:
-                logger.error("Handle '%s' in new tree order is not in project tree", tHandle)
+                if tHandle not in tmpOrder:
-
+                    logger.warning("Handle '%s' in old tree order is not in new order", tHandle)
-        # Do a reverse lookup to check for items that will be lost
-        # This is mainly for debugging purposes
-        for tHandle in self._treeOrder:
-            if tHandle not in tmpOrder:
-                logger.warning("Handle '%s' in old tree order is not in new tree order", tHandle)
 
         # Save the temp list
         self._treeOrder = tmpOrder

novelwriter/guimain.py

@@ -833,17 +833,8 @@ class GuiMain(QMainWindow):
         tStart = time()
 
         self.projView.saveProjectTasks()
-        self.theProject.index.clearIndex()
+        self.theProject.index.rebuildIndex()
-
+        self.projView.populateTree()
-        for tItem in self.theProject.tree:
-            if tItem is None:  # pragma: no cover
-                continue  # This is a bug trap
-
-            logger.debug("Indexing '%s'", tItem.itemName)
-            if self.theProject.index.reIndexHandle(tItem.itemHandle):
-                # Update Word Counts
-                self.projView.propagateCount(tItem.itemHandle, tItem.wordCount, countChildren=True)
-                self.projView.setTreeItemValues(tItem.itemHandle)
 
         tEnd = time()
         self.setStatus(

sample/nwProject.nwx

@@ -1,6 +1,6 @@
 <?xml version='1.0' encoding='utf-8'?>
-<novelWriterXML appVersion="2.0-rc1" hexVersion="0x020000c1" fileVersion="1.5" timeStamp="2022-11-07 13:00:48">
+<novelWriterXML appVersion="2.0-rc2" hexVersion="0x020000c2" fileVersion="1.5" timeStamp="2022-11-10 17:56:30">
-  <project id="e2be99af-f9bf-4403-857a-c3d1ac25abea" saveCount="1448" autoCount="237" editTime="69737">
+  <project id="e2be99af-f9bf-4403-857a-c3d1ac25abea" saveCount="1450" autoCount="237" editTime="69742">
     <name>Sample Project</name>
     <title>Sample Project</title>
     <author>Jane Smith</author>

tests/test_core/test_core_index.py

@@ -90,7 +90,7 @@ def testCoreIndex_LoadSave(monkeypatch, prjLipsum, mockGUI, tstPaths):
     assert theIndex._tagsIndex._tags == {}
     assert theIndex._itemIndex._items == {}
 
-    # No folder for sloading
+    # No folder for loading
     with monkeypatch.context() as mp:
         mp.setattr("novelwriter.core.storage.NWStorage.getMetaFile", lambda *a: None)
         assert theIndex.loadIndex() is False
@@ -108,6 +108,13 @@ def testCoreIndex_LoadSave(monkeypatch, prjLipsum, mockGUI, tstPaths):
     assert str(theIndex._tagsIndex.packData()) == tagIndex
     assert str(theIndex._itemIndex.packData()) == itemsIndex
 
+    # Rebuild index
+    theIndex.clearIndex()
+    theIndex.rebuildIndex()
+
+    assert str(theIndex._tagsIndex.packData()) == tagIndex
+    assert str(theIndex._itemIndex.packData()) == itemsIndex
+
     # Check File
     copyfile(projFile, testFile)
     assert cmpFiles(testFile, compFile)

tests/test_core/test_core_project.py

@@ -230,7 +230,7 @@ def testCoreProject_Open(monkeypatch, caplog, mockGUI, fncPath, mockRnd):
         assert "The file format of your project is about to be" in mockGUI.lastQuestion[1]
         mockGUI.askResponse = True
 
-    # Won't convert legacy file
+    # Won't open project from newer version
     with monkeypatch.context() as mp:
         mp.setattr(ProjectXMLReader, "hexVersion", property(lambda *a: 0x99999999))
         mockGUI.askResponse = False
@@ -245,6 +245,18 @@ def testCoreProject_Open(monkeypatch, caplog, mockGUI, fncPath, mockRnd):
 
     assert theProject.closeProject()
 
+    # Trigger an index rebuild
+    with monkeypatch.context() as mp:
+        mp.setattr(ProjectXMLReader, "state", property(lambda *a: XMLReadState.WAS_LEGACY))
+        mp.setattr("novelwriter.core.index.NWIndex.loadIndex", lambda *a: True)
+        mockGUI.askResponse = True
+        theProject.index._indexBroken = True
+        assert theProject.openProject(fncPath) is True
+        assert "The file format of your project is about to be" in mockGUI.lastQuestion[1]
+        assert theProject.index._indexBroken is False
+
+    assert theProject.closeProject()
+
 # END Test testCoreProject_Open
 
 

tests/test_core/test_core_projectxml.py

@@ -38,6 +38,13 @@ class MockProject:
         pass
 
 
+@pytest.fixture(scope="function", autouse=True)
+def mockVersion(monkeypatch):
+    monkeypatch.setattr("novelwriter.__version__", "2.0-rc1")
+    monkeypatch.setattr("novelwriter.__hexversion__", "0x020000c1")
+    return
+
+
 @pytest.mark.core
 def testCoreProjectXML_ReadCurrent(monkeypatch, tstPaths, fncPath):
     """Test reading the current XML file format.

tests/test_core/test_core_tree.py

@@ -24,6 +24,7 @@ import random
 
 from pathlib import Path
 
+from mock import causeOSError
 from tools import readFile
 
 from novelwriter.enum import nwItemClass, nwItemType, nwItemLayout
@@ -230,6 +231,36 @@ def testCoreTree_BuildTree(mockGUI, mockItems):
 # END Test testCoreTree_BuildTree
 
 
+@pytest.mark.core
+def testCoreTree_PackUnpack(mockGUI, mockItems):
+    """Test packing and unpacking data.
+    """
+    theProject = NWProject(mockGUI)
+    theTree = NWTree(theProject)
+
+    aHandles = []
+    for tHandle, pHandle, nwItem in mockItems:
+        aHandles.append(tHandle)
+        theTree.append(tHandle, pHandle, nwItem)
+        theTree.updateItemData(tHandle)
+
+    assert len(theTree) == len(mockItems)
+
+    # Pack
+    tree = theTree.pack()
+    for i, (tHandle, pHandle, nwItem) in enumerate(mockItems):
+        assert tree[i]["itemAttr"]["handle"] == tHandle
+
+    # Unpack
+    theTree.clear()
+    assert len(theTree) == 0
+    assert theTree.handles() == []
+    assert theTree.unpack(tree) is True
+    assert theTree.handles() == aHandles
+
+# END Test testCoreTree_PackUnpack
+
+
 @pytest.mark.core
 def testCoreTree_Methods(mockGUI, mockItems):
     """Test various class methods.
@@ -272,6 +303,13 @@ def testCoreTree_Methods(mockGUI, mockItems):
     assert theTree.findRoot(nwItemClass.NOVEL) == "a000000000001"
     assert theTree.findRoot(nwItemClass.CHARACTER) == "a000000000004"
 
+    # Iter roots
+    roots = list(theTree.iterRoots(None))
+    assert roots[0][0] == "a000000000001"
+    assert roots[1][0] == "a000000000002"
+    assert roots[2][0] == "a000000000003"
+    assert roots[3][0] == "a000000000004"
+
     # Add a fake item to root and check that it can handle it
     theTree._treeRoots["0000000000000"] = NWItem(theProject)
     assert theTree.findRoot(nwItemClass.WORLD) is None
@@ -363,7 +401,7 @@ def testCoreTree_Stats(mockGUI, mockItems):
 
 
 @pytest.mark.core
-def testCoreTree_Reorder(mockGUI, mockItems):
+def testCoreTree_Reorder(caplog, mockGUI, mockItems):
     """Test changing tree order.
     """
     theProject = NWProject(mockGUI)
@@ -384,12 +422,16 @@ def testCoreTree_Reorder(mockGUI, mockItems):
     theTree.setOrder(bHandle)
     assert theTree.handles() == bHandle
 
+    caplog.clear()
     theTree.setOrder(bHandle + ["stuff"])
     assert theTree.handles() == bHandle
+    assert "Handle 'stuff' in new tree order is not in old order" in caplog.text
 
+    caplog.clear()
     theTree._treeOrder.append("stuff")
     theTree.setOrder(bHandle)
     assert theTree.handles() == bHandle
+    assert "Handle 'stuff' in old tree order is not in new order" in caplog.text
 
 # END Test testCoreTree_Reorder
 
@@ -421,6 +463,11 @@ def testCoreTree_ToCFile(monkeypatch, mockGUI, mockItems, tmpPath):
     theProject._storage._runtimePath = None
     assert theTree.writeToCFile() is False
 
+    theProject._storage._runtimePath = tmpPath
+    with monkeypatch.context() as mp:
+        mp.setattr("builtins.open", causeOSError)
+        assert theTree.writeToCFile() is False
+
     theProject._storage._runtimePath = tmpPath
     assert theTree.writeToCFile() is True