.. _docs_technical_storage:

******************
How Data is Stored
******************

.. _version control: https://en.wikipedia.org/wiki/Version_control

This chapter contains details of how novelWriter stores and handles the project data.


Overview
========

The files of a novelWriter project are stored in a dedicated project folder. The project structure
is kept in a file at the root of this folder called ``nwProject.nwx``. All the document files and
associated meta data are stored in other folders below the project folder.

This way of storing data was chosen for several reasons.

Firstly, all the text you add to your project is saved directly to your project folder in separate
files. Only the project structure and the text you are currently editing is stored in memory at any
given time, which means there is a smaller risk of losing data if the application or your computer
crashes.

Secondly, having multiple small files means it is very easy to synchronise them between computers
with standard file synchronisation tools.

Thirdly, if you use `version control`_ software to track the changes to your project, the file
formats used for the files are well suited. All the JSON documents have line breaks and indents as
well, which makes it easier to track them with version control software.

.. note::

   Since novelWriter has to keep track of a bunch of files and folders when a project is open, it
   may not run well on some virtual file systems. A file or folder must be accessible with exactly
   the path it was saved or created with. An example where this is not the case is the way Google
   Drive is mapped on Linux Gnome desktops using gvfs/gio.

.. caution::

   You should not add additional files to the project folder yourself. Nor should you, as a rule,
   manually edit files within it. If you really must manually edit the text files, e.g. with some
   automated task you want to perform, you need to rebuild the Project Index when you open
   the project again.

   Editing text files in the ``content`` folder is less risky as these are just plain text. Editing
   the main project XML file, however, may make the project file unreadable and you may crash
   novelWriter and lose project structure information and project settings.


Project Structure
=================

All novelWriter files are written with utf-8 encoding. Since Python automatically converts Unix
line endings to Windows line endings on Windows systems, novelWriter does not make any adaptations
to the formatting on Windows systems. This is handled entirely by the Python standard library.
Python also handles this when working on the same files on both Windows and Unix-based operating
systems.


Main Project File
-----------------

The project itself requires a dedicated folder for storing its files, where novelWriter will create
its own "file system" where the project's folder and file hierarchy is described in a project XML
file. This is the main project file in the project's root folder with the name ``nwProject.nwx``.
This file also contains all the meta data required for the project (except the index data), and a
number of related project settings.

If this file is lost or corrupted, the structure of the project is lost, although not the text
itself. It is important to keep this file backed up, either through the built-in backup tool, or
your own backup solution.

The project XML file is indent-formatted, and is suitable for diff tools and version control since
most of the file will stay static, although a timestamp is set in the meta section on line 2, and
various meta data entries incremented, on each save.

A full project file format specification is available under "More Documents".


Project Documents
=================

All the project documents are saved in a subfolder of the main project folder named ``content``.
Each document has a file handle based on a 52 bit random number, represented as a hexadecimal
string. The documents are saved with a filename assembled from this handle and the file extension
``.nwd``.

If you wish to find the file system location of a document in the project, you can either look it
up in the project XML file, select **Show File Details** from the **Document** menu when having the
document open in the editor, or look in the ``ToC.txt`` file in the root of the project folder. The
``ToC.txt`` file has a list of all documents in the project, referenced by their label, and where
they are saved.

The reason for this cryptic file naming is to avoid issues with file naming conventions and
restrictions on different operating systems, and also to have a file name that does not depend on
what you name the document within the project, or changes it to. This is particularly useful when
using a versioning system.

Each document file contains a plain text version of the text from the editor. The file can in
principle be edited in any text editor, and is suitable for diffing and version control if so
desired. Just make sure the file remains in utf-8 encoding, otherwise unicode characters may
become mangled when the file is opened in novelWriter again.

Editing these files is generally not recommended. The reason for this is that the index will not be
automatically updated when doing so, which means novelWriter doesn't know you've altered the file.
If you *do* edit a file in this manner, you should rebuild the index when you next open the project
in novelWriter.

The first lines of the file may contain some meta data starting with the characters ``%%~``. These
lines are mainly there to restore some information if the file is lost from the main project file,
and the information may be helpful if you do open the file in an external editor as it contains the
document label and the document class and layout. The lines can be deleted without any consequences
to the rest of the content of the file, and will be added back the next time the document is saved
in novelWriter.


The File Saving Process
-----------------------

When saving the project file, or any of the documents, the data is first saved to a temporary file.
If successful, the old data file is then removed, and the temporary file replaces it. This ensures
that the previously saved data is only replaced when the new data has been successfully saved to
the storage medium.


Project Meta Data
=================

The project folder contains a subfolder named ``meta``, containing a number of files. The meta
folder contains semi-important files. That is, they can be lost with only minor impact to the
project. All files in this folder are JSON or JSON Lines files, although some other files may
remain from earlier versions of novelWriter as they haven't all been JSON files in the past.

If you use version control software on your project, you can exclude this folder, although you may
want to track the session log file and the custom words list.


The Project Index
-----------------

Between writing sessions, the project index is saved in a JSON file in ``meta/index.json``.
This file is not critical. If it is lost, it can be completely rebuilt from within novelWriter from
the **Tools** menu.

The index is maintained and updated whenever a document or note is saved in the editor. It contains
all references and tags in documents and notes, as well as the location of all headers in the
project, and the word counts within each header section.

The integrity of the index is checked when the file is loaded. It is possible to corrupt the index
if the file is manually edited and manipulated, so the check is important to avoid sudden crashes
of novelWriter. If the file contains errors, novelWriter will automatically build it anew. If the
check somehow fails and novelWriter keeps crashing, you can delete the file manually and rebuild
the index. If this too fails, you have likely encountered a bug.


Build Definitions
-----------------

The build definitions from the **Manuscript Build** tool are kept in the ``meta/builds.json`` file.
If this file is lost, all custom build definitions are lost too.


Cached GUI Options
------------------

A file named ``meta/options.json`` contains the latest state of various GUI buttons, switches,
dialog window sizes, column sizes, etc, from the GUI. These are the GUI settings that are specific
to the project. Global GUI settings are stored in the main config file.

The file is not critical, but if it is lost, all such GUI options will revert back to their default
settings.


Custom Word List
----------------

A file named ``meta/userdict.json`` contains all the custom words you've added to the project for
spell checking purposes. The content of the file can be edited from the **Tools** menu. If you lose
this file, all your custom spell check words will be lost too.


Session Stats
-------------

The writing progress is saved in the ``meta/sessions.jsonl`` file. This file records the length
and word counts of each writing session on the given project. The file is used by the **Writing
Statistics** tool. If this file is lost, the history it contains is also lost, but it has otherwise
no impact on the project.

Each session is recorded as a JSON object on a single line of the file. Each session record is
appended tot he file.
