Python docstrings and Sphinx

2023-08-25 9 min by Lukas Atkinson Python (9) Documentation (2)

Docstrings are great. Sphinx is great. But they don't go great together.

One of the truly great Python features is the docstring. Every object, and that also means every method, can have an associated string containing documentation. This is comparable to doc comments in Java, C#, or Haskell, but it makes the documentation a first-class object of the language, and greatly improves the result of metaprogramming when done correctly. It can be introspected at run time with the help() function.

Although PEP-257 tries to set standards for docstring conventions, these standards are both very vague and little respected in practice. There also isn't any kind of markup language present, and the docstring contents are just plain text. This is in stark contrast to Javadoc, C# XML comments, or Doxygen.

Sphinx

Sphinx is a document processing engine primarily targeted at Python documentation. but it makes more sense to see it as a dumbed-down LaTeX. You could conceivably use it to write a book, and publish it as a website, ebook, and PDF. Sphinx uses the Restructured Text (ReST) as a lightweight markup language. ReST looks a bit like Markdown and has the same intention (human-readable plain-text), but is generally very different. Unlike Markdown, ReST is designed for extensibility and can define various “text roles”, which work a bit like macros. While there are some minor details that I dislike, Sphinx+ReST are a capable and mature authoring tool.

Sphinx implements many roles useful for Python documentation, e.g. :class:`SomeClass` creates a link to the reference documentation of that class.

This combination works very well for tutorials, overview documentation, and other technical documents – far better than Markdown, actually! But can it autogenerate reference docs from source files? Kind of, but not really.

Reference docs with Sphinx

For reference documentation, my expectation is that I point a documentation tool at my source tree, and twiddle my thumbs until the tool has extracted all reference docs from the source files and put them into indexed and cross-referenced HTML files. That is how Doxygen and Javadoc work. It is great.

But that is not how Sphinx does it. Sphinx expects you to write your reference docs by hand. Now that is insane. This is understandable if you're documenting the Python core because C does not have docstrings. But when documenting Python code, this is not a good idea. By separating your code from your reference docs in a separate document, it is harder to keep them in sync. It also duplicates any effort that goes into the docstrings.

Creating stub documents for Sphinx

Luckily, the sphinx.ext.autodoc extension exists. You can now plonk an autoclass or automodule directive into your documentation, and Sphinx will work its magic to fill in all nested classes and methods from the docstrings. That's more like it! However, this still requires you to specify which modules you want documented and create the appropriate files. And now you need to use ReST markup in your docstrings.

The problem of writing the autodoc stubs for all your modules can be automated with the sphinx-apidoc tool. This doesn't actually create the documentation, just the stubs that will then be filled when you actually run the sphinx-build. This means you'll either have to make the apidoc invocation part of your build process, or have to check in the generated files, and re-run the apidoc tool whenever you rename, move, delete, or add a module.

I've tried to automate this by writing a Makefile. It puts the autogenerated reference docs into an api directory. In my docs, I will then have to manually link to the top-level autogenerated modules in order to include the reference docs in the toctree.

So if the project directory is . and the Python package lives in the ./your_module_name/ directory, this Makefile will generate the ./docs/api/your_module_name.rst stub:

NAME = your_module_name
SETUPPY = python setup.py
AUTODOC_DEFAULTS = members,show-inheritance
DOCS = ./docs

.PHONY: docs clean

docs: $(DOCS)/api/$(NAME).rst
  @ # can't check doc coverage and create html output in a single pass :(
  $(SETUPPY) build_sphinx -b coverage
  $(SETUPPY) build_sphinx -b html
  @ # Write out the coverage information,
  @ # which is a text file listing uncovered methods.
  @ cat ./build/sphinx/coverage/python.txt

$(DOCS)/api/%.rst:
  @ [ -d $(DOCS)/api ] || mkdir $(DOCS)/api
  SPHINX_APIDOC_OPTIONS='$(AUTODOC_DEFAULTS)' \
    sphinx-apidoc -o $(DOCS)/api $* \
      --force --separate --module-first --no-toc

clean:
  @ rm -rf build/
  @ rm -rf $(DOCS)/api

Note that if I have several top-level packages in my source tree, I'll need multiple autodoc invocations.

(Note: this code was last tested in 2017. Things might be better now.)

Human-readable docstrings with Sphinx

So, regarding the ReST syntax in our docstrings, can we get rid of that?

The problem is that the ReST roles do not look very appealing as plain text. As a very contrived example, a method may be documented as:

def double_each(xs):
  """Double each number.

  :param xs: The numbers to be doubled.
  :type xs: list(int)
  :returns: A new list containing the doubled numbers.
  :rtype: list(int)
  :throws ValueError: when the input is empty.
  """

  if not xs:
    raise ValueError("input was empty")

  return [2 * x for x in xs]

Yikes! This works well as a Sphinx document, but if we inspect the function in the REPL with help(double_each), the output will look incomprehensible.

This can be lessened with the sphinx.ext.napoleon extension. With this extension active, we can write docs in the “Google style” that used indentation to mark these roles. The result is mostly readable in plaintext:

Double each number.

Args:
  xs (list(int)): The numbers to be doubled.
Returns:
  list(int): A new list containing the doubled numbers.
Throws:
  ValueError: when the input is empty.

The Google style has one notable restriction: If a function returns a tuple, the syntax does not provide a good way to document the members of that tuple. In this case, you can switch to the alternative Numpy style that is also supported by Napoleon.

With Napoleon, it is possible to write plaintext-readable ReST in most cases. This is not the case if you use cross-references to other functions or classes in your docstring, but that can't be helped. Unless the help() system starts rendering ReST, a bit of markup in our docstrings is unavoidable.

MkDocs

MkDocs, and especially the mkdocs-material project, provides an alternative to Sphinx. No ReStructured Text, just Markdown (albeit with some extensions).

While I consider Sphinx' content-tree concept more intuitive than MkDocs' navigation, individual MkDocs pages are more pleasurable to author.

There is also a simpler approach to including docstrings. The relevant extension for this is called mkdocstrings. Similar to Sphinx-Autodoc, the documentation must include a stub file for rendering the docs of a module, class, or other item.

Creating stub documents for MkDocs

However, mkdocstrings provides a recipy for auto-generating these stubs. This workflow isn't particularly pretty, but it's flexible and easily adapted to the concrete project.

First, configure the core gen-files plugin to invoke a build script during the documentation build process, and enable support for SUMMARY.md files (as is common for similar tools like mdbook):

plugins:
- gen-files:
    scripts:
      - docs/gen_api_docs.py
- literate-nav:
    nav_file: SUMMARY.md

Create that script with content like the following (taken with minimal edits from the recipe):

"""Generate the code reference pages and navigation."""

from pathlib import Path
import mkdocs_gen_files

nav = mkdocs_gen_files.Nav()

for path in sorted(Path("src").rglob("*.py")):
    module_path = path.relative_to("src").with_suffix("")
    doc_path = path.relative_to("src").with_suffix(".md")
    full_doc_path = Path("reference", doc_path)

    parts = tuple(module_path.parts)

    if parts[-1] == "__init__":
        parts = parts[:-1]

    nav[parts] = doc_path.as_posix()

    with mkdocs_gen_files.open(full_doc_path, "w") as fd:
        ident = ".".join(parts)
        fd.write(f"::: {ident}")

    mkdocs_gen_files.set_edit_path(full_doc_path, path)

with mkdocs_gen_files.open("reference/SUMMARY.md", "w") as nav_file:
    nav_file.writelines(nav.build_literate_nav())

This script loads all paths matching the glob src/**.py, then creates corresponding reference/**.md files with stub content like:

::: foo.bar

This stub content can be adjusted as desired.

Note mkdocs_gen_files.open(...). This creates a virtual/in-memory file, without writing anything to disk.

This script has complete control over the generated stubs, so could also use a different file layout.

And while resolving the stubs to actual content, the layout of the auto-generated reference material can be controlled using Jinja templates. For example, I once used such template inheritance to insert a list of sub-modules into each module, without showing the full submodule contents.

Doctests in MkDocs

It is fairly common in Python documentation to show a doctest, a code snippet that mimics a REPL session. These doctests can also be executed by Pytest, as a normal part of the test suite. This is great for executable documentation.

Here is a typical example of a Python function with a docstring that contains doctests:

def foo(x):
  """
  This function does something.

  Example: positive inputs:

  >>> foo(4)
  1234

  Example: negative inputs:

  >>> foo(-2)
  42

  """
  return 1234 if x > 0 else 42

Unfortunately, this is one feature that is much more cumbersome with MkDocs than with Sphinx. Doctests are a built-in ReST syntax. But they are not part of Markdown, and MkDocs does not offer a corresponding syntax extensoin.

Instead, it is necessary to wrap the doctest in an ordinary (fenced) code block:

```pycon
>>> foo(4)
1234
```

This will render correctly, but breaks the test when executing it: the doctest block must end with an empty line.

So correct version of the Python function with MkDocs-compatible Markdown syntax would be:

def foo(x):
  """
  This function does something.

  Example: positive inputs:

  ```pycon
  >>> foo(4)
  1234

  ```

  Example: negative inputs:

  ```pycon
  >>> foo(-2)
  42

  ```
  """
  return 1234 if x > 0 else 42

This is a bit annoying, but aside from that MkDocs offers a pretty convenient way to create an API documentation based on the docstrings in the code.