Skip to content

python.rst

Latest commit

 

History

History
1427 lines (1152 loc) · 67 KB

python.rst

File metadata and controls

1427 lines (1152 loc) · 67 KB
Raw

Python docs

breadcrumb:{filename}/documentation.rst Doc generators
summary:A modern, mobile-friendly Sphinx-alike Python documentation generator with a first-class search functionality
footer:
.. note-dim::
    :class: m-text-center

    `« Doxygen C++ theme <{filename}/documentation/doxygen.rst>`_ | `Doc generators <{filename}/documentation.rst>`_

A modern, mobile-friendly Sphinx-alike Python documentation generator with a first-class search functionality. Generated by inspecting Python modules and using either embedded docstrings or external :abbr:`reST <reStructuredText>` files to populate the documentation.

One of the design goals is providing a similar user experience to the Doxygen documentation theme.

.. button-success:: https://doc.magnum.graphics/python/

    Live demo

    doc.magnum.graphics

Contents

Basic usage

The base is contained in a single Python script and related style/template files, for advanced features such as math rendering it'll make use of internals of some m.css plugins. Clone :gh:`the m.css GitHub repository <mosra/m.css$master/documentation>` and look into the documentation/ directory:

git clone git://github.com/mosra/m.css
cd m.css/documentation

The script requires Python 3.6 and depends on Jinja2 for templating and docutils for :abbr:`reST <reStructuredText>` markup rendering. You can install the dependencies via pip or your distribution package manager, in most cases you'll probably have them already installed:

# You may need sudo here
pip3 install docutils jinja2

Next, you need a configuration file which tells the script what modules to inspect, how to name the project and where to put the output. In this example, we'll generate documentation for the Python builtin math module:

PROJECT_TITLE = "Python math"
INPUT_MODULES = ['math']

Now, run the script and pass path to the configuration file to it:

./python.py path/to/conf.py

This will generate an output/ directory next to the conf.py file and fill it with the generated output. Open index.html to see the result.

Features

  • Theme tailored from scratch for Python-specific language features
  • Uses code inspection to query modules, classes, data, functions and their signatures, does not rely on error-prone source code parsing
  • Does not force the documentation writer to explicitly list all symbols in order to have them documented
  • Can use both in-code docstrings and external :abbr:`reST <reStructuredText>` files to describe the APIs, giving the user a control over the code size vs documentation verbosity tradeoff

Configuration

Together with the above PROJECT_TITLE and INPUT_MODULES variables mentioned above, the configuration file supports the following variables. The options are similar to the Doxygen config, but free of the Doxygen-specific naming and constraints.

Variable Description
PROJECT_TITLE: str Project title. Rendered in top navbar, page title and fine print. If not set, "My Python Project" is used.
PROJECT_SUBTITLE: str Project subtitle. If set, appended in a thinner font to PROJECT_TITLE.
PROJECT_LOGO: str URL of an image to use as a log in the top navbar. Default is none.
MAIN_PROJECT_URL: str If set and PROJECT_SUBTITLE is also set, then PROJECT_TITLE in the top navbar will link to this URL and PROJECT_SUBTITLE to the documentation main page, similarly as shown here.
INPUT: str Base input directory. If not set, config file base dir is used. Relative paths are relative to config file base dir.
OUTPUT: str Where to save the output. Relative paths are relative to INPUT; if not set, output/ is used.
INPUT_MODULES: List[Any] List of modules to generate the docs from. Values can be either strings or module objects. See Module inspection for more information.
INPUT_PAGES: List[str] List of :abbr:`reST <reStructuredText>` files for standalone pages. See Pages for more information.
THEME_COLOR: str Color for <meta name="theme-color" />, corresponding to the CSS style. If empty, no <meta> tag is rendered. See Theme selection for more information.
FAVICON: str Favicon URL, used to populate <link rel="icon" />. If empty, no <link> tag is rendered. Relative paths are searched relative to INPUT and to the python.py script dir as a fallback. See Theme selection for more information.
STYLESHEETS: List[str] List of CSS files to include. Relative paths are searched relative to INPUT and to the python.py script dir as a fallback. See Theme selection for more information.
HTML_HEADER: str HTML code to put at the end of the <head> element. Useful for linking arbitrary JavaScript code or, for example, adding <link> CSS stylesheets with additional properties and IDs that are otherwise not possible with just STYLESHEETS.
EXTRA_FILES: List[str] List of extra files to copy (for example additional CSS files that are @imported from the primary one). Relative paths are searched relative to INPUT and to the python.py script dir as a fallback.
LINKS_NAVBAR1: List[Any] Left navbar column links. See Navbar links for more information.
LINKS_NAVBAR2: List[Any] Right navbar column links. See Navbar links for more information.
PAGE_HEADER: str :abbr:`reST <reStructuredText>` markup to put at the top of every page. If not set, nothing is added anywhere. The {url} placeholder is replaced with current file URL.
FINE_PRINT: str :abbr:`reST <reStructuredText>` markup to put into the footer. If not set, a default generic text is used. If empty, no footer is rendered at all.
FORMATTED_METADATA: List[str] Which meatadata fields should be formatted in documentation pages. By default only the summary field is.
PLUGINS: List[str] List of plugins to use. See Plugins for more information.
PLUGIN_PATHS: List[str] Additional plugin search paths. Relative paths are relative to INPUT.
CLASS_INDEX_EXPAND_LEVELS How many levels of the class index tree to expand. 0 means only the top-level symbols are shown. If not set, 1 is used.
CLASS_INDEX_EXPAND_INNER Whether to expand inner classes in the class index. If not set, False is used.
NAME_MAPPING: Dict[str, str] Additional name mapping in addition to what's figured out from the __all__ members
PYBIND11_COMPATIBILITY: bool Enable some additional tricks for better compatibility with pybind11. If not set, False is used. See pybind11 compatibility for more information.
ATTRS_COMPATIBILITY: bool Enable some additional tricks for better compatibility with attrs. If not set, False is used. See attrs compatibility for more information.
SEARCH_DISABLED: bool Disable search functionality. If this option is set, no search data is compiled and the rendered HTML does not contain search-related UI or support. If not set, False is used.
SEARCH_DOWNLOAD_BINARY Download search data as a binary to save bandwidth and initial processing time. If not set, False is used. See Search options for more information.
SEARCH_HELP: str :abbr:`reST <reStructuredText>` markup to display as help text on empty search popup. If not set, a default message is used. Has effect only if SEARCH_DISABLED is not True.
SEARCH_BASE_URL: str Base URL for OpenSearch-based search engine suggestions for web browsers. See Search options for more information. Has effect only if SEARCH_DISABLED is not True.
SEARCH_EXTERNAL_URL: str URL for external search. The {query} placeholder is replaced with urlencoded search string. If not set, no external search is offered. See Search options for more information. Has effect only if SEARCH_DISABLED is not True.
DOCUTILS_SETTINGS: Dict[Any] Additional docutils settings. Key/value pairs as described in the docs.
URL_FORMATTER: Callable Function for creating filenames and URLs for modules, classes, pages and index pages. See Custom URL formatters for more information.
ID_FORMATTER: Callable Function for creating link anchors for module and class members. See Custom URL formatters for more information.

Theme selection

By default, the dark m.css theme together with documentation-theme-specific additions is used, which corresponds to the following configuration:

STYLESHEETS = [
    'https://fonts.googleapis.com/css?family=Source+Sans+Pro:400,400i,600,600i%7CSource+Code+Pro:400,400i,600',
    '../css/m-dark+documentation.compiled.css']
THEME_COLOR = '#22272e'
FAVICON = 'favicon-dark.png'

If you have a site already using the m-dark.compiled.css file, there's another file called m-dark.documentation.compiled.css, which contains just the documentation-theme-specific additions so you can reuse the already cached m-dark.compiled.css file from your main site:

STYLESHEETS = [
    'https://fonts.googleapis.com/css?family=Source+Sans+Pro:400,400i,600,600i%7CSource+Code+Pro:400,400i,600',
    '../css/m-dark.compiled.css',
    '../css/m-dark.documentation.compiled.css']
THEME_COLOR = '#22272e'
FAVICON = 'favicon-dark.png'

If you prefer the light m.css theme instead, use the following configuration (and, similarly, you can use m-light.compiled.css together with m-light.documentation.compiled-css in place of m-light+documentation.compiled.css:

STYLESHEETS = [
    'https://fonts.googleapis.com/css?family=Libre+Baskerville:400,400i,700,700i%7CSource+Code+Pro:400,400i,600',
    '../css/m-light+documentation.compiled.css']
THEME_COLOR = '#cb4b16'
FAVICON = 'favicon-light.png'

See the CSS files section below for more information about customizing the CSS files.

Navbar links

The LINKS_NAVBAR1 and LINKS_NAVBAR2 options define which links are shown on the top navbar, split into left and right column on small screen sizes. These options take a list of (title, path, sub) tuples --- title is the link title; path is either one of 'index', 'pages', 'modules' or 'classes' (linking to the main page or page / module / class index path), a full URL (pasted as-is) or a path to a particular page or module/class (in the form of ['module', 'sub', 'ClassName'] for module.sub.ClassName, which then gets formatted according to URL formatting rules); and sub is an optional submenu, containing (title, path) tuples, with path being interpreted the same way.

By default the variables are defined like following --- there's just three items in the left column, with no submenus and the right column is empty:

LINKS_NAVBAR1 = [
    ('Pages', 'pages', []),
    ('Modules', 'modules', []),
    ('Classes', 'classes', [])]
LINKS_NAVBAR2 = []

A menu item is highlighted if a page with the same path is the current page. The path can be also a full URL --- if it contains a scheme prefix (such as https://), then it's taken as-is, without conversion.

Search options

Symbol search is implemented using JavaScript Typed Arrays and does not need any server-side functionality to perform well --- the client automatically downloads a tightly packed binary containing search data and performs search directly on it.

However, due to restrictions of Chromium-based browsers, it's not possible to download data using XMLHttpRequest when served from a local file-system. Because of that, the search defaults to producing a Base85-encoded representation of the search binary and loading that asynchronously as a plain JavaScript file. This results in the search data being 25% larger, but since this is for serving from a local filesystem, it's not considered a problem. If your docs are accessed through a server (or you don't need Chrome support), set the SEARCH_DOWNLOAD_BINARY option to True. The search data are by default fetched from the current directory on the webserver, if you want to supply a different location, set it to a string and provide a custom URL formatter.

The site can provide search engine metadata using the OpenSearch specification. On supported browsers this means you can add the search field to search engines and search directly from the address bar. To enable search engine metadata, point SEARCH_BASE_URL to base URL of your documentation, for example:

SEARCH_BASE_URL = 'https://doc.magnum.graphics/magnum/'

In general, even without the above setting, appending ?q={query}#search to the URL will directly open the search popup with results for {query}.

.. note-info::

    OpenSearch also makes it possible to have autocompletion and search results
    directly in the browser address bar. However that requires a server-side
    search implementation and is not supported at the moment.

If SEARCH_EXTERNAL_URL is specified, full-text search using an external search engine is offered if nothing is found for given string or if the user has JavaScript disabled. It's recommended to restrict the search to a particular domain or add additional keywords to the search query to filter out irrelevant results. Example, using Google search engine and restricting the search to a subdomain:

SEARCH_EXTERNAL_URL = 'https://google.com/search?q=site:doc.magnum.graphics+{query}'

Custom URL formatters

The URL_FORMATTER option allows you to control how all filenames and generated URLs look like. It takes an entry type and a "path" as a list of strings (so for example my.module.Class is represented as ['my', 'module', 'Class']), returning a tuple of a filename and an URL. Those can be the same, but also different (for example a file getting saved into my/module/Class/index.html but the actual URL being https://docs.my.module/Class/). The default implementation looks like this, producing both filenames and URLs in the form of my.module.Class.html:

The type is an enum, if you don't want to fiddle with imports, compare type.name against a string, which is one of 'PAGE', 'MODULE', 'CLASS', 'SPECIAL' or 'STATIC'. The 'SPECIAL' is for index pages and in that case the path has always just one item, one of 'pages', 'modules' or 'classes'. The 'STATIC' is for static data such as images or CSS files and the path is absolute input filename including the extension and except for search data (which are generated on-the-fly) it always exists. If the static path is an URL, the URL formatter is not called.

The ID_FORMATTER handles formatting of anchors on a page. Again it takes an entry type (which in this case is always one of 'ENUM', 'ENUM_VALUE', 'FUNCTION', 'PROPERTY', 'DATA' or, in case of pybind11 code, 'OVERLOADED_FUNCTION'. The second parameter is again a path, being always just one item except for 'ENUM_VALUE' (in which case it's enum name and value name together) and for 'OVERLOADED_FUNCTION', in which case it contains also a llist of argument types. The default implementation simply returns the the path concatenated with dashes:

def default_id_formatter(type: EntryType, path: List[str]) -> str:
    return '-'.join(path)

Module inspection

By default, if a module contains the __all__ attribute, all names listed there are exposed in the documentation. Otherwise, all module (and class) members are extracted using inspect.getmembers(), skipping names imported from elsewhere and undocumented underscored names. Additionally, class data members with type annotations (but with no values) are pulled out from __annotations__, allowing you to expose (and document) also fields that might otherwise only be populated from __init__():

class MyClass:
    a_float: float
    string_value: str

Detecting if a module is a submodule of the current package or if it's imported from elsewhere is tricky, the script thus includes only submodules that have their __package__ property the same or one level below the parent package. If a module's __package__ is empty, it's assumed to be a plain module (instead of a package) and since those can't have submodules, all found submodules in it are ignored.

.. block-success:: Overriding the set of included names, module reorganization

    In case the autodetection includes more than you want or, conversely, you
    need to include names that would otherwise be excluded (such as underscored
    names), you can temporarily override the :py:`__all__` attribute when
    generating the docs. For example, the following will list just the
    :py:`pow()` and :py:`log()` funtions from the :py:`math` module, ignoring
    the rest:

    .. code:: py

        import math
        math.__all__ = ['pow', 'log']

        INPUT_MODULES = [math]

    In other cases, especially when native modules are involved, the inspected
    name locations might not be what you want. By putting the names into
    :py:`__all__` you tell the script it should map the inspected location to
    the one provided. Note you should also hide the original location from the
    script to avoid duplicate definitons (unless it's underscored, in which
    case it'll get ignored automatically).

    .. code:: py

        # module math

        from _native_math import fast_sin as sin
        from _native_math import fast_cos as cos
        __all__ = ['sin', 'cos']

    Additionally, for mapping types of external libraries where the
    autodetection from :py:`__all__` can't be performed, you can use the
    :py:`NAME_MAPPING` option:

    .. code:: py

        NAME_MAPPING = {
            'fastmath._native.Vector3': 'fastmath.Vector3',
            'fastmath._native.Quaternion': 'fastmath.Quaternion',
            # or, equivalently, if the mapping is the same for all members:
            'fastmath._native': 'fastmath'
        }

Docstrings

By default, the first paragraph of a module-level, class-level and function-level docstring is used as a doc summary, copied as-is to the output without formatting it in any way. What follows is put (again without formatting) paragraph-by-paragraph into detailed docs.

"""Module summary

First paragraph of module detailed docs."""

class Foo:
    """Class summary"""

    def bar(self):
        """Function summary"""

Using just docstrings, however, comes with a few limitations:

  • Class and module-level variables can't have a docstring attached due to how Python works
  • Because not every Python API can be documented using docstrings, the output contains everything, including undocumented names
  • Instance variables added inside __init__() are not extracted, as this would require parsing Python code directly (which is what Sphinx has to do to support these). You can work around this by adding annotated "declarations" to the class as shown above, however no docstrings can be specified for those either.

To overcome the limitations, externally-supplied documentation provides means to document names that can't have a docstring attached, and together with the m.sphinx plugin expanding formatting capabilities beyond plain text.

Function and variable annotations

The script uses inspect.signature() to query function parameter / return type annotations together with default values and displays them in the output. Similar is for module and class variables, extracted from the __annotations__ property. If a variable type implements __repr__(), a repr() of it is printed as the value, otherwise the value is omitted.

from typing import Tuple, List

def foo(a: str, be_nice: bool = True) -> Tuple[int, str]:
    pass

SETTINGS: List[Tuple[str, bool]] = []

For better readability, if the function signature contains type annotations or a default value, the arguments are printed each on one line. Otherwise, to avoid wasting vertical space, the arguments are listed on a single line.

Similarly to how the builtin help() in Python 3.7 started annotating boundaries between position-only, position-or-keyword and keyword-only arguments with / and *, the same is done here --- it's especially helpful for native functions, where you can for example call math.sin(0.3) but not math.sin(x=0.3), because the x argument is positional-only. Currently, positional-only arguments are possible only with native functions, PEP570 adds them for pure Python functions as well.

In some cases, especially when documenting native functions, the signature can't be extracted and the function signature shows just an ellipsis (…) instead of the actual argument list.

Class methods, static methods, dunder methods, properties

Methods decorated with @classmethod are put into a "Class methods" section, @staticmethods into a "Static methods" section. Double-underscored methods explicitly implemented in the class are put into a "Special methods" section, otherwise they're ignored --- by default, Python adds a large collection of dunder methods to each class and the only way to know if the method is user-provided or implicit is by checking the docstring.

class MyClass:
    @classmethod
    def a_classmethod(cls):
        """A class method"""

    @staticmethod
    def a_staticmethod():
        """A static method"""

    def __init__(self, foo, bar):
        """A constructor"""

Properties added to classes either using the @property decorator or created with the property() builtin are added to the "Properties" section. Each property is annotated with :label-flat-success:`get set del` if it has a getter, a setter and a deleter or with :label-flat-warning:`get` and other variants if it has just some. The docstring and type annotation is extracted from the property getter.

from typing import Tuple

class MyClass:
    @property
    def a_read_write_property(self) -> Tuple[int, int]:
        """A read-write tuple property"""

    @a_read_write_property.setter
    def a_read_write_property(self, a):
        # Docstring and type annotation taken from the getter, no need to
        # have it repeated here too
        pass

Documenting enum values

Python supplies an implicit docstrings for enums derived from enum.Enum and enum values implicitly inherit the docstring of the enum class. If either is detected to be the case, docstring of the enum or the value is ignored. While it's possible to document enum classes the usual way, there's a non-obvious way to document enum values as well.

import enum

class MyEnum(enum.Enum):
    """My enum"""

    ZERO = 0
    TWO = 3
    CONSISTENCY = -73

MyEnum.ZERO.__doc__ = "Zero value"
MyEnum.TWO.__doc__ = "Three, but named TWO for compatibility"

The documentation output for enums includes enum value values and the class it was derived from, so it's possible to know whether it's an enum or a flag.

Including underscored names in the output

By default, names starting with an underscore (except for __dunder__ methods) are treated as private and not listed in the output. One way to expose them is to list them in __all__, however that works for module content only. For exposing general underscored names, you either need to provide a docstring or external documentation content (and in case of plain data, external documentation content is the only option).

Note that at the point where modules and classes are crawled for members, docstrings are not parsed yet --- so e.g. a data documentation via a :data: option of the .. py:class:: m.sphinx directive won't be visible to the initial crawl and thus the data will stay hidden.

Sometimes, however, you'll want the inverse --- keeping an underscored name hidden, even though it has a docstring. Solution is to remove the docstring while generating the docs, directly in the conf.py file during module import:

import mymodule
mymodule._private_thing.__doc__ = None

INPUT_MODULES = [mymodule]

Pages

In addition to documentation generated by inspecting particular module, it's possible to add dedicated documentation pages. Content is written in :abbr:`reST <reStructuredText>` (see Writing reST content for a short introduction) and taken from files specified in INPUT_PAGES. Filenames are interpreted relative to configuration file path, output filename is input basename with extension replaced to .html. In particular, content of a index.rst file is used for the documentation main page. Example:

INPUT_PAGES = ['pages/index.rst']
My Python library
=================

:summary: Welcome on the main page!

This is a documentation of the mypythonlib module. You can use it like
this:

.. code:: py

    import mypythonlib
    mypythonlib.foo()

Apart from :summary:, the page can have any number of metadata, with all of them exposed as properties of page in the output templates. Fields listed in FORMATTED_METADATA (the :summary: is among them) are expected to be formatted as :abbr:`reST <reStructuredText>` and exposed as HTML, otherwise as a plain text.

All referenced images are expected to have either an absolute URL or be relative to INPUT, the ones with relative paths are then copied directly to OUTPUT with the leading dirs stripped from the path.

Plugins

The :abbr:`reST <reStructuredText>` content is not limited to just the builtin functionality and it's possible to extend it via plugins eiter from m.css itself or 3rd party ones. See documentation of each plugin to see its usage; the m.htmlsanity plugin is used unconditionally while all others are optional. For example, enabling the common m.css plugins might look like this:

PLUGINS = ['m.code', 'm.components', 'm.dox']

External documentation content

Because it's often not feasible to have the whole documentation stored in Python docstrings, the generator allows you to supply documentation from external files. Similarly to pages, the INPUT_DOCS setting is a list of :abbr:`reST <reStructuredText>` files that contain documentation for particular names using custom directives. A set of custom directives is provided by the m.sphinx plugin --- see its documentation for detailed description of all features. Below is a simple example of using it to document a class:

PLUGINS = ['m.sphinx']
INPUT_DOCS = ['docs.rst']
.. py:class:: mymodule.sub.Class
    :summary: A pretty class

    This class is *pretty*.

pybind11 compatibility

C++ bindings generated using pybind11 do not have all information accessible through introspection and thus the script has to do a few pybind11-specific workarounds to generate expected output. This behavior is not enabled by default as it might have unwanted consequences in pure Python code, enable it using the PYBIND11_COMPATIBILITY option.

Function signatures, property annotations

For reasons explained in :gh:`pybind/pybind11#990`, pybind11 is not able to provide function signatures through introspection and thus the script falls back to parsing argument names, type annotations and default values from the docstring instead. By default, unless py::arg() is used, function arguments are positional-only (shown as arg0, arg1, ...) and marked as such in the output.

Similarly, property types are extracted from getter docstrings.

Unlike Python, pybind11 has a builtin support for overloaded functions --- depending on types passed to a function, it dispatches to a particular C++ overload. The overloads are expanded in the output as well, meaning you can see one function mentioned more than once with different signatures.

Because static methods in pybind11 are not decorated with @staticmethod, they are detected based on presence of self as the first parameter --- if it's there, it's an instance method, otherwise it's a static method.

.. block-warning:: Limitations

    The static / instance method autodetection may fail when you name the first
    argument of a static method as :cpp:`py::arg("self")`. Don't do that |wink|

    The signature parsing can't handle all cases and, especially when templated
    C++ type names leak through, it may fail to extract the argument names. If
    that happens, the function signature shows just an ellipsis (``…``). On the
    other hand, encountering a pure C++ type in a Python function signature
    most probably points to a problem with the bindings as the type can't be
    expressed with Python code.

Enums

Enums in pybind11 are not derived from enum.Enum, but rather are plain classes. The only reliable way to detect a pybind11 enum is by looking for a __members__ member, which is a dict providing string names and their corresponding values. With pybind 2.2, it's only possible to document the enum class itself, not the values.

.. note-info::

    pybind 2.3 supports docstrings for enum values (see
    :gh:`pybind/pybind11#1160`). Support for this feature is not done on the
    script side yet.

attrs compatibility

If a codebase is using the attrs package and the ATTRS_COMPATIBILITY option is enabled, the script is able to extract the (otherwise inaccessible by normal means) information about attributes defined using attr.ib() or via the @attr.s(auto_attribs=True) decorator. Note that attributes of classes using @attr.s(slots=True) are visible even without the compatibility enabled.

In all cases, there's no possibility of adding in-source docstrings for any of these and you need to supply the documentation with the .. py:property:: directive as described in External documentation content.

Additionally, various dunder methods that say just "Automatically created by attrs." in their docstring are implicitly hidden from the output if this option is enabled. In order to show them again, override the docstring to something meaningful.

Command-line options

./python.py [-h] [--templates TEMPLATES] [--debug] conf

Arguments:

  • conf --- configuration file

Options:

  • -h, --help --- show this help message and exit
  • --templates TEMPLATES --- template directory. Defaults to the templates/python/ subdirectory if not set.
  • --debug --- verbose logging output. Useful for debugging.

Implementing custom plugins

Third-party plugins can be loaded from paths specified in PLUGIN_PATHS. Custom plugins need to implement a registration function named register_mcss(). It gets passed the following named arguments and the plugin might or might not use them.

Keyword argument Content
mcss_settings Dict containing all m.css settings
jinja_environment Jinja2 environment. Useful for adding new filters etc.
module_doc_contents Module documentation contents
class_doc_contents Class documentation contents
enum_doc_contents Enum documentation contents
enum_value_doc_contents Enum documentation contents
function_doc_contents Function documentation contents
property_doc_contents Property documentation contents
data_doc_contents Data documentation contents
hooks_post_crawl Hooks to call after the initial name crawl
hooks_pre_scope Hooks to call on scope enter
hooks_post_scope Hooks to call on scope exit
hooks_docstring Hooks to call when parsing a docstring
hooks_pre_page Hooks to call before each page gets rendered
hooks_post_run Hooks to call at the very end of the script run

The module_doc_contents, class_doc_contents, enum_doc_contents, enum_value_doc_contents, function_doc_contents, property_doc_contents and data_doc_contents variables are Dict[str, Dict[str, str]], where the first level is a name and second level are key/value pairs of the actual HTML documentation content. Plugins that parse extra documentation inputs (such as m.sphinx) are supposed to add to the dict, which is then used to fill the actual documentation contents. The following corresponds to the documentation source shown in the External documentation content section below. Note that the dict can already have existing entries added from elsewhere, so it's important to avoid fully overwriting it:

docs = class_doc_contents.setdefault('mymodule.sub.Class', {})
docs['summary'] = "A pretty class"
docs['details'] = "This class is *pretty*."

The hooks_post_crawl, hooks_docstring, hooks_pre_page and hooks_post_run variables are lists of functions. Plugins that need to do something at specific points of the execution are supposed to add functions to the list.

The hooks_post_crawl is called once gathering of all names is done. It gets passed the following arguments:

Keyword argument Content
name_map

Map with all gathered module, class, enum, function, property, data and page metadata. Plugins are allowed to read from it (for example to serialize them to a file for searching or linking from other projects) as well as write to it (for example to allow linking to names from external projects). Key is a name, value has at the following properties:

Property Description
type Entry type. Same as the enum passed to custom URL formatters.
object Object which the entry documents. This property being None means this entry is external [1]; if not present at all there's filename instead.
filename File this entry documents. Present only for EntryType.PAGE or EntryType.SPECIAL.
path Path. Equivalent to key.split('.').
url URL to the entry documentation, formatted with custom URL formatters.
css_classes List of CSS classes to add to the <a> tag. Internal entries usually have ['m-doc'] while exteral have ['m-doc-external'].
[1]As this distinguishes between internal and external entries, new entries added by the plugin need to have object set to None so the script as well as other plugins can correctly distinguish them.

The hooks_pre_scope and hooks_post_scope get called before entering and after leaving a name scope (page, module, class, enum, enum value, function, property or data), and are meant mainly to aid with context-sensitive linking. Those scopes can be nested and can be called successively for the same scope --- for example, when rendering module docs, hooks_pre_scope gets called first for the module scope, but then another hooks_pre_scope gets called when rendering a summary for reference to an inner class. Then, hooks_post_scope gets called in reverse order. The plugins are expected to implement a stack-like data structure for maintaining information about current scope. Both of those functions get passed the following arguments:

Keyword argument Content
type Type of the scope that's being entered or exited. Same as the enum passed to custom URL formatters.
path Path of the module / class / function / enum / enum value / data scope that's being entered or exited. A list of names, '.'.join(path) is equivalent to the fully qualified name.
param_names In case of functions, list of parameter names. This argument is not present otherwise.

Hooks listed in hooks_docstring are called when docstrings are parsed, and always preceded by a corresponding hooks_pre_scope call. The first listed hook gets the raw docstring only processed by inspect.cleandoc() and each following gets the output of the previous. When a hook returns an empty string, hooks later in the list are not called. String returned by the last hook is processed, if any, the same way as if no hooks would be present --- it gets partitioned into summary and content and those put to the output as-is, each paragraph wrapped in <p> tags. The hooks are free to do anything with the docstring --- extracting metadata from it and returning it as-is, transpiling it from one markup language to another, or fully consuming it, populating the *_doc_contents variables mentioned above and returning nothing back. Each hook gets passed the following arguments:

Keyword argument Content
type Name type. Same as the enum passed to custom URL formatters.
path Path of the module / class / function / enum / enum value / data containing the docstring. A list of names, '.'.join(path) is equivalent to the fully qualified name.
signature Signature of a function, for distinguishing between particular overloads. In a form of (param1: type1, param2: type2).
doc Docstring content. Always non-empty --- once a hook returns nothing back, no further hooks are called.

The hooks_pre_page is called before each page of output gets rendered. Can be used for example for resetting some internal counter for page-wide unique element IDs. The hooks_post_run is called after the whole run is done, useful for example to serialize cached internal state. Currently, those two functions get no arguments passed.

Registration function for a plugin that needs to query the OUTPUT setting might look like this --- the remaining keyword arguments will collapse into the **kwargs parameter. See code of various m.css plugins for actual examples. The below example shows registration of a hypothetic HTML validator plugin --- it saves the output path from settings and registers a post-run hook that validates everything in given output directory.

output_dir = None

…

def _validate_output():
    validate_all_html_files(output_dir)

def register_mcss(mcss_settings, hooks_post_run, **kwargs):
    global output_dir
    output_dir = mcss_settings['OUTPUT']
    hooks_post_run += [_validate_output]

Customizing the template

The rest of the documentation explains how to customize the builtin template to better suit your needs. Each documentation file is generated from one of the template files that are bundled with the script. However, it's possible to provide your own Jinja2 template files for customized experience as well as modify the CSS styling.

CSS files

By default, compiled CSS files are used to reduce amount of HTTP requests and bandwidth needed for viewing the documentation. However, for easier customization and debugging it's better to use the unprocessed stylesheets. The STYLESHEETS option lists all files that go to the <link rel="stylesheet" /> in the resulting HTML markup, while EXTRA_FILES list the indirectly referenced files that need to be copied to the output as well. Below is an example configuration corresponding to the dark theme:

STYLESHEETS = [
    'https://fonts.googleapis.com/css?family=Source+Sans+Pro:400,400i,600,600i%7CSource+Code+Pro:400,400i,600',
    '../css/m-dark.css',
    '../css/m-documentation.css']
EXTRA_FILES = [
    '../css/m-grid.css',
    '../css/m-components.css',
    '../css/pygments-dark.css',
    '../css/pygments-console.css']
THEME_COLOR = '#22272e'

After making desired changes to the source files, it's possible to postprocess them back to the compiled version using the postprocess.py utility as explained in the CSS themes documentation. In case of the dark theme, the m-dark+documentation.compiled.css and m-dark.documentation.compiled.css files are produced like this:

cd css
./postprocess.py m-dark.css m-documentation.css -o m-dark+documentation.compiled.css
./postprocess.py m-dark.css m-documentation.css --no-import -o m-dark.documentation.compiled.css

Output templates

Each output file is rendered with one of these templates:

Filename Use
module.html Module documentation
class.html Class documentation
page.html Explicit documentation pages, including the main page

Each template gets passed all configuration values from the Configuration table as-is, together with a URL variable with URL of given output file. In addition to builtin Jinja2 filters, the format_url filter returns either a path formatted according to custom URL formatters, if the path is relative; or a full URL, if the argument is an absolute URL. It's useful in cases like this:

{% for css in HTML_EXTRA_STYLESHEET %}
<link rel="stylesheet" href="{{ css|format_url|e }}" />
{% endfor %}

The actual page contents are provided in a page object, which has the following properties. All exposed data are meant to be passed directly to the HTML markup without any additional escaping.

Property Description
page.summary Doc summary
page.filename File name [4]
page.url File URL [4]
page.breadcrumb List of (title, URL) tuples for breadcrumb navigation.
page.content Detailed documentation, if any

Each module page, rendered with module.html, has the following additional properties:

Property Description
page.prefix_wbr Fully-qualified symbol prefix for given compound with trailing . with <wbr/> tag after every ..
page.modules List of inner modules. See Module properties for details.
page.classes List of classes. See Class properties for details.
page.enums List of enums. See Enum properties for details.
page.functions List of module-level functions. See Function properties for details.
page.data List of module-level data. See Data properties for details.
page.has_enum_details If there is at least one enum with full description block [3]
page.has_function_details If there is at least one function (or method, in case of classes) with full description block [3]
page.has_data_details If there is at least one data with full description block [3]

Each class page, rendered with class.html, has the following additional properties:

Property Description
page.classmethods List of class methods (annotated with @classmethod). See Function properties for details.
page.staticmethods List of static methods (annotated with @staticmethod). See Function properties for details.
page.methods List of methods. See Function properties for details.
page.exceptions List of exceptions, does not overlap with py:page.classes. See Class properties for details.
page.dunder_methods List of double-underscored special functions. See Function properties for details.
page.properties List of properties. See Property properties for details.
page.has_property_details If there is at least one property with full description block [3]

Explicit documentation pages rendered with class.html have additional properties taken from input metadata. If given metadata is listed in FORMATTED_METADATA, it's rendered into HTML, otherwise it's exposed as plain text.

Module properties

Property Description
module.url URL of detailed module documentation
module.name Module name
module.summary Doc summary

Class properties

Property Description
class_.url URL of detailed class documentation
class_.name Class name
class_.is_exception Marks exceptions
class_.summary Doc summary

Enum properties

Property Description
enum.name Enum name
enum.id Enum ID [5]
enum.summary Doc summary
enum.content Detailed documentation, if any
enum.base Base class from which the enum is derived. Set to None if no base class information is available.
enum.base_link Like enum.base, but with cross-linked types
enum.values List of enum values
enum.has_details If there is enough content for the full description block. [3]
enum.has_value_details If the enum values have description. Impies enum.has_details.

Every item of enum.values has the following properties:

Property Description
value.name Value name
value.id Value ID [5]
value.value Value value. Set to None if no value is available.
value.content Value documentation, if any

Function properties

Property Description
function.name Function name
function.id Function ID [5]
function.summary Doc summary
function.content Detailed documentation, if any
function.type Function return type annotation [2]
function.type_link Like function.type, but with cross-linked types
function.params List of function parameters. See below for details.
function.exceptions List of exceptions raised by this function. See below for details.
function.has_complex_params Set to True if the parameter list should be wrapped on several lines for better readability (for example when it contains type annotations or default arguments). Set to False when wrapping on multiple lines would only occupy too much vertical space.
function.has_param_details If the function parameters are documented
function.return_value Return value documentation. Can be empty.
function.has_details If there is enough content for the full description block [3]
function.is_classmethod Set to True if the function is annotated with @classmethod, False otherwise.
function.is_staticmethod Set to True if the function is annotated with @staticmethod, False otherwise.

The function.params is a list of function parameters and their description. Each item has the following properties:

Property Description
param.name Parameter name
param.type Parameter type annotation [2]
param.type_link Like param.type, but with cross-linked types
param.default Default parameter value, if any
param.kind Parameter kind, a string equivalent to one of the inspect.Parameter.kind values
param.content Detailed documentation, if any

In some cases (for example in case of native APIs), the parameters can't be introspected. In that case, the parameter list is a single entry with name set to "..." and the rest being empty.

The function.exceptions is a list of exceptions types and descriptions. Each item has the following properties:

Property Description
exception.type Exception type
exception.type_link Like exception, but with a cross-linked type
exception.content Detailed documentation

Property properties

Property Description
property.name Property name
property.id Property ID [5]
property.type Property getter return type annotation [2]
property.type_link Like property.type, but with cross-linked types
property.summary Doc summary
property.content Detailed documentation, if any
property.exceptions List of exceptions raised when accessing this property. Same as function.exceptions described in function properties.
property.is_gettable If the property is gettable
property.is_settable If the property is settable
property.is_deletable If the property is deletable with del
property.has_details If there is enough content for the full description block [3]

Data properties

Property Description
data.name Data name
data.id Data ID [5]
data.type Data type
data.type_link Like data.type_link, but with cross-linked types
data.summary Doc summary
data.content Detailed documentation, if any
data.value Data value representation
data.has_details If there is enough content for the full description block [3]

Index page templates

The following index pages are provided, showing a expandable tree of the contents:

Filename Use
classes.html Class listing
modules.html Module listing
pages.html Page listing

Each template is passed all configuration values from the Configuration table as-is, together with an URL, as above. The navigation tree is provided in an index object, which has the following properties:

Property Description
index.classes List of all modules + classes
index.pages List of all pages

The form of each list entry is the same:

Property Description
i.kind Entry kind (one of 'module', 'class', 'exception' or 'page')
i.name Name
i.url URL of the file with detailed documentation
i.summary Doc summary
i.has_nestable_children If the list has nestable children (i.e., dirs or namespaces)
i.children Recursive list of child entries

Module/class list is ordered in a way that all modules are before all classes.

[2](1, 2, 3) i.type is extracted out of function annotation. If the types aren't annotated, the annotation is empty.
[3](1, 2, 3, 4, 5, 6, 7, 8) page.has_*_details and i.has_details are True if there is detailed description, function parameter documentation or documented enum value listing that makes it worth to render the full description block. If False, the member should be included only in the summary listing on top of the page to avoid unnecessary repetition.
[4](1, 2) page.filename and page.url is generated by an URL formatter, see Custom URL formatters for more information
[5](1, 2, 3, 4, 5) i.id is an ID used for linking to given entry on a page. Generated by an anchor formatter, see Custom URL formatters for more information.