autodoc-documented type aliases can't be referenced from type annotations

[godlygeek]

Describe the bug

When using autodoc to document type aliases in a module, references to those aliases in the signature of a function cannot be resolved, even when using from __future__ import annotations and autodoc_type_aliases.

How to Reproduce

Create module.py with these contents:

from __future__ import annotations
import pathlib

#: Any type of path
pathlike = str | pathlib.Path


def read_file(path: pathlike) -> bytes:
    """Read a file and return its contents."""
    with open(path, "rb") as f:
        return f.read()

and index.rst with these contents:

.. automodule:: module
   :members:
   :member-order: bysource

and then run Sphinx, enabling autodoc and using autodoc_type_aliases:

$ python -m sphinx -aE -C -D 'extensions=sphinx.ext.autodoc' -D 'autodoc_type_aliases.pathlike=pathlike' . output

Expected behavior

On the module.read_file(path: pathlike) → bytes line, pathlike should be a link to the module.pathlike type alias, but it is not a link at all.

Running with nitpicky mode shows:

module.py:docstring of module.read_file:1: WARNING: py:class reference target not found: pathlike

This is because autodoc is generating a py:attr entry for pathlike, and Sphinx is trying to resolve a py:class entry instead.

Your project

See "how to reproduce"

Screenshots

No response

OS

Linux

Python version

3.10.6

Sphinx version

5.1.1

Sphinx extensions

sphinx.ext.autodoc

Extra tools

No response

Additional context

I'm working around this with a hack in my docs/conf.py:

TYPE_ALIASES = ["pathlike", "filelike"]

def resolve_type_aliases(app, env, node, contnode):
    """Resolve :class: references to our type aliases as :attr: instead."""
    if (
        node["refdomain"] == "py"
        and node["reftype"] == "class"
        and node["reftarget"] in TYPE_ALIASES
    ):
        return app.env.get_domain("py").resolve_xref(
            env, node["refdoc"], app.builder, "attr", node["reftarget"], node, contnode
        )


def setup(app):
    app.connect("missing-reference", resolve_type_aliases)

[electric-coder]

Running with nitpicky mode shows:

module.py:docstring of module.read_file:1: WARNING: py:class reference target not found: pathlike

I was looking for a Python 3.9 workaround (not using TypeVar) if a solution couldn't be found, and this comment allowed me to find a way to get the hyperlink to the type alias working in the signature.

This is because autodoc is generating a py:attr entry for pathlike, and Sphinx is trying to resolve a py:class entry instead.

autodoc is wrong here to begin with, a module level type alias (at least up until Python 3.9 without TypeVar) is a py:data role. It makes sense that Sphinx itself is trying to resolve the role as py:class because types outside of the standard library used in signatures would always be classes.

A type alias (not using TypeVar) would be in reST:

.. data:: pathlike 
    :type: str | pathlib.Path

So to get the hyperlink to resolve I changed the declaration to:

.. class:: pathlike 

The important part is giving Sphinx the .. py:class directive it expects. Together with setting autodoc_type_aliases in conf.py the hyperlink then works.

autodoc_type_aliases = {
    'pathlike ': 'module.pathlike ',  # works
}

However, the workaround leaves you with a problem: in the documentation the type alias now appears as a class. So to have it display with the proper type (as a stop gap measure) you would have to:

1. redeclare pathlike as .. data:: pathlike using reST
2. add the :noindex: option to the .. data:: pathlike declaration
3. Finally, use CSS to make the .. class:: pathlike declaration invisible:

.. class:: pathlike 

.. data:: pathlike 
    :type: str | pathlib.Path
    :noindex:

This does get the documentation with hyperlinks and cross-references working as intended. But it requires manually writing reST for each type alias declaration. From the end user's perspective the API shows correctly (the only telltale sign would the type in the index, but that could also be manually overridden.)

[cjw296]

In my case, I have a type annotation to an alias called Evaluator used as follows:

class Region:
    """
    Parsers should yield instances of this class for each example they
    discover in a documentation source file.
    
    :param start: 
        The character position at which the example starts in the
        :class:`~sybil.document.Document`.
    
    :param end: 
        The character position at which the example ends in the
        :class:`~sybil.document.Document`.
    
    :param parsed: 
        The parsed version of the example.
    
    :param evaluator: 
        The callable to use to evaluate this example and check if it is
        as it should be.
    """

    def __init__(self, start: int, end: int, parsed: Any, evaluator: Evaluator):
        #: The start of this region within the document's :attr:`~sybil.Document.text`.
        self.start: int = start
        #: The end of this region within the document's :attr:`~sybil.Document.text`.
        self.end: int = end
        #: The parsed version of this region. This only needs to have meaning to
        #: the :attr:`evaluator`.
        self.parsed: Any = parsed
        #: The :any:`Evaluator` for this region.
        self.evaluator: Evaluator = evaluator

This is defined in sybil/typing.py as:

#: The signature for an evaluator. See :ref:`developing-parsers`.
Evaluator = Callable[['sybil.Example'], Optional[str]]

This is documented in a .rst file as follows:

.. autoclass:: sybil.typing.Evaluator

Now, for the __init__ parameter usage, I get this rendering:

class sybil.Region(start: int, end: int, parsed: Any, evaluator: Callable[[sybil.Example], Optional[str]])

However, for the attribute, I get:

evaluator: Evaluator
The Evaluator for this region.

Note the lack of linking in Evaluator. I get the following warning:

sybil/region.py:docstring of sybil.Region.evaluator:1: WARNING: py:class reference target not found: Evaluator

I can find no way to fix this, I tried:

• .. autodata:: sybil.typing.Evaluator instead of .. autoclass:: sybil.typing.Evaluator
• Both autoclass and autodata but with autodoc_type_aliases = {'Evaluator': 'sybil.typing.Evaluator'} in my conf.py

Since I have nitpicky on and treating warnings as thing to cause a doc build to fail, the only workaround I can find is to put this in my conf.py:

nitpick_ignore = [('py:class', 'Evaluator')]

So, three problems:

1. Evaluator isn't linked and I get a "py:class reference target not found: Evaluator" warning (which I believe is the crux of this github issue?)
2. Where Evaluator is used in a method parameter type annotation, it is replaced by the type annotation rather than just the text Evaluator linked to its autoclass definition. Is anyone aware of a github issue already open for this?
3. I've noticed that Any isn't link to anywhere in the Python docs from my Sphinx sounds, again, is there an issue already open for this?

If I've missed any information that would be useful in making progress on this, please let me know!

[mantasu]

To add on top of #10785 (comment), here's how to properly document and hide the dummy class reference using Python 3.12:

1. Declare your aliases in some file, e.g., in my_package/utils.py, using new type statement (see PEP 695)
2. At the top of the file add a docstring describing your aliases (use :data: directive with :noindex:). For each alias add a dummy :class: directive with the same name (as mentioned above)

An example python file may look like this:

"""
.. class:: BytesOrStr
 
.. data:: BytesOrStr
    :noindex:
    :type: typing.TypeAliasType
    :value: str | bytes
    
    Type alias for bytes or string.

    Bound:
        :class:`str` | :class:`bytes`

.. class:: FilePath
    
.. data:: FilePath
    :noindex:
    :type: typing.TypeAliasType
    :value: BytesOrStr | os.PathLike

    Type alias for a file path.

    Bound:
        :class:`str` | :class:`bytes` | :class:`os.PathLike`
"""
import os

type BytesOrStr = str | bytes
type FilePath = BytesOrStr | os.PathLike
# More type aliases ...

4. In conf.py add an alias dictionary TYPE_ALIAS mapping from type names to their module paths
5. Modify HTML to delete class directives but copy their properties (including ID) to corresponding data directives. This can be done using build-finished event.

Here is an example of what could be appended to conf.py (I'm using sphinx 7.2.6 and pydata-sphinx-theme 0.15.1, other versions may have different HTML layouts):

from pathlib import Path
from bs4 import BeautifulSoup, Tag

# Define alias paths
TYPE_ALIASES = {
    "BytesOrStr": "my_package.utils.",
    "FilePath": "my_package.utils.",
}

def keep_only_data(soup: BeautifulSoup):
    def has_children(tag: Tag, txt1: str, txt2: str):
        if tag.name != "dt":
            return False

        # Get the prename and name elements of the signature
        ch1 = tag.select_one("span.sig-prename.descclassname span.pre")
        ch2 = tag.select_one("span.sig-name.descname span.pre")

        return ch1 and ch2 and ch1.string == txt1 and ch2.string == txt2

    for alias, module in TYPE_ALIASES.items():
        if dt := soup.find("dt", id=f"{module}{alias}"):
            # Copy class directive's a
            a = dt.find("a").__copy__()
            dt.parent.decompose()
        else:
            continue

        if dt := soup.find(lambda tag: has_children(tag, module, alias)):
            # ID and a for data directive
            dt["id"] = f"{module}{alias}"
            dt.append(a)

def edit_html(app, exception):
    if app.builder.format != "html":
        return

    for pagename in app.env.found_docs:
        if not isinstance(pagename, str):
            continue

        with (Path(app.outdir) / f"{pagename}.html").open("r") as f:
            # Parse HTML using BeautifulSoup html parser
            soup = BeautifulSoup(f.read(), "html.parser")
            keep_only_data(soup)

        with (Path(app.outdir) / f"{pagename}.html").open("w") as f:
            # Write back HTML
            f.write(str(soup))

def setup(app):
    app.connect("build-finished", edit_html)

[Shoeboxam]

In my case, I only encounter this issue if I have from __future__ import annotations in the file. I avoided the issue by just using Union[A, B] instead of the future import and A | B.

[tanrbobanr]

Thought I'd add another temporary solution while we wait for this to get added. I created a small extension to autodoc that does something similar to what was done by @mantasu (#10785 (comment), except:

1. Edits the doctree itself, so that it is not dependent on the internals of the HTML builder.
2. Adds the autoalias directive to handle the creation and linkage of the py:class directive used for generating the references.

Feel free to use it in your own project (although hopefully this gets fixed soon so it won't be necessary).

Example:

.. autoalias:: path.to.my.alias

The code

import dataclasses
from collections import defaultdict
from collections.abc import Generator, Sequence
from typing import Any, Union

from docutils.nodes import Node, document
from docutils.statemachine import StringList
from sphinx import addnodes
from sphinx.application import Sphinx
from sphinx.domains.python import PyClasslike, PyObject, PyVariable
from sphinx.ext.autodoc import SUPPRESS, DataDocumenter, Documenter, Options
from sphinx.util import inspect
from sphinx.util.typing import ExtensionMetadata, restify


class ExtendedPyObjectMixin(PyObject):
    option_spec = {
        "alias-id": int,
    }

    def handle_signature(
        self, sig: str, signode: addnodes.desc_signature
    ) -> tuple[str, str]:
        # set alias-id on the signode parent (desc node) if alias-id is
        # available
        if "alias-id" in self.options:
            signode.parent.attributes["alias-id"] = self.options["alias-id"]
        return super().handle_signature(sig, signode)


class ExtendedPyVariable(ExtendedPyObjectMixin, PyVariable):
    option_spec = {
        **ExtendedPyObjectMixin.option_spec,
        **PyVariable.option_spec,
    }

    def get_signature_prefix(self, sig: str) -> Sequence[Node]:
        # `alias` prefeix
        if "alias-id" in self.options:
            return addnodes.desc_annotation("alias ", "alias ")
        return ()


class ExtendedPyClassLike(ExtendedPyObjectMixin, PyClasslike):
    option_spec = {
        **ExtendedPyObjectMixin.option_spec,
        **PyClasslike.option_spec,
    }


class AliasDocumenter(DataDocumenter):
    objtype = "alias"
    directivetype = "data"
    priority = DataDocumenter.priority + 2

    @classmethod
    def can_document_member(
        cls, member: Any, membername: str, isattr: bool, parent: Any
    ) -> bool:
        return inspect.isgenericalias(member)

    def add_alias_id(self) -> None:
        """Adds the :alias-id: option as a new line."""
        alias_id = hash((
            self.module,
            self.modname,
            self.real_modname,
            self.object,
            self.object_name,
            self.directivetype,
            self.objtype,
        ))
        self.add_line(f"   :alias-id: {alias_id}", self.get_sourcename())

    def add_directive_header(self, sig: str) -> None:
        self.options = Options(self.options)
        self.options["annotation"] = SUPPRESS
        self.options["no-index"] = True
        super().add_directive_header(sig)
        self.add_alias_id()

    def add_content(self, more_content: Union[StringList, None]) -> None:
        sourcename = self.get_sourcename()

        # format alias portion
        if getattr(self.config, "autodoc_typehints_format") == "short":
            alias = restify(self.object, "smart")
        else:
            alias = restify(self.object)

        self.add_line(f"An alias of {alias}.", sourcename)
        self.add_line("", sourcename)
        self.add_line("", sourcename)

        Documenter.add_content(self, more_content)

        # add py:class immediately after, using
        # `self.directive.result.append` directly since we need to
        # dedent
        self.directive.result.append(
            f"{self.indent[:-3]}.. py:class:: {self.fullname}", sourcename
        )
        self.add_alias_id()


def replace_obj_attrs(source: object, target: object) -> None:
    for k, v in source.__dict__.items():
        setattr(target, k, v)


@dataclasses.dataclass(eq=False)
class Alias:
    py_data: Union[addnodes.desc, None] = None
    py_class: Union[addnodes.desc, None] = None

    @staticmethod
    def get_index(node: addnodes.desc) -> addnodes.index:
        return node.parent.children[node.parent.index(node) - 1]

    @staticmethod
    def get_signature(node: addnodes.desc) -> addnodes.desc_signature:
        return node.children[
            node.first_child_matching_class(addnodes.desc_signature)
        ]

    def apply(self) -> None:
        if self.py_data is None or self.py_class is None:
            return

        py_class_index = self.get_index(self.py_class)

        # replace index
        replace_obj_attrs(py_class_index, self.get_index(self.py_data))

        # set ids
        self.get_signature(self.py_data).attributes["ids"] = (
            self.get_signature(self.py_class).attributes["ids"]
        )

        # remove class and its index
        self.py_data.parent.remove(self.py_class)
        self.py_data.parent.remove(py_class_index)


def is_alias(node: Node) -> bool:
    if not isinstance(node, addnodes.desc):
        return False

    if node.attributes.get("alias-id") is None:
        return False

    return True


def get_alias_nodes(doctree: document) -> Generator[addnodes.desc, None, None]:
    yield from doctree.findall(is_alias)


def merge_aliases(app: Sphinx, doctree: document) -> None:
    aliases: defaultdict[int, Alias] = defaultdict(Alias)

    # find aliases
    for node in get_alias_nodes(doctree):
        domain: Union[str, None] = node.attributes.get("domain")
        objtype: Union[str, None] = node.attributes.get("objtype")
        alias_id: int = node.attributes["alias-id"]

        if domain != "py":
            raise ValueError(f"Invalid domain for alias: {domain}")

        if objtype == "data":
            aliases[alias_id].py_data = node
        elif objtype == "class":
            aliases[alias_id].py_class = node
        else:
            raise ValueError(f"Invalid object type for alias: {objtype}")

    # apply aliases
    for alias in aliases.values():
        alias.apply()


def setup(app: Sphinx) -> ExtensionMetadata:
    app.setup_extension("sphinx.ext.autodoc")
    app.add_directive_to_domain("py", "data", ExtendedPyVariable)
    app.add_directive_to_domain("py", "class", ExtendedPyClassLike)
    app.add_autodocumenter(AliasDocumenter, override=True)
    app.connect("doctree-read", merge_aliases)

    return {"parallel_read_safe": True}

[AA-Turner]

hopefully this gets fixed soon

Unless I’ve missed it there’s no outstanding PR for this issue. It’s likely to be resolved faster if there is one.

A

[tanrbobanr]

hopefully this gets fixed soon

Unless I’ve missed it there’s no outstanding PR for this issue. It’s likely to be resolved faster if there is one.

A

Yeah. If I had the time (and the knowledge of Sphinx's internal systems), I might attempt to make a PR, but right now I have neither :(