11_documentation_building.rst

Building documentation

Airflow uses Sphinx to build the documentation. As of Airflow 3 when we split the distributions into multiple packages, and moved the documentation to inside those distribution building the documentation is way easier than it was on Airflow 2 and you can usually iterate on documentation way faster, including automatically refreshing the documentation in your browser when using sphinx-autobuild.

The current distributions we have that have sphinx-buildable documentation are:

Documentation in separate distributions:

• airflow-core/docs - documentation for Airflow Core
• providers/**/docs - documentation for Providers
• chart/docs - documentation for Helm Chart
• task-sdk/docs - documentation for Task SDK (new format not yet published)
• airflow-ctl/docs - documentation for Airflow CLI (future)

Documentation for general overview and summaries not connected with any specific distribution:

• docker-stack-docs - documentation for Docker Stack'
• providers-summary-docs - documentation for provider summary page

Each of the distributions have a conf.py file in the root of the documentation and there are various configuration parameters for sphinx configured in the conf.py files. A number of common functions in those conf.py files are imported from deve-common distribution, from doc package, you can also find sphinx_ext folder there that keeps extensions used during our documentation build.

Building documentation with uv in local venv

Prerequisites

• The uv as mandatory tour

First of all. you need to have uv installed. You can very easily build the documentation in your local virtualenv using uv command. Because some dependencies are not easy to be installed on all OS-es, the docs group of dependencies is not installed by default when you run uv sync or uv run - so you need to pass --group docs when you want to run command locally that installs the documentation dependencies.

• Python 3.11

Warning

At least until we manage to workaround it you need to uese at most Python 3.11 to build documentation for some packages (samba and google).

When you try to build docs locally (at least on MacOS) with Python 3.12, you get TypeError: __type_params__ must be set to a tuple error related to This CPython issue

You should run uv python pin 3.11 to pin to Python 3.11 in order to build documentation.

uv python pin 3.11

• Enchant system dependency

Warning

There is a prerequisite needed to build the documentation locally. Airflow's Sphinx configuration depends on enchant C-library that needs to be installed and configured to be used by pyenchant Python distribution package. You can install it as explained in the pyenchant documentation. Usually you use your package manager on Linux or brew on MacOS. Sometimes you might also need to configure it so that pyenchant can find the dictionary files. You will get error like:

Could not import extension sphinxcontrib.spelling (exception: The 'enchant' C library was not
found and maybe needs to be installed

The easiest way is to find where your enchant library is (usually libenchant-2.dylib) and set the PYENCHANT_LIBRARY_PATH environment variable to that path in your .zshrc or .bashrc - depending which terminal you use and restart your terminal. For example on MacOS, brew install enchant or brew reinstall enchant will print the path where enchant is installed and usually the library is in the lib subfolder of it. An example of line added to one of the .rc files:

export PYENCHANT_LIBRARY_PATH=/opt/homebrew/Cellar/enchant/2.8.2/lib/libenchant-2.dylib

Also for some of the providers you might have trouble installing dependencies. The uv run command will automatically install the needed dependencies for you but some of them might need extra system dependencies. You can always revert to breeze way of building the documentation in this case (see below) in case you have problems with uv run command not being able to install the dependencies.

Once you have enchant installed and configured, you can build the documentation very easily.

Building the documentation

In Airflow 3 the documentation is placed closely to where source code is placed (in most distribution packages it is in the doc subfolder of a distribution you want to build the documentation for. Some distributions do not have docs``subfolder - when they are ``doc-only. The build-doc script is installed automatically from devel-common` distribution and available to run using ``uv run command (or directly build-docs inside the breeze container). This script will automatically detect which distribution you are in and build the documentation for it.

Note

You can also run all those commands directly (without uv run --group-doc) inside breeze container, because there all dependencies are already installed and there is no need to sync dependencies.

cd YOUR_DISTRIBUTION_FOLDER
uv run --group docs build-docs

Example:

cd providers/fab
uv run --group docs build-docs

or

cd airflow-core
uv run --group docs build-docs

When you iterate on the documentation rather than wait for a complete build to pass you might want to run a documentation server that will expose the built documentation and enable auto-refreshing the documentation in your browser (including refreshing the browser when you change the documentation files):

cd YOUR_DISTRIBUTION_FOLDER
uv run --group docs build-docs --autobuild

Example:

cd providers/fab
uv run --group docs build-docs --autobuild

Building docs for multiple distributions at the same time

Sometimes - when you have references between distributions and you make changes in both distributions. The common problem when you build such documentation might be when you want to - at the same time - modify documentation for several distributions and refer to the new documentation from the other distribution and that might not work because the documentation for the other distribution is not built yet.

If you just build one distribution, the links to the other distribution will not work. And the bad thing might be that you have circular references and the links might be needed in both directions.

In order to fix that problem you should run the same build-docs command and specify the packages you want to build together. The build-docs command will attempt to build the docs up to three times in order to make sure that references between the packages are resolved.

Note that this will not work with --autobuild option because the --autobuild option will only build the documentation for single distribution at a time. You will see an error if you try to use --autobuild with multiple distributions. But once the "other" package is build, you will be able ot use --autobuild for the other package (until you use --clean-build option).

You can run this command in the folder of the package you want to build - if the package needs another package, usually local virtualenv created by uv will include sources for the other package. But in case you refer to a package that has no direct dependency, some dependencies might not be installed, so you might need to run it from the root of Airflow repository - when you run uv run there - all dependencies for all packages are installed and available.

Example of building both amazon and google provider docs from amazon provider dir:

cd providers/amazon
uv run --group docs build-docs amazon google

Building documentation for all distributions

If you want to build all documentation at once, you can run the command from the root of the Airflow repository and by default it will build all documentation in parallel.

cd YOUR_ROOT_OF_AIRFLOW_REPO
uv run --group docs build-docs

Complete command has the following parameters:

Building documentation in Breeze

This used to be the default way of building documentation - but this method requires things to start and run in Docker containers and is a bit slower than using uv - especially when iterating with changes. and especially comparing it to sphinx-autobuild. However, the breeze CI container is guaranteed to have all the dependencies installed that should allow to build all documentation for all distributions we have and it is independent on the host / local environment including the OS you are using. You can always fall-back to this method if - for some reason - your local documentation building is failing.

Basic command is breeze build-docs which except --auto-build has very similar options as the uv run build-docs command.. You can also specify a number of options like selecting which distribution packages you want to build and which kind of build to run (--doc-only od --spelling-only) or request to --clean the build directory before building the documentation.

For example:

breeze build-docs --doc-only --clean fab

Will build fab provider documentation and clean inventories and other build artifacts before.

You can also use breeze build-docs --help to see available options and head to breeze documentation to learn more about the breeze command and it's options.

Hints and issues when building documentation

We are using Sphinx to build our documentation, and while powerful, Sphinx is known from speaking riddles. Sometimes the errors printed by Sphinx build commands are a bit cryptic and do not tell you exactly how to solve them.

Here are typical issues and guidelines on how to solve them:

WARNING: document isn't included in any toctree

This is by far most common and most cryptic error printed by Sphinx. What it really means is that the document that this error refers to is not linked from anywhere else in any of the other documents. By default all the documents that we generate, should be somehow reachable by the user from the main index, but apparently there is some document that is not reachable, so no other document refer to it either via index and Sphinx complains about this.

There might be several reasons for this:

• the document is referred to in some other documents with :ref: or :doc: but there is a typo and the name is wrong. Or the reference does not use properly formatted syntax for those directives.
• the document should be referred by some :toctree: directive that walks through some of the sub-folders of regular folders or the _api folders automatically generated by the autoapi extension. For example the below toctree will build index with references to all the files that are direct sub-directories of operators/airbyte and connections folders.

.. toctree::
 :hidden:
 :maxdepth: 1
 :caption: Guides

 Operators <operators/airbyte>
 Connection types <connections>

Including examples and other files

When you include examples using .. exampleinclude:: or .. include:: directive, you should use relative reference only to the files that are in the same distribution in one of the subfolders - when you know that this file is relative to your documentation file. For example:

``.. include:: ../_partials/prerequisite_tasks.rst``

This one will include prerequisite_tasks.rst file from _partials subdirectory of the parent folder of the folder where the file you include it from is.

However, when you need to refer to files outside of your "docs" tree, it is better to start your reference with /. Such reference start from the "top" of your documentation tree in your distribution (usually docs folder in your distribution. This way, when we move the distributions around, we should be able to easily move the whole distribution around by knowing how the distribution moved relatively to the included file. For example:

``.. include:: /../../../devel-common/src/sphinx_exts/includes/sections-and-options.rst``