Skip to content

Commit 8c36139

Browse files
github-actions[bot]kaxil
authored andcommitted
[v3-2-test] Add FAQ entry for API server memory growth with gunicorn worker recycling (#65036) (#65037)
* Add FAQ entry for API server memory growth with gunicorn fix Users experiencing API server memory growth from accumulated serialized DAG cache entries can use gunicorn with rolling worker restarts as the recommended solution. This FAQ entry documents the configuration and cross-references the dag version inflation FAQ. * Add version availability note for gunicorn support (cherry picked from commit 74283fb) Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
1 parent 83f2ba1 commit 8c36139

1 file changed

Lines changed: 37 additions & 0 deletions

File tree

airflow-core/docs/faq.rst

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -666,6 +666,43 @@ try pausing the Dag again, or check the console or server logs if the
666666
issue recurs.
667667

668668

669+
API Server
670+
^^^^^^^^^^
671+
672+
.. _faq:api-server-memory-growth:
673+
674+
How to prevent API server memory growth?
675+
-----------------------------------------
676+
677+
The API server caches serialized Dag objects in memory. Over time, as Dag versions accumulate
678+
(see :ref:`faq:dag-version-inflation`), this cache grows and can consume several gigabytes of memory.
679+
680+
The recommended solution (available since Airflow 3.2.0) is to use **gunicorn** with **rolling worker
681+
restarts**. Gunicorn periodically recycles worker processes, releasing all accumulated memory. It also
682+
uses ``preload`` + ``fork``, so workers share read-only memory pages via copy-on-write, reducing overall
683+
memory usage by 40-50% compared to uvicorn's multiprocess mode.
684+
685+
To enable gunicorn with worker recycling:
686+
687+
.. code-block:: ini
688+
689+
[api]
690+
server_type = gunicorn
691+
# Restart each worker every 12 hours (43200 seconds)
692+
worker_refresh_interval = 43200
693+
worker_refresh_batch_size = 1
694+
695+
This requires the ``apache-airflow-core[gunicorn]`` extra to be installed.
696+
697+
See :ref:`config:api__server_type`, :ref:`config:api__worker_refresh_interval`, and
698+
:ref:`config:api__worker_refresh_batch_size` for the full configuration reference.
699+
700+
.. note::
701+
702+
Worker recycling handles memory growth from *any* source, not just the Dag cache. It is the
703+
recommended approach for production API server deployments.
704+
705+
669706
MySQL and MySQL variant Databases
670707
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
671708

0 commit comments

Comments
 (0)