init

docs/drivers/freedreno.rst

@@ -0,0 +1,746 @@
+Freedreno
+=========
+
+Freedreno GLES and GL driver for Adreno 2xx-6xx GPUs.  It implements up to
+OpenGL ES 3.2 and desktop OpenGL 4.5.
+
+See the `Freedreno Wiki
+<https://gitlab.freedesktop.org/freedreno/freedreno/-/wikis/home>`__ for more
+details.
+
+Turnip
+------
+
+Turnip is a Vulkan 1.3 driver for Adreno 6xx GPUs.
+
+The current set of specific chip versions supported can be found in
+:file:`src/freedreno/common/freedreno_devices.py`.  The current set of features
+supported can be found rendered at `Mesa Matrix <https://mesamatrix.net/>`__.
+There are no plans to port to a5xx or earlier GPUs.
+
+Hardware architecture
+---------------------
+
+Adreno is a mostly tile-mode renderer, but with the option to bypass tiling
+("gmem") and render directly to system memory ("sysmem").  It is UMA, using
+mostly write combined memory but with the ability to map some buffers as cache
+coherent with the CPU.
+
+.. toctree::
+   :glob:
+
+   freedreno/hw/*
+
+Hardware acronyms
+^^^^^^^^^^^^^^^^^
+
+.. glossary::
+
+  Cluster
+    A group of hardware registers, often with multiple copies to allow
+    pipelining.  There is an M:N relationship between hardware blocks that do
+    work and the clusters of registers for the state that hardware blocks use.
+
+  CP
+    Command Processor.  Reads the stream of state changes and draw commands
+    generated by the driver.
+
+  PFP
+    Prefetch Parser.  Adreno 2xx-4xx CP component.
+
+  ME
+    Micro Engine. Adreno 2xx-4xx CP component after PFP, handles most PM4 commands.
+
+  SQE
+    a6xx+ replacement for PFP/ME.  This is the microcontroller that runs the
+    microcode (loaded from Linux) which actually processes the command stream
+    and writes to the hardware registers.  See `qrisc
+    <https://gitlab.freedesktop.org/mesa/mesa/-/blob/main/src/freedreno/qrisc/README.rst>`__.
+
+  ROQ
+    DMA engine used by the SQE for reading memory, with some prefetch buffering.
+    Mostly reads in the command stream, but also serves for
+    ``CP_MEMCPY``/``CP_MEM_TO_REG`` and visibility stream reads.
+
+  SP
+    Shader Processor.  Unified, scalar shader engine.  One or more, depending on
+    GPU and tier.
+
+  TP
+    Texture Processor.
+
+  UCHE
+    Unified L2 Cache.  32KB on A330, unclear how big now.
+
+  CCU
+    Color Cache Unit.
+
+  VSC
+    Visibility Stream Compressor
+
+  PVS
+    Primitive Visibility Stream
+
+  FE
+    Front End?  Index buffer and vertex attribute fetch cluster.  Includes PC,
+    VFD, VPC.
+
+  VFD
+    Vertex Fetch and Decode
+
+  VPC
+    Varying/Position Cache?  Hardware block that stores shaded vertex data for
+    primitive assembly.
+
+  HLSQ
+    High Level Sequencer.  Manages state for the SPs, batches up PS invocations
+    between primitives, is involved in preemption.
+
+  PC_VS
+    Cluster where varyings are read from VPC and assembled into primitives to
+    feed GRAS.
+
+  VS
+    Vertex Shader. Responsible for generating VS/GS/tess invocations
+
+  GRAS
+    Rasterizer. Responsible for generating PS invocations from primitives, also
+    does LRZ
+
+  PS
+    Pixel Shader.
+
+  RB
+    Render Backend.  Performs both early and late Z testing, blending, and
+    attachment stores of output of the PS.
+
+  GMEM
+    Roughly 128KB-1MB of memory on the GPU (SKU-dependent), used to store
+    attachments during tiled rendering
+
+  LRZ
+    Low Resolution Z.  A low resolution area of the depth buffer that can be
+    initialized during the binning pass to contain the worst-case (farthest) Z
+    values in a block, and then used to early reject fragments during
+    rasterization.
+
+Cache hierarchy
+^^^^^^^^^^^^^^^
+
+The a6xx GPUs have two main caches: CCU and UCHE.
+
+UCHE (Unified L2 Cache) is the cache behind the vertex fetch, VSC writes,
+texture L1, LRZ, and storage image accesses (``ldib``/``stib``).  Misses and
+flushes access system memory.
+
+The CCU is the separate cache used by 2D blits and sysmem render target access
+(and also for resolves to system memory when in GMEM mode).  Its memory comes
+from a carveout of GMEM controlled by ``RB_CCU_CNTL``, with a varying amount
+reserved based on whether we're in a render pass using GMEM for attachment
+storage, or we're doing sysmem rendering.  Cache entries have the attachment
+number and layer mixed into the cache tag in some way, likely so that a
+fragment's access is spread through the cache even if the attachments are the
+same size and alignments in address space.  This means that the cache must be
+flushed and invalidated between memory being used for one attachment and another
+(notably depth vs color, but also MRT color).
+
+The Texture Processors (TP) additionally have a small L1 cache (1KB on A330,
+unclear how big now) before accessing UCHE. This cache is used for normal
+sampling like ``sam``` and ``isam`` (and the compiler will make read-only
+storage image access through it as well).  It is not coherent with UCHE (may get
+stale results when you ``sam`` after ``stib``), but must get flushed per draw or
+something because you don't need a manual invalidate between draws storing to an
+image and draws sampling from a texture.
+
+The command processor (CP) does not read from either of these caches, and
+instead uses FIFOs in the ROQ to avoid stalls reading from system memory.
+
+Draw states
+^^^^^^^^^^^
+
+Since the SQE is not a fast processor, and tiled rendering means that many draws
+won't even be used in many bins, since a5xx state updates can be batched up into
+"draw states" that point to a fragment of CP packets.  At draw time, if the draw
+call is going to actually execute (some primitive is visible in the current
+tile), the SQE goes through the ``GROUP_ID``\s and for any with an update since
+the last time they were executed, it executes the corresponding fragment.
+
+Starting with a6xx, states can be tagged with whether they should be executed
+at draw time for any of sysmem, binning, or tile rendering.  This allows a
+single command stream to be generated which can be executed in any of the modes,
+unlike pre-a6xx where we had to generate separate command lists for the binning
+and rendering phases.
+
+Note that this means that the generated draw state has to always update all of
+the state you have chosen to pack into that ``GROUP_ID``, since any of your
+previous state changes in a previous draw state command may have been skipped.
+
+Pipelining (a6xx+)
+^^^^^^^^^^^^^^^^^^
+
+Most CP commands write to registers.  In a6xx+, the registers are located in
+clusters corresponding to the stage of the pipeline they are used from (see
+``enum tu_stage`` for a list). To pipeline state updates and drawing, registers
+generally have two copies ("contexts") in their cluster, so previous draws can
+be working on the previous set of register state while the next draw's state is
+being set up. You can find what registers go into which clusters by looking at
+:command:`crashdec` output in the ``regs-name: CP_MEMPOOL`` section.
+
+As SQE processes register writes in the command stream, it sends them into a
+per-cluster queue stored in ``CP_MEMPOOL``.  This allows the pipeline stages to
+process their stream of register updates and events independent of each other
+(so even with just 2 contexts in a stage, earlier stages can proceed on to later
+draws before later stages have caught up).
+
+Each cluster has a per-context bit indicating that the context is done/free.
+Register writes will stall on the context being done.
+
+During a 3D draw command, SQE generates several internal events flow through the
+pipeline:
+
+- ``CP_EVENT_START`` clears the done bit for the context when written to the
+  cluster
+- ``PC_EVENT_CMD``/``PC_DRAW_CMD``/``HLSQ_EVENT_CMD``/``HLSQ_DRAW_CMD`` kick off
+  the actual event/drawing.
+- ``CONTEXT_DONE`` event completes after the event/draw is complete and sets the
+  done flag.
+- ``CP_EVENT_END`` waits for the done flag on the next context, then copies all
+  the registers that were dirtied in this context to that one.
+
+The 2D blit engine has its own ``CP_2D_EVENT_START``, ``CP_2D_EVENT_END``,
+``CONTEXT_DONE_2D``, so 2D and 3D register contexts can do separate context
+rollover.
+
+Because the clusters proceed independently of each other even across draws, if
+you need to synchronize an earlier cluster to the output of a later one, then
+you will need to ``CP_WAIT_FOR_IDLE`` after flushing and invalidating any
+necessary caches.
+
+Also, note that some registers are not banked at all, and will require a
+``CP_WAIT_FOR_IDLE`` for any previous usage of the register to complete.
+
+In a2xx-a4xx, there weren't per-stage clusters, and instead there were two
+register banks that were flipped between per draw.
+
+Bindless/Bindful Descriptors (a6xx+)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Starting with a6xx++, cat5 (texture) and cat6 (image/SSBO/UBO) instructions are
+extended to support bindless descriptors.
+
+In the old bindful model, descriptors are separate for textures, samplers,
+UBOs, and IBOs (combined descriptor for images and SSBOs), with separate
+registers for the memory containing the array of descriptors, and/or different
+``STATE_TYPE`` and ``STATE_BLOCK`` for ``CP_LOAD_STATE``/``_FRAG``/``_GEOM``
+to pre-load the descriptors into cache.
+
+- textures - per-shader-stage
+   - registers: ``SP_xS_TEX_CONST``/``SP_xS_TEX_COUNT``
+   - state-type: ``ST6_CONSTANTS``
+   - state-block: ``SB6_xS_TEX``
+- samplers - per-shader-stage
+   - registers: ``SP_xS_TEX_SAMP``
+   - state-type: ``ST6_SHADER``
+   - state-block: ``SB6_xS_TEX``
+- UBOs - per-shader-stage
+   - registers: none
+   - state-type: ``ST6_UBO``
+   - state-block: ``SB6_xS_SHADER``
+- IBOs - global across shader 3d stages, separate for compute shader
+   - registers: ``SP_IBO``/``SP_IBO_COUNT`` or ``SP_CS_IBO``/``SP_CS_IBO_COUNT``
+   - state-type: ``ST6_SHADER``
+   - state-block: ``ST6_IBO`` or ``ST6_CS_IBO`` for compute shaders
+   - Note, unlike per-shader-stage descriptors, ``CP_LOAD_STATE6`` is used,
+     as opposed to ``CP_LOAD_STATE6_GEOM`` or ``CP_LOAD_STATE6_FRAG``
+     depending on shader stage.
+
+.. note::
+   For the per-shader-stage registers and state-blocks the ``xS`` notation
+   refers to per-shader-stage names, ex. ``SP_FS_TEX_CONST`` or ``SB6_DS_TEX``
+
+Textures and IBOs (images) use *basically* the same 64byte descriptor format
+with some exceptions (for ex, for IBOs cubemaps are handles as 2d array).
+SSBOs are just untyped buffers, but otherwise use the same descriptors and
+instructions as images.  Samplers use a 16byte descriptor, and UBOs use an
+8byte descriptor which packs the size in the upper 15 bits of the UBO address.
+
+In the bindless model, descriptors are split into 5 descriptor sets, which are
+global across shader stages (but as with bindful IBO descriptors, separate for
+3d stages vs compute stage).  Each HW descriptor is an array of descriptors
+of configurable size (each descriptor set can be configured for a descriptor
+pitch of 8bytes or 64bytes).  Each descriptor can be of arbitrary format (ie.
+UBOs/IBOs/textures/samplers interleaved), it's interpretation by the HW is
+determined by the instruction that references the descriptor.  Each descriptor
+set can contain at least 2^^16 descriptors.
+
+The HW is configured with the base address of the descriptor set via an array
+of "BINDLESS_BASE" registers, ie ``SP_BINDLESS_BASE[n]``/``HLSQ_BINDLESS_BASE[n]``
+for 3d shader stages, or ``SP_CS_BINDLESS_BASE[n]``/``HLSQ_CS_BINDLESS_BASE[n]``
+for compute shaders, with the descriptor pitch encoded in the low bits.
+Which of the descriptor sets is referenced is encoded via three bits in the
+instruction.  The address of the descriptor is calculated as::
+
+   descriptor_addr = (BINDLESS_BASE[n] & ~0x3) +
+                     (idx * 4 * (2 << BINDLESS_BASE[n] & 0x3))
+
+
+.. note::
+   Turnip reserves one descriptor set for internal use and exposes the other
+   four for the application via the Vulkan API.
+
+Software Architecture
+---------------------
+
+Freedreno and Turnip use a shared core for shader compiler, image layout, and
+register and command stream definitions.  They implement separate state
+management and command stream generation.
+
+.. toctree::
+   :glob:
+
+   freedreno/*
+
+.. _freedreno-gpu-devcoredump:
+
+GPU devcoredump
+^^^^^^^^^^^^^^^^^^
+
+A kernel message from DRM of "gpu fault" can mean any sort of error reported by
+the GPU (including its internal hang detection).  If a fault in GPU address
+space happened, you should expect to find a message from the iommu, with the
+faulting address and a hardware unit involved:
+
+.. code-block:: text
+
+  *** gpu fault: ttbr0=000000001c941000 iova=000000010066a000 dir=READ type=TRANSLATION source=TP|VFD (0,0,0,1)
+
+On a GPU fault or hang, a GPU core dump is taken by the DRM driver and saved to
+``/sys/devices/virtual/devcoredump/**/data``.  You can cp that file to a
+:file:`crash.devcore` to save it, otherwise the kernel will expire it
+eventually. Echo 1 to the file to free the core early, as another core won't be
+taken until then.
+
+Once you have your core file, you can use :command:`crashdec -f crash.devcore`
+to decode it.  The output will have ``ESTIMATED CRASH LOCATION`` where we
+estimate the CP to have stopped.  Note that it is expected that this will be
+some distance past whatever state triggered the fault, given GPU pipelining, and
+will often be at some ``CP_REG_TO_MEM`` (which waits on previous WFIs) or
+``CP_WAIT_FOR_ME`` (which waits for all register writes to land) or similar
+event. You can try running the workload with ``TU_DEBUG=flushall`` or
+``FD_MESA_DEBUG=flush`` to try to close in on the failing commands.
+
+You can also find what commands were queued up to each cluster in the
+``regs-name: CP_MEMPOOL`` section.
+
+If ``ESTIMATED CRASH LOCATION`` doesn't exist you could find ``CP_SQE_STAT``,
+though going here is the last resort and likely won't be helpful.
+
+.. code-block::
+
+  indexed-registers:
+    - regs-name: CP_SQE_STAT
+      dwords: 51
+  	 PC: 00d7                                <-------------
+  	PKT: CP_LOAD_STATE6_FRAG
+  	$01: 70348003		$11: 00000000
+  	$02: 20000000		$12: 00000022
+
+The ``PC`` value is an instruction address in the current firmware.
+You would need to disassemble the firmware (/lib/firmware/qcom/aXXX_sqe.fw) via:
+
+.. code-block:: sh
+
+  qrisc-disasm -v a650_sqe.fw > a650_sqe.fw.disasm
+
+Now you should search for PC value in the disassembly, e.g.:
+
+.. code-block::
+
+  l018:	00d1: 08dd0001  add $addr, $06, 0x0001
+       	00d2: 981ff806  mov $data, $data
+       	00d3: 8a080001  mov $08, 0x0001 << 16
+       	00d4: 3108ffff  or $08, $08, 0xffff
+       	00d5: 9be8f805  and $data, $data, $08
+       	00d6: 9806e806  mov $addr, $06
+       	00d7: 9803f806  mov $data, $03           <------------- HERE
+       	00d8: d8000000  waitin
+       	00d9: 981f0806  mov $01, $data
+
+
+Command Stream Capture
+^^^^^^^^^^^^^^^^^^^^^^
+
+During Mesa development, it's often useful to look at the command streams we
+send to the kernel.  We have an interface for the kernel to capture all
+submitted command streams:
+
+.. code-block:: sh
+
+  cat /sys/kernel/debug/dri/0/rd > cmdstream &
+
+By default, command stream capture does not capture texture/vertex/etc. data.
+You can enable capturing all the BOs with:
+
+.. code-block:: sh
+
+  echo Y > /sys/module/msm/parameters/rd_full
+
+Note that, since all command streams get captured, it is easy to run the system
+out of memory doing this, so you probably don't want to enable it during play of
+a heavyweight game.  Instead, to capture a command stream within a game, you
+probably want to cause a crash in the GPU during a frame of interest so that a
+single GPU core dump is generated.  Emitting ``0xdeadbeef`` in the CS should be
+enough to cause a fault.
+
+``fd_rd_output`` facilities provide support for generating the command stream
+capture from inside Mesa. Different ``FD_RD_DUMP`` options are available:
+
+- ``enable`` simply enables dumping the command stream on each submit for a
+  given logical device. When a more advanced option is specified, ``enable`` is
+  implied as specified.
+- ``combine`` will combine all dumps into a single file instead of writing the
+  dump for each submit into a standalone file.
+- ``full`` will dump every buffer object, which is necessary for replays of
+  command streams (see below).
+- ``trigger`` will establish a trigger file through which dumps can be better
+  controlled. Writing a positive integer value into the file will enable dumping
+  of that many subsequent submits. Writing -1 will enable dumping of submits
+  until disabled. Writing 0 (or any other value) will disable dumps.
+
+Output dump files and trigger file (when enabled) are placed under ``/tmp`` or
+``/data/local/tmp`` under Android by default, but that location can be adjusted
+through ``FD_RD_DUMP_PATH``. ``FD_RD_DUMP_TESTNAME`` can be used to specify a
+more descriptive prefix for the output or trigger files.
+
+Dumping can be limited to specific ranges of frames or submits. For example,
+``FD_RD_DUMP_SUBMITS=120-140,160,165`` will dump command streams only for the
+specified submits. Similarly, ``FD_RD_DUMP_FRAMES`` can be set to specify for
+which frames any submitted command stream should be dumped.
+
+Functionality is generic to any Freedreno-based backend, but is currently only
+integrated in the MSM backend of Turnip. Using the existing ``TU_DEBUG=rd``
+option will translate to ``FD_RD_DUMP=enable``.
+
+Capturing Hang RD
++++++++++++++++++
+
+Devcore file doesn't contain all submitted command streams, only the hanging one.
+Additionally it is geared towards analyzing the GPU state at the moment of the crash.
+
+Alternatively, it's possible to obtain the whole submission with all command
+streams via ``/sys/kernel/debug/dri/0/hangrd``:
+
+.. code-block:: sh
+
+  sudo cat /sys/kernel/debug/dri/0/hangrd > logfile.rd // Do the cat _before_ the expected hang
+
+The format of hangrd is the same as in ordinary command stream capture.
+``rd_full`` also has the same effect on it.
+
+Replaying Command Stream
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+``replay`` tool allows capturing and replaying ``rd`` to reproduce GPU faults.
+Especially useful for transient GPU issues since it has much higher chances to
+reproduce them.
+
+Dumping rendering results or even just memory is currently unsupported.
+
+- Replaying command streams requires kernel with ``MSM_INFO_SET_IOVA`` support.
+- Requires ``rd`` capture to have full snapshots of the memory (``rd_full`` is enabled).
+
+Replaying is done via ``replay`` tool:
+
+.. code-block:: sh
+
+  ./replay test_replay.rd
+
+More examples:
+
+.. code-block:: sh
+
+  ./replay --first=start_submit_n --last=last_submit_n test_replay.rd
+
+.. code-block:: sh
+
+  ./replay --override=0 test_replay.rd
+
+.. _freedreno-editing-command-stream:
+
+Editing Command Stream (a6xx+)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+While replaying a fault is useful in itself, modifying the capture to
+understand what causes the fault could be even more useful.
+
+``rddecompiler`` decompiles a single cmdstream from ``rd`` into compilable C source.
+Given the address space bounds the generated program creates a new ``rd`` which
+could be used to override cmdstream with 'replay'. Generated ``rd`` is not replayable
+on its own and depends on buffers provided by the source ``rd``.
+
+C source could be compiled by putting it into src/freedreno/decode/generate-rd.cc.
+
+The workflow would look like this:
+
+1. Find the cmdstream № you want to edit;
+2. Decompile it:
+
+.. code-block:: sh
+
+  ./rddecompiler -s %cmd_stream_n% example.rd > src/freedreno/decode/generate-rd.cc
+
+3. Edit the command stream;;
+4. Compile and deploy freedreno tools;
+5. Plug the generator into cmdstream replay:
+
+.. code-block:: sh
+
+  ./replay --override=%cmd_stream_№%
+
+6. Repeat 3-5.
+
+GPU Hang Debugging
+^^^^^^^^^^^^^^^^^^
+
+Not a guide for how to do it but mostly an enumeration of methods.
+
+Useful ``TU_DEBUG`` (for Turnip) options to narrow down the hang cause:
+
+``sysmem``, ``gmem``, ``nobin``, ``forcebin``, ``noubwc``, ``nolrz``, ``flushall``, ``syncdraw``, ``rast_order``
+
+Useful ``FD_MESA_DEBUG`` (for Freedreno) options:
+
+``sysmem``, ``gmem``, ``nobin``, ``noubwc``, ``nolrz``, ``notile``, ``dclear``, ``ddraw``, ``flush``, ``inorder``, ``noblit``
+
+Useful ``IR3_SHADER_DEBUG`` options:
+
+``nouboopt``, ``spillall``, ``nopreamble``, ``nofp16``
+
+Use Graphics Flight Recorder to narrow down the place which hangs,
+use our own breadcrumbs implementation in case of unrecoverable hangs.
+
+In case of faults use RenderDoc to find the problematic command. If it's
+a draw call, edit shader in RenderDoc to find whether it culprit is a shader.
+If yes, bisect it.
+
+If editing the shader messes the assembly too much and the issue becomes unreproducible
+try editing the assembly itself via ``IR3_SHADER_OVERRIDE_PATH``.
+
+If fault or hang is transient try capturing an ``rd`` and replay it. If issue
+is reproduced - bisect the GPU packets until the culprit is found.
+
+Do the above if culprit is not a shader.
+
+The hang recovery mechanism in Kernel is not perfect, in case of unrecoverable
+hangs check whether the kernel is up to date and look for unmerged patches
+which could improve the recovery.
+
+GPU Breadcrumbs
++++++++++++++++
+
+Breadcrumbs described below are available only in Turnip.
+
+Freedreno has simpler breadcrumbs, in debug build writes breadcrumbs
+into ``CP_SCRATCH_REG[6]`` and per-tile breadcrumbs into ``CP_SCRATCH_REG[7]``,
+in this way they are available in the devcoredump. TODO: generalize Tunip's
+breadcrumbs implementation.
+
+This is a simple implementations of breadcrumbs tracking of GPU progress
+intended to be a last resort when debugging unrecoverable hangs.
+For best results use Vulkan traces to have a predictable place of hang.
+
+For ordinary hangs as a more user-friendly solution use GFR
+"Graphics Flight Recorder".
+
+Or breadcrumbs implementation aims to handle cases where nothing can be done
+after the hang. In-driver breadcrumbs also allow more precise tracking since
+we could target a single GPU packet.
+
+While breadcrumbs support gmem, try to reproduce the hang in a sysmem mode
+because it would require much less breadcrumb writes and syncs.
+
+Breadcrumbs settings:
+
+.. code-block:: sh
+
+  TU_BREADCRUMBS=%IP%:%PORT%,break=%BREAKPOINT%:%BREAKPOINT_HITS%
+
+``BREAKPOINT``
+  The breadcrumb starting from which we require explicit ack.
+``BREAKPOINT_HITS``
+  How many times breakpoint should be reached for break to occur.
+  Necessary for a gmem mode and re-usable cmdbuffers in both of which
+  the same cmdstream could be executed several times.
+
+A typical work flow would be:
+
+- Start listening for breadcrumbs on a remote host:
+
+.. code-block:: sh
+
+   nc -lkvup $PORT | stdbuf -o0 xxd -pc -c 4 | awk -Wposix '{printf("%u:%u\n", "0x" $0, a[$0]++)}'
+
+- Start capturing command stream;
+- Replay the hanging trace with:
+
+.. code-block:: sh
+
+   TU_BREADCRUMBS=$IP:$PORT,break=-1:0
+
+- Increase hangcheck period:
+
+.. code-block:: sh
+
+   echo -n 60000 > /sys/kernel/debug/dri/0/hangcheck_period_ms
+
+- After GPU hang note the last breadcrumb and relaunch trace with:
+
+.. code-block:: sh
+
+   TU_BREADCRUMBS=%IP%:%PORT%,break=%LAST_BREADCRUMB%:%HITS%
+
+- After the breakpoint is reached each breadcrumb would require
+  explicit ack from the user. This way it's possible to find
+  the last packet which didn't hang.
+
+- Find the packet in the decoded cmdstream.
+
+Debugging random failures
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In most cases random GPU faults and rendering artifacts are caused by some kind
+of undefined behavior that falls under the following categories:
+
+- Usage of a stale reg value;
+- Usage of stale memory (e.g. expecting it to be zeroed when it is not);
+- Lack of the proper synchronization.
+
+Finding instances of stale reg reads
+++++++++++++++++++++++++++++++++++++
+
+Turnip has a debug option to stomp the registers with invalid values to catch
+the cases where stale data is read.
+
+.. code-block:: sh
+
+  MESA_VK_ABORT_ON_DEVICE_LOSS=1 \
+  TU_DEBUG_STALE_REGS_RANGE=0x00000c00,0x0000be01 \
+  TU_DEBUG_STALE_REGS_FLAGS=cmdbuf,renderpass \
+  ./app
+
+.. envvar:: TU_DEBUG_STALE_REGS_RANGE
+
+  the reg range in which registers would be stomped. Add ``inverse`` to the
+  flags in order for this range to specify which registers NOT to stomp.
+
+.. envvar:: TU_DEBUG_STALE_REGS_FLAGS
+
+  ``cmdbuf``
+    stomp registers at the start of each command buffer.
+  ``renderpass``
+    stomp registers before each render pass.
+  ``inverse``
+    changes ``TU_DEBUG_STALE_REGS_RANGE`` meaning to
+    "regs that should NOT be stomped".
+
+The best way to pinpoint the reg which causes a failure is to bisect the regs
+range. In case when a fail is caused by combination of several registers
+the ``inverse`` flag may be set to find the reg which prevents the failure.
+
+Runtime toggling of ``TU_DEBUG`` options
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In some cases, it is useful to toggle ``TU_DEBUG`` options at runtime, such as
+when assessing the performance impact of a particular option. This can be done
+by setting the ``TU_DEBUG_FILE`` environment variable to a file path, and writing
+the desired ``TU_DEBUG`` options to that file. The driver will check the file at
+startup and apply all debug options (runtime and non-runtime) but then will
+continue to monitor the file for changes and only apply runtime options.
+
+The folder containing the file should exist prior to running the application, and
+deleting the folder during runtime will result in the driver no longer picking up
+any changes even if the folder is recreated.
+
+Additionally, not all ``TU_DEBUG`` options can be toggled at runtime, the following
+are supported at the moment: ``nir``, ``nobin``, ``sysmem``, ``gmem``, ``forcebin``,
+``layout``, ``nolrz``, ``nolrzfc``, ``perf``, ``flushall``, ``syncdraw``,
+``rast_order``, ``unaligned_store``, ``log_skip_gmem_ops``, ``3d_load``, ``fdm``,
+``noconcurrentresolves``, ``noconcurrentunresolves``, ``nobinmerging``.
+
+Some of these options will behave differently when toggled at runtime, for example:
+``nolrz`` will still result in LRZ allocation which would not happen if the option
+was set in the environment variable.
+
+Autotune
+^^^^^^^^
+
+Turnip supports dynamically selecting between SYSMEM and GMEM rendering with the
+autotune system, the behavior of which can be controlled with the following
+environment variables:
+
+.. envvar:: TU_AUTOTUNE_ALGO
+
+  Selects the algorithm used for autotuning. Supported values are:
+
+  ``bandwidth``
+    Estimates the bandwidth usage of rendering in SYSMEM and GMEM modes, and chooses
+    the one with lower estimated bandwidth. This is the default algorithm.
+
+  ``profiled``
+    Dynamically profiles the RP timings in SYSMEM and GMEM modes, and uses that to
+    move a probability distribution towards the optimal choice over time. This
+    algorithm tends to be far more accurate than the bandwidth algorithm at choosing
+    the optimal rendering mode but may result in larger FPS variance due to being
+    based on a probability distribution with random sampling.
+
+  ``profiled_imm``
+    Similar to ``profiled``, but only profiles the first few instances of a RP
+    and then sticks to the chosen mode for subsequent instances. This is meant
+    for single-frame traces run multiple times in a CI where this algorithm can
+    immediately chose the optimal rendering mode for each RP.
+
+  ``prefer_sysmem``/``prefer_gmem``
+    Always chooses SYSMEM/GMEM rendering. This is useful for games that work
+    better in one mode over the other due to their rendering patterns, setting
+    this is better than using ``TU_DEBUG=sysmem``/``TU_DEBUG=gmem`` when done
+    for performance reasons, since these still allow the other mode to be used
+    in some high-confidence cases as well as letting ``TU_AUTOTUNE_FLAGS`` still
+    be applied.
+
+  The algorithm can be set via the driconf option ``tu_autotune_algorithm`` as well.
+
+.. envvar:: TU_AUTOTUNE_FLAGS
+
+  Modifies the behavior of the selected algorithm. Supported flags are:
+
+  ``big_gmem``
+    Always chooses GMEM rendering if the amount of draw calls in the render pass
+    is greater than a certain threshold. Larger RPs generally benefit more from
+    GMEM rendering due to less overhead from tiling. This tends to lead to worse
+    performance in most cases, so it's only useful for testing.
+
+  ``tune_small``
+    Enables tuning for small render passes (those with a small number of draw
+    calls). By default, small RPs always use SYSMEM mode as they generally don't
+    benefit from GMEM rendering due to the overhead of tiling.
+
+  ``preempt_optimize``
+    Tries to keep non-preemptible time in the render pass is below a certain
+    threshold. This is useful for systems with GPU-based compositors where long
+    non-preemptible times can lead to missed frame deadlines, causing noticeable
+    stuttering. This flag will reduce the performance of the render pass in order
+    to improve overall system responsiveness, it should not be used unless the
+    rest of the system is affected by preemption delays.
+    
+    This is done by measuring the time between a preemption request and preemption
+    actually occurring, if it is above a threshold we force those RPs to use GMEM
+    rendering, where non-preemptible times are driven by tile size which we
+    progressively reduce until the non-preemptible time is below the threshold.
+
+    It should be noted that this will occupy the last two CP performance counters
+    which may interfere with other profiling tools (such as `fdperf`).
+
+  Multiple flags can be combined by separating them with commas, e.g.
+  ``TU_AUTOTUNE_FLAGS=big_gmem,tune_small``.
+
+  If no flags are specified, the default behavior is used.