init

docs/drivers/amd/function-calls.rst

@@ -0,0 +1,173 @@
+:orphan:
+
+.. _aco-fn-calls:
+
+Function call support in RADV/ACO
+=================================
+
+ACO supports function calls inside shaders - given a function signature and ABI, shaders can call
+an arbitrary function, even via only a function pointer (i.e. with an unknown function definition).
+
+This function call support is useful for implementing ray tracing pipelines (by representing individual RT shaders
+as callable functions), but it also has potential use cases in GPGPU/Compute workloads.
+
+This page serves to document the concepts involved in implementing function calls as well as an overview of the
+implementation components.
+
+Function call representation
+----------------------------
+
+In NIR, function calls are represented by a `nir_call_instr`. The instruction takes a `nir_function` representing
+the function being called, as well as SSA defs for each call parameter.
+NIR can also represent "indirect calls", i.e. calls where the function being called is
+unknown - instead, the instruction takes an SSA def containing a function pointer to the callee. In this case, the
+`nir_function` only serves to provide information about the function signature, i.e. how many and which parameters
+the function takes.
+
+Call instructions do not have return values - instead, return values are represented by so-called "return parameters".
+Instead of an SSA value, these parameters are derefs, and the return value is written into the deref when the callee
+returns. Return parameters can double as input parameters, too - the callee can read the previous value of the deref
+before (potentially) overwriting it with a new value.
+
+ACO's representation of function calls follows this very closely. Calls are described by the `p_call` pseudo-instruction.
+The operands to this instruction are a function pointer (i.e. the address of the callee), followed by the call
+parameters. Return parameters are handled differently, though: While the initial value of the return parameter is passed
+as an operand, the call instruction produces new definitions that refer to the SSA values of the return parameters after
+the function call returns. There is a special NIR intrinsic ``load_return_param_amd`` that can be used to access these
+new definitions when lowering return parameter derefs to SSA form.
+
+.. _div-calls:
+
+Divergent calls
+---------------
+
+On CPUs, a call instruction will only ever jump to a single address. However, GPUs are SIMT, and the value of a function
+pointer may be divergent, i.e. different threads try calling different functions within the same call instruction. AMD
+hardware executes one instruction for all threads in lockstep, so the multiple callees have to be executed one after
+the other.
+
+This is handled by RADV in ``radv_nir_lower_call_abi``. In addition to the (non-divergent) function pointer to jump to,
+``radv_nir_lower_call_abi`` prepends another parameter representing the (potentially divergent) function pointer for all
+lanes. For callable functions, ``radv_nir_lower_call_abi`` wraps the function body in a condition that verifies that the
+current thread's (divergent) pointer matches the (non-divergent) pointer that is currently being executed. This serves
+to "mask off" all threads that wanted to jump to a different function than what is currently executing. At the very end,
+``radv_nir_lower_call_abi`` inserts some code deciding whether to jump to the next callee or to return.
+
+.. _stack:
+
+Stack
+-----
+
+Supporting arbitrary function calls also means supporting recursion, and recursive functions need a stack.
+AMD hardware provides instructions for accessing a per-thread scratch memory area in VRAM, and ACO uses this per-thread
+scratch memory to set up its stack.
+
+The stack frame for a function consists of all scratch memory allocated for this function in NIR, as well as space to
+spill VGPRs if that is required. ACO adds a stack pointer as a parameter to every function - this stack pointer is added
+to the offset inside the scratch space for all scratch loads/stores to make sure they don't overwrite stack frames of
+caller functions.
+
+ACO's call instructions take two stack-related operands: The current (caller) stack pointer and the caller's stack size.
+When converting the call instruction to hardware instructions, ACO will add the caller stack size to the stack pointer
+for the duration of the call (and subtract it again afterwards). This allows us to re-use the same stack pointer after
+the call.
+
+Implicit/System Parameters
+--------------------------
+
+In addition to parameters defined by the function signature, both RADV and ACO will insert additional parameters while
+lowering calls. This is an overview of which lowering passes add which parameters.
+
+Parameters added by ``radv_nir_lower_call_abi`` (see :ref:`Divergent calls <div-calls>`):
+- "Uniform"/Non-divergent callee pointer
+- Divergent function pointer
+
+Parameters added by ACO: (see :ref:`Stack <stack>`)
+- Stack pointer (uniform)
+
+ABI Definition
+--------------
+
+The ABI (Application Binary Interface) defines specifics about the interaction between the function caller and the
+callee (e.g. assignment of registers to parameters or register preservation). In ACO, the primary purpose of the ABI is
+to define which register ranges are "preserved" (i.e. never overwritten by the callee) or "clobbered" (i.e. potentially
+overwritten by the callee).
+
+The caller can use preserved register ranges to store temporaries that are live across a call, and the callee can use
+clobbered register ranges to store its own temporaries. If the callee wants to use registers from a preserved range,
+then it needs to back up the value contained in the preserved register beforehand, and restore it when it's done using
+the preserved register. Similarly, if there are not enough preserved registers for the caller to store all its
+temporaries, the caller will need to spill excess temporaries to the stack.
+
+ACO has to cater to different needs when defining ABIs: On one side, ray tracing traversal shaders demand to free up
+the entire register file for the callee (Ray traversal is a really hot loop, so we don't want to spill anything at all).
+Besides some parameters like the invocation ID, these shaders should be able to overwrite almost anything. On the other
+side, RT traversal shaders should not be required to free up the register file when calling any-hit/intersection shaders
+as this would also cause spilling during traversal. GPGPU compute workloads could fall anywhere between these extremes,
+so a middle-ground solution is desirable for these.
+
+ACO's way of defining an ABI divides the register file into "blocks" (``struct aco::ABI::RegisterBlock``). Each block
+consists of a fixed number of preserved and clobbered registers, and a boolean determining whether the preserved or
+clobbered registers come first in the block. Preserved and clobbered register ranges are defined by
+repeating these blocks for as long as there are unassigned registers.
+
+Some examples of preserved/clobbered register ranges using this approach::
+
+   For all examples, there are 108 SGPRs and 128 VGPRs to assign.
+
+   RegisterBlock:
+      clobbered_size: {16 sgpr, 16 vgpr}
+      preserved_size: {16 sgpr, 16 vgpr}
+      clobbered_first: false
+   results in:
+      v0-v15:    preserved
+      v16-v31:   clobbered
+      v32-v47:   preserved
+      v48-v63:   clobbered
+      v64-v79:   preserved
+      v80-v95:   clobbered
+      v96-v111:  preserved
+      v112-v127: clobbered
+
+      s0-s15:   preserved
+      s16-s31:  clobbered
+      s32-s47:  preserved
+      s48-s63:  clobbered
+      s64-s79:  preserved
+      s80-s95:  clobbered
+      s96-s108: preserved
+
+   RegisterBlock:
+      clobbered_size: {128 sgpr, 256 vgpr}
+      preserved_size: {80 sgpr, 80 vgpr}
+      clobbered_first: false
+   results in:
+      v0-v79:   preserved
+      v80-v127: clobbered
+
+      s0-s79:   preserved
+      s80-s108: clobbered
+
+An alternating preserved-clobbered-preserved pattern is useful for generic compute workloads, because the ratio of
+preserved to clobbered registers is roughly the same, no matter how many registers are used by the shaders.
+
+The latter example where the lower part of the register file is preserved and only some registers high up in the
+register file are clobbered is suitable for any-hit/intersection shaders - traversal shader temporaries can live in the
+preserved part low in the register file.
+
+This block assignment is optional - if no ``RegisterBlock`` is given, the ABI defines the entire register range as
+clobbered-by-default, although parameters that are not marked as clobbered via ``ACO_NIR_PARAM_ATTRIB_DISCARDABLE``
+will continue being preserved.
+
+Parameter Register Assignment
+-----------------------------
+
+If a ``RegisterBlock`` defines preserved and clobbered ranges, then parameters are assigned registers from either range
+depending on ``ACO_NIR_PARAM_ATTRIB_DISCARDABLE`` - if parameters are marked as clobbered with this attribute, then they
+are assigned a register in a clobbered range, otherwise they are assigned in a register in a preserved range. The order
+of the parameters in the register file is not necessarily the same order as in the function signature - they may get
+reordered if it's beneficial to fill gaps or for alignment.
+
+If there is no ``RegisterBlock``, then registers will be assigned based on alignment only.
+
+If there is no more space for a parameter in any of its corresponding register ranges, it will be moved to the stack.

docs/drivers/amd/hang-debugging.rst

@@ -0,0 +1,85 @@
+:orphan:
+
+.. _radv-debug-hang:
+
+Debugging GPU hangs with RADV
+=============================
+
+UMR (optional)
+--------------
+
+UMR is needed for dumping a lot of useful information. Clone, build and install
+`UMR <https://gitlab.freedesktop.org/tomstdenis/umr>`__. Do not forget to run
+``chmod +s $(which umr)`` so RADV can actually access UMR.
+
+UMR needs to access some kernel debug interfaces:
+
+.. code-block:: sh
+
+   chmod 777 /sys/kernel/debug
+   chmod -R 777 /sys/kernel/debug/dri
+
+Secure boot has to be disabled as well.
+
+Generating and analyzing hang reports
+-------------------------------------
+
+With UMR installed, you can now set ``RADV_DEBUG=hang`` which makes RADV insert
+trace markers and synchronization and check for hangs. The hang report will be
+saved to ``~/radv_dumps_<pid>_<time>``. Inside the directory of the hang report,
+there are a couple of files:
+
+* ``*.spv``: SPIR-V binaries of the pipeline that was bound when the hang
+  occurred.
+* ``addr_binding_report.log``: VK_EXT_address_binding_report logs.
+* ``app_info.log``: ``VkApplicationInfo`` fields.
+* ``bo_history.log``: A list of every GPU memory allocation and deallocation.
+  If the GPU hang was caused by a page fault, you can use
+  `radv_check_va.py <https://gitlab.freedesktop.org/mesa/mesa/-/blob/main/src/amd/vulkan/tools/radv_check_va.py>`__
+  to figure out if address is invalid or used after the memory was deallocated.
+* ``bo_ranges.log``: Address ranges that were valid at the time of submission.
+* ``dmesg.log``: Output of ``dmesg``, if available.
+* ``gpu_info.log``: Fields of ``radeon_info``.
+* ``pipeline.log``: IR of the shaders that were bound during the hang as well as
+  programm counters of waves executing said shaders and bound descriptors.
+* ``registers.log``: Various GPU state registers.
+* ``trace.log``: An annotated list of the command stream that caused the hang.
+  the commands that hung come after
+  ``!!!!! This is the last trace point that was reached by the CP !!!!!``.
+* ``umr_ring.log``: Similar to ``trace.log``.
+* ``umr_waves.log``: A list of waves that were active at the time of the hang,
+  including register values.
+* ``vm_fault.log``: The page fault address if a page fault occurred.
+
+Note: By default, the backend IR (ACO or LLVM) and the disassembly should be
+dumped to ``pipeline.log``. But due to shaders caching, you might need
+``RADV_DEBUG=hang,nocache`` to get SPIR-V and NIR in the GPU hang report.
+
+Debugging Steam games
+---------------------
+
+Steam games require a bit more work so RADV can access UMR: In your Steam library,
+make sure **Tools** is checked and search for **Steam Linux Runtime**.
+Under **Properties** -> **Installed Files**, click **Browse**, open
+``_v2-entry-point`` and add
+
+.. code-block:: sh
+
+   shift 2
+   exec "$@"
+
+at the top of the file. Hang debugging can be enabled by selecting the faulting
+game and adding ``RADV_DEBUG=hang %command%`` under **Properties** -> **General**
+-> **LAUNCH OPTIONS**.
+
+Debugging hangs without RADV_DEBUG=hang
+---------------------------------------
+
+In some situations, ``RADV_DEBUG=hang`` wouldn't be able to generate a GPU hang
+report, like for synchronization issues (because it enables
+``RADV_DEBUG=syncshaders`` behind the scene). An alternative solution is to
+disable GPU recovery by adding ``amdgpu.gpu_recovery=0`` to your kernel command
+line options. And then invoke UMR manually with
+``umr --by-pci <pci_id> -O bits,halt_waves -go 0 -wa <ring> -go 1 2>&1`` for
+dumping the waves and ``umr --by-pci <pci_id> -RS <ring> 2>&1`` for dumping the
+rings once the GPU hang occurred.

docs/drivers/amd/hw/pops.rst

@@ -0,0 +1,477 @@
+Primitive Ordered Pixel Shading
+===============================
+
+Primitive Ordered Pixel Shading (POPS) is the feature available starting from
+GFX9 that provides the Fragment Shader Interlock or Fragment Shader Ordering
+functionality.
+
+It allows a part of a fragment shader — an ordered section (or a critical
+section) — to be executed sequentially in rasterization order for different
+invocations covering the same pixel position.
+
+This article describes how POPS is set up in shader code and the registers. The
+information here is currently provided for architecture generations up to GFX11.
+
+Note that the information in this article is **not official** and may contain
+inaccuracies, as well as incomplete or incorrect assumptions. It is based on the
+shader code output of the Radeon GPU Analyzer for Rasterizer Ordered View usage
+in Direct3D shaders, AMD's Platform Abstraction Library (PAL), ISA references,
+and experimentation with the hardware.
+
+Shader code
+-----------
+
+With POPS, a wave can dynamically execute up to one ordered section. It is fine
+for a wave not to enter an ordered section at all if it doesn't need ordering on
+its execution path, however.
+
+The setup of the ordered section consists of three parts:
+
+1. Entering the ordered section in the current wave — awaiting the completion of
+   ordered sections in overlapped waves.
+2. Resolving overlap within the current wave — intrawave collisions (optional
+   and GFX9–10.3 only).
+3. Exiting the ordered section — resuming overlapping waves trying to enter
+   their ordered sections.
+
+GFX9–10.3: Entering the ordered section in the wave
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Awaiting the completion of ordered sections in overlapped waves is performed by
+setting the POPS packer hardware register, and then polling the volatile
+``pops_exiting_wave_id`` ALU operand source until its value exceeds the newest
+overlapped wave ID for the current wave.
+
+The information needed for the wave to perform the waiting is provided to it via
+the SGPR argument ``COLLISION_WAVEID``. Its loading needs to be enabled in the
+``SPI_SHADER_PGM_RSRC2_PS`` and ``PA_SC_SHADER_CONTROL`` registers (note that
+the POPS arguments specifically need to be enabled not only in ``RSRC`` unlike
+various other arguments, but in ``PA_SC_SHADER_CONTROL`` as well).
+
+The collision wave ID argument contains the following unsigned values:
+
+* [31]: Whether overlap has occurred.
+* [29:28] (GFX10+) / [28] (GFX9): ID of the packer the wave should be associated
+  with.
+* [25:16]: Newest overlapped wave ID.
+* [9:0]: Current wave ID.
+
+The 2020 RDNA and RDNA 2 ISA references contain incorrect offsets and widths of
+the fields, possibly from an early development iteration, but the meanings of
+them are accurate there.
+
+The wait must not be performed if the "did overlap" bit 31 is set to 0,
+otherwise it will result in a hang. Also, the bit being set to 0 indicates that
+there are *both* no wave overlap *and no intrawave collisions* for the current
+wave — so if the bit is 0, it's safe for the wave to skip all of the POPS logic
+completely and execute the contents of the ordered section simply as usual with
+unordered access as a potential additional optimization. The packer hardware
+register, however, may be set even without overlap safely — it's the wait loop
+itself that must not be executed if it was reported that there was no overlap.
+
+The packer ID needs to be passed to the packer hardware register using
+``s_setreg_b32`` so the wave can poll ``pops_exiting_wave_id`` on that packer.
+
+On GFX9, the ``MODE`` (1) hardware register has two bits specifying which packer
+the wave is associated with:
+
+* [25]: The wave is associated with packer 1.
+* [24]: The wave is associated with packer 0.
+
+Initially, both of these bits are set 0, meaning that POPS is disabled for the
+wave. If the wave needs to enter the ordered section, it must set bit 24 to 1 if
+the packer ID in ``COLLISION_WAVEID`` is 0, or set bit 25 to 1 if the packer ID
+is 1.
+
+Starting from GFX10, the ``POPS_PACKER`` (25) hardware register is used instead,
+containing the following fields:
+
+* [2:1]: Packer ID.
+* [0]: POPS enabled for the wave.
+
+Initially, POPS is disabled for a wave. To start entering the ordered section,
+bits 2:1 must be set to the packer ID from ``COLLISION_WAVEID``, and bit 0 needs
+to be set to 1.
+
+The wave IDs, both in ``COLLISION_WAVEID`` and ``pops_exiting_wave_id``, are
+10-bit values wrapping around on overflow — consecutive waves are numbered 1022,
+1023, 0, 1… This wraparound needs to be taken into account when comparing the
+exiting wave ID and the newest overlapped wave ID.
+
+Specifically, until the current wave exits the ordered section, its ID can't be
+smaller than the newest overlapped wave ID or the exiting wave ID. So
+``current_wave_id + 1`` can be subtracted from 10-bit wave IDs to remap them to
+monotonically increasing unsigned values. In this case, the largest value,
+0xFFFFFFFF, will correspond to the current wave, 10-bit values up to the current
+wave ID will be in a range near 0xFFFFFFFF growing towards it, and wave IDs from
+before the last wraparound will be near 0 increasing away from it. Subtracting
+``current_wave_id + 1`` is equivalent to adding ``~current_wave_id``.
+
+GFX9 has an off-by-one error in the newest overlapped wave ID: if the 10-bit
+newest overlapped wave ID is greater than the 10-bit current wave ID (meaning
+that it's behind the last wraparound point), 1 needs to be added to the newest
+overlapped wave ID before using it in the comparison. This was corrected in
+GFX10.
+
+The exiting wave ID (not to be confused with "exited" — the exiting wave ID is
+the wave that will exit the ordered section next) is queried via the
+``pops_exiting_wave_id`` ALU operand source, numbered 239. Normally, it will be
+one of the arguments of ``s_add_i32`` that remaps it from a wrapping 10-bit wave
+ID to monotonically increasing one.
+
+It's a volatile operand, and it needs to be read in a loop until its value
+becomes greater than the newest overlapped wave ID (after remapping both to
+monotonic). However, if it's too early for the current wave to enter the ordered
+section, it needs to yield execution to other waves that may potentially be
+overlapped — via ``s_sleep``. GFX9 requires a finite amount of delay to be
+specified, AMD uses 3. Starting from GFX10, exiting the ordered section wakes up
+the waiting waves, so the maximum delay of 0xFFFF can be used.
+
+In pseudocode, the entering logic would look like this::
+
+   bool did_overlap = collision_wave_id[31];
+   if (did_overlap) {
+      if (gfx_level >= GFX10) {
+         uint packer_id = collision_wave_id[29:28];
+         s_setreg_b32(HW_REG_POPS_PACKER[2:0], 1 | (packer_id << 1));
+      } else {
+         uint packer_id = collision_wave_id[28];
+         s_setreg_b32(HW_REG_MODE[25:24], packer_id ? 0b10 : 0b01);
+      }
+
+      uint current_10bit_wave_id = collision_wave_id[9:0];
+      // Or -(current_10bit_wave_id + 1).
+      uint wave_id_remap_offset = ~current_10bit_wave_id;
+
+      uint newest_overlapped_10bit_wave_id = collision_wave_id[25:16];
+      if (gfx_level < GFX10 &&
+          newest_overlapped_10bit_wave_id > current_10bit_wave_id) {
+         ++newest_overlapped_10bit_wave_id;
+      }
+      uint newest_overlapped_wave_id =
+         newest_overlapped_10bit_wave_id + wave_id_remap_offset;
+
+      while (!(src_pops_exiting_wave_id + wave_id_remap_offset >
+               newest_overlapped_wave_id)) {
+         s_sleep(gfx_level >= GFX10 ? 0xFFFF : 3);
+      }
+   }
+
+The SPIR-V fragment shader interlock specification requires an invocation — an
+individual invocation, not the whole subgroup — to execute
+``OpBeginInvocationInterlockEXT`` exactly once. However, if there are multiple
+begin instructions, or even multiple begin/end pairs, under divergent
+conditions, a wave may end up waiting for the overlapped waves multiple times.
+Thankfully, it's safe to set the POPS packer hardware register to the same
+value, or to run the wait loop, multiple times during the wave's execution, as
+long as the ordered section isn't exited in between by the wave.
+
+GFX11: Entering the ordered section in the wave
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Instead of exposing wave IDs to shaders, GFX11 uses the "export ready" wave
+status flag to report that the wave may enter the ordered section. It's awaited
+by the ``s_wait_event`` instruction, with the bit 0 ("don't wait for
+``export_ready``") of the immediate operand set to 0. On GFX11 specifically, AMD
+passes 0 as the whole immediate operand.
+
+The "export ready" wait can be done multiple times safely.
+
+GFX9–10.3: Resolving intrawave collisions
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+On GFX9–10.3, it's possible for overlapping fragment shader invocations to be
+placed not only in different waves, but also in the same wave, with the shader
+code making sure that the ordered section is executed for overlapping
+invocations in order.
+
+This functionality is optional — it can be activated by enabling loading of the
+``INTRAWAVE_COLLISION`` SGPR argument in ``SPI_SHADER_PGM_RSRC2_PS`` and
+``PA_SC_SHADER_CONTROL``.
+
+The lower 8 or 16 (depending on the wave size) bits of ``INTRAWAVE_COLLISION``
+contain the mask of whether each quad in the wave starts a new layer of
+overlapping invocations, and thus the ordered section code for them needs to be
+executed after running it for all lanes with indices preceding that quad index
+multiplied by 4. The rest of the bits in the argument need to be ignored — AMD
+explicitly masks them out in shader code (although this is not necessary if the
+shader uses "find first 1" to obtain the start of the next set of overlapping
+quads or expands this quad mask into a lane mask).
+
+For example, if the intrawave collision mask is 0b0000001110000100, or
+``(1 << 2) | (1 << 7) | (1 << 8) | (1 << 9)``, the code of the ordered section
+needs to be executed first only for quads 1:0 (lanes 7:0), then only for quads
+6:2 (lanes 27:8), then for quad 7 (lanes 31:28), then for quad 8 (lanes 35:32),
+and then for the remaining quads 15:9 (lanes 63:36).
+
+This effectively causes the ordered section to be executed as smaller
+"sub-subgroups" within the original subgroup.
+
+However, this is not always compatible with the execution model of SPIR-V or
+GLSL fragment shaders, so enabling intrawave collisions and wrapping a part of
+the shader in a loop may be unsafe in some cases. One particular example is when
+the shader uses subgroup operations influenced by lanes outside the current
+quad. In this case, the code outside and inside the ordered section may be
+executed with different sets of active invocations, affecting the results of
+subgroup operations. But in SPIR-V and GLSL, fragment shader interlock is not
+supposed to modify the set of active invocations in any way. So the intrawave
+collision loop may break the results of subgroup operations in unpredictable
+ways, even outside the driver's compiler infrastructure. Even if the driver
+splits the subgroup exactly at ``OpBeginInvocationInterlockEXT`` and makes the
+lane subsets rejoin exactly at ``OpEndInvocationInterlockEXT``, the application
+and the compilers that created the source shader are still not aware of that
+happening — the input SPIR-V or GLSL shader might have already gone through
+various optimizations, such as common subexpression elimination which might
+have considered a subgroup operation before ``OpBeginInvocationInterlockEXT``
+and one after it equivalent.
+
+The idea behind reporting intrawave collisions to shaders is to reduce the
+impact on the parallelism of the part of the shader that doesn't depend on the
+ordering, to avoid wasting lanes in the wave and to allow the code outside the
+ordered section in different invocations to run in parallel lanes as usual. This
+may be especially helpful if the ordered section is small compared to the rest
+of the shader — for instance, a custom blending equation in the end of the usual
+fragment shader for a surface in the world.
+
+However, whether handling intrawave collisions is preferred is not a question
+with one universal answer. Intrawave collisions are pretty uncommon without
+multisampling, or when using sample interlock with multisampling, although
+they're highly frequent with pixel interlock with multisampling, when adjacent
+primitives cover the same pixels along the shared edge (though that's an
+extremely expensive situation in general). But resolving intrawave collisions
+adds some overhead costs to the shader. If intrawave overlap is unlikely to
+happen often, or even more importantly, if the majority of the shader is inside
+the ordered section, handling it in the shader may cause more harm than good.
+
+GFX11 removes this concept entirely, instead overlapping invocations are always
+placed in different waves.
+
+GFX9–10.3: Exiting the ordered section in the wave
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To exit the ordered section and let overlapping waves resume execution and enter
+their ordered sections, the wave needs to send the ``ORDERED_PS_DONE`` message
+(7) using ``s_sendmsg``.
+
+If the wave has enabled POPS by setting the packer hardware register, it *must
+not* execute ``s_endpgm`` without having sent ``ORDERED_PS_DONE`` once, so the
+message must be sent on all execution paths after the packer register setup.
+However, if the wave exits before having configured the packer register, sending
+the message is not required, though it's still fine to send it regardless of
+that.
+
+Note that if the shader has multiple ``OpEndInvocationInterlockEXT``
+instructions executed in the same wave (depending on a divergent condition, for
+example), it must still be ensured that ``ORDERED_PS_DONE`` is sent by the wave
+only once, and especially not before any awaiting of overlapped waves.
+
+Before the message is sent, all counters for memory accesses that need to be
+primitive-ordered, both writes and (in case something after the ordered section
+depends on the per-pixel data, for instance, the tail blending fallback in
+order-independent transparency) reads, must be awaited. Those may include
+``vm``, ``vs``, and in some cases ``lgkm`` (though normally primitive-ordered
+memory accesses will be done through VMEM with divergent addresses, not SMEM, as
+there's no synchronization between fragments at different pixel coordinates, but
+it's still technically possible for a shader, even though pointless and
+nonoptimal, to explicitly perform them in a waterfall loop, for instance, and
+that must work correctly too). Without that, a race condition will occur when
+the newly resumed waves start accessing the memory locations to which there
+still are outstanding accesses in the current wave.
+
+Another option for exiting is the ``s_endpgm_ordered_ps_done`` instruction,
+which combines waiting for all the counters, sending the ``ORDERED_PS_DONE``
+message, and ending the program. Generally, however, it's desirable to resume
+overlapping waves as early as possible, including before the export, as it may
+stall the wave for some time too.
+
+GFX11: Exiting the ordered section in the wave
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The overlapping waves are resumed when the wave performs the last export (with
+the ``done`` flag).
+
+The same requirements for awaiting the memory access counters as on GFX9–10.3
+still apply.
+
+Memory access requirements
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The compiler needs to ensure that entering the ordered section implements
+acquire semantics, and exiting it implements release semantics, in the fragment
+interlock memory scope for ``UniformMemory`` and ``ImageMemory`` SPIR-V storage
+classes.
+
+A fragment interlock memory scope instance includes overlapping fragment shader
+invocations executed by commands inside a single subpass. It may be considered a
+subset of a queue family memory scope instance from the perspective of memory
+barriers.
+
+Fragment shader interlock doesn't perform implicit memory availability or
+visibility operations. Shaders must do them by themselves for accesses requiring
+primitive ordering, such as via ``coherent`` (``queuefamilycoherent``) in GLSL
+or ``MakeAvailable`` and ``MakeVisible`` in at least the ``QueueFamily`` scope
+in SPIR-V.
+
+On AMD hardware, this means that the accessed memory locations must be made
+available or visible between waves that may be executed on any compute unit — so
+accesses must go directly to the global L2 cache, bypassing L0$ via the GLC flag
+and L1$ via DLC.
+
+However, it should be noted that memory accesses in the ordered section may be
+expected by the application to be done in primitive order even if they don't
+have the GLC and DLC flags. Coherent access not only bypasses, but also
+invalidates the lower-level caches for the accessed memory locations. Thus,
+considering that normally per-pixel data is accessed exclusively by the
+invocation executing the ordered section, it's not necessary to make all reads
+or writes in the ordered section for one memory location to be GLC/DLC — just
+the first read and the last write: it doesn't matter if per-pixel data is cached
+in L0/L1 in the middle of a dependency chain in the ordered section, as long as
+it's invalidated in them in the beginning and flushed to L2 in the end.
+Therefore, optimizations in the compiler must not simply assume that only
+coherent accesses need primitive ordering — and moreover, the compiler must also
+take into account that the same data may be accessed through different bindings.
+
+Export requirements
+^^^^^^^^^^^^^^^^^^^
+
+With POPS, on all hardware generations, the shader must have at least one
+export, though it can be a null or an ``off, off, off, off`` one.
+
+Also, even if the shader doesn't need to export any real data, the export
+skipping that was added in GFX10 must not be used, and some space must be
+allocated in the export buffer, such as by setting ``SPI_SHADER_COL_FORMAT`` for
+some color output to ``SPI_SHADER_32_R``.
+
+Without this, the shader will be executed without the needed synchronization on
+GFX10, and will hang on GFX11.
+
+Drawing context setup
+---------------------
+
+Configuring POPS
+^^^^^^^^^^^^^^^^
+
+Most of the configuration is performed via the ``DB_SHADER_CONTROL`` register.
+
+To enable POPS for the draw,
+``DB_SHADER_CONTROL.PRIMITIVE_ORDERED_PIXEL_SHADER`` should be set to 1.
+
+On GFX9–10.3, ``DB_SHADER_CONTROL.POPS_OVERLAP_NUM_SAMPLES`` controls which
+fragment shader invocations are considered overlapping:
+
+* For pixel interlock, it must be set to 0 (1 sample).
+* If sample interlock is sufficient (only synchronizing between invocations that
+  have any common sample mask bits), it may be set to
+  ``PA_SC_AA_CONFIG.MSAA_EXPOSED_SAMPLES`` — the number of sample coverage mask
+  bits passed to the shader which is expected to use the sample mask to
+  determine whether it's allowed to access the data for each of the samples. As
+  of April 2023, PAL for some reason doesn't use non-1x
+  ``POPS_OVERLAP_NUM_SAMPLES`` at all, even when using Direct3D Rasterizer
+  Ordered Views or ``GL_INTEL_fragment_shader_ordering`` with sample shading
+  (those APIs tie the interlock granularity to the shading frequency — Vulkan
+  and OpenGL fragment shader interlock, however, allows specifying the interlock
+  granularity independently of it, making it possible both to ask for finer
+  synchronization guarantees and to require stronger ones than Direct3D ROVs can
+  provide). However, with MSAA, on AMD hardware, pixel interlock generally
+  performs *massively*, sometimes prohibitively, slower than sample interlock,
+  because it causes fragment shader invocations along the common edge of
+  adjacent primitives to be ordered as they cover the same pixels (even though
+  they don't cover any common samples). So it's highly desirable for the driver
+  to provide sample interlock, and to set ``POPS_OVERLAP_NUM_SAMPLES``
+  accordingly, if the shader declares that it's enough for it via the execution
+  mode.
+
+On GFX11, when POPS is enabled, ``DB_SHADER_CONTROL.OVERRIDE_INTRINSIC_RATE`` is
+used in place of ``DB_SHADER_CONTROL.POPS_OVERLAP_NUM_SAMPLES`` from the earlier
+architecture generations (and has a different bit offset in the register), and
+``DB_SHADER_CONTROL.OVERRIDE_INTRINSIC_RATE_ENABLE`` must be set to 1. The GFX11
+blending performance workaround overriding the intrinsic rate must not be
+applied if POPS is used in the draw — the intrinsic rate override must be used
+solely to control the interlock granularity in this case.
+
+No explicit flushes/synchronization are needed when changing the pipeline state
+variables that may be involved in POPS, such as the rasterization sample count.
+POPS automatically keeps synchronizing invocations even between draws with
+different sample counts (invocations with common coverage mask bits are
+considered overlapping by the hardware, regardless of what those samples
+actually are — only the indices are important).
+
+Also, on GFX11, POPS uses ``DB_Z_INFO.NUM_SAMPLES`` to determine the coverage
+sample count, and it must be equal to ``PA_SC_AA_CONFIG.MSAA_EXPOSED_SAMPLES``
+even if there's no depth/stencil target.
+
+Hardware bug workarounds
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+Early revisions of GFX9 — ``CHIP_VEGA10`` and ``CHIP_RAVEN`` — contain a
+hardware bug that may result in a hang, and need a workaround to be enabled.
+Specifically, if POPS is used with 8 or more rasterization samples, or with 8 or
+more depth/stencil target samples, ``DB_DFSM_CONTROL.POPS_DRAIN_PS_ON_OVERLAP``
+must be set to 1 for draws that satisfy this condition. In PAL, this is the
+``waMiscPopsMissedOverlap`` workaround. It results in slightly lower performance
+in those cases, increasing the frame time by around 1.5 to 2 times in
+`nvpro-samples/vk_order_independent_transparency <https://github.com/nvpro-samples/vk_order_independent_transparency>`_
+on the RX Vega 10, but it's required in a pretty rare case (8x+ MSAA) and is
+mandatory to ensure stability.
+
+Also, even though ``DB_DFSM_CONTROL.POPS_DRAIN_PS_ON_OVERLAP`` is not required
+on chips other than the ``CHIP_VEGA10`` and ``CHIP_RAVEN`` GFX9 revisions, if
+it's enabled for some reason on GFX10.1 (``CHIP_NAVI10``, ``CHIP_NAVI12``,
+``CHIP_NAVI14``), and the draw uses POPS,
+``DB_RENDER_OVERRIDE2.PARTIAL_SQUAD_LAUNCH_CONTROL`` must be set to
+``PSLC_ON_HANG_ONLY`` to avoid a hang (see ``waStalledPopsMode`` in PAL).
+
+Out-of-order rasterization interaction
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This is a largely unresearched topic currently. However, considering that POPS
+is primarily the functionality of the Depth Block, similarity to the behavior of
+out-of-order rasterization in depth/stencil testing may possibly be expected.
+
+If the shader specifies an ordered interlock execution mode, out-of-order
+rasterization likely must not be enabled implicitly.
+
+As of April 2023, PAL doesn't have any rules specifically for POPS in the logic
+determining whether out-of-order rasterization can be enabled automatically.
+Some of the POPS usage cases may possibly be covered by the rule that always
+disables out-of-order rasterization if the shader writes to Unordered Access
+Views (storage resources), though fragment shader interlock can be used for
+read-only purposes too (for ordering between draws that only read per-pixel data
+and draws that may write it), so that may be an oversight.
+
+Explicitly enabled relaxed rasterization order modifies the concept of
+rasterization order itself in Vulkan, so from the point of view of the
+specification of fragment shader interlock, relaxed rasterization order should
+still be applicable regardless of whether the shader requests ordered interlock.
+PAL also doesn't make any POPS-specific exceptions here as of April 2023.
+
+Variable-rate shading interaction
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+On GFX10.3, enabling ``DB_SHADER_CONTROL.PRIMITIVE_ORDERED_PIXEL_SHADER`` forces
+the shading rate to be 1x1, thus the
+``fragmentShadingRateWithFragmentShaderInterlock`` Vulkan device property must
+be false.
+
+On GFX11, by default, POPS itself can work with non-1x1 shading rates, and the
+``fragmentShadingRateWithFragmentShaderInterlock`` property must be true.
+However, if ``PA_SC_VRS_SURFACE_CNTL_1.FORCE_SC_VRS_RATE_FINE_POPS`` is set,
+enabling POPS will force 1x1 shading rate.
+
+The widest interlock granularity available on GFX11 — with the lowest possible
+Depth Block intrinsic rate, 1x — is per-fine-pixel, however. There's no
+synchronization between coarse fragment shader invocations if they don't cover
+common fine pixels, so the ``fragmentShaderShadingRateInterlock`` Vulkan device
+feature is not available.
+
+Additional configuration
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+These are some largely unresearched options found in the register declarations.
+PAL doesn't use them, so it's unknown if they make any significant difference.
+No effect was found in `nvpro-samples/vk_order_independent_transparency <https://github.com/nvpro-samples/vk_order_independent_transparency>`_
+during testing on GFX9 ``CHIP_RAVEN`` and GFX11 ``CHIP_NAVI31``.
+
+* ``DB_SHADER_CONTROL.EXEC_IF_OVERLAPPED`` on GFX9–10.3.
+* ``PA_SC_BINNER_CNTL_0.BIN_MAPPING_MODE = BIN_MAP_MODE_POPS`` on GFX10-11.5.
+  This field is reserved on GFX12+ and should be set to 0.

docs/drivers/amd/live-var-analysis.rst

@@ -0,0 +1,164 @@
+:orphan:
+
+.. _aco-live-var-analysis:
+
+Description of live variable analysis in ACO
+============================================
+
+Operand flags
+-------------
+
+Operands can have several different flags which are set by live variable analysis:
+
+* ``isKill``: this operand is not live after the instruction
+* ``isFirstKill``: this is the first instance of a temporary in the instruction with ``isKill=true``
+* ``isLateKill``: this operand cannot use the same registers as any definition
+* ``isClobbered``: this operand will use some of the same registers as a definition
+* ``isCopyKill``: this operand must use a different register than earlier instances of the same temporary in the
+  instruction's operand list
+
+Note that:
+
+* ``isFirstKill=true`` requires that the operand is also ``isKill=true``.
+* If ``isKill=true``, then ``isLateKill=true`` requires that the operand is also ``isFirstKill=true`` or
+  ``isCopyKill=true``.
+* ``isCopyKill=true`` is incompatible with ``isFirstKill=true`` in the same operand.
+* ``isLateKill=true`` cannot be used with ``isClobbered=true`` for the same temporary in an instruction.
+* If ``isCopyKill=true``, then the operand will also have ``isKill=true``, even if the temporary is live after the
+  instruction.
+
+Operands and definitions can be "tied", indicated by ``get_tied_defs()``, meaning that the must use the same registers:
+
+* Operands tied to definitions have ``isClobbered=true``.
+* If a clobbered operand has ``isKill=false``, it will be moved to a different register.
+* If two operands of the same temporary are tied to different definitions of the same instruction, the second of those
+  operands will have ``isCopyKill=true``.
+
+There also exists the ``isVectorAligned`` flag. This can be used to define a vector of operands, starting with
+``isVectorAligned=true`` and ending with ``isVectorAligned=false``, which must be placed in consecutive registers.
+
+To prevent issues with register allocation, an operand being part of a vector means:
+
+* Unless a vector is tied to a definition, it will also have ``isLateKill=true``, so that a partially killed vectors
+  are not required to share the same space with definitions.
+* If two operands of the same temporary are both part of vectors in the same instruction, the second of those operands
+  will have ``isCopyKill=true``.
+
+Register demand calculation
+---------------------------
+
+``live_in``
+   temporaries live before the instruction
+
+``live_out``
+   temporaries live after the instruction
+
+``live_through``
+   temporaries live both before and after the instruction
+
+``live_definitions``
+   temporaries defined and used later (definitions where ``isKill=false``)
+
+``dead_definitions``
+   temporaries defined and not used later (definitions where ``isKill=true``)
+
+``early_kill_operands``
+   temporaries killed which are not marked late kill (operands where ``isFirstKill=true && isLateKill=false``)
+
+``late_kill_operands``
+   temporaries killed which are marked late kill (operands where ``isFirstKill=true && isLateKill=true``)
+
+``first_kill_operands``
+   temporaries killed by the instruction (operands where ``isFirstKill=true``)
+
+   ``early_kill_operands + late_kill_operands``
+
+``copied_operands``
+   operands which are either clobbered but not killed, or copy-kill (operands where
+   ``isCopyKill=true || (isClobbered=true && isKill=false)``)
+
+``early_kill_copies``
+   ``copied_operands`` which are not marked late kill (operands where
+   ``(isCopyKill=true && isLateKill=false) || (isClobbered=true && isKill=false)``)
+
+``late_kill_copies``
+   ``copied_operands`` which are marked late kill (operands where ``(isCopyKill=true && isLateKill=true)``)
+
+``live_out``
+   ``live_through + live_definitions``
+
+   ``live_in - first_kill_operands + live_definitions``
+
+``live_in``
+   ``live_out - live_definitions + first_kill_operands``
+
+   ``live_through + first_kill_operands``
+
+``live_through``
+   ``live_out - live_definitions``
+
+   ``live_in - first_kill_operands``
+
+Breakdown of register demand stages using ``live_in``:
+
+* stage 0: before instruction: ``live_in``
+* stage 1:     setup operands: ``live_in + early_kill_copies + late_kill_copies``
+* stage 2: during instruction: ``live_in - early_kill_operands + late_kill_copies``
+* stage 3:  write definitions: ``live_in - early_kill_operands + late_kill_copies + live_definitions + dead_definitions``
+* stage 4:  after instruction: ``live_in - early_kill_operands - late_kill_operands + live_definitions``
+
+Breakdown of register demand stages using ``live_through``:
+
+* stage 0: before instruction: ``live_through + late_kill_operands + early_kill_operands``
+* stage 1:     setup operands: ``live_through + late_kill_operands + early_kill_operands + early_kill_copies + late_kill_copies``
+* stage 2: during instruction: ``live_through + late_kill_operands + late_kill_copies``
+* stage 3:  write definitions: ``live_through + late_kill_operands + late_kill_copies + live_definitions + dead_definitions``
+* stage 4:  after instruction: ``live_through + live_definitions``
+
+Breakdown of register demand stages using ``live_out``:
+
+* stage 0: before instruction: ``live_out - live_definitions + late_kill_operands + early_kill_operands``
+* stage 1:     setup operands: ``live_out - live_definitions + late_kill_operands + early_kill_operands + early_kill_copies + late_kill_copies``
+* stage 2: during instruction: ``live_out - live_definitions + late_kill_operands + late_kill_copies``
+* stage 3:  write definitions: ``live_out + dead_definitions + late_kill_operands + late_kill_copies``
+* stage 4:  after instruction: ``live_out``
+
+If instruction B immediately follows instruction A, then stage 0 of instruction B equals stage 4 of instruction A.
+``Instruction::register_demand`` is ``max(stage1, stage3)``, which is equal to the maximum of all stages.
+
+There are a few helper functions for examining how an instruction changes register demand:
+
+``get_live_changes()``
+   This is the register demand change from killed temporaries and live definitions.
+
+   ``live_definitions - first_kill_operands``
+
+   equal to ``live_out - live_in``
+
+``get_temp_registers()``
+   This is the temporary increase in register demand needed for copy-kill operands, late-kill operands, clobbered
+   operands, and dead definitions.
+
+   ``max(early_kill_operands + late_kill_operands + early_kill_copies + late_kill_copies - live_definitions, late_kill_operands + late_kill_copies + dead_definitions)``
+
+   equal to ``register_demand - live_out``
+
+``get_temp_reg_changes()``
+   Since ``register_demand`` is ``max(stage1, stage3)``, this can be used to know what's the effect of marking an
+   operand killed will be.
+
+   ``live_definitions + dead_definitions - early_kill_operands - early_kill_copies``
+
+   equal to ``stage3 - stage1``
+
+They can be used as follows:
+
+* ``register_demand(a) = live_in(a) + live_changes(a) + temp_registers(a)``
+* ``register_demand(a) = live_out(a) + temp_registers(a)``
+
+Assuming ``stage4(a)==stage0(b)``:
+
+* ``register_demand(a) = register_demand(b) - temp_registers(b) - live_changes(b) + temp_registers(a)``
+* ``register_demand(b) = register_demand(a) - temp_registers(a) + live_changes(b) + temp_registers(b)``
+
+(note that ``max(a + b, a + c) - max(b, c) = a`` and ``a + max(b, c) = max(a + b, a + c)``)