intel-hard/intel-hard-graphics
2
0 Fork 0
mirror of https://github.com/elwrcl/intel-hard-graphics.git synced 2026-11-01 11:18:03 +03:00
Code Issues Packages Projects Releases Wiki Activity

init

Browse Source
This commit is contained in:
elwrcl committed 2026-06-06 13:25:15 +03:00
1 parent 04d4dc8b09
commit 818cab85dc
12511 files changed
+6156641
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/drivers/freedreno/fdm.rst
+286
View File
@@ -0,0 +1,286 @@
Fragment Density Map
====================
``VK_EXT_fragment_density_map`` is an extension which is intended to allow
users to render parts of the screen at a lower resolution. It is designed to be
implemented on tiled rendering GPU architectures such as Adreno, and the
intention is that it is implemented by rendering some of the tiles at a lower
resolution and scaling them up when resolving to system memory or when sampling
the resulting image. This inherently means that it is "all or nothing," that
is, it must be enabled or disabled for the entire render pass. While the idea is
simple, the implementation in turnip is very subtle with lots of
interactions with various different features. This page attempts to document
the main principles behind the implementation.
Coordinate Space Soup
---------------------
In order to render a tile at lower resolution, we have to override the user's
viewport and scissor for each tile depending on the scaling factor provided by
the user. This becomes complicated fast, so let's start by defining a few
coordinate spaces that we'll have to work with.
Framebuffer space
^^^^^^^^^^^^^^^^^
This is the space of the final rendered image. From the user's perspective
everything is specified in this space, and fragments created by the rasterizer
appear to be larger than 1 pixel. But this is not what actually happens in the
hardware, it is a fiction created by the driver. The other spaces below are
what the hardware actually "sees".
GMEM Space
^^^^^^^^^^
This space exists whenever tiled rendering/GMEM is used, even without FDM. It
is the space used to access GMEM, with the origin at the upper left of the
tile. The hardware automatically transforms rendering space into GMEM space
whenever GMEM is accessed using the various ``*_WINDOW_OFFSET`` registers. The
origin of this space in rendering space, or the value of ``*_WINDOW_OFFSET``,
will be called :math:`b_{cs}`, the common bin start, for reasons that are
explained below. When using FDM, coordinates in this space must be multiplied
by the scaling factor :math:`s` derived from the fragment density map, or
equivalently divided by the fragment area (as defined by the Vulkan
specification), with the origin still at the upper left of the tile. For
example, if :math:`s_x = 1/2`, then the bin is half as wide as it would've been
without FDM and all coordinates in this space must be divided by 2.
Rendering space
^^^^^^^^^^^^^^^
This is the space in which the hardware rasterizer operates and produces
fragments. Normally this is the same as framebuffer space, but with FDM it is
not. We transform the viewport and scissor from framebuffer space to
rendering space by patching them per-tile in the driver and then when we
resolve the tile we scale the resulting tile back to the correct resolution by
blitting from the rendering space source to the framebuffer space destination.
In order to come up with the correct transform from framebuffer space to
rendering space, it has to shrink the coordinates by :math:`s` while
mapping the original bin start in framebuffer space :math:`b_s` to
:math:`b_{cs}`. Since :math:`b_{cs}` is entirely defined by the driver when
programming ``*_WINDOW_OFFSET``, one tempting way to do this is to just
multiply by :math:`s` and define :math:`b_{cs} = b_s * s`. It turns out,
however, that this doesn't work. A key requirement is to handle cases where the
same scene is rendered in multiple different views at the same time using
``VK_KHR_multiview``, as in VR use-cases, and in this case we want :math:`s` to
vary per view, but :math:`b_{cs}` is always the same for every view because
there is only one ``*_WINDOW_OFFSET`` register for all layers (hence the name).
We follow the blob by leaving :math:`b_{cs}` the same regardless of whether FDM
is enabled or not. This means that normally :math:`b_s = b_{cs}`, although this
is not the case if ``VK_EXT_fragment_density_map_offset`` is in use and the
bins are shifted per-view. Since the coordinates need to be scaled by :math:`s`,
we know that the transform needs to look like :math:`x' = s * x + o`, where
only the offset :math:`o` is free. Plugging in the constraint that :math:`b_s`
maps to :math:`b_{cs}`, we get that :math:`b_{cs} = s * b_s + o` or
:math:`o = b_{cs} - s * b_s`. This is the function computed by
``tu_fdm_per_bin_offset`` and used to calculate the transform for the viewport,
scissor, and ``gl_FragCoord``. One critical thing is that the offset must be an
integer, or in other words the framebuffer space bin start :math:`b_s` must be
a multiple of :math:`1 / s`. This is a natural constraint anyway, because if
it wasn't the case then the bin would start in the middle of a fragment which
isn't possible to handle correctly.
Subsampled Space
^^^^^^^^^^^^^^^^
When using subsampled images, this is the space where the bin is stored in the
underlying subsampled image. When sampling from a subsampled image, the driver
inserts shader code to transform from framebuffer space to subsampled space
using metadata written when rendering to the image.
Accesses towards the edge of a bin may partially bleed into its neighboring bin
with linear or bicubic sampling. If its neighbor has a different scale or isn't
adjacent in subsampled space, we will sample the incorrect data or empty space
and return a corrupted result. In order to handle this, we need to insert an
"apron" around problematic edges and corners. This is done by blitting from the
nearest neighbor of each bin after the renderpass.
Subsampled space is normally scaled down similar to rendering space, which is
the point of subsampled images in the first place, but the origin of the bin
is up to the driver. The driver chooses the origin of each bin when rendering a
given render pass and then encodes it in the metadata used when sampling the
image. Bins that require an apron must be far enough away from each other that
their aprons don't intersect, and all of the bins must be contained within the
underlying image.
Even when subsampled images are in use, not all bins may be subsampled. For
example, there may not be enough space to insert aprons around every bin. When
this is the case, subsampled space is not scaled like rendering space, that is
we expand the bin when resolving similar to non-subsampled images, however the
origin of the bin may still differ from framebuffer space origin.
The algorithm used by turnip used to calculate the bin layout in subsampled
space is to start with a "default" layout of the bins and then recursively
solve conflicts caused by bins whose aprons are too close together. The first
strategy used is to shift one of the bins over by a certain amount. The second
fallback strategy is to un-subsample both neighboring bins, making them
expanded so that they touch each other and there is no apron.
One natural choice for the "default" layout is to just use rendering space.
That is, start each bin at :math:`b_cs` by default. That mostly works, except
for two problems. The first is easier to solve, and has to do with the border
when sampling: it is allowed to use border colors with subsampled images, and
when that happens and the framebuffer covers the entire image, it is expected
that sampling around the edge correctly blends the border color and the edge
pixel. In order for that to happen, bins that touch or intersect the edge of
the framebuffer in framebuffer space have to be shifted over so that their edge
touches the framebuffer edge in subsampled space too.
Doing this also allows an optimization: because we are storing the tile's
contents one to one from GMEM to system memory instead of scaling it up, we can
use the dedicated resolve engine instead of GRAS to resolve the tile to system
memory. Normally GRAS has to be used with non-subsampled images to scale up the
bin when resolving. However this doesn't work for tiles around the right and
bottom edge where we have to shift over the tile to align to the edge. This
also gets a bit tricky when the tile is shifted to avoid apron conflicts, because
normally the resolve engine would write the tile directly without shifting.
However there is a trick we can use to avoid falling back to GRAS: by
overriding ``RB_RESOLVE_WINDOW_OFFSET``, we can effectively apply an offset by
telling the resolve engine that the tile was rendered somewhere else. This
means that the shift amount has to be aligned to the alignment of
``RB_RESOLVE_WINDOW_OFFSET``, which is ``tile_align_*`` in the device info.
The other problem with making subsampled space equal rendering space is that
with an FDM offset, rendering space can be arbitrarily larger than framebuffer
space, and we may overflow the attachments by up to the size of a tile. The API
is designed to allow the driver to allocate extra slop space in the image in
this case, because there are image create flags for subsampled and FDM offset,
however the maximum tile size is far too large and images would take up
far too much memory if we allocated enough slop space for the largest
possible tile. An alternative is to use a hybrid of framebuffer space and
rendering space: shift over the tiles by :math:`b_o` so that their origin
is :math:`b_s` instead of :math:`b_cs`, but leave them scaled down. This
requires no slop space whatsoever, because the bins are shifted inside the
original image, but we can no longer use the resolve engine as the tile offsets
are no longer aligned to ``tile_align_*``. So in the driver we combine both
approaches: we calculate an aligned offset :math:`b_o'` which is :math:`b_o`
aligned down to ``tile_align_*`` and shift over the tiles by subtracting
:math:`b_o'` instead of :math:`b_o`. This requires slop space, but only
:math:`b_o - b_o'` slop space is required, which must be less than
``tile_align_*``. As usual the first row/column are not shifted over in x/y
respectively.
Here is an example of what a subsampled image looks like in memory, in this
case without any FDM offset:
.. figure:: subsampled_annotated.jpg
:alt: Example of a subsampled image
Note how some of the bins are shifted over to make space for the apron. After
applying the coordinate transform when sampling, this is the final image:
.. figure:: subsampled_final.jpg
:alt: Example of a subsampled image after coordinate transform
When ``VK_EXT_custom_resolve`` and subsampled images are used together, the
custom resolve subpass writes directly to the subsampled image. This means that
it needs to use subsampled space instead of rendering space, which in practice
means replacing :math:`b_{cs}` with the origin of the bin in the subsampled
image.
Viewport and Scissor Patching
-----------------------------
In order to have :math:`s` differ per view, we have to be able to override the
viewport per view. That is, we need to transform the viewport for each view
differently. If there is only one viewport, then we duplicate the user's
viewport for each view and transform it using the :math:`b_s` and :math:`s` for
that view, and we set a "per-view viewport" bit to select the viewport per view
instead of using the default viewport 0. When
``VK_VALVE_fragment_density_map_layered`` is in use, we instead have to insert
shader code to achieve the same thing.
If the user specifies multiple viewports but they are per-view because
``VK_QCOM_multiview_per_view_viewport`` is enabled, then we can just set the
per-view viewport bit and transform each user viewport individually by the
corresponding scale. But if the user explicitly writes ``gl_ViewportIndex``,
then there is nothing we can do and we have to make :math:`s` the same for all
views by conservatively taking the minimum. Then we apply :math:`s` to all of
the user-specified viewports.
Because the bin size is now per-view, the usual mechanism of
``*_WINDOW_SCISSOR`` for clipping fragments outside the bin doesn't work.
Instead the driver needs to intersect the transformed user-specified scissor
with the transformed rendering-space bin coordinates, effectively replacing
``*_WINDOW_SCISSOR``.
Fragment density map offset
---------------------------
In order to "properly" implement ``VK_EXT_fragment_density_map_offset``, we
need to add an extra row/column of bins at the end and then shift the binning
grid up and to the left by an offset :math:`b_o`. This offset is based on the
user's offset but has the opposite sign, i.e. when shifting the FDM to the left
we have to shift the binning grid to the right, and once the user's offset
becomes large enough then we "wrap around" and shift over the scaling factor
:math:`s` to the next bin. This has to happen per-view. In turnip the function
that computes :math:`b_o` is called ``tu_bin_offset``. Each tile then gets an
offseted start :math:`b_s = b_{cs} - b_o` except for the first row/column which
only shrink in height/width respectively.
If we cannot make :math:`s` per-view, then we also cannot make :math:`b_s`
per-view and so we cannot shift the bins over. Therefore we fall back to only
shifting where :math:`s` is sampled from, which produces jittery and jarring
transitions when a bin suddenly changes resolution.
Bin merging
-----------
FDM shrinks the size of the bin in GMEM, which results in a lot of wasteful
unused extra space in GMEM. a7xx mitigates this by introducing "bin merging".
If two tiles next to each other have the same scaling for each view, then we
combine them into one tile, as long as the combined size in rendering space
isn't larger than the original size of an unscaled bin in framebuffer space. We
can even merge larger groups of tiles. The only hardware feature needed for
this to work is the ability to merge the visibility streams for the tiles,
which was added on a7xx by a new bitmask in ``CP_SET_BIN_DATA5`` and variants.
Only bins within the same visibility stream/VSC pipe can be merged.
Hardware scaling registers and LRZ
----------------------------------
One disadvantage of FDM on a6xx is that low-resolution tiles cannot use
LRZ, because the LRZ hardware is not aware of the transform between framebuffer
space and rendering space and applies the framebuffer-space LRZ values to the
rendering-space fragments. In order to fix this, a740 adds new offset and scale
registers. The offset :math:`o'` is applied to fragment coordinates during
rasterization *after* LRZ, so that viewport, scissor, and LRZ are in a
new "LRZ space" while the other operations (resolves and unresolves, and
attachment writes) still happen in the rendering space which is now offset.
:math:`o'` is specified for each layer. The scale :math:`s` is the same as
before, and it is used to multiply the fragment area covered by each LRZ pixel.
Without ``VK_EXT_fragment_density_map_offset``, we can simply make LRZ space
equal to framebuffer space scaled down by :math:`s`. That is, we can set
:math:`o'` to what :math:`o` was before and then set :math:`o` to 0, only
scaling down the viewport but not shifting it and letting the hardware handle
the shift. Then LRZ pixels will be scaled up appropriately and everything will
work. However, this doesn't work if there is a bin offset :math:`b_o`. In order
to make binning work, we shift the viewport and scissor by :math:`b_o` when
binning. Unfortunately the offset registers do not have any effect when
binning, so rendering space and LRZ space have to be the same when binning, and
the visibility stream is generated from rendering space. This means that LRZ
space also has to be shifted over compared to framebuffer space, and the LRZ
buffer must be overallocated when FDM offset might be used with it (which is
signalled by ``VK_IMAGE_CREATE_FRAGMENT_DENSITY_MAP_OFFSET_BIT_EXT``) because
the LRZ image will be shifted by :math:`b_o`.
In order for LRZ to work, LRZ space when rendering must be equal to LRZ space
when binning scaled down by :math:`s`. The origin of LRZ space when binning is
:math:`-b_o`, and this must be mapped to 0. The transform from
framebuffer space to LRZ space is :math:`x' = x * s + o`, and the transform
from framebuffer space to rendering space is :math:`x'' = x * s + o + o'`.
We get that :math:`o + o' = b_{cs} - b_s * s`, similar to before, and
:math:`0 = -b_o * s + o` so that :math:`o = b_o * s` and finally
:math:`o' = b_{cs} - b_s * s - b_o * s`, or after rearranging
:math:`o' = b_{cs} - (b_s + b_o) * s`. For all tiles except those in the first
row or column, this simplifies to :math:`o' = b_{cs} - b_{cs} * s` because
:math:`b_{cs} = b_s + b_o`. For tiles in the first row or column, :math:`b_s`
and :math:`b_{cs}` are both 0 in one of the coordinates, so it becomes
:math:`o' = -b_o * s` in that coordinate. This isn't representable in hardware,
both because it is negative (which can be worked around by artifically
shifting :math:`b_{cs}`) but more importantly because it may not meet the
alignment requirements for the hardware register (which is currently 8 pixels).
We have to just disable LRZ in this case.
docs/drivers/freedreno/hw/lrz.rst
+153
View File
@@ -0,0 +1,153 @@
Low Resolution Z Buffer
=======================
This doc is based on a6xx HW reverse engineering, a5xx should be similar to
a6xx before gen3.
Low Resolution Z buffer is very similar to a depth prepass that helps
the HW to avoid executing the fragment shader on those fragments that will
be subsequently discarded by the depth test afterwards.
The interesting part of this feature is that it allows applications
to submit the vertices in any order.
Citing official Adreno documentation:
::
[A Low Resolution Z (LRZ)] pass is also referred to as draw order independent
depth rejection. During the binning pass, a low resolution Z-buffer is constructed,
and can reject LRZ-tile wide contributions to boost binning performance. This LRZ
is then used during the rendering pass to reject pixels efficiently before testing
against the full resolution Z-buffer.
Limitations
-----------
There are two main limitations of LRZ:
- Since LRZ is an early depth test, such test cannot be used when late-z is required;
- LRZ buffer could be formed only in one direction, changing depth comparison directions
without disabling LRZ would lead to a malformed LRZ buffer.
Pre-a650 (before gen3)
----------------------
The direction is fully tracked on CPU. In render pass LRZ starts with
unknown direction, the direction is set first time when depth write occurs
and if it does change afterwards then the direction becomes invalid and LRZ is
disabled for the rest of the render pass.
Since the direction is not tracked by the GPU, it's impossible to know whether
LRZ is enabled during construction of secondary command buffers.
For the same reason, it's impossible to reuse LRZ between render passes.
A650+ (gen3+)
-------------
Now LRZ direction can be tracked on GPU. There are two parts:
- Direction byte which stores current LRZ direction - ``GRAS_LRZ_CNTL.DIR``.
- Parameters of the last used depth view - ``GRAS_LRZ_DEPTH_VIEW``.
The idea is the same as when LRZ tracked on CPU: when ``GRAS_LRZ_CNTL``
is used, its direction is compared to the previously known direction
and direction byte is set to disabled when directions are incompatible.
Additionally, to reuse LRZ between render passes, ``GRAS_LRZ_CNTL`` checks
if the current value of ``GRAS_LRZ_DEPTH_VIEW`` is equal to the value
stored in the buffer. If not, LRZ is disabled. This is necessary
because depth buffer may have several layers and mip levels, while the
LRZ buffer represents only a single layer + mip level.
A7XX
-------------
A7XX introduces the concept of bidirectional LRZ where there are two LRZ
buffers, one for each direction. This way LRZ doesn't need to be disabled
when the direction changes, by default, this behavior is disabled but the
LRZ buffers have to be allocated with this space in mind as fast clears
will always write metadata for both.
Additionally, there are now two seperate LRZ buffers (on top of one for
each direction, a total of four) - due to concurrent binning, one can be
used for binning and the other for rendering concurrently. These can be
flipped between via the `LRZ_FLIP_BUFFER` event which can be put inside
a conditional block for either the BV or BR.
LRZ Fast-Clear
--------------
The LRZ fast-clear buffer is initialized to zeroes and read/written
when ``GRAS_LRZ_CNTL.FC_ENABLE`` is set. It appears to store 1b/block.
``0`` means block has original depth clear value, and ``1`` means that the
corresponding block in LRZ has been modified.
LRZ fast-clear conservatively clears LRZ buffer. At the point where LRZ is
written the LRZ block which corresponds to a single fast-clear bit is cleared:
- To ``0.0`` if depth comparison is ``GREATER``
- To ``1.0`` if depth comparison is ``LESS``
This way it's always valid to fast-clear.
On A7XX, the original depth clear value can be specified exactly allowing for
fast-clear to any value rather than just ``1.0`` or ``0.0``.
LRZ Feedback
-------------
Some draws do write depth but cannot contribute to LRZ during the BINNING pass
e.g. when fragment shader has "discard" in it, however they can contribute to LRZ
during the RENDERING pass via LRZ feedback mechanism. This may allow the draws
that follow to depth test against the updated LRZ, this is especially important
if such "bad" draws were at the start of the renderpass.
LRZ feedback happens during the RENDERING pass when ``LRZ_FEEDBACK_ZMODE_MASK``
is set, if draw has a6xx_ztest_mode that has corresponding flag set in
``LRZ_FEEDBACK_ZMODE_MASK`` - its depth values would be used for feedback.
LRZ feedback alongside with LRZ testing also works during sysmem rendering.
LRZ Precision
-------------
LRZ always uses ``Z16_UNORM``. The epsilon for it is ``1.f / (1 << 16)`` which is
not enough to represent all values of ``Z32_UNORM`` or ``Z32_FLOAT``.
This especially raises questions in context of fast-clear, if fast-clear
uses a value which cannot be precisely represented by LRZ - we wouldn't
be able to round it in the correct direction since direction is tracked
on GPU.
However, it seems that depth comparisons with LRZ values have some "slack"
and nothing special should be done for such depth clear values.
How it was tested:
- Clear ``Z32_FLOAT`` attachment to ``1.f / (1 << 17)``
- LRZ buffer contains all zeroes.
- Do draws and check whether all samples are passing:
- ``OP_GREATER`` with ``(1.f / (1 << 17) + float32_epsilon)`` - passing;
- ``OP_GREATER`` with ``(1.f / (1 << 17) - float32_epsilon)`` - not passing;
- ``OP_LESS`` with ``(1.f / (1 << 17) - float32_epsilon)`` - samples;
- ``OP_LESS`` with ``(1.f / (1 << 17) + float32_epsilon)``- not passing;
- ``OP_LESS_OR_EQ`` with ``(1.f / (1 << 17) + float32_epsilon)`` - not passing.
In all cases resulting LRZ buffer is all zeroes and LRZ direction is updated.
LRZ Caches
----------
``LRZ_FLUSH`` flushes and invalidates LRZ caches, there are two caches:
- Cache for fast-clear buffer;
- Cache for direction byte + depth view params.
They could be cleared by ``LRZ_CLEAR``. To become visible in GPU memory
the caches should be flushed with ``LRZ_FLUSH`` afterwards.
``GRAS_LRZ_CNTL`` reads from these caches.
docs/drivers/freedreno/ir3-notes.rst
+403
View File
@@ -0,0 +1,403 @@
IR3 NOTES
=========
Some notes about ir3, the compiler and machine-specific IR for the shader ISA introduced with Adreno 3xx. The same shader ISA is present, with some small differences, in Adreno 4xx.
Compared to the previous generation a2xx ISA (ir2), the a3xx ISA is a "simple" scalar instruction set. However, the compiler is responsible, in most cases, to schedule the instructions. The hardware does not try to hide the shader core pipeline stages. For a common example, a common (cat2) ALU instruction takes four cycles, so a subsequent cat2 instruction which uses the result must have three intervening instructions (or NOPs). When operating on vec4's, typically the corresponding scalar instructions for operating on the remaining three components could typically fit. Although that results in a lot of edge cases where things fall over, like:
::
ADD TEMP[0], TEMP[1], TEMP[2]
MUL TEMP[0], TEMP[1], TEMP[0].wzyx
Here, the second instruction needs the output of the first group of scalar instructions in the wrong order, resulting in not enough instruction spots between the ``add r0.w, r1.w, r2.w`` and ``mul r0.x, r1.x, r0.w``. Which is why the original (old) compiler which merely translated nearly literally from TGSI to ir3, had a strong tendency to fall over.
So the current compiler instead, in the frontend, generates a directed-acyclic-graph of instructions and basic blocks, which go through various additional passes to eventually schedule and do register assignment.
For additional documentation about the hardware, see wiki: `a3xx ISA
<https://github.com/freedreno/freedreno/wiki/A3xx-shader-instruction-set-architecture>`__.
External Structure
------------------
``ir3_shader``
A single vertex/fragment/etc shader from gallium perspective (i.e.
maps to a single TGSI shader), and manages a set of shader variants
which are generated on demand based on the shader key.
``ir3_shader_key``
The configuration key that identifies a shader variant. I.e. based
on other GL state (two-sided-color, render-to-alpha, etc) or render
stages (binning-pass vertex shader) different shader variants are
generated.
``ir3_shader_variant``
The actual HW shader generated based on input TGSI and shader key.
``ir3_compiler``
Compiler frontend which generates ir3 and runs the various backend
stages to schedule and do register assignment.
The IR
------
The ir3 IR maps quite directly to the hardware, in that instruction opcodes map directly to hardware opcodes, and that dst/src register(s) map directly to the hardware dst/src register(s). But there are a few extensions, in the form of meta_ instructions. And additionally, for normal (non-const, etc) src registers, the ``IR3_REG_SSA`` flag is set and ``reg->instr`` points to the source instruction which produced that value. So, for example, the following TGSI shader:
::
VERT
DCL IN[0]
DCL IN[1]
DCL OUT[0], POSITION
DCL TEMP[0], LOCAL
1: DP3 TEMP[0].x, IN[0].xyzz, IN[1].xyzz
2: MOV OUT[0], TEMP[0].xxxx
3: END
eventually generates:
.. graphviz::
digraph G {
rankdir=RL;
nodesep=0.25;
ranksep=1.5;
subgraph clusterdce198 {
label="vert";
inputdce198 [shape=record,label="inputs|<in0> i0.x|<in1> i0.y|<in2> i0.z|<in4> i1.x|<in5> i1.y|<in6> i1.z"];
instrdcf348 [shape=record,style=filled,fillcolor=lightgrey,label="{mov.f32f32|<dst0>|<src0> }"];
instrdcedd0 [shape=record,style=filled,fillcolor=lightgrey,label="{mad.f32|<dst0>|<src0> |<src1> |<src2> }"];
inputdce198:<in2>:w -> instrdcedd0:<src0>
inputdce198:<in6>:w -> instrdcedd0:<src1>
instrdcec30 [shape=record,style=filled,fillcolor=lightgrey,label="{mad.f32|<dst0>|<src0> |<src1> |<src2> }"];
inputdce198:<in1>:w -> instrdcec30:<src0>
inputdce198:<in5>:w -> instrdcec30:<src1>
instrdceb60 [shape=record,style=filled,fillcolor=lightgrey,label="{mul.f|<dst0>|<src0> |<src1> }"];
inputdce198:<in0>:w -> instrdceb60:<src0>
inputdce198:<in4>:w -> instrdceb60:<src1>
instrdceb60:<dst0> -> instrdcec30:<src2>
instrdcec30:<dst0> -> instrdcedd0:<src2>
instrdcedd0:<dst0> -> instrdcf348:<src0>
instrdcf400 [shape=record,style=filled,fillcolor=lightgrey,label="{mov.f32f32|<dst0>|<src0> }"];
instrdcedd0:<dst0> -> instrdcf400:<src0>
instrdcf4b8 [shape=record,style=filled,fillcolor=lightgrey,label="{mov.f32f32|<dst0>|<src0> }"];
instrdcedd0:<dst0> -> instrdcf4b8:<src0>
outputdce198 [shape=record,label="outputs|<out0> o0.x|<out1> o0.y|<out2> o0.z|<out3> o0.w"];
instrdcf348:<dst0> -> outputdce198:<out0>:e
instrdcf400:<dst0> -> outputdce198:<out1>:e
instrdcf4b8:<dst0> -> outputdce198:<out2>:e
instrdcedd0:<dst0> -> outputdce198:<out3>:e
}
}
(after scheduling, etc, but before register assignment).
Internal Structure
~~~~~~~~~~~~~~~~~~
``ir3_block``
Represents a basic block.
TODO: currently blocks are nested, but I think I need to change that
to a more conventional arrangement before implementing proper flow
control. Currently the only flow control handles is if/else which
gets flattened out and results chosen with ``sel`` instructions.
``ir3_instruction``
Represents a machine instruction or meta_ instruction. Has pointers
to dst register (``regs[0]``) and src register(s) (``regs[1..n]``),
as needed.
``ir3_register``
Represents a src or dst register, flags indicate const/relative/etc.
If ``IR3_REG_SSA`` is set on a src register, the actual register
number (name) has not been assigned yet, and instead the ``instr``
field points to src instruction.
In addition there are various util macros/functions to simplify manipulation/traversal of the graph:
``foreach_src(srcreg, instr)``
Iterate each instruction's source ``ir3_register``\s
``foreach_src_n(srcreg, n, instr)``
Like ``foreach_src``, also setting ``n`` to the source number (starting
with ``0``).
``foreach_ssa_src(srcinstr, instr)``
Iterate each instruction's SSA source ``ir3_instruction``\s. This skips
non-SSA sources (consts, etc), but includes virtual sources (such as the
address register if `relative addressing`_ is used).
``foreach_ssa_src_n(srcinstr, n, instr)``
Like ``foreach_ssa_src``, also setting ``n`` to the source number.
For example:
.. code-block:: c
foreach_ssa_src_n(src, i, instr) {
unsigned d = delay_calc_srcn(ctx, src, instr, i);
delay = MAX2(delay, d);
}
TODO probably other helper/util stuff worth mentioning here
.. _meta:
Meta Instructions
~~~~~~~~~~~~~~~~~
**input**
Used for shader inputs (registers configured in the command-stream
to hold particular input values, written by the shader core before
start of execution. Also used for connecting up values within a
basic block to an output of a previous block.
**output**
Used to hold outputs of a basic block.
**flow**
TODO
**phi**
TODO
**collect**
Groups registers which need to be assigned to consecutive scalar
registers, for example ``sam`` (texture fetch) src instructions (see
`register groups`_) or array element dereference
(see `relative addressing`_).
**split**
The counterpart to **collect**, when an instruction such as ``sam``
writes multiple components, splits the result into individual
scalar components to be consumed by other instructions.
.. _`flow control`:
Flow Control
~~~~~~~~~~~~
TODO
.. _`register groups`:
Register Groups
~~~~~~~~~~~~~~~
Certain instructions, such as texture sample instructions, consume multiple consecutive scalar registers via a single src register encoded in the instruction, and/or write multiple consecutive scalar registers. In the simplest example:
::
sam (f32)(xyz)r2.x, r0.z, s#0, t#0
for a 2d texture, would read ``r0.zw`` to get the coordinate, and write ``r2.xyz``.
Before register assignment, to group the two components of the texture src together:
.. graphviz::
digraph G {
{ rank=same;
collect;
};
{ rank=same;
coord_x;
coord_y;
};
sam -> collect [label="regs[1]"];
collect -> coord_x [label="regs[1]"];
collect -> coord_y [label="regs[2]"];
coord_x -> coord_y [label="right",style=dotted];
coord_y -> coord_x [label="left",style=dotted];
coord_x [label="coord.x"];
coord_y [label="coord.y"];
}
The frontend sets up the SSA ptrs from ``sam`` source register to the ``collect`` meta instruction, which in turn points to the instructions producing the ``coord.x`` and ``coord.y`` values. And the grouping_ pass sets up the ``left`` and ``right`` neighbor pointers to the ``collect``\'s sources, used later by the `register assignment`_ pass to assign blocks of scalar registers.
And likewise, for the consecutive scalar registers for the destination:
.. graphviz::
digraph {
{ rank=same;
A;
B;
C;
};
{ rank=same;
split_0;
split_1;
split_2;
};
A -> split_0;
B -> split_1;
C -> split_2;
split_0 [label="split\noff=0"];
split_0 -> sam;
split_1 [label="split\noff=1"];
split_1 -> sam;
split_2 [label="split\noff=2"];
split_2 -> sam;
split_0 -> split_1 [label="right",style=dotted];
split_1 -> split_0 [label="left",style=dotted];
split_1 -> split_2 [label="right",style=dotted];
split_2 -> split_1 [label="left",style=dotted];
sam;
}
.. _`relative addressing`:
Relative Addressing
~~~~~~~~~~~~~~~~~~~
Most instructions support addressing indirectly (relative to address register) into const or gpr register file in some or all of their src/dst registers. In this case the register accessed is taken from ``r<a0.x + n>`` or ``c<a0.x + n>``, i.e. address register (``a0.x``) value plus ``n``, where ``n`` is encoded in the instruction (rather than the absolute register number).
Note that cat5 (texture sample) instructions are the notable exception, not
supporting relative addressing of src or dst.
Relative addressing of the const file (for example, a uniform array) is relatively simple. We don't do register assignment of the const file, so all that is required is to schedule things properly. I.e. the instruction that writes the address register must be scheduled first, and we cannot have two different address register values live at one time.
But relative addressing of gpr file (which can be as src or dst) has additional restrictions on register assignment (i.e. the array elements must be assigned to consecutive scalar registers). And in the case of relative dst, subsequent instructions now depend on both the relative write, as well as the previous instruction which wrote that register, since we do not know at compile time which actual register was written.
Each instruction has an optional ``address`` pointer, to capture the dependency on the address register value when relative addressing is used for any of the src/dst register(s). This behaves as an additional virtual src register, i.e. ``foreach_ssa_src()`` will also iterate the address register (last).
Note that ``nop``\'s for timing constraints, type specifiers (i.e.
``add.f`` vs ``add.u``), etc, omitted for brevity in examples
::
mova a0.x, hr1.y
sub r1.y, r2.x, r3.x
add r0.x, r1.y, c<a0.x + 2>
results in:
.. graphviz::
digraph {
rankdir=LR;
sub;
const [label="const file"];
add;
mova;
add -> mova;
add -> sub;
add -> const [label="off=2"];
}
The scheduling pass has some smarts to schedule things such that only a single ``a0.x`` value is used at any one time.
To implement variable arrays, the NIR registers are stored as an ``ir3_array``,
which will be register allocated to consecutive hardware registers. The array
access uses the id field in the ``ir3_register`` to map to the array being
accessed, and the offset field for the fixed offset within the array. A NIR
indirect register read such as:
::
decl_reg vec2 32 r0[2]
...
vec2 32 ssa_19 = mov r0[0 + ssa_9]
results in:
::
0000:0000:001: shl.b hssa_19, hssa_17, himm[0.000000,1,0x1]
0000:0000:002: mov.s16s16 hr61.x, hssa_19
0000:0000:002: mov.u32u32 ssa_21, arr[id=1, offset=0, size=4, ssa_12], address=_[0000:0000:002: mov.s16s16]
0000:0000:002: mov.u32u32 ssa_22, arr[id=1, offset=1, size=4, ssa_12], address=_[0000:0000:002: mov.s16s16]
Array writes write to the array in ``instr->regs[0]->array.id``. A NIR indirect
register write such as:
::
decl_reg vec2 32 r0[2]
...
r0[0 + ssa_12] = mov ssa_13
results in:
::
0000:0000:001: shl.b hssa_29, hssa_27, himm[0.000000,1,0x1]
0000:0000:002: mov.s16s16 hr61.x, hssa_29
0000:0000:001: mov.u32u32 arr[id=1, offset=0, size=4, ssa_17], c2.y, address=_[0000:0000:002: mov.s16s16]
0000:0000:004: mov.u32u32 arr[id=1, offset=1, size=4, ssa_31], c2.z, address=_[0000:0000:002: mov.s16s16]
Note that only cat1 (mov) can do indirect write, and thus NIR register stores
may need to introduce an extra mov.
ir3 array accesses in the DAG get serialized by the ``instr->barrier_class`` and
containing ``IR3_BARRIER_ARRAY_W`` or ``IR3_BARRIER_ARRAY_R``.
Shader Passes
-------------
After the frontend has generated the use-def graph of instructions, they are run through various passes which include scheduling_ and `register assignment`_. Because inserting ``mov`` instructions after scheduling would also require inserting additional ``nop`` instructions (since it is too late to reschedule to try and fill the bubbles), the earlier stages try to ensure that (at least given an infinite supply of registers) that `register assignment`_ after scheduling_ cannot fail.
Note that we essentially have ~256 scalar registers in the
architecture (although larger register usage will at some thresholds
limit the number of threads which can run in parallel). And at some
point we will have to deal with spilling.
.. _flatten:
Flatten
~~~~~~~
In this stage, simple if/else blocks are flattened into a single block with ``phi`` nodes converted into ``sel`` instructions. The a3xx ISA has very few predicated instructions, and we would prefer not to use branches for simple if/else.
.. _`copy propagation`:
Copy Propagation
~~~~~~~~~~~~~~~~
Currently the frontend inserts ``mov``\s in various cases, because certain categories of instructions have limitations about const regs as sources. And the CP pass simply removes all simple ``mov``\s (i.e. src-type is same as dst-type, no abs/neg flags, etc).
The eventual plan is to invert that, with the front-end inserting no ``mov``\s and CP legalize things.
.. _grouping:
Grouping
~~~~~~~~
In the grouping pass, instructions which need to be grouped (for ``collect``\s, etc) have their ``left`` / ``right`` neighbor pointers setup. In cases where there is a conflict (i.e. one instruction cannot have two unique left or right neighbors), an additional ``mov`` instruction is inserted. This ensures that there is some possible valid `register assignment`_ at the later stages.
.. _depth:
Depth
~~~~~
In the depth pass, a depth is calculated for each instruction node within its basic block. The depth is the sum of the required cycles (delay slots needed between two instructions plus one) of each instruction plus the max depth of any of its source instructions. (meta_ instructions don't add to the depth). As an instruction's depth is calculated, it is inserted into a per block list sorted by deepest instruction. Unreachable instructions and inputs are marked.
TODO: we should probably calculate both hard and soft depths (?) to
try to coax additional instructions to fit in places where we need
to use sync bits, such as after a texture fetch or SFU.
.. _scheduling:
Scheduling
~~~~~~~~~~
After the grouping_ pass, there are no more instructions to insert or remove. Start scheduling each basic block from the deepest node in the depth sorted list created by the depth_ pass, recursively trying to schedule each instruction after its source instructions plus delay slots. Insert ``nop``\s as required.
.. _`register assignment`:
Register Assignment
~~~~~~~~~~~~~~~~~~~
TODO
docs/drivers/freedreno/subsampled_annotated.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.6 MiB

docs/drivers/freedreno/subsampled_final.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.9 MiB