init

docs/ci/LAVA.rst

@@ -0,0 +1,129 @@
+LAVA CI
+=======
+
+`LAVA <https://www.lavasoftware.org/>`__ is a system for functional
+testing of boards including deploying custom bootloaders and kernels.
+This is particularly relevant to testing Mesa because we often need
+to change kernels for UAPI changes (and this lets us do full testing
+of a new kernel during development), and our workloads can easily
+take down boards when mistakes are made (kernel oopses, OOMs that
+take out critical system services).
+
+Available LAVA labs
+-------------------
+- Collabora `[dashboard] <https://lava.collabora.dev/scheduler/device_types>`__ (without authentication only health check jobs are displayed)
+- Lima [dashboard not available]
+
+Mesa-LAVA software architecture
+-------------------------------
+
+The gitlab-runner will run on some host that has access to the LAVA
+lab, with tags like "mesa-ci-x86-64-lava-$DEVICE_TYPE" to control only
+taking in jobs for the hardware that the LAVA lab contains.  The
+gitlab-runner spawns a Docker container with lavacli in it, and
+connects to the LAVA lab using a predefined token to submit jobs under
+a specific device type.
+
+The LAVA instance manages scheduling those jobs to the boards present.
+For a job, it will deploy the kernel, device tree, and the ramdisk
+containing the CTS.
+
+Deploying a new Mesa-LAVA lab
+-----------------------------
+
+You'll want to start with setting up your LAVA instance and getting
+some boards booting using test jobs.  Start with the stock QEMU
+examples to make sure your instance works at all.  Then, you'll need
+to define your actual boards.
+
+The device type in lava-gitlab-ci.yml is the device type you create in
+your LAVA instance, which doesn't have to match the board's name in
+``/etc/lava-dispatcher/device-types``.  You create your boards under
+that device type and the Mesa jobs will be scheduled to any of them.
+Instantiate your boards by creating them in the UI or at the command
+line attached to that device type, then populate their dictionary
+(using an "extends" line probably referencing the board's template in
+``/etc/lava-dispatcher/device-types``).  Now, go find a relevant
+health check job for your board as a test job definition, or cobble
+something together from a board that boots using the same boot_method
+and some public images, and figure out how to get your boards booting.
+
+Once you can boot your board using a custom job definition, it's time
+to connect Mesa CI to it.  Install gitlab-runner and register as a
+shared runner (you'll need a GitLab admin for help with this).  The
+runner *must* have a tag (like "mesa-ci-x86-64-lava-rk3399-gru-kevin")
+to restrict the jobs it takes or it will grab random jobs from tasks
+across ``gitlab.freedesktop.org``, and your runner isn't ready for
+that.
+
+The Docker image will need access to the LAVA instance.  If it's on a
+public network it should be fine.  If you're running the LAVA instance
+on localhost, you'll need to set ``network_mode="host"`` in
+``/etc/gitlab-runner/config.toml`` so it can access localhost.  Create a
+gitlab-runner user in your LAVA instance, log in under that user on
+the web interface, and create an API token.  Copy that into a
+``lavacli.yaml``:
+
+.. code-block:: yaml
+
+   default:
+      token: <token contents>
+      uri: <URL to the instance>
+      username: gitlab-runner
+
+Add a volume mount of that ``lavacli.yaml`` to
+``/etc/gitlab-runner/config.toml`` so that the Docker container can
+access it.  You probably have a ``volumes = ["/cache"]`` already, so now it would be::
+
+   volumes = ["/home/anholt/lava-config/lavacli.yaml:/root/.config/lavacli.yaml", "/cache"]
+
+Note that this token is visible to anybody that can submit MRs to
+Mesa!  It is not an actual secret.  We could just bake it into the
+GitLab CI YAML, but this way the current method of connecting to the
+LAVA instance is separated from the Mesa branches (particularly
+relevant as we have many stable branches all using CI).
+
+Now it's time to define your test jobs in the driver-specific
+gitlab-ci.yml file, using the device-specific tags.
+
+Caching downloads
+-----------------
+
+To improve the runtime for downloading traces during traces job runs, you will
+want a pass-through HTTP cache.  On your runner box, install nginx:
+
+.. code-block:: sh
+
+   sudo apt install nginx libnginx-mod-http-lua
+
+Add the server setup files:
+
+.. literalinclude:: fdo-cache
+   :name: /etc/nginx/sites-available/fdo-cache
+   :caption: /etc/nginx/sites-available/fdo-cache
+
+.. literalinclude:: uri-caching.conf
+   :name: /etc/nginx/snippets/uri-caching.conf
+   :caption: /etc/nginx/snippets/uri-caching.conf
+
+Edit the listener addresses in fdo-cache to suit the ethernet interface that
+your devices are on.
+
+Enable the site and restart nginx:
+
+.. code-block:: sh
+
+   sudo rm /etc/nginx/sites-enabled/default
+   sudo ln -s /etc/nginx/sites-available/fdo-cache /etc/nginx/sites-enabled/fdo-cache
+   sudo systemctl restart nginx
+
+   # First download will hit the internet
+   wget http://localhost/cache/?uri=https://s3.freedesktop.org/mesa-tracie-public/itoral-gl-terrain-demo/demo-v2.trace
+   # Second download should be cached.
+   wget http://localhost/cache/?uri=https://s3.freedesktop.org/mesa-tracie-public/itoral-gl-terrain-demo/demo-v2.trace
+
+The trace runner script automatically sets the caching proxy, so there's no
+need to modify anything in the Mesa CI YAML files.
+Add ``LAVA_HTTP_CACHE_URI=http://localhost/cache/?uri=`` to your ``config.toml``
+runner environment lines and you can use it for cached artifact downloads
+instead of going all the way to freedesktop.org on each job.

docs/ci/bare-metal.rst

@@ -0,0 +1,10 @@
+Bare-metal CI
+=============
+
+Bare-metal support is being removed from Mesa, and adding new devices is
+no longer supported.
+
+Please consider using one of the following alternatives:
+
+- `CI-tron <https://docs.ci-tron.dev>`__
+- :doc:`LAVA`

docs/ci/docker.rst

@@ -0,0 +1,74 @@
+Docker CI
+=========
+
+For LLVMpipe and Softpipe CI, we run tests in a container containing
+VK-GL-CTS, on the shared GitLab runners provided by `freedesktop
+<https://www.freedesktop.org>`__
+
+Software architecture
+---------------------
+
+The Docker containers are rebuilt using the shell scripts under
+.gitlab-ci/container/ when the FDO\_DISTRIBUTION\_TAG changes in
+.gitlab-ci.yml. The resulting images are around 1 GB, and are
+expected to change approximately weekly (though an individual
+developer working on them may produce many more images while trying to
+come up with a working MR!).
+
+gitlab-runner is a client that polls gitlab.freedesktop.org for
+available jobs, with no inbound networking requirements.  Jobs can
+have tags, so we can have DUT-specific jobs that only run on runners
+with that tag marked in the GitLab UI.
+
+Since dEQP takes a long time to run, we mark the job as "parallel" at
+some level, which spawns multiple jobs from one definition, and then
+deqp-runner.sh takes the corresponding fraction of the test list for
+that job.
+
+To reduce dEQP runtime (or avoid tests with unreliable results), a
+deqp-runner.sh invocation can provide a list of tests to skip.  If
+your driver is not yet conformant, you can pass a list of expected
+failures, and the job will only fail on tests that aren't listed (look
+at the job's log for which specific tests failed).
+
+DUT requirements
+----------------
+
+In addition to the general :ref:`CI-job-user-expectations`, using
+Docker requires:
+
+* DUTs must have a stable kernel and GPU reset (if applicable).
+
+If the system goes down during a test run, that job will eventually
+time out and fail (default 1 hour).  However, if the kernel can't
+reliably reset the GPU on failure, bugs in one MR may leak into
+spurious failures in another MR.  This would be an unacceptable impact
+on Mesa developers working on other drivers.
+
+* DUTs must be able to run Docker
+
+The Mesa gitlab-runner based test architecture is built around Docker,
+so that we can cache the Debian package installation and CTS build
+step across multiple test runs.  Since the images are large and change
+approximately weekly, the DUTs also need to be running some script to
+prune stale Docker images periodically in order to not run out of disk
+space as we rev those containers (perhaps `this script
+<https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2980#note_169233611>`__).
+
+Note that Docker doesn't allow containers to be stored on NFS, and
+doesn't allow multiple Docker daemons to interact with the same
+network block device, so you will probably need some sort of physical
+storage on your DUTs.
+
+* DUTs must be public
+
+By including your device in .gitlab-ci.yml, you're effectively letting
+anyone on the internet run code on your device.  Docker containers may
+provide some limited protection, but how much you trust that and what
+you do to mitigate hostile access is up to you.
+
+* DUTs must expose the DRI device nodes to the containers.
+
+Obviously, to get access to the HW, we need to pass the render node
+through.  This is done by adding ``devices = ["/dev/dri"]`` to the
+``runners.docker`` section of /etc/gitlab-runner/config.toml.

docs/ci/fdo-cache

@@ -0,0 +1,92 @@
+proxy_cache_path /var/cache/nginx/ levels=1:2 keys_zone=my_cache:10m max_size=50g inactive=2w use_temp_path=off;
+
+server {
+	listen 10.42.0.1:80 default_server;
+	listen 127.0.0.1:80 default_server;
+	listen [::]:80 default_server;
+	resolver 8.8.8.8;
+
+	root /var/www/html;
+
+	# Add index.php to the list if you are using PHP
+	index index.html index.htm index.nginx-debian.html;
+
+	server_name _;
+
+	location / {
+		# First attempt to serve request as file, then
+		# as directory, then fall back to displaying a 404.
+		try_files $uri $uri/ =404;
+	}
+
+	location /tmp {
+		# Lava server http artifacts to the clients; e.g. for the deploy action
+		alias /var/lib/lava/dispatcher/tmp;
+	}
+
+	proxy_cache my_cache;
+
+	# Wait for the cache creation when multiple query are done for the same file
+	proxy_cache_lock on;
+	proxy_cache_lock_age 30m;
+	proxy_cache_lock_timeout 1h;
+
+	location /force_cache {
+		internal;
+		# On some setups the cache headers will indicate to nginx that the
+		# artifacts shouldn't be cached, however if we know that that is not valid
+		# for lava usage this endpoint allows caching to be forced instead
+		proxy_cache_valid 200 48h;
+		proxy_ignore_headers Cache-Control Set-Cookie expires;
+		include snippets/uri-caching.conf;
+	}
+
+	location /fdo_cache {
+		internal;
+		# As the auth information in the query is being dropped, use
+		# the minimal possible cache validity, such that in practise
+		# every requests gets revalidated. This avoids
+		# unauthenticated downloads from our cache as the cache key doesn't
+		# include auth info
+		proxy_cache_valid 200 1s;
+		proxy_cache_revalidate on;
+		proxy_ignore_headers Cache-Control Set-Cookie expires;
+		set_by_lua_block $cache_key {
+			-- Set the cache key to the uri with the query stripped
+			local unescaped =  ngx.unescape_uri(ngx.var.arg_uri);
+			local it,err = ngx.re.match(unescaped, "([^?]*).*")
+			if not it then
+				-- Fallback on the full uri as key if the regexp fails
+				return ngx.var.arg_uri;
+			end
+			return it[1]
+		}
+		proxy_cache_key $cache_key;
+		include snippets/uri-caching.conf;
+	}
+
+	location /cache {
+		# Gitlabs http server puts everything as no-cache even though
+		# the artifacts URLS don't change.
+		if ($arg_uri ~*  /.*gitlab.*artifacts(\/|%2F)raw/ ) {
+			rewrite ^ /force_cache;
+		}
+
+		# fd.o's object storage has an embedded signature for
+		# authentication as part of its query. So use an adjusted cache key
+		# without the query
+		if ($arg_uri ~*  .*your-objectstorage.com(\/|%2F)fdo-opa(\/|%2F)) {
+			rewrite ^ /fdo_cache;
+		}
+
+		# Set a really low validity together with cache revalidation; Our goal
+		# for caching isn't to lower the number of http requests but to
+		# lower the amount of data transfer. Also for some test
+		# scenarios (typical manual tests) the file at a given url
+		# might get modified so avoid confusion by ensuring
+		# revalidations happens often.
+		proxy_cache_valid 200 10s;
+		proxy_cache_revalidate on;
+		include snippets/uri-caching.conf;
+	}
+}

docs/ci/index.rst

@@ -0,0 +1,410 @@
+Continuous Integration
+======================
+
+GitLab CI
+---------
+
+GitLab provides a convenient framework for running commands in response to Git pushes.
+We use it to test merge requests (MRs) before merging them (pre-merge testing),
+as well as post-merge testing, for everything that hits ``main``
+(this is necessary because we still allow commits to be pushed outside of MRs,
+and even then the MR CI runs in the forked repository, which might have been
+modified and thus is unreliable).
+
+The CI runs a number of tests, from trivial build-testing to complex GPU rendering:
+
+- Build testing for a number of configurations and platforms
+- Sanity checks (``meson test``)
+- Most drivers are also tested using several test suites, such as the
+  `Vulkan/GL/GLES conformance test suite <https://github.com/KhronosGroup/VK-GL-CTS>`__,
+  `Piglit <https://gitlab.freedesktop.org/mesa/piglit>`__, and others.
+- Replay of application traces
+
+A typical run takes between 20 and 30 minutes, although it can go up very quickly
+if the GitLab runners are overwhelmed, which happens sometimes. When it does happen,
+not much can be done besides waiting it out, or cancel it.
+
+It is a good practice to check the ``Marge`` 
+`queue <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests?assignee_username=marge-bot>`__ 
+to evaluate if it is the right moment to trigger some testing jobs. ``Marge`` 
+is configured to pick the MRs by assignment time, and this sort option is not 
+available in the Web UI. The `marge_queue <#marge-queue>`__ CLI tool 
+provides the list sorted by assignment time. The recommended way to manage the 
+trigger of those jobs is using our :abbr:`crnm (bin/ci/ci_run_n_monitor.sh)` 
+`cli tool <#running-specific-ci-jobs>`__.
+
+Due to limited resources, we currently do not run the CI automatically
+on every push; instead, we only run it automatically once the MR has
+been assigned to ``Marge``, our merge bot.
+
+If you're interested in the details, the main configuration file is ``.gitlab-ci.yml``,
+and it references a number of other files in ``.gitlab-ci/``.
+
+If the GitLab CI doesn't seem to be running on your fork (or MRs, as they run
+in the context of your fork), you should check the "Settings" of your fork.
+Under "CI / CD" → "General pipelines", make sure "Custom CI config path" is
+empty (or set to the default ``.gitlab-ci.yml``), and that the
+"Project-based pipeline visibility" box is checked.
+
+If a specific CI farm is failing for reasons unrelated to your changes, make an
+MR to disable the farm following the `farm management <#farm-management>`__
+instructions.
+
+If you're having other issues with the GitLab CI, your best bet is to ask
+about it on ``#freedesktop`` on OFTC and tag `Daniel Stone
+<https://gitlab.freedesktop.org/daniels>`__ (``daniels`` on IRC) or
+`Eric Engestrom <https://gitlab.freedesktop.org/eric>`__ (``eric_engestrom`` on
+IRC).
+
+The three GitLab CI systems currently integrated are:
+
+
+.. toctree::
+   :maxdepth: 1
+
+   bare-metal
+   LAVA
+   docker
+
+Farm management
+---------------
+
+.. note::
+   Never mix disabling/re-enabling a farm with any change that can affect a job
+   that runs in another farm!
+
+When the farm starts failing for any reason (power, network, out-of-space), it needs to be disabled by pushing separate MR with
+
+.. code-block:: sh
+
+   git mv .ci-farms{,-disabled}/$farm_name
+
+Find the GitLab handle of the farm's admin in ``.gitlab-ci/farm-rules.yml`` and
+ping them on the MR. MRs to disable farms do not need to go through review, and
+can be assigned to ``Marge`` directly.
+
+After farm restore functionality can be enabled by pushing a new merge request,
+which contains
+
+.. code-block:: sh
+
+   git mv .ci-farms{-disabled,}/$farm_name
+
+.. warning::
+   Pushing (``git push``) directly to ``main`` is forbidden; this change must
+   be sent as a :ref:`Merge Request <merging>`.
+
+Application traces replay
+-------------------------
+
+The CI replays application traces with various drivers in two different jobs. The first
+job replays traces listed in ``src/<driver>/ci/traces-<driver>.yml`` files and if any
+of those traces fail the pipeline fails as well. The second job replays traces listed in
+``src/<driver>/ci/restricted-traces-<driver>.yml`` and it is allowed to fail. This second
+job is only created when the pipeline is triggered by ``marge-bot`` or any other user that
+has been granted access to these traces.
+
+A traces YAML file also includes a ``download-url`` pointing to a MinIO
+instance where to download the traces from. While the first job should always work with
+publicly accessible traces, the second job could point to an URL with restricted access.
+
+Restricted traces are those that have been made available to Mesa developers without a
+license to redistribute at will, and thus should not be exposed to the public. Failing to
+access that URL would not prevent the pipeline to pass, therefore forks made by
+contributors without permissions to download non-redistributable traces can be merged
+without friction.
+
+As an aside, only maintainers of such non-redistributable traces are responsible for
+ensuring that replays are successful, since other contributors would not be able to
+download and test them by themselves.
+
+Those Mesa contributors that believe they could have permission to access such
+non-redistributable traces can request permission to Daniel Stone <[email protected]>.
+
+gitlab.freedesktop.org accounts that are to be granted access to these traces will be
+added to the OPA policy for the MinIO repository as per
+https://gitlab.freedesktop.org/freedesktop/helm-gitlab-infra/-/commit/a3cd632743019f68ac8a829267deb262d9670958 .
+
+So the jobs are created in personal repositories, the name of the user's account needs
+to be added to the rules attribute of the GitLab CI job that accesses the restricted
+accounts.
+
+.. toctree::
+   :maxdepth: 1
+
+   local-traces
+
+Intel CI
+--------
+
+The Intel CI is not yet integrated into the GitLab CI.
+For now, special access must be manually given (file a issue in
+`the Intel CI configuration repo <https://gitlab.freedesktop.org/Mesa_CI/mesa_jenkins>`__
+if you think you or Mesa would benefit from you having access to the Intel CI).
+Results can be seen on `mesa-ci.01.org <https://mesa-ci.01.org>`__
+if you are *not* an Intel employee, but if you are you
+can access a better interface on
+`mesa-ci-results.jf.intel.com <http://mesa-ci-results.jf.intel.com>`__.
+
+The Intel CI runs a much larger array of tests, on a number of generations
+of Intel hardware and on multiple platforms (X11, Wayland, DRM & Android),
+with the purpose of detecting regressions.
+Tests include
+`Crucible <https://gitlab.freedesktop.org/mesa/crucible>`__,
+`VK-GL-CTS <https://github.com/KhronosGroup/VK-GL-CTS>`__,
+`dEQP <https://android.googlesource.com/platform/external/deqp>`__,
+`Piglit <https://gitlab.freedesktop.org/mesa/piglit>`__,
+`Skia <https://skia.googlesource.com/skia>`__,
+`VkRunner <https://github.com/Igalia/vkrunner>`__,
+`WebGL <https://github.com/KhronosGroup/WebGL>`__,
+and a few other tools.
+A typical run takes between 30 minutes and an hour.
+
+If you're having issues with the Intel CI, your best bet is to ask about
+it on ``#dri-devel`` on OFTC and tag `Nico Cortes
+<https://gitlab.freedesktop.org/ngcortes>`__ (``ngcortes`` on IRC).
+
+.. _CI-job-user-expectations:
+
+CI job user expectations
+------------------------
+
+To make sure that testing of one vendor's drivers doesn't block
+unrelated work by other vendors, we require that a given driver's test
+farm produces a spurious failure no more than once a week.  If every
+driver had CI and failed once a week, we would be seeing someone's
+code getting blocked on a spurious failure daily, which is an
+unacceptable cost to the project.
+
+To ensure that, driver maintainers with CI enabled should watch the Flakes panel
+of the `CI flakes dashboard
+<https://ci-stats-grafana.freedesktop.org/d/Ae_TLIwVk/mesa-ci-quality-false-positives?orgId=1>`__,
+particularly the "Flake jobs" pane, to inspect jobs in their driver where the
+automatic retry of a failing job produced a success a second time.
+Additionally, most CI reports test-level flakes to an IRC channel, and flakes
+reported as NEW are not expected and could cause spurious failures in jobs.
+Please track the NEW reports in jobs and add them as appropriate to the
+``-flakes.txt`` file for your driver.
+
+Additionally, the test farm needs to be able to provide a short enough
+turnaround time that we can get our MRs through marge-bot without the pipeline
+backing up.  As a result, we require that the test farm be able to handle a
+whole pipeline's worth of jobs in less than 15 minutes (to compare, the build
+stage is about 10 minutes).  Given boot times and intermittent network delays,
+this generally means that the test runtime as reported by deqp-runner should be
+kept to 10 minutes.
+
+If a test farm is short the HW to provide these guarantees, consider dropping
+tests to reduce runtime.  dEQP job logs print the slowest tests at the end of
+the run, and Piglit logs the runtime of tests in the results.json.bz2 in the
+artifacts.  Or, you can add the following to your job to only run some fraction
+(in this case, 1/10th) of the dEQP tests.
+
+.. code-block:: yaml
+
+   variables:
+      DEQP_FRACTION: 10
+
+to just run 1/10th of the test list.
+
+For Collabora's LAVA farm, the `device types
+<https://lava.collabora.dev/scheduler/device_types>`__ page can tell you how
+many boards of a specific tag are currently available by adding the "Idle" and
+"Busy" columns.  For bare-metal, a gitlab admin can look at the `runners
+<https://gitlab.freedesktop.org/admin/runners>`__ page.  A pipeline should
+probably not create more jobs for a board type than there are boards, unless you
+clearly have some short-runtime jobs.
+
+If a HW CI farm goes offline (network dies and all CI pipelines end up
+stalled) or its runners are consistently spuriously failing (disk
+full?), and the maintainer is not immediately available to fix the
+issue, please push through an MR disabling that farm's jobs according
+to the `Farm Management <#farm-management>`__ instructions.
+
+Personal runners
+----------------
+
+Mesa's CI is currently run primarily on packet.net's m1xlarge nodes
+(2.2Ghz Sandy Bridge), with each job getting 8 cores allocated.  You
+can speed up your personal CI builds (and marge-bot merges) by using a
+faster personal machine as a runner.  You can find the gitlab-runner
+package in Debian, or use GitLab's own builds.
+
+To do so, follow `GitLab's instructions
+<https://docs.gitlab.com/ci/runners/runners_scope/#create-a-project-runner-with-a-runner-authentication-token>`__
+to register your personal GitLab runner in your Mesa fork.  Then, tell
+Mesa how many jobs it should serve (``concurrent=``) and how many
+cores those jobs should use (``FDO_CI_CONCURRENT=``) by editing these
+lines in ``/etc/gitlab-runner/config.toml``, for example:
+
+.. code-block:: toml
+
+   concurrent = 2
+
+   [[runners]]
+     environment = ["FDO_CI_CONCURRENT=16"]
+
+
+Docker caching
+--------------
+
+The CI system uses Docker images extensively to cache
+infrequently-updated build content like the CTS.  The `freedesktop.org
+CI templates
+<https://gitlab.freedesktop.org/freedesktop/ci-templates/>`__ help us
+manage the building of the images to reduce how frequently rebuilds
+happen, and trim down the images (stripping out manpages, cleaning the
+apt cache, and other such common pitfalls of building Docker images).
+
+When running a container job, the templates will look for an existing
+build of that image in the container registry under
+``MESA_IMAGE_TAG``.  If it's found it will be reused, and if
+not, the associated ``.gitlab-ci/containers/<jobname>.sh`` will be run
+to build it.  So, when developing any change to container build
+scripts, you need to update the associated ``MESA_IMAGE_TAG`` to
+a new unique string.  We recommend using the current date plus some
+string related to your branch (so that if you rebase on someone else's
+container update from the same day, you will get a Git conflict
+instead of silently reusing their container)
+
+When developing a given change to your Docker image, you would have to
+bump the tag on each ``git commit --amend`` to your development
+branch, which can get tedious.  Instead, you can navigate to the
+`container registry
+<https://gitlab.freedesktop.org/mesa/mesa/container_registry>`__ for
+your repository and delete the tag to force a rebuild.  When your code
+is eventually merged to main, a full image rebuild will occur again
+(forks inherit images from the main repo, but MRs don't propagate
+images from the fork into the main repo's registry).
+
+Building locally using CI docker images
+---------------------------------------
+
+It can be frustrating to debug build failures on an environment you
+don't personally have.  If you're experiencing this with the CI
+builds, you can use Docker to use their build environment locally.  Go
+to your job log, and at the top you'll see a line like::
+
+   Pulling docker image registry.freedesktop.org/anholt/mesa/debian/android_build:2020-09-11
+
+We'll use a volume mount to make our current Mesa tree be what the
+Docker container uses, so they'll share everything (their build will
+go in _build, according to ``meson-build.sh``).  We're going to be
+using the image non-interactively so we use ``run --rm $IMAGE
+command`` instead of ``run -it $IMAGE bash`` (which you may also find
+useful for debug).  Extract your build setup variables from
+.gitlab-ci.yml and run the CI meson build script:
+
+.. code-block:: sh
+
+   IMAGE=registry.freedesktop.org/anholt/mesa/debian/android_build:2020-09-11
+   sudo docker pull $IMAGE
+   sudo docker run --rm -v `pwd`:/mesa -w /mesa $IMAGE env PKG_CONFIG_PATH=/usr/local/lib/aarch64-linux-android/pkgconfig/:/android-ndk-r21d/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/aarch64-linux-android/pkgconfig/ GALLIUM_DRIVERS=freedreno UNWIND=disabled EXTRA_OPTION="-D android-stub=true -D llvm=disabled" DRI_LOADERS="-D glx=disabled -D gbm=disabled -D egl=enabled -D platforms=android" CROSS=aarch64-linux-android ./.gitlab-ci/meson-build.sh
+
+All you have left over from the build is its output, and a _build
+directory.  You can hack on mesa and iterate testing the build with:
+
+.. code-block:: sh
+
+   sudo docker run --rm -v `pwd`:/mesa $IMAGE meson compile -C /mesa/_build
+
+Running specific CI jobs
+------------------------
+
+You can use ``bin/ci/ci_run_n_monitor.py`` to run specific CI jobs. It
+will automatically take care of running all the jobs yours depends on,
+and cancel the rest to avoid wasting resources.
+
+See ``bin/ci/ci_run_n_monitor.py --help`` for all the options.
+
+**Target jobs**
+
+The ``--target`` argument takes a regex that you can use to select the
+jobs names you want to run, e.g. ``--target 'zink.*'`` will run all the
+Zink jobs, leaving the other drivers' jobs free for others to use.
+
+Note that in fork pipelines, GitLab only adds the jobs for the files that have
+changed **since the last push**, so you might not get the jobs you expect.
+You can work around that by adding a dummy change in a file core to what you're
+working on and then making a new push with that change, and removing that change
+before you create the MR.
+
+**GitLab token**
+
+The ``--token`` argument is used to provide a GitLab token with rights to
+interact with the pipeline. Using the argument, one can provide the value or
+the name of the file having the value. If the argument is not provided, then
+it checks if the value of ``$XDG_CONFIG_HOME`` has a valid directory (if not,
+then uses ``$HOME/.config``), and there is a file called ``gitlab-token`` that
+contains a token. The token required to work with this tool needs ``api``
+scope permissions.
+
+.. note::
+    To create that token, refer to
+    `create-a-personal-access-token <https://docs.gitlab.com/user/profile/personal_access_tokens/#create-a-personal-access-token>`_
+    and select the ``api`` scope. The token will only be shown once after creation,
+    so make sure you store it securely.
+
+Marge queue
+-----------
+
+You can use ``bin/ci/marge_queue.sh`` to check how long the Marge queue is. As
+mentioned, the merge flow is to assign MR to the ``Marge`` bot, to serialize
+the verification and merge. Looking at the
+`merge requests assigned to Marge <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests?assignee_username=marge-bot>`__
+you can evaluate the size of the queue, since the ``marge_queue`` tool provides
+sorted and summarized information about those MR in queue.
+
+The tool requires a GitLab token as described in the
+`crnm <#running-specific-ci-jobs>`__ section. It outputs the current queue
+sorted by the ``assigned at`` to ``Marge``. It can also be used as an active
+wait for another action in a pipe, using the ``--wait`` until the queue is
+empty. The return code corresponds to the number of MRs in the queue, so when
+it returns ``0``, one can, for example, start the ``crnm`` tool on a certain
+pipeline.
+
+Conformance Tests
+-----------------
+
+Some conformance tests require a special treatment to be maintained on GitLab CI.
+This section lists their documentation pages.
+
+.. toctree::
+  :maxdepth: 1
+
+  skqp
+
+
+Updating GitLab CI Linux Kernel
+-------------------------------
+
+GitLab CI usually runs a bleeding-edge kernel. The following documentation has
+instructions on how to uprev Linux Kernel in the GitLab CI ecosystem.
+
+.. toctree::
+  :maxdepth: 1
+
+  kernel
+
+Structured tagging
+------------------
+
+Some build scripts can be tagged with a deterministic tag to allow for
+testing and validation of the build output. This section lists the
+documentation pages for the structured tagging feature.
+
+.. toctree::
+  :maxdepth: 1
+
+  structured-tagging
+
+Reusing CI scripts for other projects
+--------------------------------------
+
+The CI scripts in ``.gitlab-ci/`` can be reused for other projects, to
+facilitate reuse of the infrastructure, our scripts can be used as tools
+to create containers and run tests on the available farms.
+
+.. envvar:: EXTRA_LOCAL_PACKAGES
+
+   Define extra Debian packages to be installed in the container.

docs/ci/kernel.rst

@@ -0,0 +1,115 @@
+Upreving Linux Kernel
+=====================
+
+Occasionally, the GitLab CI needs a Linux Kernel update to enable new kernel
+features, device drivers, bug fixes etc to CI jobs.
+Kernel uprevs in GitLab CI are relatively simple, but prone to lots of
+side-effects since many devices from different platforms are involved in the
+pipeline.
+
+Kernel repository
+-----------------
+
+The Linux Kernel used in the GitLab CI is stored at the following repository:
+https://gitlab.freedesktop.org/gfx-ci/linux
+
+It is common that Mesa kernel brings some patches that were not merged on the
+Linux mainline, that is why Mesa has its own kernel version which should be used
+as the base for newer kernels.
+
+So, one should base the kernel uprev from the last tag used in the Mesa CI,
+please refer to ``.gitlab-ci/image-tags.yml`` ``KERNEL_TAG`` variable.
+Every tag has a standard naming: ``vX.YZ-for-mesa-ci-<commit_short_SHA>``, which
+can be created via the command:
+
+:code:`git tag vX.YZ-for-mesa-ci-$(git rev-parse --short HEAD)`
+
+Building Kernel
+---------------
+
+The kernel files are loaded from the artifacts uploaded to S3 from gfx-ci/linux.
+
+Updating Kconfigs
+^^^^^^^^^^^^^^^^^
+
+When a Kernel uprev happens, it is worth compiling and cross-compiling the
+Kernel locally, in order to update the Kconfigs accordingly.  Remember that the
+resulting Kconfig is a merge between *Mesa CI Kconfig* and *Linux tree
+defconfig* made via ``merge_config.sh`` script located at Linux Kernel tree.
+
+Kconfigs location
+"""""""""""""""""
+
++------------+------------------------------------------------------+-------------------------------------+
+| Platform   | Mesa CI Kconfig location                             | Linux tree defconfig                |
++============+======================================================+=====================================+
+| arm        | kernel/configs/mesa3d-ci_arm.config\@gfx-ci/linux    | arch/arm/configs/multi_v7_defconfig |
++------------+------------------------------------------------------+-------------------------------------+
+| arm64      | kernel/configs/mesa3d-ci_arm64.config\@gfx-ci/linux  | arch/arm64/configs/defconfig        |
++------------+------------------------------------------------------+-------------------------------------+
+| x86-64     | kernel/configs/mesa3d-ci_x86_64.config\@gfx-ci/linux | arch/x86/configs/x86_64_defconfig   |
++------------+------------------------------------------------------+-------------------------------------+
+
+Updating image tags
+-------------------
+
+Every kernel uprev should update the following tag:
+
+:code:`.gitlab-ci/image-tags.yml` tag
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+- **KERNEL_TAG** to use the new kernel
+
+Development routine
+-------------------
+
+1. Compile the newer kernel locally for each platform.
+2. Compile device trees for ARM platforms
+3. Update Kconfigs. Are new Kconfigs necessary? Is CONFIG_XYZ_BLA deprecated? Does the ``merge_config.sh`` override an important config?
+4. Push a new development branch to `Kernel repository`_ based on the latest kernel tag used in GitLab CI
+5. Hack ``build-kernel.sh`` script to clone kernel from your development branch
+6. Update image tags. See `Updating image tags`_
+7. Run the entire CI pipeline, all the automatic jobs should be green. If some job is red or taking too long, you will need to investigate it and probably ask for help.
+
+When the Kernel uprev is stable
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+1. Push a new tag to Mesa CI `Kernel repository`_
+2. Update KERNEL_URL ``debian/x86_test-gl`` job definition
+3. Open a merge request, if it is not opened yet
+
+Tips and Tricks
+---------------
+
+Compare pipelines
+^^^^^^^^^^^^^^^^^
+
+To have the most confidence that a kernel uprev does not break anything in Mesa,
+it is suggested that one runs the entire CI pipeline to check if the update affected the manual CI jobs.
+
+Step-by-step
+""""""""""""
+
+1. Create a local branch in the same git ref (should be the main branch) before branching to the kernel uprev kernel.
+2. Push this test branch
+3. Run the entire pipeline against the test branch, even the manual jobs
+4. Now do the same for the kernel uprev branch
+5. Compare the job results. If a CI job turned red on your uprev branch, it means that the kernel update broke the test. Otherwise, it should be fine.
+
+Bare-metal custom kernels
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Some CI jobs have support to plug in a custom kernel by simply changing a variable.
+This is great, since rebuilding the kernel and rootfs may takes dozens of minutes.
+
+For example, Freedreno jobs ``gitlab.yml`` manifest support a variable named
+``BM_KERNEL``. If one puts a gz-compressed kernel URL there, the job will use that
+kernel to boot the Freedreno bare-metal devices. The same works for ``BM_DTB`` in
+the case of device tree binaries.
+
+Careful reading of the job logs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Sometimes a job may turn to red for reasons unrelated to the kernel update, e.g.
+LAVA ``tftp`` timeout, problems with the freedesktop servers etc.
+So it is important to see the reason why the job turned red, and retry it if an
+infrastructure error has happened.

docs/ci/local-traces.rst

@@ -0,0 +1,45 @@
+Running traces on a local machine
+=================================
+
+Prerequisites
+-------------
+- Install `Apitrace <https://apitrace.github.io/>`__
+- Install `Renderdoc <https://renderdoc.org/>`__ (only needed for some traces)
+- Download and compile `Piglit <https://gitlab.freedesktop.org/mesa/piglit>`__ and install his `dependencies <https://gitlab.freedesktop.org/mesa/piglit#2-setup>`__
+- Download traces you want to replay from `traces-db <https://gitlab.freedesktop.org/gfx-ci/tracie/traces-db/>`__
+
+Running single trace
+--------------------
+A simple run to see the output of the trace can be done with
+
+.. code-block:: sh
+
+   apitrace replay -w name_of_trace.trace
+
+For more information, look into the `Apitrace documentation <https://github.com/apitrace/apitrace/blob/master/docs/USAGE.markdown>`__.
+
+For comparing checksums use:
+
+.. code-block:: sh
+
+   cd piglit/replayer
+   export PIGLIT_SOURCE_DIR="../"
+   ./replayer.py compare trace -d test path/name_of_trace.trace 0 # replace with expected checksum
+
+
+Simulating CI trace job
+-----------------------
+
+Sometimes it's useful to be able to test traces on your local machine instead of the Mesa CI runner. To simulate the CI environment as closely as possible.
+
+Download the YAML file from your driver's ``ci/`` directory and then change the path in the YAML file from local proxy or MinIO to the local directory (URL-like format ``file://``)
+
+.. code-block:: sh
+
+   # The PIGLIT_REPLAY_DEVICE_NAME has to match name in the YAML file.
+   export PIGLIT_REPLAY_DEVICE_NAME='your_device_name'
+   export PIGLIT_REPLAY_DESCRIPTION_FILE='path_to_mesa_traces_file.yml'
+   ./piglit run -l verbose --timeout 300 -j10 replay ~/results/
+
+
+Note: For replaying traces, you may need to allow higher GL and GLSL versions. You can achieve that by setting  ``MESA_GLSL_VERSION_OVERRIDE`` and ``MESA_GL_VERSION_OVERRIDE``.

docs/ci/skqp.rst

@@ -0,0 +1,33 @@
+SkQP
+====
+
+`SkQP <https://skia.org/docs/dev/testing/skqp/>`__ stands for SKIA Quality
+Program conformance tests.  Basically, it has sets of rendering tests and unit
+tests to ensure that `SKIA <https://skia.org/>`__ is meeting its design specifications on a specific
+device.
+
+The rendering tests have support for GL, GLES and Vulkan backends and test some
+rendering scenarios.
+And the unit tests check the GPU behavior without rendering images, using any of the GL/GLES or Vulkan drivers.
+
+SkQP reports
+------------
+
+SkQP generates reports after finishing its execution, and deqp-runner collects
+them in the job artifacts results directory under the test name.  Click the
+'Browse' button from a failing job to get to them.
+
+SkQP failing tests
+------------------
+
+SkQP rendering tests will have a range of pixel values allowed for the driver's
+rendering for a given test.  This can make the "expected" image in the result
+output look rather strange, but you should be able to make sense of it knowing
+that.
+
+In SkQP itself, testcases can have increased failing pixel thresholds added to
+them to keep CI green when the rendering is "correct" but out of normal range.
+However, we don't support changing the thresholds in our testing.  Because any
+driver rendering not meeting the normal thresholds will trigger Android CTS
+failures, we treat them as failures and track them as expected failures the
+```*-fails.txt`` file.`

docs/ci/structured-tagging.rst

@@ -0,0 +1,359 @@
+================================
+Structured Tagging for CI Builds
+================================
+
+This document explains the new structural tagging system integrated into our CI pipeline. Structural
+tagging ensures that every build and test job is tied to a unique, reproducible build state by
+computing a deterministic tag from each component's build script (and its relevant inputs, such as
+patches) and verifying that tag at both build and test time.
+
+Overview
+--------
+Structural tagging enhances container and rootfs tag management with an automated, end-to-end
+process. Key aspects include:
+
+Deterministic Tag Generation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+During the build, the system calculates a tag (an MD5 hash) from the contents of the build script
+plus any extra files that affect the build. This tag represents the "structure" of the build.
+
+YAML-Declared Tags
+~~~~~~~~~~~~~~~~~~
+For each component, a tag is declared in the YAML configuration file (located at
+``.gitlab-ci/conditional-build-image-tags.yml``). The declared value is used to verify that the
+build script has not changed without an accompanying update to the tag.
+
+Dual-Phase Verification
+~~~~~~~~~~~~~~~~~~~~~~~
+* **Build Time:** The build script calls the helper function ``ci_build_time_tag_check`` immediately after calculating the tag. This function compares the calculated tag against the declared tag and writes the computed tag to a file (located at ``/mesa-ci-build-tag/``), ensuring early detection of mismatches.
+
+* **Test Time:** Later, test scripts invoke ``ci_tag_test_time_check`` to read the stored tag and confirm that the tests run against the expected build version. A mismatch here will cause the test job to fail immediately.
+
+Automated Tag Updates
+~~~~~~~~~~~~~~~~~~~~~
+A helper script ``bin/ci/update_tag.py`` is provided to list, check, and update tags for all or
+individual components. This tool is intended to simplify the process of keeping the declared tags
+synchronized with the build scripts.
+
+How it works
+------------
+
+The mechanism considers a "component" any build script that follows the
+``.gitlab-ci/container/build-*.sh`` pattern and also uses the ``ci_tag_*`` prefixed functions provided by
+``.gitlab-ci/setup_test_env.sh``.
+
+Suppose that SkQP just received the structured tagging support.
+Let's look how the build and test phases work.
+
+   .. graphviz::
+      :caption: Structured Tagging
+
+      digraph StructuredTagging {
+         rankdir=TD;
+         node [style=filled, fillcolor=lightgray];
+
+         // =========================
+         // Build Phase Subgraph
+         // =========================
+         subgraph cluster_build {
+            label="Build Phase";
+            style=dashed;
+
+            // Define nodes with descriptive IDs and labels.
+            tag_decl [
+                  label="SKQP_TAG declared in\n.gitlab-ci/conditional-build-image-tags.yml",
+                  shape=note, fillcolor=white
+            ];
+            calc_tag [
+                  label="build-skqp.sh:\nCalculate tag (build script content + patches)",
+                  shape=box, style="rounded,filled", fillcolor=lightgray
+            ];
+            early_check [
+                  label="container_pre_build.sh:\nCheck all structured tags early",
+                  shape=box, style="rounded,filled", fillcolor=lightgray
+            ];
+            validate_tag [
+                  label="Validate calculated tag\nagainst declared SKQP_TAG",
+                  shape=box, style="rounded,filled", fillcolor=lightgray
+            ];
+            build_decision [
+                  label="Is calculated tag equal\nto declared SKQP_TAG?",
+                  shape=diamond, fillcolor=lightyellow
+            ];
+            fail_build [
+                  label="Fail build",
+                  shape=rounded, fillcolor=salmon
+            ];
+            write_tag [
+                  label="Write calculated tag to\n/var/tmp/mesa-ci-build-tag/SKQP_TAG",
+                  shape=box, style="rounded,filled", fillcolor=lightblue
+            ];
+            compile_skqp [
+                  label="Compile SkQP",
+                  shape=box, fillcolor=palegreen
+            ];
+
+            // Define edges for the build phase.
+            tag_decl -> calc_tag;
+            calc_tag -> validate_tag;
+            early_check -> validate_tag;
+            validate_tag -> build_decision;
+            build_decision -> fail_build [label="No"];
+            build_decision -> write_tag [label="Yes"];
+            write_tag -> compile_skqp;
+         }
+
+         // =========================
+         // Test Phase Subgraph
+         // =========================
+         subgraph cluster_test {
+            label="Test Phase";
+            style=dashed;
+
+            // Define nodes with descriptive IDs and labels.
+            skqp_running [
+                  label="Is SKQP running?",
+                  shape=diamond, fillcolor=lightyellow
+            ];
+            ci_var_include [
+                  label="image-tags.yml:\nincludes\ncontainer-builds-image-tags.yml",
+                  shape=note, fillcolor=white
+            ];
+            ci_var [
+                  label="CI Variable\n(CONDITIONAL_BUILD_SKQP_TAG)\nfrom container-builds-image-tags.yml",
+                  shape=note, fillcolor=white
+            ];
+            ci_var_extends [
+                  label="This job extends\n.container-builds-skqp\nMaking SKQP_TAG=CONDITIONAL_BUILD_SKQP_TAG",
+                  shape=note, fillcolor=white
+            ];
+            check_tag [
+                  label="deqp-runner.sh:\nPull tag in /var/tmp/mesa-ci-build-tag/SKQP_TAG",
+                  shape=box, style="rounded,filled", fillcolor=lightblue
+            ];
+            decision [
+                  label="Is calculated tag equal\nto declared SKQP_TAG?",
+                  shape=diamond, fillcolor=lightyellow
+            ];
+            proceed_test [
+                  label="Proceed the test job",
+                  shape=box, fillcolor=palegreen
+            ];
+            fail_test [
+                  label="Fail test",
+                  shape=box, fillcolor=salmon
+            ];
+
+            // Define edges for the test phase.
+            skqp_running -> check_tag [label="Yes"];
+            skqp_running -> proceed_test [label="No"];
+            check_tag -> decision;
+            decision -> proceed_test [label="Yes"];
+            decision -> fail_test [label="Mismatch"];
+            ci_var_extends -> decision;
+            ci_var -> ci_var_extends;
+            ci_var_include -> ci_var;
+         }
+      }
+
+
+Build-Time Checks
+~~~~~~~~~~~~~~~~~
+During the build phase:
+
+* **Tag Calculation:**
+   In the component's build script (named following the convention ``build-<component>.sh``), the
+   function ``_ci_calculate_tag`` computes an MD5 hash based on:
+
+   - The build script's contents.
+   - Any additional files (e.g. patches) that affect the build.
+
+* **Validation:**
+   The build script calls ``ci_tag_build_time_check`` to verify that the current value of the component's
+   tag (passed in as an environment variable) matches the tag calculated by the build script.
+
+* **Failure on Mismatch:**
+   If the tags do not match, the build is aborted. This prevents any accidental use of stale or mismatched artifacts.
+
+* **Early checks:**
+   Right now, the `container_pre_build.sh` script is responsible for checking the structured tagging
+   in all registered components. So, we can check quickly, before the component's build starts, if
+   the tag is correct.
+
+* **Tag writing:**
+   The build script writes the computed tag into a new file the structured tagging directory, namely
+   ``/mesa-ci-build-tag/<component>_TAG``.
+
+Test-Time Checks
+~~~~~~~~~~~~~~~~
+In the test scripts (for example, in ``.gitlab-ci/deqp-runner.sh``):
+
+* **Verification:**
+   The test job retrieves the tag written into the artifact (e.g. from ``/mesa-ci-build-tag/DEQP_RUNNER_TAG``) and then calls:
+
+   .. code-block:: bash
+
+      ci_tag_test_time_check "DEQP_RUNNER_TAG"
+
+* **Purpose:**
+   This check ensures that the tests are run against the exact build that was produced. If a mismatch is found, the test job fails immediately.
+
+   .. note::
+      Even when the developer forgets to update the ``image-tags.yml`` file when needed, the test job
+      will fail if the tag is not correct, given that the ``conditional-build-image-tags.yml``
+      file is properly updated.
+
+
+Adding a New Component Tag
+--------------------------
+To integrate structured tagging for a new component (for example, ``my_component``), follow these steps:
+
+1. **Modify the Build Script:**
+
+   - In your build script (e.g. ``.gitlab-ci/container/build-my-component.sh``), map out the external files that can affect the build output.
+     *Tip:* You can mimic the approach in ``build-angle.sh`` early variable declaration to get the tag.
+   - Immediately after calculating the tag, add a validation step:
+
+      .. code-block:: bash
+
+         PATCH_FILES=("...")
+
+         ci_tag_build_time_check "MY_COMPONENT_TAG" "${PATCH_FILES[@]}"
+
+2. **If the component is run in a DUT job, update the passthrough script:**
+
+   On ``.gitlab-ci/common/generate-env.sh``:
+
+   .. code-block:: bash
+
+      VARS=(
+         ...
+         MY_COMPONENT_TAG
+         ...
+      )
+
+3. **Update the CI YAMLs:**
+
+   - In your conditional image tags file (e.g. ``.gitlab-ci/conditional-build-image-tags.yml``), add an entry for your component:
+
+      .. code-block:: yaml
+
+         variables:
+               CONDITIONAL_BUILD_MY_COMPONENT_TAG: <initial-tag-value>
+
+   - Now we need to update the build related YAMLs to include the new component tag. In ``.gitlab-ci/container/gitlab-ci.yml``, add a new hidden job:
+
+      .. code-block:: yaml
+
+         .container-builds-my-component:
+            variables:
+               MY_COMPONENT_TAG: "${CONDITIONAL_BUILD_MY_COMPONENT_TAG}"
+
+   - It is time to modify the job that builds the component image to include the new component tag. Let's suppose that only the ``debian/arm64_test-gl`` job builds the component image. We need to add the new component tag to the job as an extension:
+
+      .. code-block:: yaml
+
+         debian/arm64_test-gl:
+            extends:
+               - .container-builds-my-component
+               - .container-builds-my-component2
+            variables:
+               # CI_BUILD_COMPONENTS is a space-separated list of components used during early tag checks
+               CI_BUILD_COMPONENTS: "my_component my_component2"
+
+      - Now, ``MY_COMPONENT_TAG`` will be used by the ``ci_tag_build_time_check`` and ``ci_tag_test_time_check`` functions, only for jobs that extend the ``.container-builds-my-component`` job.
+      - And the ``CI_BUILD_COMPONENTS`` variable will be swept to perform the early checks.
+
+   .. warning::
+      Do not forget to update your main image tags file (e.g. ``.gitlab-ci/image-tags.yml``) if necessary, check the header comments of the modified files for more details.
+
+   .. note::
+      Also, note that the main image tags file (``.gitlab-ci/image-tags.yml``) does not define the
+      conditional build tags directly.
+      Instead, it **retrieves** values such as ``MY_COMPONENT_TAG`` from the `includes` directive of the
+      ``.gitlab/container-builds-image-tags.yml`` file. This setup ensures centralized management of
+      tag values and maintains consistency across various components and jobs.
+
+Updating Component Tags with the Helper Script
+----------------------------------------------
+The helper script ``bin/ci/update_tag.py`` assists with tag management. Its key functionalities include:
+
+* **Listing Available Components:**
+
+  .. code-block:: bash
+
+      ./bin/ci/update_tag.py --list
+
+* **Updating All Component Tags:**
+
+  .. code-block:: bash
+
+      ./bin/ci/update_tag.py --all
+
+* **Updating a Specific Component Tag:**
+
+  .. code-block:: bash
+
+      ./bin/ci/update_tag.py --include my_component
+      ./bin/ci/update_tag.py --include 'my_component.*'
+
+* **Running a Check:**
+
+  .. code-block:: bash
+
+      ./bin/ci/update_tag.py --check my_component
+      ./bin/ci/update_tag.py --check my_component1 --check my_component2
+
+This script uses the same underlying functions as in the build scripts to generate the deterministic tag and then updates the YAML file accordingly.
+Ensure that your python environment has the requirements installed, see ``bin/ci/requirements.txt`` for the list of dependencies.
+
+Limitations
+-----------
+The current implementation has some known limitations:
+
+* **Local Utility Script Constraints:**
+
+   When running the update/tagging utility locally, the build inputs used by the build script (such
+   as environment variables defined in the YAML) are not automatically applied. For example, if the
+   tag calculation relies on a variable like ``EXTRA_MESON_ARGS``, you must manually set or mock its
+   value locally to generate the correct tag. Otherwise, the computed tag may be incorrect, and you
+   might need to run the actual build job (and extract the expected tag from the error message) to
+   verify the value. Future improvements may leverage tools like gitlab-ci-local to better reproduce
+   the YAML environment locally.
+
+* **Timing Sensitivity:**
+
+   If the build script is modified after the early check (performed by the utility script) but before the actual build job runs, the calculated tag will differ from the declared tag. This discrepancy will block the build consistently until the YAML declaration is updated.
+
+* **Manual Update Requirement:**
+
+      In this initial version, updating the ``image-tags.yml``  must be done manually. If this file is
+      not updated, the build scripts will not be validated properly.
+      However, the test-time check will still catch mismatches and abort the job, ensuring that any
+      issues do not go unnoticed.
+
+Troubleshooting and FAQ
+-----------------------
+
+* **Tag Mismatch Errors:**
+   If you encounter a tag mismatch error, verify that:
+
+   - The build script and its additional inputs (patches, environment variables, etc.) are current.
+   - The declared tag in ``.gitlab-ci/conditional-build-image-tags.yml`` has been updated accordingly. Use the update helper if necessary.
+
+* **Local Testing Challenges:**
+   When running the update utility locally, ensure that you mock any YAML-dependent variables (e.g., EXTRA_MESON_ARGS) to simulate the CI environment.
+
+Conclusion
+----------
+The new structural tagging system provides a robust, automated method to ensure that every CI build
+is uniquely identified and that tests run against the correct build state. By integrating
+deterministic tag calculation with dual-phase verification and a dedicated update helper script,
+this system minimizes human error and streamlines the CI process.
+
+.. note::
+   Be aware of the current limitations, especially around local testing and the manual update
+   requirement, as you integrate and use structural tagging. Future improvements are planned to
+   address these issues.
+
+Happy tagging!

docs/ci/uri-caching.conf

@@ -0,0 +1,44 @@
+set $proxy_authorization '';
+
+set_by_lua $proxyuri '
+	local unescaped =  ngx.unescape_uri(ngx.var.arg_uri);
+	local it, err = ngx.re.match(unescaped, "(https?://)(.*@)?([^/]*)(/.*)?");
+	if not it then
+		-- Hack to cause nginx to return 404
+		return "http://localhost/404"
+	end
+
+	local scheme = it[1];
+	local authstring = it[2];
+	local host = it[3];
+	local query = it[4];
+
+	if ngx.var.http_authorization and ngx.var.http_authorization ~= "" then
+		ngx.var.proxy_authorization = ngx.var.http_authorization;
+	elseif authstring then
+		auth = string.sub(authstring, 0, -2);
+		auth64 = ngx.encode_base64(auth);
+		ngx.var.proxy_authorization = "Basic " .. auth64;
+	end
+
+	-- Default to / if none is set to avoid using the request_uri query
+	if not query then
+		query = "/";
+	end
+
+	return scheme .. host .. query;
+';
+
+# Rewrite the location header to redirect back to this server. Do
+# this using lua header filtering to allow for url encoding the original
+# location header for use as a query parameter.
+header_filter_by_lua_block {
+	if ngx.header.location then
+		ngx.header.location = "/cache?uri=" .. ngx.escape_uri(ngx.header.location);
+	end
+}
+
+add_header X-GG-Cache-Status $upstream_cache_status;
+proxy_set_header Authorization $proxy_authorization;
+
+proxy_pass $proxyuri;