Self hosted Gitlab Community Edition upgrade and CI_JOB_TOKEN access control behavior change

Table of Contents

General introduction 📚

At IN2P3-CC, we’re hosting our own instance of Gitlab Community Edition. It was recently upgraded to the latest Gitlab version available.

I’m used to create plenty of specialized utilities containers to help me in my day to day CI journey.

The latest container I had created is a container that embeds python poetry in order to use it in all my Python based projects. I’m not a huge fan of Python 🐍 but eh… it’s used everywhere around me so 🤷. But still, Python type hints just makes me 🤯.

How I discovered the new default behavior ?

Step 1 - Create the project and the container image

As usual, I’ve created a new Project in our Gitlab instance. The project was created after the Gitlab Community Edition upgrade took place.

As usual, my C.I / C.D jobs create multiple images automatically, such as:

  • gitlab-registry.example.org/containers/poetry/python3.11/rockylinux9:1
  • gitlab-registry.example.org/containers/poetry/python3.11/rockylinux9:1.5
  • gitlab-registry.example.org/containers/poetry/python3.11/rockylinux9:1.5.1

If you’re not familiar with rockylinux, it is a RedHat based O.S. With almalinux, it is one of the two main projects that has emerged after the announced CentOS end-of-life.

Step 2 - Use the container image in a usual C.I job

In another python project, I’m trying to rely on the gitlab-registry.example.org/containers/poetry/python3.11/rockylinux9:1.5 container image to build my python application.

Here is a sample .gitlab-ci.yml file I’ve used:

---
stages:
  - "build"

build-app:
  stage: "build"
  image: "gitlab-registry.example.org/containers/poetry/python3.11/rockylinux9:1.5"
  script:
    - "poetry build"
    - "poetry install"

Nothing new, I’m quite used to do those things. No reason to fail, I can go grab a coffee ☕… or so I thought.

Step 3 - The cryptic error from my C.I job

Here is the error that appeared in my C.I job:

Running with gitlab-runner 15.x.y (436155cb)
  on runner@gitlab.example.org 98238a4c, system ID: x_fdsfkdsf89
Preparing the "docker" executor 00:04
Using Docker executor with image gitlab-registry.example.org/containers/poetry/python3.11/rockylinux9:1.5 ...
Authenticating with credentials from job payload (GitLab Registry)
Pulling docker image gitlab-registry.example.org/containers/poetry/python3.11/rockylinux9:1.5 ...
WARNING: Failed to pull image with policy "always": Error response from daemon: pull access denied for gitlab-registry.example.org/containers/poetry/python3.11/rockylinux9, repository does not exist or may require 'docker login': denied: requested access to the resource is denied (manager.go:237:0s)
ERROR: Job failed: failed to pull image "gitlab-registry.example.org/containers/poetry/python3.11/rockylinux9:1.5" with specified policies [always]: Error response from daemon: pull access denied for gitlab-registry.example.org/containers/poetry/python3.11/rockylinux9, repository does not exist or may require 'docker login': denied: requested access to the resource is denied (manager.go:237:0s)

The only useful line is

failed to pull image [...]: Error response from daemon: pull access denied

Step 4 - Whaaaatttt ?!

After many tries, all the new projects I was creating were unable to use this container image. They were all failing with the same pull access denied error.

I was able to use other container images, from previous projects in the same namespace.

Something was just making me crazy here ! Something was just making me crazy here !

I had created almost 10 new gitlab repositories to try, tag new images, push new images and test. And I always obtained the same pull access denied when the container image was originating from a newly created gitlab project.

What was happening ?

How gitlab CI_JOB_TOKEN works ?

When the C.I job executes, it automatically authenticates to the Gitlab registry using the CI_JOB_TOKEN generated for the job.

This token inherits the permissions of the user that has triggered the pipeline.

Discovering a parameter that has been recently enabled by default

By walking through the Settings -> C.I / C.D menu of my poetry container project, I discovered something interesting:

CI_JOB_TOKEN access control setting of my poetry container project CI_JOB_TOKEN access control setting of my poetry container project

The menu states that it “Control how the CI_JOB_TOKEN CI/CD variable is used for API access between projects” and the Allow access to this project with a CI_JOB_TOKEN is enabled. It also has an explicit list of whitelisted projects that can access my poetry container project using a CI_JOB_TOKEN.

Easy fix

I’ve discovered that the Allow access to this project with a CI_JOB_TOKEN whitelist has been enabled by default for any new project in latest Gitlab Community Edition versions.

So an easy fix is to whitelist my python application project in the poetry container project C.I / C.D -> Token Access setting.

CI_JOB_TOKEN access control setting of my poetry container project CI_JOB_TOKEN access control setting of my poetry container project

Once the whitelist is done, all I have to do is restart my pipelines and observe 👀…

IT WORKS ! IT WORKS !

Change history

Enabled by default since v15.9 Enabled by default since v15.9

So it seems that this was enabled by default since Gitlab v15.9.

It seems that this is already possible to automate this whitelisting using Gitlab APIs.