@marcaurele

Keep your Nomad job's image up to date with Renovate

2024-03-24T00:00:00+00:00

I run a Nomad cluster at home for my internal services which grows from time to time. It reached a level now that staying up to date with the different services’ updates is too much work, and too manual for me. I neither run my service with latest</code> as you do not have any control on the version. I had to find a way to automate MRs/PRs to propose updates whenever a new tag is pushed. Having discovered the power of Renovate</a> (highly recommended over Dependabot)</em>, and having it already setup to push updates in my development projects, I wanted to leverage it for my infrastructure configuration.</p>

EDITS: Renovate can be configured to directly parse the HCL file to update the image</code> line, see the edits from the last paragraph.</em></p>

Nomad job definitions🔗</a></h2>
With Nomad’s job definition, the image</code> attribute contains the link to the registry and the tag to use. This syntax is not recognized by Renovate bot as it cannot parse the HCL file.</p>
Extract of a Grafana + Prometheus service job definition:</p>
job</span> "grafana"</span> {</span></span>
  type</span>        =</span> "service"</span></span>
</span>
  group</span> "grafana"</span> {</span></span>
</span>
    task</span> "grafana"</span> {</span></span>
      driver</span> =</span> "docker"</span></span>
</span>
      config</span> {</span></span>
        image</span>           =</span> "docker.io/grafana/grafana-oss:10.4.1"</span></span>
        ports</span>           =</span> [</span>"http"</span>]</span></span>
        readonly_rootfs</span> =</span> true</span></span>
        cap_drop</span>        =</span> [</span>"ALL"</span>]</span></span>
        ...</span></span>
      }</span></span></code></pre>
Renovate will simply ignore the content of the file and cannot update the image url despite being a standard container image syntax. One way of solving this issue would have been to write the dependency manager</a> for Nomad HCL files for the project. I have to admit I did not want to learn enough TypeScript in order to propose this change and took another easier approach.</p>
The trick🔗</a></h2>
Instead of directly including the image link in the job file, we can leverage the HCL functions</a> to retrieve the container image definition from another file that Renovate (or any other dependency update manager like Dependabot) can parse and update. So the trick is to use a companion Dockerfile</code> where there is only the FROM</code> line pointing to the image(s) we want to use in the job, and use the HCL functions to read its content. For the Grafana service, the content of the Dockerfile.grafana</code> is as follow:</p>
FROM</span> docker.io/grafana/grafana-oss:10.4.1 </span>as</span> grafana</span></span>
FROM</span> quay.io/prometheus/prometheus:v2.51.0 </span>as</span> prometheus</span></span></code></pre>
It contains a as</code> to put the image definition into a map</em>, rather than using a simple list with indexes (not very readable).</p>
The files are organized with the job definitions inside jobs/</code> and the companion Dockerfile</code> in the sub-directory jobs/images/</code>:</p>
...</span></span>
├── jobs</span></span>
│   ├── grafana.hcl</span></span>
│   ├── images</span></span>
│   │   ├── Dockerfile.grafana</span></span>
...</span></span></code></pre>
Strangely the Nomad HCL2</a> language is only a subset of the Terraform HCL</a> one which contains more functions. Nevertheless there is enough in the Nomad HCL to do the job. The Nomad HCL code to read the image from the file:</p>
locals</span> {</span></span>
  container_file</span> =</span> "images/Dockerfile.grafana"</span></span>
  container_images</span> =</span> {</span></span>
    for</span> word</span> in</span> [</span></span>
      for</span> line</span> in</span> compact</span>(</span>split</span>(</span></span>
        "</span>\n</span>"</span>,</span></span>
        fileexists</span>(</span>"./jobs/</span>${</span>local</span>.</span>container_file</span>}</span>"</span>)</span> ?</span> file</span>(</span>"./jobs/</span>${</span>local</span>.</span>container_file</span>}</span>"</span>)</span> :</span> file</span>(</span>"./</span>${</span>local</span>.</span>container_file</span>}</span>"</span>)</span></span>
      )) </span>:</span> split</span>(</span>" "</span>,</span> trimspace</span>(</span>line</span>))</span></span>
    ] </span>:</span> word[</span>3</span>] </span>=></span> word[</span>1</span>]</span></span>
  }</span> # FROM image as name -> {"name" = "image"}</span></span>
}</span></span></code></pre>
The code uses fileexists()</code></a> because the file()</code></a> function will be executed within the same path as where the nomad</code> will be executed. As I can be in the root or inside the jobs</code> directory when I ran commands, it requires to handle both case. It’s something you can simplify. The compact(split())</code> is to remove empty lines from the Dockerfile</code> in case there are any at the end, or spaces between images. For information, the syntax does not support any comments added to the file Dockerfile</code>.</p>
It allows to reference the image later in the job definition with the local variable:</p>
    task</span> "grafana"</span> {</span></span>
      driver</span> =</span> "docker"</span></span>
</span>
      config</span> {</span></span>
        image</span>           =</span> local</span>.</span>container_images</span>[</span>"grafana"</span>]</span></span>
        ports</span>           =</span> [</span>"http"</span>]</span></span>
        readonly_rootfs</span> =</span> true</span></span></code></pre>
Renovate is now able to open merge requests to keep up to date the Dockerfile</code> used for each service:</p>
</p>
Direct update with Renovate (edits)🔗</a></h2>
Update thanks to _duncan_</a>.</em></p>
_duncan_ pointed out to me</a> that the docker datasource could be configured to update directly the HCL file for the line containing the image</code> attribute. A custom manager should be added to the configuration:</p>
{</span></span>
  "</span>$schema</span>":</span> "https://docs.renovatebot.com/renovate-schema.json"</span>,</span></span>
  "</span>customManagers</span>": [</span></span>
    {</span></span>
      "</span>customType</span>":</span> "regex"</span>,</span></span>
      "</span>datasourceTemplate</span>":</span> "docker"</span>,</span></span>
      "</span>fileMatch</span>": [</span>"</span>\\</span>.hcl$"</span>],</span></span>
      "</span>matchStrings</span>": [</span></span>
        "</span>\\</span>s*image</span>\\</span>s*=</span>\\</span>s*</span>\\\"</span>(?<depName>.*?):(?<currentValue>.*?)</span>\\\"</span>"</span></span>
      ]</span></span>
    }</span></span>
  ]</span></span>
}</span></span></code></pre>
It works brilliantly.</em> No need to read a Dockerfile</code> to extract the image tag value.</p>

One week shadowing Wayne Haber, Director of Engineering at GitLab 2024-01-10T00:00:00+00:00 Summary🔗</a></h2> Last December I spent one week shadowing Wayne Haber</a>, Engineering Director of Secure, Govern and Growth at GitLab</a>. While I’m not a GitLab employee, this opportunity was possible thanks to the openness of GitLab</a> in describing how they work</a> and their amount of information available to the public</a>.</p> What motivated me?🔗</a></h2> During one of my previous experiences, I asked other managers to be able to shadow how they were running their planning team meetings. I wanted to get a sense on how engineers were discussing their priorities and on-going work together, seeing as well how company projects were handled by each of them. I found the experience very rewarding as you are usually working in a single team and cannot experience different environments easily at the same company. For this reason, when I came across a post from Wayne about his shadow program</a>, I immediately kept the link in my One Day1</a></sup></em> list to pick it up when I will have more time. I was interested to see what could be different from managing managers rather than ICs, how TeamOps was used on a day to day basis at GitLab (openness and async communication) and how the groups Secure, Govern and Growth were handled by Wayne. So back in October when I had a medical leave after an injury the item came back to my sight and I sent a request to Wayne who quickly accepted it. I was in!</a></p> I have to admit, I really love GitLab and are following them since many years because of their openness.</p> Preparation🔗</a></h2> What I like from people I met working at GitLab is that they are leveraging their tool in good way to keep with the async communication. For the shadow program with Wayne, this means having a full issue with all the items</a> to be done: prior, during and after the shadowing week. It was also used as a reference ticket to link all the contributions I would do which were driven by the shadowing week.</p> </p> Things cannot be clearer! I had already done the TeamOps free training</a> from GitLab back in November 2022 when it came out eager to understand better all their async best practices. To get a better sense of the actual subjects and people, I watched 2-3 of the latest “Sec growth senior leaders development staff meeting”</em> you can find on the GitLab Youtube channel</a>, which I recommend for people on-boarded in the program. We had a first video call with Wayne to sync on the shadow program, what was to be prepared before the week started and what were my expectations of the shadowing week.</p> On the Friday prior to the start, Wayne shared with me a file with his notes for the coming week containing his priorities and the meetings overview I would be participating in. As you could imagine, I was eager to have more meetings but not all can be public with an external person. GitLab employees, who participate in the program, do have a better experience as they can participate in more meetings.</p> The week🔗</a></h2> I’m living in Switzerland, meaning 5 hours ahead of Wayne’s time who is based in the US, in Atlanta, GA. Despite being able to do most of the meetings along side my job as the overlap is short, I took all afternoons off in order to be able to dive into topics, navigate the handbook and be fully available for any meeting. On Monday afternoon, my time, was the first meeting with Wayne to get a sense of the topics for the week, explaining as well the content and structure of the shared notes file.</p> I had the opportunity to meet 8 GitLab employees, including Wayne, and 2 contractors during that week over Zoom calls, from direct reports, skip levels, to Wayne’s manager. As a manager myself, all the meetings were about topics I would expect. Even if there is a lot of open topics, meetings are mainly to address private ones so they would last up to about 30 minutes. It was interesting to hear about some of their current technical challenges and how they approach solutions. I could also experience how Wayne was introducing the GitLab way of working to new contractors starting on a project to ensure a smooth communication with the team, which is a major ingredient in remote only company.</p> What I liked most of course, is that thanks to GitLab openness, many topics can be followed publicly with their issues. You need Wayne’s input to find the needle in the hay stack, there are so many! I wonder how their CI handle it all.</p> A shared problematic, common to many or all software companies, was the speed of getting the merge requests in which popped up recently. I caught this in a previous senior leaders staff meeting, but sadly my week was not one with the team to be able to follow up on that.</p> Through the dedicated shadow program’s Slack channel, Wayne would post links and comments on topics and issues he was working on, or was going to read, seeking to have any sort of feedback on them. I spent a good amount of time reading the merge requests comments, following the links to get more context. My humble contributions were mainly proof reading to catch typos. As a reader of the Handbook, it feels better to have a clear content.</p> At one occasion I could feel some stress from Wayne when he knew less than 24 hours prior to a meeting that he had to cover for his manager at the CTO OKR monthly review to report his groups as well as all the other ones under his manager. It was a chase to find the right people on top of the time zone differences for the items missing updates since the last report. There you can see the scale of GitLab.</p> One recommendation I would do to future shadows, is to get in the 1-1 meetings you are invited ahead of time. It provides opportunities to do a coffee chat discussion with the other person, as well as checking each person GitLab repo to find some information about them (location, README). It’s a good source to find common topics to talk about. It made me catch that one of the meeting was with someone in Switzerland which I would not have guessed otherwise.</p> What it is not about🔗</a></h2> not about technical topics, but if some of the current challenges ring a bell to you and you have an open source committer’s mind, you will get the direct pointer to help them out.</li> not about getting hired as Wayne pointed it out in his shadow program page</a>.</li> it’s not intense and if you cannot attend most of the meetings for whatever reason, it will not affect Wayne’s week in a bad way.</li> </ul> Observations🔗</a></h2> Wayne looks well organized with his habits and is taking extra time and effort to duplicate things for the shadow with disclosable information.</li> I could observed his servant leadership line with the listening, empathy, building community, persuasion principles to name a few.</li> GitLab, as many other IT companies, had to realign its workforce, and despite the difficult past moments, everyone was engaged and happy to contribute.</li> </ol> What’s next🔗</a></h2> This shadow week gave me more confidence in how am I doing my leadership work with my team. It made me think through my weekly habits to seek more effectiveness and in ways to push more for asynchronous habits within my team. Last but not least, I will seek to participate in a mentor+mentee program</a> to have regular exchange with peers from other companies.</p> Thank you Wayne</a> for the opportunity and GitLab / Sid Sijbrandij</a> for the openness of GitLab which allow to access many information, making this Shadow program</a> possible.</p> Blog post source and merge request comments available at https://gitlab.com/marcaurele/gitlab-shadow-program/</a></em></p> I follow the Inbox-Zero method and keep those items with a dedicated flag in it, avoiding having too many tools. ↩</a></p> </li> </ol> </section>