Page MenuHomePhabricator
Create Task
Maniphest T362408

Migration to containerd and away from docker
Open, HighPublic
Actions

Description

Per T269684 we need to move away from docker. In February 2024, the serviceops team announced the results of the evaluation of the candidate replacement engines. Results and criteria have been documented in Kubernetes/CRE. The chosen container runtime engine was containerd. This task describes the plan for the migration and tracks the migration process itself

Plan

containerd upgrade

  1. We 'll probably need a new profile profile::containerd or similar.
  2. Create proper cgroups config for containerd (https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd)
  3. Handle pulling of restricted images with containerd (provide authentication credentials etc)
  4. Test integration with dragonfly/dfget

for the actual upgrade

  1. Run some workers (4 in codfw as a start) with bookworm, to surface potential OS related issues
    1. wikikube-worker2085 (R440)
    2. wikikube-worker2086 (R440)
    3. wikikube-worker2088 (Supermicro)
    4. wikikube-worker2089 (R450)
  2. Create puppetization for the configuration required by kubernetes
  3. Reimage some nodes with bookworm + containerd (>=1.6)
  4. Upgrade all clusters to the newer containerd, rolling-reimage of nodes

nerdctl

Docker has a relatively user friendly CLI. containerd doesn't. the ctr tool it ships with is a lower level, albeit useful tool. nerdctl, is a CLI released by the containerd project that is CLI compatible with docker CLI

  1. Package nerdctl. Probably utilizing our Upstream binaries policy to avoid the onus of having to build every since dependency
  2. Use puppet to install the package and populate a nerdctl configuration file /etc/nerdctl/nerdctl.toml to default to namespace k8s.io
  3. Test and approve.

crictl

Kubernetes build crictl/cri-tools https://github.com/kubernetes-sigs/cri-tools/tree/master to interact with a CRI the way kubelet would. In my initial tests with nerctl it did not completely honor all containerd configuration (like registry mirrors and authentication we require for dragonfly). So I decided to also package cricrl and have it installed on all nodes.

Kubelet (the above are a prereq)

  1. Amend puppet to have behind a feature flag the following 2 parameters
--container-runtime-endpoint=unix:///run/containerd/containerd.sock 
--container-runtime=remote

Metrics

  • Replace kubelet_docker_operations_* with kubelet_runtime_operations_*

Log processing

Parsing of logs does not work properly with containerd nodes. Logs that usually have the k8s_docker_log_field_parsed tag don't have it anymore:

T377132: containerd logs are not properly parsed during ingestion to logstash

Things to do after all k8s nodes have been migrated off of docker

  1. Remove puppet classes no longer in use (if there are any)
  2. Ensure all profile::docker::engine related hiera keys are gone (as well as profile::kubernetes::node::docker_kubernetes_user_password)

How to migrate to containerd

https://wikitech.wikimedia.org/wiki/Kubernetes/Administration/containerd_migration

Details

SubjectRepoBranchLines +/-
k8s.pool-depool-node: Add support for multiple nodesoperations/cookbooksmaster+161 -106
Migrate wikikube-worker208[5689] to containerdoperations/puppetproduction+5 -16
Add a cookbook to roll-reimage stacked k8s control planesoperations/cookbooksmaster+305 -0
etcd::v3: Don't set trusted-ca-file if client-cert-auth is falseoperations/puppetproduction+8 -0
etcd::v3: Ensure trusted-ca-file is not set on first puppet run with 3.4operations/puppetproduction+2 -2
wikikube: Remove explicit container_runtime configoperations/puppetproduction+0 -2
wikikube: Prepare clusters for containerd workersoperations/puppetproduction+24 -9
containerd: Remove container log line length limitoperations/puppetproduction+4 -0
dragonfly::dfdaemon: Refactor docker integrationoperations/puppetproduction+27 -23
dragonfly::dfdaemon: Enable by default when profile is includedoperations/puppetproduction+1 -5
Remove role kubernetes::staging::worker_containerdlabs/privatemaster+0 -1
cumin/aliases: Merge worker_containerd back to worker roleoperations/puppetproduction+4 -4
kubernetes: Create profile::kubernetes::container_runtimeoperations/puppetproduction+56 -82
Merge worker_containerd back to worker rolelabs/privatemaster+2 -1
cumin/aliases: Remove P{O:kubernetes::staging::worker}operations/puppetproduction+1 -1
Migrate kubestage1004 to containerdoperations/puppetproduction+1 -53
Migrate kubestage1003 to containerdoperations/puppetproduction+7 -3
k8s/kubelet: Remove absent containerd specific systemd overrideoperations/puppetproduction+0 -7
k8s/kubelet: Make kubelet.service depend on container runtimeoperations/puppetproduction+1 -1
k8s/kubelet: Make kubelet.service depend on container runtimeoperations/puppetproduction+14 -3
Remove kubelet systemd unit dependency to docker.serviceoperations/debs/kubernetesv1.23+10 -3
wikikube-staging-codfw: Migrate kubestage2002 to containerdoperations/puppetproduction+1 -1
wikikube-staging-codfw: Migrate kubestage2002 to containerdoperations/puppetproduction+1 -6
kubernetes/staging: Add role master_stacked_containerdoperations/puppetproduction+60 -0
containerd: Enable unprivileged icmp and binding to ports < 1024operations/puppetproduction+15 -0
cumin/aliases: Add containerd roles to wikikube aliasesoperations/puppetproduction+4 -4
kubelet: Remove --pod-infra-container-image when using containerdoperations/puppetproduction+4 -3
kubelet/containerd: Fix registry authenticationoperations/puppetproduction+17 -5
kubernetes::worker_containerd: Fix registry_auth hiera keylabs/privatemaster+2 -2
kubelet/containerd: Fix runc config and kubelet systemd unitoperations/puppetproduction+12 -5
Initial commit of containerd puppet codeoperations/puppetproduction+317 -8
kubernetes::worker_containerd: Fix registry_auth hiera keylabs/privatemaster+2 -2
kubernetes::worker_containerd: Add registry 'secrets'labs/privatemaster+2 -0
Rename kubernets2009,2010,2035,2054, reimage to bookwormoperations/puppetproduction+10 -10
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Mon, Oct 7, 2:10 PM

Change #1078417 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] wikikube-staging-codfw: Migrate kubestage2002 to containerd

https://gerrit.wikimedia.org/r/1078417

gerritbot added a comment.Mon, Oct 7, 2:13 PM

Change #1078417 merged by JMeybohm:

[operations/puppet@production] wikikube-staging-codfw: Migrate kubestage2002 to containerd

https://gerrit.wikimedia.org/r/1078417

ops-monitoring-bot added a comment.Mon, Oct 7, 2:18 PM

Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host kubestage2002.codfw.wmnet with OS bookworm

JMeybohm updated the task description. (Show Details)Mon, Oct 7, 2:25 PM
JMeybohm updated the task description. (Show Details)
Maintenance_bot removed a project: Patch-For-Review.Mon, Oct 7, 2:30 PM
JMeybohm updated the task description. (Show Details)Mon, Oct 7, 2:36 PM
ops-monitoring-bot added a comment.Mon, Oct 7, 4:16 PM

Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host kubestage2002.codfw.wmnet with OS bookworm completed:

  • kubestage2002 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410071443_jayme_2779980_kubestage2002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Mon, Oct 7, 4:19 PM

Change #1078447 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/debs/[email protected]] Remove kubelet systemd unit dependency to docker.service

https://gerrit.wikimedia.org/r/1078447

gerritbot added a project: Patch-For-Review.Mon, Oct 7, 4:19 PM
gerritbot added a comment.Mon, Oct 7, 4:31 PM

Change #1078450 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] k8s/kubelet: Make kubelet.service depend on container runtime

https://gerrit.wikimedia.org/r/1078450

gerritbot added a comment.Mon, Oct 7, 4:31 PM

Change #1078451 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] k8s/kubelet: Remove absent containerd specific systemd override

https://gerrit.wikimedia.org/r/1078451

gerritbot added a comment.Tue, Oct 8, 8:53 AM

Change #1078447 merged by JMeybohm:

[operations/debs/[email protected]] Remove kubelet systemd unit dependency to docker.service

https://gerrit.wikimedia.org/r/1078447

Stashbot added a comment.Tue, Oct 8, 9:25 AM

Mentioned in SAL (#wikimedia-operations) [2024-10-08T09:25:48Z] <jayme> imported kubernetes 1.23.14-4 to component/kubernetes123 (buster, bullseye, bookworm) - T362408

gerritbot added a comment.Tue, Oct 8, 10:09 AM

Change #1078450 merged by JMeybohm:

[operations/puppet@production] k8s/kubelet: Make kubelet.service depend on container runtime

https://gerrit.wikimedia.org/r/1078450

gerritbot added a comment.Tue, Oct 8, 10:15 AM

Change #1078626 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] k8s/kubelet: Make kubelet.service depend on container runtime

https://gerrit.wikimedia.org/r/1078626

gerritbot added a comment.Tue, Oct 8, 10:19 AM

Change #1078626 merged by JMeybohm:

[operations/puppet@production] k8s/kubelet: Make kubelet.service depend on container runtime

https://gerrit.wikimedia.org/r/1078626

Stashbot added a comment.Tue, Oct 8, 10:36 AM

Mentioned in SAL (#wikimedia-operations) [2024-10-08T10:36:33Z] <jayme> updated kubernetes 1.23.14-3 -> 1.23.14-4 on P:kubernetes::node - T362408

JMeybohm updated the task description. (Show Details)Tue, Oct 8, 10:44 AM
ops-monitoring-bot added a comment.Tue, Oct 8, 10:45 AM

Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host kubestage2002.codfw.wmnet with OS bookworm

gerritbot added a comment.Tue, Oct 8, 10:55 AM

Change #1078451 merged by JMeybohm:

[operations/puppet@production] k8s/kubelet: Remove absent containerd specific systemd override

https://gerrit.wikimedia.org/r/1078451

ops-monitoring-bot added a comment.Tue, Oct 8, 11:28 AM

Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host kubestage2002.codfw.wmnet with OS bookworm completed:

  • kubestage2002 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410081109_jayme_2939229_kubestage2002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Maintenance_bot removed a project: Patch-For-Review.Tue, Oct 8, 11:30 AM
ops-monitoring-bot added a comment.Tue, Oct 8, 11:35 AM

Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host kubestage2001.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Tue, Oct 8, 12:19 PM

Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host kubestage2001.codfw.wmnet with OS bookworm completed:

  • kubestage2001 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410081201_jayme_2952057_kubestage2001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Tue, Oct 8, 1:53 PM

Change #1078677 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] Migrate kubestage1003 to containerd

https://gerrit.wikimedia.org/r/1078677

gerritbot added a project: Patch-For-Review.Tue, Oct 8, 1:53 PM

Change #1078678 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] Migrate kubestage1004 to containerd

https://gerrit.wikimedia.org/r/1078678

JMeybohm added a comment.Thu, Oct 10, 7:31 AM

There are some hardware refreshes planned which should go Bookworm + containerd right away:

  • {T376171}
  • {T376185}
  • {T376170}
gerritbot added a comment.Thu, Oct 10, 9:07 AM

Change #1078677 merged by JMeybohm:

[operations/puppet@production] Migrate kubestage1003 to containerd

https://gerrit.wikimedia.org/r/1078677

ops-monitoring-bot added a comment.Thu, Oct 10, 9:14 AM

Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host kubestage1003.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Thu, Oct 10, 9:50 AM

Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host kubestage1003.eqiad.wmnet with OS bookworm completed:

  • kubestage1003 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410100933_jayme_3362138_kubestage1003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Thu, Oct 10, 11:59 AM

Change #1078678 merged by JMeybohm:

[operations/puppet@production] Migrate kubestage1004 to containerd

https://gerrit.wikimedia.org/r/1078678

ops-monitoring-bot added a comment.Thu, Oct 10, 12:00 PM

Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host kubestage1004.eqiad.wmnet with OS bookworm

Clement_Goubert subscribed.Thu, Oct 10, 12:02 PM
In T362408#10216554, @JMeybohm wrote:

There are some hardware refreshes planned which should go Bookworm + containerd right away:

  • {T376171}
  • {T376185}
  • {T376170}

As well as expansions:

  • {T376307}
  • {T376665}
Maintenance_bot removed a project: Patch-For-Review.Thu, Oct 10, 12:31 PM
ops-monitoring-bot added a comment.Thu, Oct 10, 12:38 PM

Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host kubestage1004.eqiad.wmnet with OS bookworm completed:

  • kubestage1004 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410101219_jayme_3392864_kubestage1004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Thu, Oct 10, 1:01 PM

Change #1079276 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] cumin/aliases: Remove P{O:kubernetes::staging::worker}

https://gerrit.wikimedia.org/r/1079276

gerritbot added a project: Patch-For-Review.Thu, Oct 10, 1:01 PM
gerritbot added a comment.Thu, Oct 10, 1:06 PM

Change #1079276 merged by JMeybohm:

[operations/puppet@production] cumin/aliases: Remove P{O:kubernetes::staging::worker}

https://gerrit.wikimedia.org/r/1079276

Maintenance_bot removed a project: Patch-For-Review.Thu, Oct 10, 1:30 PM
ops-monitoring-bot added a comment.Thu, Oct 10, 4:04 PM

Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host kubestage1003.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Thu, Oct 10, 4:40 PM

Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host kubestage1003.eqiad.wmnet with OS bookworm completed:

  • kubestage1003 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410101623_jayme_3430289_kubestage1003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Mon, Oct 14, 8:47 AM

Change #1079935 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] kubernetes: Create profile::kubernetes::container_runtime

https://gerrit.wikimedia.org/r/1079935

gerritbot added a project: Patch-For-Review.Mon, Oct 14, 8:47 AM
gerritbot added a comment.Mon, Oct 14, 9:54 AM

Change #1079955 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[labs/private@master] Merge worker_containerd values back into worker

https://gerrit.wikimedia.org/r/1079955

gerritbot added a comment.Mon, Oct 14, 10:02 AM

Change #1079955 merged by JMeybohm:

[labs/private@master] Merge worker_containerd back to worker role

https://gerrit.wikimedia.org/r/1079955

JMeybohm mentioned this in rLPRI68b0f377be2e: Merge worker_containerd back to worker role.Mon, Oct 14, 10:03 AM
gerritbot added a comment.Mon, Oct 14, 10:09 AM

Change #1079935 merged by JMeybohm:

[operations/puppet@production] kubernetes: Create profile::kubernetes::container_runtime

https://gerrit.wikimedia.org/r/1079935

gerritbot added a comment.Mon, Oct 14, 10:11 AM

Change #1079960 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] cumin/aliases: Merge worker_containerd back to worker role

https://gerrit.wikimedia.org/r/1079960

gerritbot added a comment.Mon, Oct 14, 10:14 AM

Change #1079960 merged by JMeybohm:

[operations/puppet@production] cumin/aliases: Merge worker_containerd back to worker role

https://gerrit.wikimedia.org/r/1079960

gerritbot added a comment.Mon, Oct 14, 10:15 AM

Change #1079961 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[labs/private@master] Remove role kubernetes::staging::worker_containerd

https://gerrit.wikimedia.org/r/1079961

gerritbot added a comment.Mon, Oct 14, 10:17 AM

Change #1079961 merged by JMeybohm:

[labs/private@master] Remove role kubernetes::staging::worker_containerd

https://gerrit.wikimedia.org/r/1079961

JMeybohm mentioned this in rLPRI45c64ec1774d: Remove role kubernetes::staging::worker_containerd.Mon, Oct 14, 10:17 AM
gerritbot added a comment.Mon, Oct 14, 10:34 AM

Change #1079970 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] wikikube: Prepare clusters for containerd workers

https://gerrit.wikimedia.org/r/1079970

JMeybohm updated the task description. (Show Details)Mon, Oct 14, 11:25 AM
gerritbot added a comment.Mon, Oct 14, 2:18 PM

Change #1080038 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] dragonfly::dfdaemon: Enable by default when profile is included

https://gerrit.wikimedia.org/r/1080038

gerritbot added a comment.Mon, Oct 14, 2:35 PM

Change #1080042 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] dragonfly::dfdaemon: Refactor docker integration

https://gerrit.wikimedia.org/r/1080042

gerritbot added a comment.Mon, Oct 14, 4:16 PM

Change #1080071 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] containerd: Remove container log line length limit

https://gerrit.wikimedia.org/r/1080071

gerritbot added a comment.Tue, Oct 15, 9:53 AM

Change #1080038 merged by JMeybohm:

[operations/puppet@production] dragonfly::dfdaemon: Enable by default when profile is included

https://gerrit.wikimedia.org/r/1080038

gerritbot added a comment.Tue, Oct 15, 9:54 AM

Change #1080042 merged by JMeybohm:

[operations/puppet@production] dragonfly::dfdaemon: Refactor docker integration

https://gerrit.wikimedia.org/r/1080042

gerritbot added a comment.Wed, Oct 16, 8:29 AM

Change #1080071 merged by JMeybohm:

[operations/puppet@production] containerd: Remove container log line length limit

https://gerrit.wikimedia.org/r/1080071

gerritbot added a comment.Wed, Oct 16, 8:29 AM

Change #1079970 merged by JMeybohm:

[operations/puppet@production] wikikube: Prepare clusters for containerd workers

https://gerrit.wikimedia.org/r/1079970

gerritbot added a comment.Wed, Oct 16, 8:47 AM

Change #1080554 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] wikikube-staging: Migrate control planes to containerd

https://gerrit.wikimedia.org/r/1080554

gerritbot added a comment.Wed, Oct 16, 9:41 AM

Change #1080554 merged by JMeybohm:

[operations/puppet@production] wikikube: Remove explicit container_runtime config

https://gerrit.wikimedia.org/r/1080554

gerritbot added a comment.Thu, Oct 17, 12:01 PM

Change #992629 had a related patch set uploaded (by JMeybohm; author: Mxmxchere):

[operations/puppet@production] etcd::v3: Don't set trusted-ca-file if client-cert-auth is false

https://gerrit.wikimedia.org/r/992629

gerritbot added a comment.Thu, Oct 17, 12:52 PM

Change #992629 merged by JMeybohm:

[operations/puppet@production] etcd::v3: Don't set trusted-ca-file if client-cert-auth is false

https://gerrit.wikimedia.org/r/992629

gerritbot added a comment.Thu, Oct 17, 5:19 PM

Change #1081224 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] etcd::v3: Ensure trusted-ca-file is not set on first puppet run with 3.4

https://gerrit.wikimedia.org/r/1081224

gerritbot added a comment.Fri, Oct 18, 9:17 AM

Change #1081377 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/cookbooks@master] Add a cookbook to roll-reimage stacked k8s control planes

https://gerrit.wikimedia.org/r/1081377

gerritbot added a comment.Fri, Oct 18, 9:33 AM

Change #1081224 merged by JMeybohm:

[operations/puppet@production] etcd::v3: Ensure trusted-ca-file is not set on first puppet run with 3.4

https://gerrit.wikimedia.org/r/1081224

ops-monitoring-bot added a comment.Fri, Oct 18, 11:00 AM

Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host kubestagemaster2005.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Fri, Oct 18, 11:43 AM

Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host kubestagemaster2005.codfw.wmnet with OS bookworm completed:

  • kubestagemaster2005 (PASS)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410181121_jayme_1191485_kubestagemaster2005.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
JMeybohm updated the task description. (Show Details)Fri, Oct 18, 2:37 PM
ops-monitoring-bot added a comment.Fri, Oct 18, 2:39 PM

Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host kubestagemaster2003.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Fri, Oct 18, 3:26 PM

Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host kubestagemaster2003.codfw.wmnet with OS bookworm completed:

  • kubestagemaster2003 (PASS)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410181502_jayme_1220197_kubestagemaster2003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Fri, Oct 18, 3:26 PM

Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host kubestagemaster2004.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Fri, Oct 18, 4:09 PM

Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host kubestagemaster2004.codfw.wmnet with OS bookworm completed:

  • kubestagemaster2003 (PASS)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410181502_jayme_1220197_kubestagemaster2003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
  • kubestagemaster2004 (PASS)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410181546_jayme_1220197_kubestagemaster2004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Fri, Oct 18, 4:10 PM

Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host kubestagemaster2005.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Fri, Oct 18, 4:54 PM

Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host kubestagemaster2005.codfw.wmnet with OS bookworm completed:

  • kubestagemaster2003 (PASS)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410181502_jayme_1220197_kubestagemaster2003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
  • kubestagemaster2004 (PASS)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410181546_jayme_1220197_kubestagemaster2004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
  • kubestagemaster2005 (PASS)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410181632_jayme_1220197_kubestagemaster2005.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
JMeybohm closed subtask T377132: containerd logs are not properly parsed during ingestion to logstash as Resolved.Fri, Oct 18, 7:47 PM
gerritbot added a comment.Mon, Oct 21, 8:23 AM

Change #1081910 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] Migrate wikikube-worker208[5689] to containerd

https://gerrit.wikimedia.org/r/1081910

ops-monitoring-bot added a comment.Mon, Oct 21, 8:47 AM

Cookbook cookbooks.sre.k8s.reimage-stacked-control-plane started by jayme@cumin1002 Reimaging k8s control planes of cluster staging-eqiad: containerd migration completed:

  • kubestagemaster1003 (PASS)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410210826_jayme_1708770_kubestagemaster1003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mon, Oct 21, 9:27 AM

Cookbook cookbooks.sre.k8s.reimage-stacked-control-plane started by jayme@cumin1002 Reimaging k8s control planes of cluster staging-eqiad: containerd migration completed:

  • kubestagemaster1003 (PASS)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410210826_jayme_1708770_kubestagemaster1003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
  • kubestagemaster1004 (PASS)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410210906_jayme_1708770_kubestagemaster1004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mon, Oct 21, 10:08 AM

Cookbook cookbooks.sre.k8s.reimage-stacked-control-plane started by jayme@cumin1002 Reimaging k8s control planes of cluster staging-eqiad: containerd migration completed:

  • kubestagemaster1003 (PASS)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410210826_jayme_1708770_kubestagemaster1003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
  • kubestagemaster1004 (PASS)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410210906_jayme_1708770_kubestagemaster1004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
  • kubestagemaster1005 (PASS)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410210946_jayme_1708770_kubestagemaster1005.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Tue, Oct 22, 11:32 AM

Change #1082191 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/cookbooks@master] k8s.pool-depool-node: Add support for multiple nodes

https://gerrit.wikimedia.org/r/1082191

gerritbot added a comment.Tue, Oct 22, 12:01 PM

Change #1081910 merged by JMeybohm:

[operations/puppet@production] Migrate wikikube-worker208[5689] to containerd

https://gerrit.wikimedia.org/r/1081910

ops-monitoring-bot added a comment.Tue, Oct 22, 12:06 PM

Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host wikikube-worker2085.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Tue, Oct 22, 12:06 PM

Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host wikikube-worker2086.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Tue, Oct 22, 12:08 PM

Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host wikikube-worker2088.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Tue, Oct 22, 12:09 PM

Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host wikikube-worker2089.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Tue, Oct 22, 12:45 PM

Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host wikikube-worker2088.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2088 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410221227_jayme_1935630_wikikube-worker2088.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Tue, Oct 22, 12:50 PM

Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host wikikube-worker2085.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2085 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410221231_jayme_1935361_wikikube-worker2085.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Tue, Oct 22, 12:53 PM

Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host wikikube-worker2086.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2086 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410221234_jayme_1935371_wikikube-worker2086.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Tue, Oct 22, 12:55 PM

Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host wikikube-worker2089.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2089 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410221237_jayme_1935895_wikikube-worker2089.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
JMeybohm updated the task description. (Show Details)Tue, Oct 22, 3:59 PM
BTullis subscribed.Tue, Oct 22, 4:15 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. � Wikimedia Foundation � Privacy Policy � Code of Conduct � Terms of Use � Disclaimer � CC-BY-SA � GPL � Credits