A tag for anything related to Kubernetes. For the discussion see T147187: Create a tag for #kubernetes.
See also:
Details
Details
Description
Today
Today
ops-monitoring-bot added a comment to T362408: Migration to containerd and away from docker.
Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host wikikube-worker2089.codfw.wmnet with OS bookworm completed:
- wikikube-worker2089 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410221237_jayme_1935895_wikikube-worker2089.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment to T362408: Migration to containerd and away from docker.
Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host wikikube-worker2086.codfw.wmnet with OS bookworm completed:
- wikikube-worker2086 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410221234_jayme_1935371_wikikube-worker2086.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment to T362408: Migration to containerd and away from docker.
Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host wikikube-worker2085.codfw.wmnet with OS bookworm completed:
- wikikube-worker2085 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410221231_jayme_1935361_wikikube-worker2085.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment to T362408: Migration to containerd and away from docker.
Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host wikikube-worker2088.codfw.wmnet with OS bookworm completed:
- wikikube-worker2088 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410221227_jayme_1935630_wikikube-worker2088.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment to T362408: Migration to containerd and away from docker.
Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host wikikube-worker2089.codfw.wmnet with OS bookworm
ops-monitoring-bot added a comment to T362408: Migration to containerd and away from docker.
Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host wikikube-worker2088.codfw.wmnet with OS bookworm
ops-monitoring-bot added a comment to T362408: Migration to containerd and away from docker.
Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host wikikube-worker2086.codfw.wmnet with OS bookworm
ops-monitoring-bot added a comment to T362408: Migration to containerd and away from docker.
Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host wikikube-worker2085.codfw.wmnet with OS bookworm
gerritbot added a comment to T362408: Migration to containerd and away from docker.
Change #1081910 merged by JMeybohm:
[operations/puppet@production] Migrate wikikube-worker208[5689] to containerd
gerritbot added a comment to T362408: Migration to containerd and away from docker.
Change #1082191 had a related patch set uploaded (by JMeybohm; author: JMeybohm):
[operations/cookbooks@master] k8s.pool-depool-node: Add support for multiple nodes
akosiaris added a comment to T376762: Remove `.cluster.local.` suffix in PTR responses.
Tracked separately in T377805
Yesterday
Yesterday
CDanis added a comment to T376762: Remove `.cluster.local.` suffix in PTR responses.
akosiaris added a comment to T376762: Remove `.cluster.local.` suffix in PTR responses.
CDanis renamed T376762: Remove `.cluster.local.` suffix in PTR responses from CoreDNS upgrade so we can rewrite `.cluster.local.` suffix in PTR responses to Remove `.cluster.local.` suffix in PTR responses.
root@db1169:~# ss -tr | grep :mysql ESTAB 0 945 db1169.eqiad.wmnet:mysql 10-67-163-233.mediawiki.mw-api-ext.svc.cluster.local:57846 ESTAB 0 0 db1169.eqiad.wmnet:mysql 10-67-163-180.mediawiki.mw-web.svc.cluster.local:54236 ESTAB 0 0 db1169.eqiad.wmnet:mysql 10-67-172-94.mediawiki.mw-web.svc.cluster.local:40890 ESTAB 0 0 db1169.eqiad.wmnet:mysql 10-67-163-151.mediawiki.mw-api-ext.svc.cluster.local:42186 ESTAB 0 0 db1169.eqiad.wmnet:mysql 10-67-133-187.mediawiki.mw-api-int.svc.cluster.local:57268 ESTAB 0 674 db1169.eqiad.wmnet:mysql 10-67-190-118.mediawiki.mw-api-ext.svc.cluster.local:42074 ESTAB 0 11 db1169.eqiad.wmnet:mysql 10-67-158-92.mediawiki.mw-api-ext.svc.cluster.local:33186 ESTAB 0 753 db1169.eqiad.wmnet:mysql 10-67-172-220.mediawiki.mw-web.svc.cluster.local:60126 ESTAB 0 0 db1169.eqiad.wmnet:mysql 10-67-189-105.mediawiki.mw-web.svc.cluster.local:32812 ESTAB 0 673 db1169.eqiad.wmnet:mysql 10-67-167-12.mediawiki.mw-web.svc.cluster.local:35374
CDanis added a comment to T376762: Remove `.cluster.local.` suffix in PTR responses.
Should we attempt this on a cluster or two? One of the stagings, and then perhaps aux?
ops-monitoring-bot added a comment to T362408: Migration to containerd and away from docker.
Cookbook cookbooks.sre.k8s.reimage-stacked-control-plane started by jayme@cumin1002 Reimaging k8s control planes of cluster staging-eqiad: containerd migration completed:
- kubestagemaster1003 (PASS)
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Set boot media to disk
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410210826_jayme_1708770_kubestagemaster1003.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment to T362408: Migration to containerd and away from docker.
Cookbook cookbooks.sre.k8s.reimage-stacked-control-plane started by jayme@cumin1002 Reimaging k8s control planes of cluster staging-eqiad: containerd migration completed:
- kubestagemaster1003 (PASS)
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Set boot media to disk
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410210826_jayme_1708770_kubestagemaster1003.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment to T362408: Migration to containerd and away from docker.
Cookbook cookbooks.sre.k8s.reimage-stacked-control-plane started by jayme@cumin1002 Reimaging k8s control planes of cluster staging-eqiad: containerd migration completed:
- kubestagemaster1003 (PASS)
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Set boot media to disk
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410210826_jayme_1708770_kubestagemaster1003.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
gerritbot added a comment to T362408: Migration to containerd and away from docker.
Change #1081910 had a related patch set uploaded (by JMeybohm; author: JMeybohm):
[operations/puppet@production] Migrate wikikube-worker208[5689] to containerd
Fri, Oct 18
Fri, Oct 18
JMeybohm closed T377132: containerd logs are not properly parsed during ingestion to logstash as Resolved.
JMeybohm closed T377132: containerd logs are not properly parsed during ingestion to logstash, a subtask of T362408: Migration to containerd and away from docker, as Resolved.
ops-monitoring-bot added a comment to T362408: Migration to containerd and away from docker.
Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host kubestagemaster2005.codfw.wmnet with OS bookworm completed:
- kubestagemaster2003 (PASS)
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Set boot media to disk
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410181502_jayme_1220197_kubestagemaster2003.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
gerritbot added a comment to T377132: containerd logs are not properly parsed during ingestion to logstash.
Change #1081401 merged by Cwhite:
[operations/puppet@production] logstash/containerd: fix regexp to match also non-json entries
ops-monitoring-bot added a comment to T362408: Migration to containerd and away from docker.
Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host kubestagemaster2005.codfw.wmnet with OS bookworm
ops-monitoring-bot added a comment to T362408: Migration to containerd and away from docker.
Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host kubestagemaster2004.codfw.wmnet with OS bookworm completed:
- kubestagemaster2003 (PASS)
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Set boot media to disk
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410181502_jayme_1220197_kubestagemaster2003.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment to T362408: Migration to containerd and away from docker.
Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host kubestagemaster2004.codfw.wmnet with OS bookworm
ops-monitoring-bot added a comment to T362408: Migration to containerd and away from docker.
Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host kubestagemaster2003.codfw.wmnet with OS bookworm completed:
- kubestagemaster2003 (PASS)
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Set boot media to disk
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410181502_jayme_1220197_kubestagemaster2003.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
JMeybohm updated the task description for T362408: Migration to containerd and away from docker.
ops-monitoring-bot added a comment to T362408: Migration to containerd and away from docker.
Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host kubestagemaster2003.codfw.wmnet with OS bookworm
gerritbot added a comment to T377132: containerd logs are not properly parsed during ingestion to logstash.
Change #1081401 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):
[operations/puppet@production] logstash/containerd: fix regexp to match also non-json entries
ops-monitoring-bot added a comment to T362408: Migration to containerd and away from docker.
Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host kubestagemaster2005.codfw.wmnet with OS bookworm completed:
- kubestagemaster2005 (PASS)
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Set boot media to disk
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410181121_jayme_1191485_kubestagemaster2005.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment to T362408: Migration to containerd and away from docker.
Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host kubestagemaster2005.codfw.wmnet with OS bookworm
gerritbot added a comment to T362408: Migration to containerd and away from docker.
Change #1081224 merged by JMeybohm:
[operations/puppet@production] etcd::v3: Ensure trusted-ca-file is not set on first puppet run with 3.4
gerritbot added a comment to T362408: Migration to containerd and away from docker.
Change #1081377 had a related patch set uploaded (by JMeybohm; author: JMeybohm):
[operations/cookbooks@master] Add a cookbook to roll-reimage stacked k8s control planes
JMeybohm added a comment to T377132: containerd logs are not properly parsed during ingestion to logstash.
This looks great, thanks!
Thu, Oct 17
Thu, Oct 17
gerritbot added a comment to T377132: containerd logs are not properly parsed during ingestion to logstash.
Change #1080603 merged by Cwhite:
[operations/puppet@production] logstash: parse new containerd log format
gerritbot added a comment to T362408: Migration to containerd and away from docker.
Change #1081224 had a related patch set uploaded (by JMeybohm; author: JMeybohm):
[operations/puppet@production] etcd::v3: Ensure trusted-ca-file is not set on first puppet run with 3.4
gerritbot added a comment to T362408: Migration to containerd and away from docker.
Change #992629 merged by JMeybohm:
[operations/puppet@production] etcd::v3: Don't set trusted-ca-file if client-cert-auth is false
gerritbot added a comment to T362408: Migration to containerd and away from docker.
Change #992629 had a related patch set uploaded (by JMeybohm; author: Mxmxchere):
[operations/puppet@production] etcd::v3: Don't set trusted-ca-file if client-cert-auth is false
gerritbot added a comment to T377132: containerd logs are not properly parsed during ingestion to logstash.
Change #1081089 merged by JMeybohm:
[operations/puppet@production] etcd::v3: Add an etcd_version fact
gerritbot added a comment to T377132: containerd logs are not properly parsed during ingestion to logstash.
Change #992629 had a related patch set uploaded (by JMeybohm; author: Mxmxchere):
[operations/puppet@production] etcd::v3: Don't set trusted-ca-file if client-cert-auth is false
gerritbot added a comment to T377132: containerd logs are not properly parsed during ingestion to logstash.
Change #1081089 had a related patch set uploaded (by JMeybohm; author: JMeybohm):
[operations/puppet@production] etcd::v3: Don't set trusted-ca-file if client-cert-auth is false
Wed, Oct 16
Wed, Oct 16
Etonkovidova moved T357122: linkrecommendation-internal regularly uses more than 95% of its memory limit from Inbox to Triaged on the Growth-Team board.