Milestone for Data Platform SRE work
Details
Details
Description
Yesterday
Yesterday
Maintenance_bot removed a project from T376406: Import the upstream ceph-csi-cephfs chart and adapt it to our needs: Patch-For-Review.
gerritbot added a comment to T376406: Import the upstream ceph-csi-cephfs chart and adapt it to our needs.
Change #1081911 merged by Brouberol:
[operations/puppet@production] ceph/server: fix typo in caps
gerritbot added a comment to T376406: Import the upstream ceph-csi-cephfs chart and adapt it to our needs.
Change #1081911 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] ceph/server: fix typo in caps
Maintenance_bot removed a project from T376406: Import the upstream ceph-csi-cephfs chart and adapt it to our needs: Patch-For-Review.
gerritbot added a comment to T376406: Import the upstream ceph-csi-cephfs chart and adapt it to our needs.
Change #1081905 merged by Brouberol:
[operations/puppet@production] ceph/server: fix the dse-k8s-csi-cephfs according to the CSI doc
gerritbot added a comment to T376406: Import the upstream ceph-csi-cephfs chart and adapt it to our needs.
Change #1081903 merged by Brouberol:
[operations/deployment-charts@master] ceph-csi-cephs: fix RBAC by granting cluster-wide permisions on PVC and storageclasses
gerritbot added a comment to T376406: Import the upstream ceph-csi-cephfs chart and adapt it to our needs.
Change #1081905 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] ceph/server: fix the dse-k8s-csi-cephfs according to the CSI doc
gerritbot added a comment to T376406: Import the upstream ceph-csi-cephfs chart and adapt it to our needs.
Change #1081903 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/deployment-charts@master] ceph-csi-cephs: fix RBAC by granting cluster-wide permisions on PVC and storageclasses
Fri, Oct 18
Fri, Oct 18
BTullis reopened T353788: Add kafka-stretch100[1-2] to the hadoop cluster as "Open".
Ah, there is a slight problem with an-worker1176. It won't take long to sort out.
The partition table on /dev/sda looks like it was created for the operating system disk, which makes sense.
We can see here that /dev/sda1 is only 1 GB in size, with 3.6TB unused in /dev/sda2
btullis@an-worker1176:~$ lsblk /dev/sda NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 3.6T 0 disk ├─sda1 8:1 0 953M 0 part /var/lib/hadoop/data/m └─sda2 8:2 0 3.6T 0 part
I think that we will have to modify the partition table and resize the file system.
xcollazo added a comment to T351388: Make Airflow SparkSQL operator set fileoutputcommitter.algorithm.version=2 to avoid concurrent write issues.
BTullis renamed T351388: Make Airflow SparkSQL operator set fileoutputcommitter.algorithm.version=2 to avoid concurrent write issues from Make Airflow SaprkSQL operator set fileoutputcommitter.algorithm.version=2 to avoid concurrent write issues to Make Airflow SparkSQL operator set fileoutputcommitter.algorithm.version=2 to avoid concurrent write issues.
CodeReviewBot added a comment to T351388: Make Airflow SparkSQL operator set fileoutputcommitter.algorithm.version=2 to avoid concurrent write issues.
Maintenance_bot removed a project from T375714: Cleanup obsolete DAGs from Datahub: Patch-For-Review.
CodeReviewBot added a comment to T375714: Cleanup obsolete DAGs from Datahub.
gerritbot added a comment to T374948: Migrate airflow webservers to Kubernetes.
Change #1081261 merged by Bking:
[labs/private@master] analytics_test_cluster: add secret
Jelto added a comment to T350793: move commons-query.wikimedia.org and query.wikidata.org to kubernetes.
The custom-config.json has been added and is now available on Wikikube:
gerritbot added a comment to T351388: Make Airflow SparkSQL operator set fileoutputcommitter.algorithm.version=2 to avoid concurrent write issues.
Change #975006 abandoned by Btullis:
[operations/puppet@production] Set a non-default mapreduce file committer algorithm for spark
Reason:
See: https://phabricator.wikimedia.org/T351388#10237936
gerritbot added a comment to T350793: move commons-query.wikimedia.org and query.wikidata.org to kubernetes.
Change #1081382 merged by jenkins-bot:
[operations/deployment-charts@master] wikidata-query-gui: fix volumeMount with subPath
gerritbot added a comment to T350793: move commons-query.wikimedia.org and query.wikidata.org to kubernetes.
Change #1081382 had a related patch set uploaded (by Jelto; author: Jelto):
[operations/deployment-charts@master] wikidata-query-gui: fix volumeMount with subPath
gerritbot added a comment to T350793: move commons-query.wikimedia.org and query.wikidata.org to kubernetes.
Change #1079466 merged by jenkins-bot:
[operations/deployment-charts@master] wikidata-query-gui: mount custom-config.json into pod
gerritbot added a comment to T350793: move commons-query.wikimedia.org and query.wikidata.org to kubernetes.
Change #1079465 merged by jenkins-bot:
[operations/deployment-charts@master] miscweb: add support to mount add confimaps
Stevemunene updated the task description for T353788: Add kafka-stretch100[1-2] to the hadoop cluster.
BTullis added a comment to T376657: Unable to find ingested tables in datahub.
When I do a search for discovery in DataHub I can see 17 hive tables returned.
I exported the results as a CSV and the entity URLs are here:
- https://datahub.wikimedia.org/dataset/urn:li:dataset:(urn:li:dataPlatform:hive,discovery.webrequest_metrics,PROD)
- https://datahub.wikimedia.org/dataset/urn:li:dataset:(urn:li:dataPlatform:hive,discovery.wikibase_rdf_subgraphs,PROD)
- https://datahub.wikimedia.org/dataset/urn:li:dataset:(urn:li:dataPlatform:hive,discovery.cirrus_index,PROD)
- https://datahub.wikimedia.org/dataset/urn:li:dataset:(urn:li:dataPlatform:hive,discovery.cirrus_index_without_content,PROD)
- https://datahub.wikimedia.org/dataset/urn:li:dataset:(urn:li:dataPlatform:hive,discovery.search_satisfaction_metrics,PROD)
- https://datahub.wikimedia.org/dataset/urn:li:dataset:(urn:li:dataPlatform:hive,discovery.wikibase_rdf,PROD)
- https://datahub.wikimedia.org/dataset/urn:li:dataset:(urn:li:dataPlatform:hive,discovery.subgraph_pair_metrics,PROD)
- https://datahub.wikimedia.org/dataset/urn:li:dataset:(urn:li:dataPlatform:hive,discovery.general_subgraph_metrics,PROD)
- https://datahub.wikimedia.org/dataset/urn:li:dataset:(urn:li:dataPlatform:hive,discovery.popularity_score,PROD)
- https://datahub.wikimedia.org/dataset/urn:li:dataset:(urn:li:dataPlatform:hive,discovery.query_clicks_daily,PROD)
- https://datahub.wikimedia.org/dataset/urn:li:dataset:(urn:li:dataPlatform:hive,discovery.subgraph_pair_query_metrics,PROD)
- https://datahub.wikimedia.org/dataset/urn:li:dataset:(urn:li:dataPlatform:hive,discovery.query_clicks_hourly,PROD)
- https://datahub.wikimedia.org/dataset/urn:li:dataset:(urn:li:dataPlatform:hive,discovery.general_query_metrics,PROD)
- https://datahub.wikimedia.org/dataset/urn:li:dataset:(urn:li:dataPlatform:hive,discovery.general_subgraph_query_metrics,PROD)
- https://datahub.wikimedia.org/dataset/urn:li:dataset:(urn:li:dataPlatform:hive,discovery.per_subgraph_metrics,PROD)
- https://datahub.wikimedia.org/dataset/urn:li:dataset:(urn:li:dataPlatform:hive,discovery.per_subgraph_query_metrics,PROD)
- https://datahub.wikimedia.org/dataset/urn:li:dataset:(urn:li:dataPlatform:hive,discovery.processed_external_sparql_query,PROD)
BTullis added a comment to T375716: Ensure the Airflow API can be reached out to from within Kubernetes and is authenticated.
Yes. Each Airflow instance will have a kerberos principal associated with it anyway. The workers will need this principal in order to access HDFS, YARN, and Hive.
We then supply each instance with a keytab which is a means of making sure that services can automatically authenticate with their given Kerberos principal.
Stevemunene moved T353788: Add kafka-stretch100[1-2] to the hadoop cluster from In Progress to Done on the Data-Platform-SRE (2024.09.28 - 2024.10.18) board.
The hosts are now set back to Active on netbox, and are now part of the hadoop cluster and are catching up to the production hosts. Keeping an eye on this
Stevemunene moved T369582: Enable prometheus metrics on the cephosd cluster from In Progress to Done on the Data-Platform-SRE (2024.09.28 - 2024.10.18) board.
Thu, Oct 17
Thu, Oct 17
gerritbot added a comment to T377490: Requesting access to airflow-analytics-product-admins for jebe.
Change #1081285 had a related patch set uploaded (by Eevans; author: Eevans):
[operations/puppet@production] Add jebe to airflow-analytics-product-admins per access request
Eevans updated the task description for T377490: Requesting access to airflow-analytics-product-admins for jebe.
odimitrijevic added a comment to T377490: Requesting access to airflow-analytics-product-admins for jebe.
Approved
gerritbot added a comment to T374948: Migrate airflow webservers to Kubernetes.
Change #1081268 had a related patch set uploaded (by Bking; author: Bking):
[operations/puppet@production] airflow: make 'secret_key' configurable
Eevans added a comment to T377490: Requesting access to airflow-analytics-product-admins for jebe.
Eevans updated the task description for T377490: Requesting access to airflow-analytics-product-admins for jebe.
bking updated the task description for T374948: Migrate airflow webservers to Kubernetes.
gerritbot added a comment to T374948: Migrate airflow webservers to Kubernetes.
Change #1081261 had a related patch set uploaded (by Bking; author: Bking):
[labs/private@master] analytics_test_cluster: add secret
Ottomata added a comment to T376882: 2024-10-10 Data Loss Incident - webrequest Hive table .
FYI, just updated Ops week page with docs on using Airflow cli to rerun tasks:
bking updated the task description for T374948: Migrate airflow webservers to Kubernetes.
mpopov added a comment to T377490: Requesting access to airflow-analytics-product-admins for jebe.
I approve membership in airflow-analytics-product-admins
Maintenance_bot removed a project from T374948: Migrate airflow webservers to Kubernetes: Patch-For-Review.
gerritbot added a comment to T374948: Migrate airflow webservers to Kubernetes.
Change #1081230 merged by jenkins-bot:
[operations/deployment-charts@master] airflow-analytic-test: disable remote logging