It seems that it is simply another case of stalled documentation, what you are looking for is in mwlog1002:/srv/mw-log/JobExecutor.log. I updated that bit on Wikitech. That information is also on logstash, however, to my knowledge, are currently lacking a beautiful dashboard for that.

In T372800#10199506, @Jhancock.wm wrote:

@jijiki hi, we got the servers in this week and are going to be racking them today. Could you update the operations and puppet repo for us? I'm hoping to get them installed and to you before the end of the week. thanks!

In the immediate term, we should determine whether to either accept this as the new normal (i.e., being "ready" for single-DC serving at steady state) or scale back down to multi-DC sizing

In T365687#10205624, @JMeybohm wrote:

Providing certs to pods

• Secrets
  • The certificates could be secrets which we can then mount as files

As per our initial discussion I would rather not do this as it created an additional (potentially manual) step.

Yes, I was just writing down our options.

• Files on workers
  • We could have those certs present on all workers and mount them via host path

Open questions

• certs expiration?
• certs renewal

That was poorly phrased, I meant, how often would we want the certificates to be renewed

Puppet and PKI will take care of both automatically. The open question here is if typha and felix are capable of detecting changes to the certs on disk and hot reloads.

Rephrased the task description accordingly

I have looked into this a little bit, but sadly I have not found yet anythat could be causing this. @TK-999 let us know if your patch improves the issue

Any other cleanup work will be attached under T371378, marking this as done

@Ladsgroup and I believe this works alright, https://wikitech-static.wikimedia.org looks as expected. I will check tomorrow if there is anything else we may have missed, and mark it as done. Further code cleanups will be worked T371378

Thanks @taavi for pointing it out, we'll try to find a temporary bandaid to keep dumps running, and then re-design it

@Jclark-ctr very sorry for missing your message. I added the hosts in site.pp, there is already a 'mc*' reference in preseed.yaml to match.

In T370755#10186060, @santhosh wrote:

@akosiaris Horizontal scaling is required, we will reach out for that separatly. Very soon.

The nature of machine translation requests and processing is, for many of upcoming usecases, the input will be same as something we already translated and output is same as long as the machine translation model does not change. And as machine translation is CPU intensive, it is a good idea to cache.

@jijiki The translation of some word/phrase/sentence is going to be same between restarts, application deployments. We aleady have a non-persisting fallback caching in our application. What we are looking for is persisting cache solution. Memcache does not help there right?

@Ladsgroup and I flipped the relevant switches today, for the time being all seems well.

The de-wikitech patch has been merged, however I am leaving this open for a few more days in case something pops up

Wikitech is using our main memcached cluster, so far so good.

LDAP extension has been removed.

Rolled out the apache change.

Jobs are running on mw-jobrunner

ATS is done