A hack we properly explore at Bumble is ClearML
From the Bumble Inc

Now certain meats for all you practitioners that want having tooling, recommendations, knowledge, the system discovering program is made with the foundations and you may tissues. Once again, the reason for the system understanding system is to abstract difficulty to view measuring resources. While somebody who has experience in dealing with such rules, hears abstraction, difficulty, especially complexity and you may calculating information, Kubernetes is the product that comes in your thoughts. , we have a private cloud, therefore we enjoys some other Kubernetes clusters that allow us to contract in order to abstract making use of the various other measuring resources. We have groups that have hundreds of GPU tips in different countries. I deploy this Kubernetes people so new supply to the information try totally abstracted to any or all that just necessary entry to GPU. Machine understanding therapists otherwise has MLEs later on need to have since the needs, okay, I would like to fool around with an incredibly larger GPU, they should following actually know or make life a headache to really availability these GPUs, to ensure all the CUDA people is actually hung accurately. Kubernetes can there be thus. They just need certainly to say, okay, I’d like good GPU, so when if it are secret, Kubernetes is going to let them have the new information they want.

