De Python a Kubeflow

Abstract

[EN] With the rapid advancement of technology development for various purposes, companies, institutions and even individuals are contracting computing capacity and storage in Cloud environments, which allow iterating any solution more quickly and efficiently, leaving such vital matters in the hands of the provider. how the maintenance of the infrastructure and its scalability. Providers such as AWS, GCP and Azure (United States) and to a lesser extent, OVH Cloud (Europe) have a large share of the market, offering solutions for practically any problem.

Applications run in these environments are often replicated and scaled through virtualization and the use of containers. When the number of these containers (replicas of the same server or a web application, for example) becomes large, the use of some orchestration tool is essential, which allows managing, updating and scaling these containers in an organized and sustainable way. . For this purpose, the most widely used tool today is Kubernetes.

On the other hand, many organizations, whether public (universities and research centers) or private (companies) are increasingly interested in deep learning, training neural networks (NN), which can have very varied functions. that allow them to solve the problems they face.

The objective of the TFG is to take an evolutionary tour of the tools that can be used to train NN, analyzing the pros and cons of each one of them. Work environments will be created and executed that allow real tests to be carried out with each of the tools used. The database with which the NN will be trained will be that of the MNIST (database of handwritten digits), being initially implemented in Python. Throughout the dissertation, this approach will evolve through the use of libraries such as NumPy, Jupyter Notebooks, TensorFlow and Keras to make the environment that enables NN training more friendly, agile and effective.

Once these tools have been analyzed, we will jump to the Cloud environment where we will be able to train the same network using KubeFlow, a set of tools that automates all the training steps using a K8S cluster as execution support. It will start with a local deployment using MiniKF, a tool that allows KubeFlow to be used on a local virtual machine or in a cloud environment. Finally, a real K8S cluster will be deployed on one of the previously mentioned providers and the Kubeflow toolset will be installed on said cluster. This open source tool will be compared with solutions from the cloud environment.

[ES] El trabajo en el campo de la inteligencia artificial y específicamente en el del aprendizaje automático con las redes neuronales como punta de lanza, está cobrando más relevancia que nunca. La aparición de herramientas que emplean modelos entrenados como GPT-3 o GPT-4 han popularizado el uso de la inteligencia artificial a través de aplicaciones como ChatGPT o Claude. Dentro de este contexto y del avance futuro de campos como la IA generativa o el reconocimiento del lenguaje natural, se resalta la relevancia y el impacto de los recursos educativos que permiten introducir de forma eficaz y didáctica a cualquier ingeniero en estos temas. Siguiendo esta línea, el trabajo propuesto aborda el análisis, configuración y uso de tecnologías que permiten entrenar y poner a punto modelos de redes neuronales, de forma robusta, escalable y eficiente en recursos.

[EN] The work in the field of artificial intelligence, specifically in machine learning with neural networks as the spearhead, is gaining more relevance than ever. The emergence of tools that employ trained models like GPT-3 or GPT-4 has popularized the use of artificial intelligence through applications like ChatGPT or Claude. Within this context and the future advancement of areas such as generative AI or natural language recognition, the significance and impact of educational resources that effectively and didactically introduce any engineer to these subjects are highlighted. Following this line, the proposed work addresses the analysis, configuration, and utilization of technologies that enable the training and fine-tuning of neural network models in a robust, scalable, and resource-efficient manner.