Shypd.aiThe largest open index of AI toolsspark-deep-learning is an open-source deep learning pipeline library for Apache Spark. It enables distributed deep learning training jobs using Horovod, making it easier to scale machine learning workloads on Spark clusters.
Trending
About
spark-deep-learning provides Deep Learning Pipelines for Apache Spark, focusing on distributed deep learning training jobs using Horovod. While the open-source version primarily supports local development, it integrates seamlessly with Databricks Runtime for Machine Learning to launch Horovod jobs as distributed Spark jobs. This allows data scientists and engineers to manage cluster setup and scale their deep learning models efficiently. The library offers functionalities for running training code across multiple processes, with options to specify the number of parallel processes for optimal resource utilization on GPU or CPU clusters. It simplifies the deployment and execution of complex deep learning tasks within the Spark ecosystem.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
