Shypd.aiThe largest open index of AI toolsPetastorm is an open-source data access library that enables single-machine or distributed training and evaluation of deep learning models directly from datasets in Apache Parquet format. It supports popular ML frameworks like Tensorflow, PyTorch, and PySpark.
Trending
About
Petastorm is an open-source data access library developed at Uber ATG, designed to facilitate the training and evaluation of deep learning models. It supports both single-machine and distributed environments, working directly with datasets stored in Apache Parquet format. The library is compatible with major Python-based machine learning frameworks including Tensorflow, PyTorch, and PySpark, and can also be utilized from pure Python code. Petastorm offers features such as selective column readout, multiple parallelism strategies (thread, process, single-threaded), N-grams readout support, row filtering, shuffling, partitioning for multi-GPU training, and local caching. It also simplifies data conversion from Spark DataFrames to TensorFlow or PyTorch datasets.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
