Research & Education

Self-attention-cv is an open-source repository offering implementations of diverse self-attention mechanisms specifically tailored for computer vision applications. Built in PyTorch, it leverages `einsum` and `einops` for efficient and flexible module creation. The repository serves as an ongoing collection of building blocks, enabling developers to integrate advanced attention models into their projects. It supports a range of computer vision tasks, including image recognition and segmentation, with examples for Multi-head attention, Axial attention, Vision Transformers (ViT), and TransUnet. It also includes various positional embedding implementations.

ZeroCostDL4Mic is a free and open-source toolbox designed to democratize deep learning in microscopy. It consists of a collection of self-explanatory Jupyter Notebooks, hosted on Google Colab, which provides the necessary computational resources at no cost. The tool features an easy-to-use graphical user interface, making it accessible for researchers with little or no coding expertise. Its primary goal is to allow users to quickly test, train, and utilize popular Deep-Learning networks for processing microscopy data. This project originated from a collaboration between the Jacquemet and Henriques laboratories and has expanded with global contributions, as acknowledged in their Nature Communications paper.

BiRefNet is an open-source project offering a powerful solution for high-resolution dichotomous image segmentation, as detailed in the CAAI AIR 2024 paper. It provides official implementations and well-trained weights for various tasks, including general image segmentation, matting, Dichotomous Image Segmentation (DIS), High-Resolution Salient Object Detection (HRSOD), and Co-Salient Object Detection (COD). The tool supports dynamic resolution ranges, from 256x256 up to 2304x2304, and demonstrates robust performance across different image sizes. Users can leverage its capabilities through Hugging Face Models for easy integration or explore online demos for inference and evaluation. BiRefNet also supports ONNX conversion for efficient deployment and has been integrated into several third-party applications and frameworks, making it accessible for both researchers and developers.

MARLlib is a comprehensive, open-source library designed for Multi-agent Reinforcement Learning (MARL), leveraging Ray and its RLlib toolkit. It offers a unified platform for researchers and developers to create, train, and evaluate MARL algorithms across a wide array of tasks and environments. Key features include support for all task modes (cooperative, collaborative, competitive, mixed), a Gym-like interface for multi-agent environments, and flexible parameter-sharing strategies. MARLlib provides 18 pre-built algorithms with an intuitive API, making it accessible even for those new to MARL. Users can customize model architectures, policy sharing, and access over a thousand released experiments. It is compatible with Linux operating systems and offers step-by-step installation or Docker-based usage.

cnn-text-classification-pytorch is an open-source implementation of Convolutional Neural Networks (CNNs) for sentence classification, built using PyTorch. This tool is based on the model described in Kim's influential paper on CNNs for Sentence Classification. It offers a practical framework for developers to perform text classification tasks, providing consistent results with the original research. The implementation has been updated to be compatible with modern PyTorch versions (2.0+), removing deprecated dependencies like `torchtext` and fixing various runtime errors. It supports datasets like MR and SST, includes options for different optimizers (Adam, Adadelta), and allows for easy training, testing, and prediction of text sentiment.

MM-EUREKA is a cutting-edge project exploring the frontiers of multimodal reasoning through rule-based reinforcement learning. It introduces powerful models such as MM-Eureka-Qwen-7B and MM-Eureka-Qwen-32B, which significantly advance performance in multidisciplinary K12 and mathematical reasoning tasks. The project has iterated on model architecture, algorithms, and data, moving from InternVL to the more robust Qwen2.5-VL base models. Key improvements include enhanced online filtering, adaptive online rollout adjustment (ADORA), and novel RL algorithms like Clipped Policy Gradient Optimization with Policy Drift (CPGD). MM-EUREKA also open-sources a comprehensive pipeline, including self-collected MMK12 datasets, to foster further research and development in multimodal AI.

DeePMD-kit is a Python/C++ package designed to facilitate the creation of deep learning-based models for interatomic potential energy and force fields, and to perform molecular dynamics simulations. It addresses the accuracy-versus-efficiency dilemma in molecular simulations by leveraging deep learning. The package is highly modularized and interfaces with popular deep learning frameworks like TensorFlow, PyTorch, JAX, and Paddle, as well as high-performance classical and quantum MD packages such as LAMMPS, i-PI, and GROMACS. It implements the Deep Potential series models, which have been successfully applied to various systems, including organic molecules, metals, and semiconductors. DeePMD-kit also supports MPI and GPU for efficient parallel and distributed computing, making it suitable for complex scientific research.

Deepgaze is an open-source computer vision library designed for human-computer interaction, providing advanced capabilities for analyzing human behavior through visual data. It leverages Convolutional Neural Networks (CNNs) for precise head pose and gaze direction estimation, which is crucial for understanding a person's focus of attention, even when eyes are obscured or far from the camera. Beyond CNN-based estimation, Deepgaze incorporates features like skin detection via backprojection, robust motion detection and tracking, and saliency map generation using the FASA algorithm. Built on OpenCV and TensorFlow, it offers optimized, state-of-the-art algorithms, making complex implementations accessible with just a few lines of code for both beginners and advanced users in computer vision and machine learning.

LimSim is a Long-term Interactive Multi-scenario traffic Simulator designed to provide continuous simulation capabilities within complex urban road networks. It features long-term traffic flow generation based on demand and route planning, diverse behavioral models for heterogeneous driving styles, and interactivity to manage sophisticated vehicle interactions. The simulator supports multiple scenarios including freeways, signalized intersections, roundabouts, and overpasses. LimSim also includes a cross-platform GUI for visualizing simulations, road networks, and ego-vehicle status. It can generate log reports and extract key scenarios for evaluation after long-term simulations. Notably, LimSim supports co-simulation with CARLA and SUMO, ensuring identical vehicle status across platforms, and LimSim++ offers Multimodal LLM support.

PyTorch-BayesianCNN provides an implementation of Bayesian Convolutional Neural Networks (CNNs) with variational inference, specifically utilizing Bayes by Backprop, within the PyTorch framework. This tool allows researchers and developers to build CNNs that can infer intractable posterior probability distributions over weights, offering a significant advantage over traditional frequentist approaches by providing uncertainty estimations. It includes two types of Bayesian layer implementations: BBB (Bayes by Backprop) and BBB_LRT (Bayes by Backprop with Local Reparametrization Trick), which enhances sampling efficiency. The repository supports standard datasets like MNIST, CIFAR10, and CIFAR100, and includes implementations of common models such as AlexNet and LeNet, making it a valuable resource for experimenting with Bayesian deep learning and understanding model uncertainty.

RoseTTAFold is a deep learning model and script package designed for the accurate prediction of protein structures and interactions. This tool is an official implementation of the RoseTTAFold architecture, which employs a 3-track neural network to achieve its predictions. It is primarily intended for research in computational biology, enabling scientists to model complex protein structures and protein-protein interactions (PPIs). The package includes scripts for installation, dependency management, and running predictions for both monomer structures and complex modeling. It also features a faster 2-track version for PPI screening, making it a versatile tool for advanced biological research.

SimGNN is a PyTorch implementation of a novel neural network approach designed for fast graph similarity computation, as detailed in the WSDM 2019 paper. It addresses the computational burden of traditional methods like Graph Edit Distance (GED) and Maximum Common Subgraph (MCS) while maintaining high performance. The tool employs a learnable embedding function to map graphs into embedding vectors, providing a global summary. A key feature is its attention mechanism, which emphasizes important nodes for specific similarity metrics. Additionally, SimGNN includes a pairwise node comparison method to supplement graph-level embeddings with fine-grained node-level information. This approach leads to better generalization on unseen graphs and offers quadratic time complexity in the worst case. Experimental results demonstrate its effectiveness and efficiency, achieving smaller error rates and significant time reductions compared to existing baselines.

TensorFlow Federated (TFF) is an open-source framework designed for machine learning and other computations on decentralized data. It specifically supports Federated Learning (FL), an approach where a shared global model is trained across many participating clients while their sensitive training data remains local. This framework enables developers to utilize included federated learning algorithms with their existing TensorFlow models and data, or to experiment with novel algorithms. TFF provides both a high-level Federated Learning (FL) API for applying federated training and evaluation, and a lower-level Federated Core (FC) API for expressing new federated algorithms. It includes a single-machine simulation runtime for experiments, making it suitable for researchers and developers exploring privacy-preserving machine learning.

Thesis is an AI-native platform designed for data science and machine learning, offering an environment where researchers can build and deploy frontier models. The platform allows ML research scientists to run experiments and train models autonomously and at scale within its datacenters. Key features include an intuitive interface for managing datasets, experiments, and models, as well as tools for exploratory data analysis (EDA) and lineage tracking for model development. Thesis aims to accelerate AI R&D, making it easier for data scientists to turn curiosity into consequential discoveries. It offers both a free Spark plan and a 'Pay as you go Ultra' option for production workloads.

Veles is a distributed platform designed for rapid deep learning application development, released under the Apache 2.0 license. It comprises several key components, including the core Veles platform, the Znicz Plugin which serves as a neural network engine, and Mastodon, a bridge facilitating integration between Veles and Java-based systems like Hadoop. Additionally, it features a SoundFeatureExtraction library for audio processing. This platform is ideal for developers and researchers looking to build and deploy deep learning applications in a distributed environment, offering tools for both model development and data processing.

Bert-Multi-Label-Text-Classification offers a PyTorch implementation of pretrained BERT and XLNET models specifically tailored for multi-label text classification. This open-source repository includes a structured codebase with modules for callbacks, configuration, dataset handling, model architecture, output management, text preprocessing, and training. Developers can fine-tune BERT models, preprocess data, and predict new data using provided scripts. The tool supports various dependencies like PyTorch, transformers, and scikit-learn, making it a robust solution for NLP tasks requiring multi-label classification.

aTrain is a powerful GUI tool designed for offline transcription of speech recordings, leveraging state-of-the-art machine learning models for high accuracy and speed. Developed by researchers at the University of Graz, it features speaker diarization to identify different speakers in a recording. A key differentiator is its commitment to privacy, processing all data locally on your device without internet uploads, ensuring GDPR compliance. It supports transcription in 99 languages and offers compatibility with popular qualitative analysis tools like MAXQDA, ATLAS.ti, and nVivo. The tool can run on both CPU and NVIDIA GPUs, with GPU support significantly reducing transcription times.

evidential-deep-learning is an open-source Python package designed to help neural networks learn their own measures of uncertainty directly from data. It provides the necessary code to reproduce the Deep Evidential Regression paper published in NeurIPS 2020, offering a general framework for evidential learning. The tool allows users to integrate evidential layers and loss functions into existing `tf.keras` model pipelines, supporting both fully connected and convolutional layers. This enables the development of models that can provide fast, scalable, and calibrated measures of uncertainty, enhancing their trustworthiness and utility. The package is compatible with Python (>=3.7) and TensorFlow (>=2.0), with PyTorch support planned.

The NVIDIA Data Loading Library (DALI) is a GPU-accelerated library designed to optimize data loading and pre-processing for deep learning applications. It offers a collection of highly optimized building blocks and an efficient execution engine, specifically tailored for processing image, video, and audio data. DALI addresses the common bottleneck of CPU-bound data pipelines by offloading these tasks to the GPU, significantly enhancing performance and scalability for training and inference. It supports various data formats and is portable across popular deep learning frameworks like TensorFlow, PyTorch, and PaddlePaddle. Key features include prefetching, parallel execution, batch processing, and extensibility for custom operators, making it a versatile solution for accelerating complex deep learning workflows.

deepnet is an open-source project providing GPU-based Python implementations of several deep learning algorithms. It supports a range of models including feed-forward neural networks, Restricted Boltzmann Machines, Deep Belief Nets, Autoencoders, Deep Boltzmann Machines, and Convolutional Neural Nets. Built upon the cudamat library by Vlad Mnih and cuda-convnet library by Alex Krizhevsky, deepnet offers a foundational resource for developers and researchers working with deep learning. Its focus on core algorithm implementations makes it a valuable tool for understanding and experimenting with these fundamental AI architectures.

tiny-dnn is a C++14 implementation of deep learning, designed for environments with limited computational resources, such as embedded systems and IoT devices. It stands out as a header-only and dependency-free framework, meaning there's nothing to install beyond a C++14 compiler. This makes it highly portable and easy to integrate into existing applications. The framework supports a variety of network layers, activation functions, loss functions, and optimization algorithms, allowing for the construction of diverse deep learning models. It offers reasonable speed without a GPU, leveraging TBB threading and SSE/AVX vectorization. Additionally, tiny-dnn can import models from Caffe and provides a simple, exception-free operational model, making it a good choice for learning neural networks.

camel_tools is a comprehensive, open-source Python toolkit developed by the CAMeL Lab at New York University Abu Dhabi, specifically designed for Arabic natural language processing. It offers a wide array of functionalities including text pre-processing, advanced morphological modeling, and specialized components for Dialect Identification, Named Entity Recognition, and Sentiment Analysis. The tool is built to be accessible for researchers and developers, with clear installation instructions for various operating systems like Linux, macOS, and Windows. It also provides options for installing necessary data packages, making it a robust solution for anyone working with the complexities of the Arabic language in NLP tasks.

Cron AI specializes in next-generation 3D perception, leveraging cutting-edge deep learning algorithms to process raw data from 3D sensors such as LiDAR. Their flagship senseEDGE platform provides unparalleled accuracy and intelligence in object detection, classification, and tracking, even in challenging environments and adverse weather conditions. It goes beyond traditional methods, offering adaptive flexibility for seamless object detection across varied settings, geographies, and sensor types. The platform is designed for easy deployment at the edge, scaling effortlessly from single-sensor solutions to complex deployments. Cron AI's technology is crucial for intelligent transportation systems, smart spaces, smart security, and automotive applications, ensuring consistent and precise results while being resource-efficient and GDPR compliant.

Stellon Labs is an AI research lab dedicated to developing powerful, tiny AI models specifically optimized for edge applications. Their focus is on creating 'frontier AI' solutions that can operate efficiently on minimal hardware, making advanced artificial intelligence accessible for devices with limited computational resources. The lab aims to push the boundaries of AI performance in constrained environments, enabling new possibilities for on-device intelligence without requiring extensive infrastructure. Their work is geared towards practical applications where low-power and small-footprint AI is crucial.