Coding & Development

Opik, built by Comet, is an open-source platform designed to streamline the entire lifecycle of LLM applications, from prototype to production. It empowers developers to evaluate, test, monitor, and optimize their models and agentic systems with comprehensive tracing of LLM calls, conversation logging, and agent activity. Key features include advanced evaluation capabilities like LLM-as-a-judge for tasks such as hallucination detection and RAG assessment, experiment management, and integration into CI/CD pipelines. Opik also offers production-ready scalable monitoring dashboards, online evaluation rules, and dedicated SDKs for prompt and agent optimization, along with guardrails for safe AI practices. It supports a wide array of frameworks and offers client SDKs for Python, TypeScript, and Ruby.

Onepanel is an open-source, end-to-end computer vision platform designed to streamline the entire computer vision lifecycle. It provides a unified environment for labeling datasets, building models, training, tuning hyperparameters, deploying, and automating computer vision workflows. The platform is built to be flexible, supporting deployment on any cloud infrastructure as well as on-premises environments. By integrating various open-source projects like Argo, Couler, CVAT, JupyterLab, and NNI, Onepanel offers a comprehensive solution for machine learning and deep learning practitioners. It aims to simplify complex computer vision tasks from data preparation to model deployment and automation.

TensorDock offers an affordable and easy-to-use cloud GPU infrastructure designed for machine learning, AI, rendering, and cloud gaming. It provides access to a global fleet of GPU servers, including high-end models like NVIDIA H100 and A100, as well as consumer GPUs like RTX 4090, at significantly lower costs than traditional cloud providers. The platform emphasizes on-demand access with no quotas or commitments, allowing users to deploy a server in just 30 seconds. TensorDock also provides CPU cloud services for scientific computing and HPC workloads, root access with KVM virtualization, and a robust API for server management. It caters to a wide range of needs, from individual researchers to AI startups, ensuring secure and reliable enterprise-grade hardware.

redis-inference-optimization is a Redis module designed for serving tensors and executing deep learning graphs. Previously known as RedisAI, this tool acts as a "workhorse" for model serving, offering support for popular Deep Learning and Machine Learning frameworks such as PyTorch, TensorFlow, TensorFlow Lite, and ONNXRuntime. It maximizes computation throughput and reduces latency by adhering to data locality principles, while simplifying the deployment and serving of graphs through Redis's robust infrastructure. Although the project is no longer actively maintained or supported, it provides a valuable reference for integrating AI inference capabilities directly within a Redis environment. Users are directed to the Redis website for current AI offerings.

Qodo-Cover is an AI-powered tool designed to automate test generation and enhance code coverage for software projects. It leverages Generative AI models to streamline development workflows by creating unit tests. The tool can be integrated into GitHub CI workflows or run locally as a CLI tool, supporting various programming languages like Python, Go, and Java. Key components include a Test Runner, Coverage Parser, Prompt Builder, and AI Caller, ensuring tests contribute to overall effectiveness and interact with LLMs for generation. It requires an OpenAI API key and a Cobertura XML code coverage report for functionality, with active development for more coverage types.

Capsolver is an AI-powered automatic CAPTCHA solver designed for web scraping, data extraction, and automation workflows. It supports a wide range of CAPTCHA types including reCAPTCHA, Cloudflare, AWS WAF, and OCR. The platform offers a robust API for integration into existing systems like Selenium, Playwright, and Puppeteer, alongside a browser extension for manual and semi-automated tasks. Capsolver emphasizes enterprise-grade security, customizable solutions, and 24/7 dedicated support for high-volume processing, making it suitable for teams requiring reliable and scalable CAPTCHA management.

Crashtify is an AI-powered incident management solution designed to supercharge incident response for Slack teams. It leverages AI to provide intelligent suggestions drawn from your own knowledge base, past incidents, and even web searches, helping teams resolve issues faster. The platform automates workflows, allowing for seamless integrations with tools like Linear, Jira, and GitHub Issues (coming soon), including automatic ticket creation and bidirectional comment syncing. Crashtify also features a powerful dashboard for managing on-call schedules, tracking incidents, and creating postmortems. Its Smart Knowledge Base learns from team expertise, ensuring relevant solutions are surfaced when needed, and custom fields allow for tailored incident forms. The system is SOC 2 Compliant and multi-tenant ready, making it suitable for various organizational needs.

Text Generation Inference (TGI) is an open-source toolkit designed for deploying and serving Large Language Models (LLMs) with high performance. Developed by Hugging Face, it's used in production for services like Hugging Chat and the Inference API. TGI supports popular open-source LLMs including Llama, Falcon, and BLOOM, offering features such as tensor parallelism for faster inference on multiple GPUs, token streaming, and continuous batching for increased throughput. It also includes optimized transformers code with Flash Attention and Paged Attention, various quantization methods (bitsandbytes, GPT-Q, AWQ, Marlin, fp8), and hardware support for Nvidia, AMD, Inferentia, Intel GPU, Gaudi, and Google TPU. While TGI is now in maintenance mode, it has influenced the development of other optimized inference engines like vLLM and SGLang, which Hugging Face now recommends.

Chonkie is an AI web monitoring platform designed for real-time topic tracking and deep research. It provides always-on intelligence without manual effort, built for teams needing continuous insights from the internet. The platform monitors various sources and summarizes key signals in a UI tailored to specific topics. Users can combine private intelligence with public data by plugging in internal documents, which Chonkie then smartly joins with web sources to provide comprehensive reports. It also allows users to ask follow-up questions for deeper dives, with every answer cited. Chonkie can transform numbers buried across tables, text, and documents into clear graphs, eliminating the need for manual spreadsheet wrangling.

Dynamik provides AI-powered solutions designed to optimize mobile work and material flows for various industries. Their flagship product, Allocator, automates scheduling and route optimization for mobile workforces, reducing travel time and increasing billable hours. It integrates seamlessly with existing ERP/FSM/TMS systems via modern APIs, allowing businesses to leverage AI without massive system overhauls. Allocator considers constraints such as task locations, skill requirements, and time windows to create efficient daily plans. Another key offering is Stackpacker, which uses AI to optimize packaging processes. Dynamik's solutions are cloud-based and aim to improve operational efficiency, reduce costs, and enhance customer service across sectors like logistics, maintenance, installation, and cleaning.

vllm-omni is a framework designed for efficient model inference and serving of omni-modality models, building upon the foundation of vLLM. It expands support beyond text-based autoregressive generation to include text, image, video, and audio data processing. The framework also accommodates non-autoregressive architectures like Diffusion Transformers (DiT) and other parallel generation models, enabling heterogeneous outputs. Key features include state-of-the-art autoregressive support through efficient KV cache management, pipelined stage execution for high throughput, and fully disaggregated architecture with dynamic resource allocation. It offers flexibility with heterogeneous pipeline abstraction, seamless integration with Hugging Face models, and support for various parallelism techniques for distributed inference. vllm-omni also provides streaming outputs and an OpenAI-compatible API server.

Labelbees AI offers enterprise AI infrastructure designed to transform real-world data into reliable signals, enabling the development and scaling of robust AI systems. It addresses common challenges in AI data pipelines, such as inconsistent annotations, temporal drift, and misalignment between data and model requirements. The platform provides a structured pipeline to ingest and normalize raw data from various sources like video, sensors, and documents, ensuring consistency and usability. It also focuses on defining clear ontologies and labeling standards, aligning data across time and interactions, and integrating domain expertise to generate high-quality ground truth data. Labelbees AI ensures continuous evaluation and feedback, maintaining data quality and consistency at scale through structured workflows, automation, and human-in-the-loop systems, making it ideal for production-ready AI.

Mooncake is an open-source serving platform designed for large language models (LLMs), notably powering Kimi by Moonshot AI. It features a KVCache-centric disaggregated architecture that separates prefill and decoding clusters, leveraging underutilized CPU, DRAM, and SSD resources to implement a disaggregated KVCache pool. The core component is the Transfer Engine (TE), which provides a unified interface for batched data transfer across various storage devices and network links, supporting multiple protocols like TCP, RDMA, and NVMe over Fabric. Mooncake also includes P2P Store for sharing temporary objects and Mooncake Store for distributed pooled KVCache, enhancing resource utilization and system performance. It integrates with leading LLM inference systems like vLLM and SGLang for prefill-decode disaggregation and hierarchical KV caching, significantly improving inference efficiency for large-scale distributed tasks.

Hailo offers breakthrough AI processors specifically designed for high-performance deep learning applications on edge devices. Their product portfolio includes AI accelerators like the Hailo-8 and Hailo-10H, which are cost-efficient, low-power co-processors for real-time inference tasks. They also provide AI Vision Processors such as the Hailo-15L and Hailo-15H, which are AI-centric camera SoCs with high-performance AI processing for image enhancement and rich video analytics, including GenAI-powered smart search. Hailo's solutions are geared towards the new era of generative AI on the edge, alongside enabling perception and video enhancement across various applications like robotics, automotive, security, industrial automation, and retail. They also offer a comprehensive AI Software Suite to support their hardware.

RTP-LLM is Alibaba's high-performance LLM inference engine, designed to accelerate large language model deployment across various applications. It is widely utilized within Alibaba Group for services like Taobao, Tmall, and Cainiao. Key features include production-proven reliability, high performance achieved through advanced CUDA kernels like PagedAttention and FlashAttention, and support for WeightOnly INT8/INT4 Quantization. The engine offers flexibility with seamless integration for HuggingFace models, multi-LoRA service deployment, multimodal input handling, and multi-machine/multi-GPU tensor parallelism. It also incorporates advanced acceleration techniques such as Contextual Prefix Cache and Speculative Decoding, making it suitable for optimizing LLM inference in complex, high-demand environments.

Gradio User History is a plugin designed to cache generated images for users within Hugging Face Spaces. This tool aims to improve the user experience by allowing the storage and retrieval of previously generated images, which can be particularly useful for applications involving iterative image generation or for users who frequently revisit their past creations. The plugin requires users to sign in with Hugging Face to access its functionality, ensuring a personalized and secure experience. While the current live website content indicates a runtime error, the intended purpose is to provide a seamless way for users to manage and access their image generation history within Gradio applications hosted on Hugging Face Spaces.

Skild AI is dedicated to developing general-purpose robotic intelligence, aiming to bridge the gap between AI in cyberspace and its application in the physical world. The company's core thesis is to create an "omni-bodied brain" capable of controlling any robot for any task, overcoming limitations of specific robot or task types. Skild AI achieves this by learning from human videos, offering a scalable solution to the robotics data problem. Their technology is applied in various real-world scenarios, including security and inspection robots for navigating unstructured environments, mobile manipulation platforms for tasks like grasping and navigation, and autonomous packing for precise and dexterous skills. This allows users to build applications via an API without delving into the complexities of the physical world.

VerifyFetch is an open-source JavaScript library designed for secure and efficient downloading of large files, particularly AI models like those used by Transformers.js and WebLLM, directly within web browsers. It addresses the limitations of native fetch() by offering streaming integrity verification with a constant 2MB memory footprint, regardless of file size, preventing browser crashes that can occur with large files. Key features include resumable downloads that persist across network failures and page reloads, chunked verification for early corruption detection, and multi-CDN failover. VerifyFetch provides drop-in integrations for popular AI frameworks, CLI tools for hash generation and CI/CD enforcement, and a Service Worker mode for automatic verification of all fetches, enhancing supply chain security for browser-based AI applications.

Hopx offers secure, isolated sandboxes for executing untrusted code, specifically designed for AI agents and autonomous systems. It leverages Linux micro-VMs that launch in approximately 100ms, providing hardware-level security and kernel isolation superior to containers. Users can run Python, JavaScript, Bash, and Go code with real-time streaming output, background execution, and full filesystem operations. Hopx supports various use cases including validating AI-generated code, running long-running jobs, CI/CD testing, data processing, and even desktop automation. The platform provides a simple SDK for integration and an MCP server for IDEs, ensuring code execution is safe, fast, and scalable.

Test-tube is a Python library designed to streamline the logging and parallelization of hyperparameter searches for Deep Learning and Machine Learning experiments. It offers framework-agnostic compatibility, supporting popular libraries like TensorFlow, Keras, PyTorch, and Scikit-learn. Key features include the ability to log experiment hyperparameters and data, visualize results with TensorBoard, and optimize hyperparameters across multiple GPUs or CPUs. It also supports parallel hyperparameter optimization on HPC clusters using SLURM, making it suitable for large-scale research and development. The library is built on the Python argparse API, ensuring ease of use for developers.

TPU-MLIR is an open-source machine learning compiler built on MLIR, specifically designed for Sophgo TPUs. It provides a comprehensive toolchain to convert pre-trained neural networks from various deep learning frameworks, including PyTorch, ONNX, TFLite, and Caffe, into optimized binary files (bmodel) that can run efficiently on TPUs. The project also supports compiling HuggingFace LLM models, with current support for Qwen2 and Llama series, and plans for more. It offers tools for model transformation, deployment, and calibration, enabling users to convert models to different quantization types like F16 and INT8, and provides auxiliary tools for model inference and bmodel manipulation.

vllm-ascend is a community-maintained hardware plugin designed to integrate vLLM with Ascend NPUs, allowing for seamless execution of large language models on Ascend hardware. It adheres to a hardware-pluggable interface, decoupling the integration of Ascend NPUs with vLLM. This plugin supports various open-source models, including Transformer-like, Mixture-of-Experts (MoE), Embedding, and Multi-modal LLMs. It is the recommended approach for supporting the Ascend backend within the vLLM community, enhancing performance for fine-tuning, evaluation, reinforcement learning (RL), and deployment scenarios. The project provides detailed documentation for getting started and contributing, with active development branches and regular releases.

Worktrunk is a command-line interface (CLI) tool built for efficient Git worktree management, specifically tailored for parallel AI agent workflows. It addresses the clunky native Git worktree UX by providing three core commands that make worktrees as straightforward as branches. Beyond the core functionality, Worktrunk offers numerous quality-of-life features, including hooks for automating local workflows, LLM commit message generation, an interactive picker for browsing worktrees, and the ability to copy build caches to skip cold starts. It also supports CI status and AI-generated summaries per branch, and allows for dev servers per worktree, making it an indispensable tool for developers managing complex, parallel development environments, especially when working with AI agents like Claude Code.

The LLM-Engineers-Handbook is an official repository and practical guide for building end-to-end LLM-based systems, developed by Paul Iusztin and Maxime Labonne. It covers essential aspects from data collection and generation to LLM training pipelines, simple RAG systems, and production-ready AWS deployment. The handbook emphasizes LLMOps best practices, including comprehensive monitoring, testing, and evaluation frameworks. It details the use of various tools and cloud services like HuggingFace, Comet ML, Opik, ZenML, AWS, MongoDB, Qdrant, and GitHub Actions. The repository provides actively maintained code, installation instructions, and guidance on setting up local development and cloud deployment environments.