# Ray > Ray is an open-source unified compute framework for scaling AI and Python workloads, including data processing, model training, model serving, hyperparameter tuning, and reinforcement learning. The full documentation lives at https://docs.ray.io/. Full page text, grouped by section, is in [llms-full.txt](https://docs.ray.io/en/latest/llms-full.txt). Request the links that follow with the HTTP header `Accept: text/markdown` to retrieve a Markdown rendering of the page. ## Overview - [Overview](https://docs.ray.io/en/latest/ray-overview/index.html): Introduction to Ray, the open-source framework for scaling AI and Python workloads — its core primitives, AI libraries, and cluster integrations. - [Ray Examples](https://docs.ray.io/en/latest/ray-overview/examples.html): Gallery of end-to-end Ray examples spanning data processing, model training, serving, and hyperparameter tuning workloads. ## Getting Started - [Getting Started](https://docs.ray.io/en/latest/ray-overview/getting-started.html): Quickstart paths for Ray: scale ML workloads with the AI libraries, parallelize Python with Ray Core, or deploy to the cloud with Ray Clusters. ## Installation - [Installing Ray](https://docs.ray.io/en/latest/ray-overview/installation.html): Install Ray and its AI libraries with pip across Linux, Windows (beta), and macOS on x86_64, ARM, and Apple silicon hardware. ## Use Cases - [Ray Use Cases](https://docs.ray.io/en/latest/ray-overview/use-cases.html): Common Ray use cases: LLMs and generative AI, batch inference, distributed training, hyperparameter tuning, reinforcement learning, and ML platforms. - [Ray for ML Infrastructure](https://docs.ray.io/en/latest/ray-air/getting-started.html): Overview of using Ray as ML infrastructure, covering how Ray Train, Ray Data, and Ray Serve compose end-to-end ML workflows on a unified compute runtime. ## Examples - [Examples](https://docs.ray.io/en/latest/ray-overview/examples/index.html): End-to-end Ray examples: multimodal AI, audio and timeseries, XGBoost, object detection, RAG, and serving LLM agents with MCP and LangChain. - [Multi-modal AI pipeline](https://docs.ray.io/en/latest/_collections/ray-overview/examples/e2e-multimodal-ai-workloads/README.html): 💻 Run this entire tutorial on Anyscale for free: https://console.anyscale.com/template-preview/image-search-and-classification or access the repository here. - [LLM training and inference](https://docs.ray.io/en/latest/_collections/ray-overview/examples/entity-recognition-with-llms/README.html): This end-to-end tutorial fine-tunes an LLM to perform batch inference and online serving at scale. While entity recognition (NER) is the main task in this tutorial, you can easily extend these end-to-end workflows to any use case. - [Audio batch inference](https://docs.ray.io/en/latest/_collections/ray-overview/examples/e2e-audio/README.html): This tutorial demonstrates a batch inference pipeline that converts raw audio files into a curated subset using two different ML models. The pipeline chains together four stages that each run on the same Ray cluster using heterogeneous resources: - [Distributed XGBoost pipeline](https://docs.ray.io/en/latest/_collections/ray-overview/examples/e2e-xgboost/README.html): These tutorials implement an end-to-end XGBoost application including: - [Time-series forecasting](https://docs.ray.io/en/latest/_collections/ray-overview/examples/e2e-timeseries/README.html): These tutorials implement an end-to-end time-series application including: - [Scalable video processing](https://docs.ray.io/en/latest/_collections/ray-overview/examples/object-detection/README.html): This tutorial builds an end-to-end face mask detection pipeline that leverages distributed fine-tuning, large-scale batch inference, video analytics, and scalable serving: - [Distributed RAG pipeline](https://docs.ray.io/en/latest/_collections/ray-overview/examples/e2e-rag/README.html): This tutorial covers end-to-end Retrieval-Augmented Generation (RAG) pipelines using Ray, from data ingestion and LLM deployment to prompt engineering, evaluation and scaling out all workloads in the application. - [Deploy MCP servers](https://docs.ray.io/en/latest/_collections/ray-overview/examples/mcp-ray-serve/README.html): This repository provides end-to-end examples for deploying and scaling Model Context Protocol (MCP) servers using Ray Serve and Anyscale Service, covering both streamable HTTP and stdio transport types: - [Build a tool-using agent](https://docs.ray.io/en/latest/_collections/ray-overview/examples/langchain_agent_ray_serve/content/README.html): This tutorial guides you through building and deploying a sophisticated, tool-using agent using LangChain, LangGraph, and Ray Serve on Anyscale. - [Build a multi-agent system with the A2A protocol](https://docs.ray.io/en/latest/_collections/ray-overview/examples/multi_agent_a2a/README.html): This tutorial guides you through building and deploying a multi-agent system where agents communicate using the A2A (Agent-to-Agent) protocol. The system is built on Ray Serve for scalable deployment, LangChain for agent orchestration, and MCP for… ## Ecosystem - [The Ray Ecosystem](https://docs.ray.io/en/latest/ray-overview/ray-libraries.html): Third-party libraries and tools that integrate with Ray for distributed execution, plus an ecosystem map showing component maturity. ## Ray Core - [What’s Ray Core?](https://docs.ray.io/en/latest/ray-core/walkthrough.html): Ray's distributed computing primitives — tasks, actors, and objects — with examples for turning Python functions and classes into distributed apps. - [Key Concepts](https://docs.ray.io/en/latest/ray-core/key-concepts.html): Ray Core primitives: tasks (remote functions), actors (stateful remote classes), objects in the distributed object store, and placement groups. - [User Guides](https://docs.ray.io/en/latest/ray-core/user-guide.html): Ray Core how-to guides: tasks, actors, objects, dependencies, scheduling, fault tolerance, design patterns, and compiled graphs. - [Tasks](https://docs.ray.io/en/latest/ray-core/tasks.html): Run Python functions asynchronously as Ray tasks: request resources, pass ObjectRefs, wait for partial results, and cancel tasks. - [Nested Remote Functions](https://docs.ray.io/en/latest/ray-core/tasks/nested-tasks.html): Call remote functions from within remote functions, and how nested tasks yield their resources while blocked. - [Actors](https://docs.ray.io/en/latest/ray-core/actors.html): Turn Python classes into stateful Ray actors: declare resources, call methods, pass handles around, use generators, and cancel actor tasks. - [Named Actors](https://docs.ray.io/en/latest/ray-core/actors/named-actors.html): Give an actor a unique name in its namespace so any job in the cluster can retrieve it, with get-or-create and actor lifetime options. - [Terminating Actors](https://docs.ray.io/en/latest/ray-core/actors/terminating-actors.html): Terminate Ray actors from a handle or from inside the actor, and how actor cleanup runs when ray.shutdown is called. - [AsyncIO / Concurrency for Actors](https://docs.ray.io/en/latest/ray-core/actors/async_api.html): Concurrency for Ray actors: async actors built on asyncio, threaded actors, ObjectRefs as futures, and setting max concurrency. - [Limiting Concurrency Per-Method with Concurrency Groups](https://docs.ray.io/en/latest/ray-core/actors/concurrency_group_api.html): Limit concurrency per actor method with concurrency groups, including the default group and choosing a group at call time. - [Utility Classes](https://docs.ray.io/en/latest/ray-core/actors/actor-utils.html): Utility classes for actors: ActorPool for load-balanced method calls and Ray Queue for message passing between tasks and actors. - [Out-of-band Communication](https://docs.ray.io/en/latest/ray-core/actors/out-of-band-communication.html): Communicate between actors outside method calls: wrapping library processes, Ray Collective, and running an HTTP server in an actor. - [Actor Task Execution Order](https://docs.ray.io/en/latest/ray-core/actors/task-orders.html): How Ray orders actor method execution, and how that differs between synchronous single-threaded actors and async or threaded actors. - [Objects](https://docs.ray.io/en/latest/ray-core/objects.html): Ray remote objects and ObjectRefs: fetching data, passing objects as arguments, closure capture, nested objects, and fault tolerance. - [Serialization](https://docs.ray.io/en/latest/ray-core/objects/serialization.html): How Ray serializes data for the Plasma object store, with NumPy zero-copy reads, read-only array pitfalls, and custom serializers. - [Object Spilling](https://docs.ray.io/en/latest/ray-core/objects/object-spilling.html): Configure where Ray spills objects once the object store fills up, including custom spill directories and spill statistics. - [Environment Dependencies](https://docs.ray.io/en/latest/ray-core/handling-dependencies.html): Ship Python and system dependencies to a Ray cluster with runtime environments, per job or per task, using local files, conda, or pip. - [Scheduling](https://docs.ray.io/en/latest/ray-core/scheduling/index.html): How Ray schedules tasks and actors onto nodes: labels, resources, and the DEFAULT, SPREAD, placement group, and node affinity strategies. - [Use labels to control scheduling](https://docs.ray.io/en/latest/ray-core/scheduling/labels.html): Learn about using labels to control how Ray schedules tasks, actors, and placement groups to nodes in your Kubernetes cluster. - [Resources](https://docs.ray.io/en/latest/ray-core/scheduling/resources.html): Ray physical and logical resources: define custom resources, set node capacity, and request fractional resources per task or actor. - [Accelerator Support](https://docs.ray.io/en/latest/ray-core/scheduling/accelerators.html): Use accelerators in Ray tasks and actors: start nodes with GPUs, request fractional accelerators, and pin to accelerator types. - [Placement Groups](https://docs.ray.io/en/latest/ray-core/scheduling/placement-group.html): Atomically reserve resources across nodes with placement groups, Ray's gang scheduling primitive, using bundles and placement strategies. - [Memory Management](https://docs.ray.io/en/latest/ray-core/scheduling/memory-management.html): How Ray manages memory: ObjectRef reference counting, inspecting usage with ray memory, and memory-aware scheduling. - [Out-Of-Memory Prevention](https://docs.ray.io/en/latest/ray-core/scheduling/ray-oom-prevention.html): Ray's memory monitor, which prevents node OOM by killing workers under memory pressure, with its retry and worker-killing policies. - [Fault tolerance](https://docs.ray.io/en/latest/ray-core/fault-tolerance.html): How Ray classifies application-level and system-level failures, and where to find the per-component fault tolerance guarantees. - [Task Fault Tolerance](https://docs.ray.io/en/latest/ray-core/fault_tolerance/tasks.html): Task fault tolerance in Ray: catch application-level errors, configure retries for failed tasks, and cancel misbehaving tasks. - [Actor Fault Tolerance](https://docs.ray.io/en/latest/ray-core/fault_tolerance/actors.html): Actor fault tolerance in Ray: process and creator failure, restarts and checkpointing, force-killing, and unavailable-actor errors. - [Object Fault Tolerance](https://docs.ray.io/en/latest/ray-core/fault_tolerance/objects.html): Object fault tolerance in Ray: lineage-based recovery from data loss, recovery from owner failure, and understanding ObjectLostError. - [Node Fault Tolerance](https://docs.ray.io/en/latest/ray-core/fault_tolerance/nodes.html): Node fault tolerance in Ray: what happens when a worker node, the head node, or an individual raylet fails. - [GCS Fault Tolerance](https://docs.ray.io/en/latest/ray-core/fault_tolerance/gcs.html): Make the GCS fault tolerant by backing cluster metadata with Redis or the alpha embedded RocksDB, with tuning guidance. - [Design Patterns & Anti-patterns](https://docs.ray.io/en/latest/ray-core/patterns/index.html): Catalog of Ray Core design patterns and anti-patterns for structuring tasks, actors, and object passing. - [Pattern: Using nested tasks to achieve nested parallelism](https://docs.ray.io/en/latest/ray-core/patterns/nested-tasks.html): Pattern: call remote functions from inside remote functions to express nested parallelism such as divide-and-conquer. - [Pattern: Using generators to reduce heap memory usage](https://docs.ray.io/en/latest/ray-core/patterns/generators.html): Pattern: yield results from a generator task instead of returning them all at once, to cap heap memory usage. - [Pattern: Using ray.wait to limit the number of pending tasks](https://docs.ray.io/en/latest/ray-core/patterns/limit-pending-tasks.html): Pattern: use ray.wait to bound the number of in-flight tasks so the submission loop doesn't outrun the cluster. - [Pattern: Using resources to limit the number of concurrently running tasks](https://docs.ray.io/en/latest/ray-core/patterns/limit-running-tasks.html): Pattern: use custom resource requirements to cap how many tasks or actors run concurrently, limiting memory pressure. - [Pattern: Using asyncio to run actor methods concurrently](https://docs.ray.io/en/latest/ray-core/patterns/concurrent-operations-async-actor.html): Pattern: use an async actor so its methods run concurrently on one worker, overlapping I/O-bound operations. - [Pattern: Using an actor to synchronize other tasks and actors](https://docs.ray.io/en/latest/ray-core/patterns/actor-sync.html): Pattern: use a single actor as a synchronization point to coordinate other tasks and actors. - [Pattern: Using a supervisor actor to manage a tree of actors](https://docs.ray.io/en/latest/ray-core/patterns/tree-of-actors.html): Pattern: use a supervisor actor to create and manage a tree of worker actors, centralizing lifecycle and failure handling. - [Pattern: Using pipelining to increase throughput](https://docs.ray.io/en/latest/ray-core/patterns/pipelining.html): Pattern: pipeline task submission and result processing so compute overlaps with data transfer and throughput rises. - [Anti-pattern: Returning ray.put() ObjectRefs from a task harms performance and fault tolerance](https://docs.ray.io/en/latest/ray-core/patterns/return-ray-put.html): Anti-pattern: returning ray.put ObjectRefs from a task adds a copy and weakens fault tolerance; return the value instead. - [Anti-pattern: Calling ray.get on task arguments harms performance](https://docs.ray.io/en/latest/ray-core/patterns/nested-ray-get.html): Anti-pattern: calling ray.get on task arguments serializes execution; pass ObjectRefs and let Ray resolve them. - [Anti-pattern: Calling ray.get in a loop harms parallelism](https://docs.ray.io/en/latest/ray-core/patterns/ray-get-loop.html): Anti-pattern: calling ray.get inside a loop blocks on each task in turn and destroys parallelism; collect the refs first. - [Anti-pattern: Calling ray.get unnecessarily harms performance](https://docs.ray.io/en/latest/ray-core/patterns/unnecessary-ray-get.html): Anti-pattern: calling ray.get before the value is needed blocks the driver and forfeits overlap between tasks. - [Anti-pattern: Processing results in submission order using ray.get increases runtime](https://docs.ray.io/en/latest/ray-core/patterns/ray-get-submission-order.html): Anti-pattern: processing results in submission order idles on slow tasks; use ray.wait to handle them as they complete. - [Anti-pattern: Fetching too many objects at once with ray.get causes failure](https://docs.ray.io/en/latest/ray-core/patterns/ray-get-too-many-objects.html): Anti-pattern: fetching too many objects at once with ray.get can exhaust the object store or heap and fail the job. - [Anti-pattern: Over-parallelizing with too fine-grained tasks harms speedup](https://docs.ray.io/en/latest/ray-core/patterns/too-fine-grained-tasks.html): Anti-pattern: splitting work into very small tasks lets per-task overhead dominate; batch work into coarser tasks. - [Anti-pattern: Redefining the same remote function or class harms performance](https://docs.ray.io/en/latest/ray-core/patterns/redefine-task-actor-loop.html): Anti-pattern: redefining the same remote function or actor class in a loop re-exports it each time, adding overhead. - [Anti-pattern: Passing the same large argument by value repeatedly harms performance](https://docs.ray.io/en/latest/ray-core/patterns/pass-large-arg-by-value.html): Anti-pattern: passing the same large argument by value to many tasks re-serializes it each time; ray.put it once instead. - [Anti-pattern: Closure capturing large objects harms performance](https://docs.ray.io/en/latest/ray-core/patterns/closure-capture-large-objects.html): Anti-pattern: capturing large objects in a remote function's closure copies them to every worker; put them in the object store instead. - [Anti-pattern: Using global variables to share state between tasks and actors](https://docs.ray.io/en/latest/ray-core/patterns/global-variables.html): Anti-pattern: global variables don't propagate between Ray workers; pass state explicitly or hold it in an actor. - [Anti-pattern: Serialize ray.ObjectRef out of band](https://docs.ray.io/en/latest/ray-core/patterns/out-of-band-object-ref-serialization.html): Anti-pattern: serializing an ObjectRef out of band escapes reference counting and can let its value be reclaimed early. - [Anti-pattern: Forking new processes in application code](https://docs.ray.io/en/latest/ray-core/patterns/fork-new-processes.html): Anti-pattern: forking processes inside a Ray task or actor breaks Ray's process management and resource accounting. - [Ray Direct Transport (RDT)](https://docs.ray.io/en/latest/ray-core/direct-transport/direct-transport.html): Ray Direct Transport moves tensors between actors without going through the object store, using collective groups over Gloo or NCCL. - [Implementing a custom tensor transport (Advanced)](https://docs.ray.io/en/latest/ray-core/direct-transport/custom-tensor-transport.html): Implement a custom tensor transport for Ray Direct Transport: metadata classes, communicator metadata, send and receive, and cleanup. - [Ray Compiled Graph (beta)](https://docs.ray.io/en/latest/ray-core/compiled-graph/ray-compiled-graph.html): Ray Compiled Graph (beta) for programming multi-GPU distributed systems with a static execution graph and low per-call overhead. - [Quickstart](https://docs.ray.io/en/latest/ray-core/compiled-graph/quickstart.html): First Ray Compiled Graph program: declare data dependencies, use asyncio, set execution timeouts, and move tensors between CPU and GPU. - [Profiling](https://docs.ray.io/en/latest/ray-core/compiled-graph/profiling.html): Profile Ray Compiled Graph execution with the PyTorch or Nsight profilers to find task-level and system overhead bottlenecks. - [Experimental: Overlapping communication and computation](https://docs.ray.io/en/latest/ray-core/compiled-graph/overlap.html): Experimental Ray Compiled Graph feature that overlaps GPU communication with computation to hide data transfer latency. - [Troubleshooting](https://docs.ray.io/en/latest/ray-core/compiled-graph/troubleshooting.html): Common Ray Compiled Graph problems: current limitations, returning NumPy arrays, and tearing down before reusing the same actors. - [Resource Isolation With Cgroup v2](https://docs.ray.io/en/latest/ray-core/resource-isolation-with-cgroupv2.html): Isolate Ray system and application resources with cgroup v2 across containers, Kubernetes, GKE, and bare-metal deployments. - [Ray Sandboxes](https://docs.ray.io/en/latest/ray-core/sandboxes.html): Execute untrusted model-generated code and agent tool calls safely with Ray Sandboxes using lightweight gVisor kernel isolation. - [Advanced topics](https://docs.ray.io/en/latest/ray-core/advanced-topics.html): Index of advanced Ray Core topics that go beyond the basic task, actor, and object APIs. - [Tips for first-time users](https://docs.ray.io/en/latest/ray-core/tips-for-first-time.html): Four performance habits for new Ray users: delay ray.get, avoid tiny tasks, don't repeatedly pass the same object, and pipeline data. - [Type hints in Ray](https://docs.ray.io/en/latest/ray-core/type-hint.html): Python type hints for Ray remote functions and actors, enabling IDE support and static type checking via ray.remote and @ray.method. - [Starting Ray](https://docs.ray.io/en/latest/ray-core/starting-ray.html): Start the Ray runtime on one machine with ray.init, from the CLI with ray start, or launch a multi-node cluster with ray up. - [Ray Generators](https://docs.ray.io/en/latest/ray-core/ray-generator.html): Ray generators that yield values incrementally from tasks and actor methods, with error handling, asyncio, cancellation, and fault tolerance. - [Dynamic generators](https://docs.ray.io/en/latest/ray-core/tasks/dynamic_generators.html): Ray remote generators with a dynamic number of returns, set by either the caller or the executor, plus exception handling and limits. - [Using Namespaces](https://docs.ray.io/en/latest/ray-core/namespaces.html): Group jobs and named actors into namespaces so actor names need only be unique per namespace, including anonymous namespaces. - [Cross-language programming](https://docs.ray.io/en/latest/ray-core/cross-language.html): Call Java from Python and Python from Java in one Ray application, covering driver setup, data serialization, and exception stacks. - [Working with Jupyter Notebooks & JupyterLab](https://docs.ray.io/en/latest/ray-core/using-ray-with-jupyter.html): Run Ray inside Jupyter Notebook and JupyterLab, covering notebook setup and connecting to an existing cluster. - [Lazy Computation Graphs with the Ray DAG API](https://docs.ray.io/en/latest/ray-core/ray-dag.html): Build lazy computation graphs with the Ray DAG API over functions, classes, and actor methods, using InputNode and MultiOutputNode. - [Miscellaneous Topics](https://docs.ray.io/en/latest/ray-core/miscellaneous.html): Assorted Ray Core topics: dynamic remote parameters, overloaded functions, inspecting cluster state, and OS tuning for large clusters. - [Authenticating Remote URIs in runtime_env](https://docs.ray.io/en/latest/ray-core/runtime_env_auth.html): Authenticate private remote URIs in runtime_env with a netrc file or HTTPS bearer tokens, on VMs and with KubeRay secrets. - [Lifetimes of a User-Spawn Process](https://docs.ray.io/en/latest/ray-core/user-spawn-processes.html): Lifetime of processes your code spawns inside Ray workers, including killing them on worker exit and zombie reaping behavior. - [Head Node Memory Management](https://docs.ray.io/en/latest/ray-core/head-node-memory-management.html): Why Ray head node memory grows and how to mitigate it: keep work off the head node, disable the dashboard, and size the head pod. - [Examples](https://docs.ray.io/en/latest/ray-core/examples/overview.html): Ray Core code examples from beginner (Monte Carlo pi) to intermediate (MapReduce, web crawler) to advanced (parameter server, pong). - [Monte Carlo Estimation of π](https://docs.ray.io/en/latest/ray-core/examples/monte_carlo_pi.html): Worked Ray Core example that estimates π by Monte Carlo sampling, combining parallel remote tasks with a progress-tracking actor. - [Internals](https://docs.ray.io/en/latest/ray-core/internals.html): Ray Core internals for advanced users: task lifecycle, streaming generators, autoscaler v2, RPC fault tolerance, object spilling, and metrics. - [Task Lifecycle](https://docs.ray.io/en/latest/ray-core/internals/task-lifecycle.html): Internals of a Ray Core task from definition through invocation, scheduling, execution, and returning its value to the caller. - [Streaming Generator](https://docs.ray.io/en/latest/ray-core/internals/streaming-generator.html): Internals of Ray streaming generator tasks: return ObjectID handling, scheduling, reporting yielded values, and backpressure. - [Autoscaler v2](https://docs.ray.io/en/latest/ray-core/internals/autoscaler-v2.html): Internals of Ray's autoscaler v2: worker group configuration, periodic reconciliation, bin packing, and the instance manager. - [RPC Fault Tolerance](https://docs.ray.io/en/latest/ray-core/internals/rpc-fault-tolerance.html): Make Ray Core RPCs fault tolerant with the retryable gRPC client, including long-polling pitfalls and idempotency requirements. - [Token Authentication](https://docs.ray.io/en/latest/ray-core/internals/token-authentication.html): How Ray token authentication works: authentication modes, token sources and precedence, and propagation across C++, Python, and HTTP. - [Metric Exporter Infrastructure](https://docs.ray.io/en/latest/ray-core/internals/metric-exporter.html): Internals of Ray's metric exporter: C++ metric registration and recording, OTLP gRPC export, and OpenTelemetry SDK integration. - [Ray Event Exporter Infrastructure](https://docs.ray.io/en/latest/ray-core/internals/ray-event-exporter.html): Internals of Ray's event exporter: event types and structure, the entity ID concept, and C++-side event recording and buffering. - [Port Service Discovery](https://docs.ray.io/en/latest/ray-core/internals/port-service-discovery.html): How Ray assigns and discovers ports across raylet, agents, and core workers, plus GCS-side discovery via the node and actor tables. - [Object Spilling](https://docs.ray.io/en/latest/ray-core/internals/object-spilling.html): Internals of Ray object spilling: primary and secondary copies, reactive and threshold-based triggers, object pinning, and data flow. - [Accelerator types](https://docs.ray.io/en/latest/ray-core/accelerator-types.html): The accelerator types Ray recognizes as resources, including NVIDIA and AMD GPUs, AWS Neuron cores, Google TPUs, and Intel Gaudi HPUs. - [Ray Direct Transport (RDT) API](https://docs.ray.io/en/latest/ray-core/api/direct-transport.html): API reference for Ray Direct Transport (RDT): usage with the core APIs, collective tensor transports, and advanced APIs. - [Sandbox API](https://docs.ray.io/en/latest/ray-core/api/sandboxes.html): API reference for Ray Sandboxes (gVisor-isolated container environments). ## Ray Data - [Ray Data: Scalable Data Processing for AI Workloads](https://docs.ray.io/en/latest/data/data.html): Scalable data processing for AI workloads — a streaming engine for batch inference, preprocessing, and ML training ingest across CPUs and GPUs. - [Ray Data Quickstart](https://docs.ray.io/en/latest/data/quickstart.html): Get started with the Ray Data Dataset API: load data from files or cloud storage, transform it, consume it, and save results. - [Key Concepts](https://docs.ray.io/en/latest/data/key-concepts.html): Core Ray Data concepts: Datasets and blocks, the lazy streaming execution model, and how data is partitioned and processed in parallel. - [User Guides](https://docs.ray.io/en/latest/data/user-guide.html): How-to guides for Ray Data tasks: loading, transforming, joining, shuffling, and iterating over data, and working with images, text, tensors, and LLMs. - [Loading Data](https://docs.ray.io/en/latest/data/loading-data.html): Load data into Ray Data from local and cloud storage, compressed files, URIs, single-node libraries, distributed DataFrames, and Hugging Face. - [Inspecting Data](https://docs.ray.io/en/latest/data/inspecting-data.html): Inspect a Ray Data Dataset's schema, row count, and sample rows or batches to understand your data before processing it. - [Transforming Data](https://docs.ray.io/en/latest/data/transforming-data.html): Transform Ray Data Datasets with map, flat_map, and map_batches, and choose the right batch format and batch size for the work. - [Aggregating data](https://docs.ray.io/en/latest/data/aggregating-data.html): Aggregate Ray Data Datasets with built-in aggregations and custom aggregators, including a worked example building a custom mean aggregator. - [Iterating over Data](https://docs.ray.io/en/latest/data/iterating-over-data.html): Iterate over a Ray Data Dataset by rows or batches, including the framework-specific batch formats used in training loops. - [Joining data](https://docs.ray.io/en/latest/data/joining-data.html): Join Ray Data Datasets on key columns using the supported join types, and tune the partition and aggregator counts. - [Shuffling data](https://docs.ray.io/en/latest/data/shuffling-data.html): Shuffle Ray Data at different granularities: file order, local buffer shuffle, map_batches shuffle, block order, and global per-epoch shuffle. - [Weighted Dataset Mixing](https://docs.ray.io/en/latest/data/mixing-data.html): Combine multiple Ray Data Datasets into one streaming dataset with weighted mixing strategies, block size control, and stopping conditions. - [Saving Data](https://docs.ray.io/en/latest/data/saving-data.html): Write Ray Data Datasets to local or cloud storage, control the output file count, write partitioned datasets, and convert back to pandas. - [Working with Images](https://docs.ray.io/en/latest/data/working-with-images.html): Read, transform, run inference on, and save large image datasets with Ray Data. - [Working with Text](https://docs.ray.io/en/latest/data/working-with-text.html): Read, transform, run inference on, and save large text datasets with Ray Data. - [Working with Tensors / NumPy](https://docs.ray.io/en/latest/data/working-with-tensors.html): Work with n-dimensional array data in Ray Data, covering fixed- and variable-shape tensor batches, transformations, and saving. - [Working with PyTorch](https://docs.ray.io/en/latest/data/working-with-pytorch.html): Use Ray Data with PyTorch: iterate torch tensors for training, integrate with Ray Train, apply built-in transforms, and migrate off DataLoader. - [Working with LLMs](https://docs.ray.io/en/latest/data/working-with-llms.html): Batch LLM inference with ray.data.llm: vLLM and SGLang engines or hosted endpoints, multi-GPU scaling, and vision, video, and audio models. - [How to avoid out-of-memory errors (OOMs)](https://docs.ray.io/en/latest/data/how-to-avoid-ooms.html): Avoid Ray Data out-of-memory errors: recognize Ray and kernel OOM kills, tune batch size, and configure memory for reads and high-memory UDFs. - [Monitoring Your Workload](https://docs.ray.io/en/latest/data/monitoring-your-workload.html): Monitor Ray Data execution with progress bars, the Ray Data dashboard, and Prometheus metrics for inputs, operators, and resource use. - [Execution Configurations](https://docs.ray.io/en/latest/data/execution-configurations.html): Configuration options that control Ray Data execution on top of the Ray Core cluster, including job-level checkpointing. - [Run multiple Datasets in one cluster](https://docs.ray.io/en/latest/data/concurrent-dataset-execution.html): Run multiple Ray Data Datasets in one cluster without contention by labeling worker nodes and tagging each Dataset with a label selector. - [End-to-end: Offline Batch Inference](https://docs.ray.io/en/latest/data/batch_inference.html): Run offline batch inference with Ray Data end to end: GPU inference, batch size tuning, job-level checkpointing, and OOM troubleshooting. - [Advanced: Performance Tips and Tuning](https://docs.ray.io/en/latest/data/performance-tips.html): Tune Ray Data performance: batch transforms, Polars sorts, read block and resource tuning, Parquet projection pushdown, and memory reduction. - [Advanced: Scaling out expensive collate functions](https://docs.ray.io/en/latest/data/scaling-collation-functions.html): Move an expensive collate function into Ray Data so it scales across the cluster, with batch size alignment and tensor serialization. - [Advanced: Read and Write Custom File Types](https://docs.ray.io/en/latest/data/custom-datasource-example.html): Read and write custom file types in Ray Data by implementing your own datasource for formats the built-in readers don't cover. - [Examples](https://docs.ray.io/en/latest/data/examples.html) - [Contributing to Ray Data](https://docs.ray.io/en/latest/data/contributing/contributing.html): How to contribute to Ray Data, including the contributor workflow and how to write tests. - [Contributing Guide](https://docs.ray.io/en/latest/data/contributing/contributing-guide.html): How to contribute to Ray Data: choose an issue, get early feedback, write good tests, verify locally, and open a reviewable pull request. - [How to write tests](https://docs.ray.io/en/latest/data/contributing/how-to-write-tests.html): Write non-flaky Ray Data tests: prefer unit tests and fixtures, avoid assuming output order, and don't depend on block count or repr output. - [Comparing Ray Data to other systems](https://docs.ray.io/en/latest/data/comparisons.html): How Ray Data compares to batch services, online-inference tools, and distributed frameworks like Spark and Daft for offline inference. - [Ray Data Benchmarks](https://docs.ray.io/en/latest/data/benchmark.html): Ray Data performance benchmarks across image, document, audio, and video workloads, with methodology and comparisons to Daft. - [Ray Data internals](https://docs.ray.io/en/latest/data/data-internals.html): Implementation internals of Ray Data for advanced users and contributors: the execution model, blocks, shuffle algorithms, scheduling, and the memory model. - [LLM batch inference with Ray Data LLM](https://docs.ray.io/en/latest/_collections/data/examples/llm_batch_inference_text/README.html): This example shows you how to run batch inference for large language models (LLMs) using Ray Data LLM APIs. In this use case, the batch inference job infers company industries from company names across a large customer dataset. - [Multimodal LLM batch inference with Ray Data LLM](https://docs.ray.io/en/latest/_collections/data/examples/llm_batch_inference_vision/README.html): This example shows you how to run batch inference for vision-language models (VLMs) using Ray Data LLM APIs. In this use case, the batch inference job generates captions for a large-scale image dataset. - [Unstructured Data Ingestion and Processing With Ray Data](https://docs.ray.io/en/latest/_collections/data/examples/unstructured_data_ingestion/README.html): Time to complete: 35 min | Difficulty: Intermediate | Prerequisites: Data engineering experience, document processing, basic natural language processing (NLP) knowledge - [Expressions API](https://docs.ray.io/en/latest/data/api/expressions.html): Reference for Ray Data expressions: use col and lit to build column-based operations for filtering and transforms, with classes and namespaces. - [API Guide for Users from Other Data Libraries](https://docs.ray.io/en/latest/data/api/from_other_data_libs.html): Map familiar pandas, PyArrow, and PyTorch Dataset and DataLoader operations onto their Ray Data equivalents. ## Ray Train - [Ray Train: Scalable Model Training](https://docs.ray.io/en/latest/train/train.html): Distributed model training and fine-tuning at scale across PyTorch, Lightning, Hugging Face Transformers, XGBoost, and JAX. - [Overview](https://docs.ray.io/en/latest/train/overview.html): Core Ray Train concepts: the training function, worker processes, ScalingConfig for CPU/GPU resources, and the Trainer class. - [PyTorch Guide](https://docs.ray.io/en/latest/train/getting-started-pytorch.html): Convert a PyTorch script to distributed training with Ray Train: TorchTrainer, sharded dataloaders, metrics, checkpoints, and GPU scaling. - [PyTorch Lightning Guide](https://docs.ray.io/en/latest/train/getting-started-pytorch-lightning.html): Convert a PyTorch Lightning script to distributed training with Ray Train: the Lightning integration, checkpointing, and ScalingConfig. - [Hugging Face Transformers Guide](https://docs.ray.io/en/latest/train/getting-started-transformers.html): Convert a Hugging Face Transformers script to distributed training with Ray Train: TorchTrainer, checkpointing, and multi-GPU ScalingConfig. - [XGBoost Guide](https://docs.ray.io/en/latest/train/getting-started-xgboost.html): Convert an XGBoost script to distributed training with Ray Train using XGBoostTrainer, with checkpointing and CPU/GPU ScalingConfig. - [JAX Guide](https://docs.ray.io/en/latest/train/getting-started-jax.html): Distribute JAX training across GPUs and TPUs with JaxTrainer: SPMD execution, ScalingConfig topology for TPU PodSlices, and CUDA setup. - [More Frameworks](https://docs.ray.io/en/latest/train/more-frameworks.html): Index of Ray Train integration guides for additional frameworks: Hugging Face Accelerate, DeepSpeed, TensorFlow/Keras, LightGBM, and Horovod. - [Hugging Face Accelerate Guide](https://docs.ray.io/en/latest/train/huggingface-accelerate.html): Run Hugging Face Accelerate training on Ray Train, including Accelerate configuration and migration off AccelerateTrainer. - [DeepSpeed Guide](https://docs.ray.io/en/latest/train/deepspeed.html): Launch DeepSpeed training across a Ray cluster with TorchTrainer, using ZeRO optimization stages to train large models efficiently. - [TensorFlow and Keras Guide](https://docs.ray.io/en/latest/train/distributed-tensorflow-keras.html): Distribute TensorFlow and Keras training with TensorflowTrainer: adapt the training function, load data, report results, and checkpoint. - [LightGBM Guide](https://docs.ray.io/en/latest/train/getting-started-lightgbm.html): Distribute LightGBM training with Ray Train: build the training function, shard data with Ray Data, and configure scale, GPUs, and storage. - [Horovod Guide](https://docs.ray.io/en/latest/train/horovod.html): Distribute training with Horovod on Ray Train by adapting your training function and configuring HorovodTrainer. - [User Guides](https://docs.ray.io/en/latest/train/user-guides.html): Ray Train how-to guides: data loading, accelerators, storage, monitoring, checkpointing, fault tolerance, elastic training, and experiment tracking. - [Data Loading and Preprocessing](https://docs.ray.io/en/latest/train/user-guides/data-loading-preprocessing.html): Load and preprocess data for Ray Train with Ray Data: ingest, transform, split across workers, and consume shards in the training function. - [Configuring Scale and Accelerators](https://docs.ray.io/en/latest/train/user-guides/using-accelerators.html): Configure Ray Train scale and accelerators: worker count, GPUs per worker, accelerator type, communication backend, and per-worker resources. - [Configuring Persistent Storage](https://docs.ray.io/en/latest/train/user-guides/persistent-storage.html): Configure persistent storage for Ray Train: cloud object storage, shared filesystems, local storage, fsspec, and S3-compatible backends. - [Monitoring and Logging Metrics](https://docs.ray.io/en/latest/train/user-guides/monitoring-logging.html): Obtain and aggregate training metrics reported from multiple Ray Train workers. - [Saving and Loading Checkpoints](https://docs.ray.io/en/latest/train/user-guides/checkpoints.html): Save and load Ray Train checkpoints: distributed checkpointing from multiple workers, upload modes, async uploads, and post-training use. - [Validating checkpoints asynchronously](https://docs.ray.io/en/latest/train/user-guides/asynchronous-validation.html): Validate checkpoints asynchronously so training continues while validation runs, with TorchTrainer and Ray Data approaches and subcluster isolation. - [Experiment Tracking](https://docs.ray.io/en/latest/train/user-guides/experiment-tracking.html): Use experiment tracking libraries such as MLflow and Weights & Biases with distributed Ray Train runs, covering credentials and per-worker logging. - [Inspecting Training Results](https://docs.ray.io/en/latest/train/user-guides/results.html): Inspect the Result object returned by trainer.fit: reported metrics, a dataframe of all metrics, saved checkpoints, and the storage location. - [Handling Failures and Node Preemption](https://docs.ray.io/en/latest/train/user-guides/fault-tolerance.html): Handle worker, node, and driver failures in Ray Train, including which checkpoint is restored and how node preemption is absorbed. - [Elastic training](https://docs.ray.io/en/latest/train/user-guides/elastic-training.html): Elastic training that adapts to changing resource availability, continuing through node preemption and scaling up as new nodes join. - [Ray Train Metrics](https://docs.ray.io/en/latest/train/user-guides/monitor-your-application.html): Prometheus metrics Ray Train exports for controller state, worker group startup, and checkpoint timing, viewable in the Ray Dashboard. - [Local Mode](https://docs.ray.io/en/latest/train/user-guides/local_mode.html): Run a Ray Train training function without launching worker actors, for faster iteration, testing, and single-node multi-GPU runs. - [Reproducibility](https://docs.ray.io/en/latest/train/user-guides/reproducibility.html): Limit nondeterminism in Ray Train with ray.train.torch.enable_reproducibility, and the caveats on fully reproducible PyTorch results. - [Hyperparameter Optimization](https://docs.ray.io/en/latest/train/user-guides/hyperparameter-optimization.html): Tune hyperparameters for a Ray Train run with Ray Tune, covering per-trial resources, concurrency limits, and metric reporting. - [Tutorials](https://docs.ray.io/en/latest/_collections/train/tutorials/README.html): This tutorial series provides hands-on learning for Ray Train and its ecosystem (Ray Data, Anyscale Workspaces).
\\ The tutorials walk through common ML workload patterns—vision, tabular, time series, generative, policy learning, and… - [Examples](https://docs.ray.io/en/latest/train/examples.html) - [Benchmarks](https://docs.ray.io/en/latest/train/benchmarks.html): Ray Train performance benchmarks: GPU image-classification throughput across cluster sizes, and parity with native PyTorch Distributed. - [Fine-tune an LLM with Ray Train and DeepSpeed](https://docs.ray.io/en/latest/_collections/train/examples/pytorch/deepspeed_finetune/README.html): This notebook combines Ray Train with DeepSpeed to efficiently scale PyTorch training across GPUs and nodes while minimizing GPU memory usage. - [Distributing your PyTorch Training Code with Ray Train and Ray Data](https://docs.ray.io/en/latest/_collections/train/examples/pytorch/distributing-pytorch/README.html): This template shows you how to distribute your PyTorch training code with Ray Train and Ray Data, getting performance and usability improvements along the way. - [Get started with PyTorch Fully Sharded Data Parallel (FSDP2) and Ray Train](https://docs.ray.io/en/latest/_collections/train/examples/pytorch/pytorch-fsdp/README.html): This template shows how to get memory and performance improvements of integrating PyTorch’s Fully Sharded Data Parallel with Ray Train. - [Profiling a Ray Train Workload with PyTorch Profiler](https://docs.ray.io/en/latest/_collections/train/examples/pytorch/pytorch-profiling/README.html): This template profiles PyTorch training code with PyTorch Profiler to identify performance bottlenecks, optimize memory usage, and monitor training efficiency in distributed environments. - [Get started with 2D Parallelism (Tensor + Data Parallelism) using DeepSpeed and Ray Train](https://docs.ray.io/en/latest/_collections/train/examples/pytorch/tensor_parallel_autotp/README.html): This template shows how to train large language models using tensor parallelism with DeepSpeed’s AutoTP and Ray Train for distributed execution. - [Get started with 2D Parallelism (Tensor + Data Parallelism) using FSDP2 and Ray Train](https://docs.ray.io/en/latest/_collections/train/examples/pytorch/tensor_parallel_dtensor/README.html): This template shows how to train large language models using tensor parallelism with PyTorch’s native DTensor (Distributed Tensor) API and Ray Train for distributed execution. - [Distributed Training with Hugging Face Accelerate](https://docs.ray.io/en/latest/train/examples/accelerate/accelerate_example.html): Runnable example of distributed training with Hugging Face Accelerate on Ray Train. - [Distributed fine-tuning of Llama 3.1 8B on AWS Trainium with Ray and PyTorch Lightning](https://docs.ray.io/en/latest/train/examples/aws-trainium/llama3.html): Fine-tune Llama 3.1 8B on AWS Trainium with Ray Train and PyTorch Lightning, covering the Docker image, cluster config, and job launch. - [Train with DeepSpeed ZeRO-3 and Ray Train](https://docs.ray.io/en/latest/train/examples/deepspeed/deepspeed_example.html): Runnable example of training a model with DeepSpeed ZeRO-3 on Ray Train. - [Run Horovod Distributed Training with PyTorch and Ray Train](https://docs.ray.io/en/latest/train/examples/horovod/horovod_example.html): Runnable example of Horovod distributed training with PyTorch on Ray Train. - [Fine-tuning of Stable Diffusion with DreamBooth and Ray Train](https://docs.ray.io/en/latest/train/examples/pytorch/dreambooth_finetuning.html): Fine-tune Stable Diffusion with DreamBooth on Ray Train, covering data loading, distributed training, scale configuration, and throughput. - [Train a PyTorch model on Fashion MNIST](https://docs.ray.io/en/latest/train/examples/pytorch/torch_fashion_mnist_example.html): Runnable example training a PyTorch model on Fashion MNIST with Ray Train. - [Training with TensorFlow and Ray Train](https://docs.ray.io/en/latest/train/examples/tf/tensorflow_mnist_example.html): Runnable example training a TensorFlow model on MNIST with Ray Train. - [Fine-tune a Text Classifier with Hugging Face Transformers](https://docs.ray.io/en/latest/train/examples/transformers/transformers_torch_trainer_basic.html): Runnable example fine-tuning a text classifier with Hugging Face Transformers on Ray Train. ## Ray Tune - [Ray Tune: Hyperparameter Tuning](https://docs.ray.io/en/latest/tune/index.html): Hyperparameter tuning and experiment execution at any scale, with algorithms like PBT and ASHA and integrations for PyTorch, XGBoost, and Optuna. - [Getting Started](https://docs.ray.io/en/latest/tune/getting-started.html): Tutorial tuning a PyTorch CNN with the Tuner API, ASHAScheduler for early stopping, and HyperOpt for Bayesian search. - [Key Concepts](https://docs.ray.io/en/latest/tune/key-concepts.html): Core Ray Tune abstractions: Trainables (Function and Class APIs), search spaces, search algorithms, schedulers, Tuner configuration, trials, and ResultGrid. - [User Guides](https://docs.ray.io/en/latest/tune/tutorials/overview.html): Ray Tune how-to guides: experiment runs, trial resources, search spaces, stopping, checkpointing, storage, fault tolerance, and callbacks. - [Running Basic Experiments](https://docs.ray.io/en/latest/tune/tutorials/tune-run.html): Run basic Tune experiments with independent trials in parallel, and how Tune compares to writing the same sweep with Ray Core tasks. - [Logging and Outputs in Tune](https://docs.ray.io/en/latest/tune/tutorials/tune-output.html): Control Tune logging and output: configure loggers, log to TensorBoard, set console verbosity, and redirect trainable logs to files. - [Setting Trial Resources](https://docs.ray.io/en/latest/tune/tutorials/tune-resources.html): Control Tune parallelism and resources: request GPUs per trial, run distributed training inside trials, and limit trial concurrency. - [Using Search Spaces](https://docs.ray.io/en/latest/tune/tutorials/tune-search-spaces.html): Define Tune search spaces, including custom distributions and conditional spaces where one parameter's range depends on another. - [How to Define Stopping Criteria for a Ray Tune Experiment](https://docs.ray.io/en/latest/tune/tutorials/tune-stopping.html): Define stopping criteria for a Tune experiment: metric thresholds, per-trial and experiment timeouts, failure limits, and scheduler early stopping. - [How to Save and Load Trial Checkpoints](https://docs.ray.io/en/latest/tune/tutorials/tune-trial-checkpoints.html): Save and load Tune trial checkpoints with the function and class APIs, including periodic checkpointing and checkpointing at termination. - [How to Configure Persistent Storage in Ray Tune](https://docs.ray.io/en/latest/tune/tutorials/tune-storage.html): Configure persistent storage for Tune with cloud object storage or NFS, plus what changes when running without external storage. - [How to Enable Fault Tolerance in Ray Tune](https://docs.ray.io/en/latest/tune/tutorials/tune-fault-tolerance.html): Enable experiment- and trial-level fault tolerance in Tune: restore a run, configure auto-resume, and recover from node and trial failures. - [Using Callbacks and Metrics](https://docs.ray.io/en/latest/tune/tutorials/tune-metrics.html): Work with Tune callbacks and metrics: hook into trial events, log custom metrics, and see which metrics Tune fills in automatically. - [Getting Data in and out of Tune](https://docs.ray.io/en/latest/tune/tutorials/tune_get_data_in_and_out.html): Pass data into Tune trainables through search spaces and tune.with_parameters, and get metrics, checkpoints, and artifacts back out. - [Deploying Tune in the Cloud](https://docs.ray.io/en/latest/tune/tutorials/tune-distributed.html): Run Tune experiments across a multi-node Ray cluster, covering launch on cloud VMs, storage options, spot instances, and failure recovery. - [Tune Architecture](https://docs.ray.io/en/latest/tune/tutorials/tune-lifecycle.html): How Tune works internally: what Tuner.fit does, the lifecycle of a trial, resource management, and the TuneController architecture. - [Scalability Benchmarks](https://docs.ray.io/en/latest/tune/tutorials/tune-scalability.html): Benchmark results for Ray Tune's overhead and throughput at scale, covering result throughput and thousands of concurrent trials. - [Ray Tune Examples](https://docs.ray.io/en/latest/tune/examples/index.html): Ray Tune examples by ML framework (PyTorch, Lightning, XGBoost, Keras), experiment trackers (W&B, MLflow), and HPO frameworks. - [PyTorch with ASHA](https://docs.ray.io/en/latest/_collections/tune/examples/tune_pytorch_asha/README.html): This notebook runs hyperparameter tuning experiments on a PyTorch CNN using Ray Tune, a scalable library for experiment management and search. The workload runs on the driver, but it sets the stage for the next tutorial where you combine Ray Tune… - [Ray Tune FAQ](https://docs.ray.io/en/latest/tune/faq.html): Common Ray Tune questions: choosing search algorithms and schedulers, early stopping, and trial parallelism tradeoffs. - [Tune Callbacks (tune.Callback)](https://docs.ray.io/en/latest/tune/api/callbacks.html): API reference for tune.Callback: the callback interface, initialization and setup, the available hooks, and stateful callbacks. - [Tune CLI (Experimental)](https://docs.ray.io/en/latest/tune/api/cli.html): Experimental tune CLI for listing, inspecting, and managing Ray Tune experiments from the command line. - [Environment variables used by Ray Tune](https://docs.ray.io/en/latest/tune/api/env.html): Reference for the environment variables that configure Ray Tune, covering storage, retries, reporting intervals, and debugging toggles. - [Tune Loggers (tune.logger)](https://docs.ray.io/en/latest/tune/api/logging.html): Reference for Tune's LoggerCallback interface and built-in loggers, plus the MLflow, Weights & Biases, Comet, and Aim integrations. - [Tune Console Output (Reporters)](https://docs.ray.io/en/latest/tune/api/reporters.html): Reference for tune.ProgressReporter and Tune's built-in reporters, which control the trial table and console output during a run. - [Tune Trial Schedulers (tune.schedulers)](https://docs.ray.io/en/latest/tune/api/schedulers.html): Reference for Tune trial schedulers that early-stop or mutate trials: ASHA, HyperBand, Median Stopping, Population Based Training, PB2, and BOHB. - [Tune Stopping Mechanisms (tune.stopper)](https://docs.ray.io/en/latest/tune/api/stoppers.html): Reference for tune.Stopper and Tune's built-in stoppers, which end trials or whole experiments on metric, time, or plateau criteria. - [Tune Search Algorithms (tune.search)](https://docs.ray.io/en/latest/tune/api/suggestion.html): Reference for Tune search algorithms, including grid and random search, Ax, BayesOpt, BOHB, HEBO, HyperOpt, Nevergrad, and Optuna. - [Training in Tune (tune.Trainable, tune.report)](https://docs.ray.io/en/latest/tune/api/trainable.html): Reference for Tune's function and class Trainable APIs, tune.report, actor reuse across trials, and per-trial resource allocation. ## Ray Serve - [Ray Serve: Scalable and Programmable Serving](https://docs.ray.io/en/latest/serve/index.html): Scalable, framework-agnostic model serving for online inference APIs, with LLM serving, model composition, response streaming, and request batching. - [Getting Started](https://docs.ray.io/en/latest/serve/getting_started.html): Hands-on tutorial: convert a HuggingFace model to a Ray Serve deployment, test it locally over HTTP, and compose two models into a single application. - [Key Concepts](https://docs.ray.io/en/latest/serve/key-concepts.html): Core Ray Serve abstractions: deployments, replicas, applications, the ingress deployment, and the DeploymentHandle API for composing deployments. - [Develop and Deploy an ML Application](https://docs.ray.io/en/latest/serve/develop-and-deploy.html): End-to-end workflow for converting an ML model to a Serve app, testing it locally with serve run, building a config file, and deploying to production. - [Deploy Compositions of Models](https://docs.ray.io/en/latest/serve/model_composition.html): Build multi-deployment apps with DeploymentHandles: call between deployments, await DeploymentResponse objects, and scale each stage independently. - [Deploy Multiple Applications](https://docs.ray.io/en/latest/serve/multi-app.html): Deploy and manage multiple independent Serve applications on one cluster using multi-application config files, the serve CLI, and the Serve dashboard. - [Model Multiplexing](https://docs.ray.io/en/latest/serve/model-multiplexing.html): Serve many model variants from a shared replica pool using the serve.multiplexed decorator and serve_multiplexed_model_id request header for per-model routing. - [Model Registry Integration](https://docs.ray.io/en/latest/serve/model-registries.html): Integrate Ray Serve with model registries such as MLflow, Hugging Face Hub, and Weights and Biases to load and serve registered models in production. - [Configure Ray Serve deployments](https://docs.ray.io/en/latest/serve/configure-serve-deployment.html): Reference for @serve.deployment parameters: num_replicas, ray_actor_options, max_ongoing_requests, autoscaling_config, and health checks. - [Set Up FastAPI and HTTP](https://docs.ray.io/en/latest/serve/http-guide.html): Configure HTTP handling in Ray Serve: Starlette request/response API, FastAPI integration, request cancellation, keep-alive timeouts, and custom HTTP adapters. - [Serving LLMs](https://docs.ray.io/en/latest/serve/llm/index.html): Deploy LLMs with Ray Serve LLM: OpenAI-compatible API, multi-model serving, tensor/pipeline parallelism, LoRA, and vLLM/SGLang backends. - [Quickstart](https://docs.ray.io/en/latest/serve/llm/quick-start.html): Deploy your first LLM with Ray Serve through OpenAiIngress, then move the same configuration to a production deployment. - [Examples](https://docs.ray.io/en/latest/serve/llm/examples.html): End-to-end tutorials for deploying LLMs with Ray Serve, organized by model size and by capability. - [Deploy a small-sized LLM](https://docs.ray.io/en/latest/_collections/serve/tutorials/deployment-serve-llm/small-size-llm/README.html): A small LLM runs on a single node with 1–2 GPUs, making it fast, inexpensive, and simple to use. It’s ideal for prototyping, lightweight applications, latency-critical use cases, cost-sensitive deployments, and environments with limited resources… - [Deploy a medium-sized LLM](https://docs.ray.io/en/latest/_collections/serve/tutorials/deployment-serve-llm/medium-size-llm/README.html): A medium LLM typically runs on a single node with 4-8 GPUs. It offers a balance between performance and efficiency. These models provide stronger accuracy and reasoning than small models while remaining more affordable and resource-friendly than… - [Deploy a large-sized LLM](https://docs.ray.io/en/latest/_collections/serve/tutorials/deployment-serve-llm/large-size-llm/README.html): A large LLM typically runs on multiple nodes with multiple GPUs, prioritizing peak quality and capability: stronger reasoning, broader knowledge, longer context windows, more robust generalization. When higher latency, complexity, and cost are… - [Deploy a vision LLM](https://docs.ray.io/en/latest/_collections/serve/tutorials/deployment-serve-llm/vision-llm/README.html): A vision LLM can interpret images as well as text, enabling tasks like answering questions about charts, analyzing photos, or combining visuals with instructions. It extends LLMs beyond language to support multimodal reasoning and richer… - [Deploy a reasoning LLM](https://docs.ray.io/en/latest/_collections/serve/tutorials/deployment-serve-llm/reasoning-llm/README.html): A reasoning LLM handles tasks that require deeper analysis or step-by-step thought. It generates intermediate reasoning before arriving at a final answer, making it better suited for situations where careful logic or structured problem-solving is… - [Deploy a hybrid reasoning LLM](https://docs.ray.io/en/latest/_collections/serve/tutorials/deployment-serve-llm/hybrid-reasoning-llm/README.html): A hybrid reasoning model provides flexibility by allowing you to enable or disable reasoning as needed. You can use structured, step-by-step thinking for complex queries while skipping it for simpler ones, balancing accuracy with efficiency… - [Deploy gpt-oss](https://docs.ray.io/en/latest/_collections/serve/tutorials/deployment-serve-llm/gpt-oss/README.html): gpt-oss is a family of open-source models designed for general-purpose language understanding and generation. The 20 B parameter variant (gpt-oss-20b) offers strong reasoning capabilities with lower latency. This makes it well-suited for local or… - [Deploy NVIDIA Nemotron-3-Super-120B-A12B-FP8](https://docs.ray.io/en/latest/_collections/serve/tutorials/deployment-serve-llm/nemotron-3-super-120b/README.html): This tutorial deploys nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8 with Ray Serve LLM on H100 GPUs. It’s a 120 B-total / 12 B-active hybrid model built for agentic workflows, long-context reasoning, and high-volume workloads. - [User Guides](https://docs.ray.io/en/latest/serve/llm/user-guides/index.html): How-to guides for deploying, scaling, and operating Ray Serve LLM, from configuration through production operations. - [Configuration reference](https://docs.ray.io/en/latest/serve/llm/user-guides/configuration.html): Reference for LLMConfig, the central Ray Serve LLM configuration object: model loading, engine arguments, accelerators, scaling, and LoRA. - [Deployment initialization](https://docs.ray.io/en/latest/serve/llm/user-guides/deployment-initialization.html): Speed up Ray Serve LLM startup: load models from Hugging Face or remote storage, handle gated models, and use the RunAI Streamer. - [Multi-LoRA deployment](https://docs.ray.io/en/latest/serve/llm/user-guides/multi-lora.html): Serve multiple fine-tuned LoRA adapters from a single Ray Serve LLM deployment, with adapter-aware request routing. - [Cross-node parallelism](https://docs.ray.io/en/latest/serve/llm/user-guides/cross-node-parallelism.html): Distribute LLM inference across GPUs and nodes with tensor and pipeline parallelism, including custom placement group bundles. - [Data parallel attention](https://docs.ray.io/en/latest/serve/llm/user-guides/data-parallel-attention.html): Deploy data parallel attention with Ray Serve LLM, covering YAML configuration, required parameters, and replica coordination. - [Fractional GPU serving](https://docs.ray.io/en/latest/serve/llm/user-guides/fractional-gpu.html): Serve multiple small models on one GPU with fractional GPU allocation, covering placement groups and memory management. - [Prefill/decode disaggregation](https://docs.ray.io/en/latest/serve/llm/user-guides/prefill-decode.html): Deploy Ray Serve LLM with separated prefill and decode phases using NIXLConnector or LMCacheConnectorV1. - [Direct streaming](https://docs.ray.io/en/latest/serve/llm/user-guides/direct-streaming.html): Lower streaming latency by routing requests straight to model replicas, bypassing the ingress proxy hop, with body-aware routers. - [Prefix-aware routing](https://docs.ray.io/en/latest/serve/llm/user-guides/prefix-aware-routing.html): Route LLM requests by prompt prefix to exploit KV cache locality, with load balance checks and imbalanced-load fallback. - [KV-aware routing](https://docs.ray.io/en/latest/serve/llm/user-guides/kv-aware-routing.html): Route LLM requests on measured KV cache overlap and token load with KVAwareRouter: installation, configuration, router comparison, and architecture. - [KV cache offloading](https://docs.ray.io/en/latest/serve/llm/user-guides/kv-cache-offloading.html): Extend KV cache capacity by offloading to CPU memory or local disk with native vLLM offloading or LMCache, and compose backends with MultiConnector. - [vLLM compatibility](https://docs.ray.io/en/latest/serve/llm/user-guides/vllm-compatibility.html): vLLM features reachable through Ray Serve LLM: embeddings, transcriptions, structured JSON output, and vision language models. - [Custom vLLM models](https://docs.ray.io/en/latest/serve/llm/user-guides/custom-vllm.html): Serve a custom vLLM model architecture with Ray Serve LLM by writing, registering, and packaging a model plugin. - [SGLang integration](https://docs.ray.io/en/latest/serve/llm/user-guides/sglang.html): Serve models with the SGLang engine through Ray Serve LLM's OpenAI-compatible API via the server_cls parameter on LLMConfig. - [TPU serving](https://docs.ray.io/en/latest/serve/llm/user-guides/tpu.html): Serve LLMs on single-host and multi-host TPU slices with Ray Serve LLM, including topology-aware placement groups and a multi-host vLLM TPU example. - [Observability and monitoring](https://docs.ray.io/en/latest/serve/llm/user-guides/observability.html): Monitor Ray Serve LLM deployments with service-level and engine metrics, a Grafana dashboard, and usage data collection. - [Architecture](https://docs.ray.io/en/latest/serve/llm/architecture/index.html): How Ray Serve LLM is built: its components, how a request flows through them, and the patterns that scale serving across GPUs and nodes. - [Architecture overview](https://docs.ray.io/en/latest/serve/llm/architecture/overview.html): Architecture overview of Ray Serve LLM: the Serve primitives it builds on, LLMServer and OpenAiIngress, and its main serving patterns. - [Core components](https://docs.ray.io/en/latest/serve/llm/architecture/core.html): Core Ray Serve LLM abstractions: the LLMEngine protocol, LLMConfig, deployment protocols, and the builder pattern for constructing deployments. - [Serving patterns](https://docs.ray.io/en/latest/serve/llm/architecture/serving-patterns/index.html): Architecture reference for Ray Serve LLM's distributed serving patterns, including data parallel attention and prefill-decode disaggregation. - [Data parallel attention](https://docs.ray.io/en/latest/serve/llm/architecture/serving-patterns/data-parallel.html): Data parallel attention architecture in Ray Serve LLM: DPServer, the gang master registry, request flow, and autoscaling behavior. - [Prefill-decode disaggregation](https://docs.ray.io/en/latest/serve/llm/architecture/serving-patterns/prefill-decode.html): Prefill-decode disaggregation architecture in Ray Serve LLM: why to split the phases, their resource profiles, and the server components. - [Request routing](https://docs.ray.io/en/latest/serve/llm/architecture/routing-policies.html): How Ray Serve LLM routes requests to replicas: Power of Two Choices, prefix-aware routing, and patterns for custom routing policies. - [Benchmarks](https://docs.ray.io/en/latest/serve/llm/benchmarks.html): How to benchmark Ray Serve LLM deployments, focusing on orchestration overhead, serving pattern effectiveness, and replica startup latency. - [Troubleshooting](https://docs.ray.io/en/latest/serve/llm/troubleshooting.html): Common Ray Serve LLM issues: gated Hugging Face models, slow model downloads, vLLM compatibility, and NIXL dependency conflicts. - [Production Guide](https://docs.ray.io/en/latest/serve/production-guide/index.html): Run Ray Serve in production on Kubernetes via KubeRay RayService or directly on a Ray cluster, covering config files, health checking, and failure recovery. - [Serve Config Files](https://docs.ray.io/en/latest/serve/production-guide/config.html): Structure of the Serve config file: proxy, HTTP, gRPC, and logging config, application config, serve build, and user_config updates. - [Deploy on Kubernetes](https://docs.ray.io/en/latest/serve/production-guide/kubernetes.html): Deploy Ray Serve on Kubernetes with the KubeRay operator and a RayService resource, covering queries, status, and config updates. - [Custom Docker Images](https://docs.ray.io/en/latest/serve/production-guide/docker.html): Build custom Docker images for Ray Serve by extending the Ray image with your application, and use them with KubeRay. - [Add End-to-End Fault Tolerance](https://docs.ray.io/en/latest/serve/production-guide/fault-tolerance.html): End-to-end Serve fault tolerance: replica health checks, constructor retries, worker node recovery, and GCS-backed head node recovery. - [Handle Dependencies](https://docs.ray.io/en/latest/serve/production-guide/handling-dependencies.html): Ship Ray Serve application dependencies with runtime environments, including per-deployment dependency sets. - [Best practices in production](https://docs.ray.io/en/latest/serve/production-guide/best-practices.html): Production best practices for Ray Serve, covering Serve CLI usage, client-side HTTP request handling, and load shedding. - [Monitor Your Application](https://docs.ray.io/en/latest/serve/monitoring.html): Monitor and debug Ray Serve applications using the Ray dashboard, serve status and serve config CLI commands, Ray logging with Loki, and built-in Serve metrics. - [Resource Allocation](https://docs.ray.io/en/latest/serve/resource-allocation.html): Assign CPUs, GPUs, fractional accelerators, and custom resources to Serve deployment replicas using ray_actor_options and placement group configurations. - [Ray Serve Autoscaling](https://docs.ray.io/en/latest/serve/autoscaling-guide.html): Configure Ray Serve autoscaling with num_replicas=auto or autoscaling_config: tune target_ongoing_requests, min_replicas, and max_replicas to match traffic. - [Asynchronous Inference](https://docs.ray.io/en/latest/serve/asynchronous-inference.html): Run long-running inference asynchronously using the @task_consumer and @task_handler APIs with Celery-backed queues, keeping HTTP responses immediate. - [Advanced Guides](https://docs.ray.io/en/latest/serve/advanced-guides/index.html): Advanced Ray Serve guides: autoscaling tuning, asyncio, performance, request batching, gRPC, replica and gang scheduling, and custom routing. - [Pass Arguments to Applications](https://docs.ray.io/en/latest/serve/advanced-guides/app-builder-guide.html): Pass arguments to Serve applications with an application builder function, supplied through serve run or a config file and typed with Pydantic. - [Advanced Ray Serve Autoscaling](https://docs.ray.io/en/latest/serve/advanced-guides/advanced-autoscaling.html): Advanced Ray Serve autoscaling: tune autoscaling_config for steady state and limits, control reaction to traffic, and scale composed models. - [Asyncio and concurrency best practices in Ray Serve](https://docs.ray.io/en/latest/serve/advanced-guides/asyncio-best-practices.html): Choose between async def and def in Serve deployments, size the threadpool, and avoid blocking I/O behind a FastAPI ingress. - [Performance Tuning](https://docs.ray.io/en/latest/serve/advanced-guides/performance.html): Tune Ray Serve performance: debug the request path, use async methods, set end-to-end request timeouts, and configure locality-based routing. - [Dynamic Request Batching](https://docs.ray.io/en/latest/serve/advanced-guides/dyn-req-batch.html): Batch concurrent requests in a Serve deployment with @serve.batch, custom batch size functions, and streaming batched responses. - [Updating Applications In-Place](https://docs.ray.io/en/latest/serve/advanced-guides/inplace-updates.html): Update a running Serve application in place with lightweight config updates, user_config changes, or a full code redeploy. - [Development Workflow](https://docs.ray.io/en/latest/serve/advanced-guides/dev-workflow.html): Recommended Ray Serve development loop: iterate locally with serve.run, send HTTP requests, use local testing mode, then test on a cluster. - [Set Up a gRPC Service](https://docs.ray.io/en/latest/serve/advanced-guides/grpc-guide.html): Serve a gRPC service with Ray Serve: define the service, enable the gRPC proxy, send requests, and use streaming and model composition. - [Replica ranks](https://docs.ray.io/en/latest/serve/advanced-guides/replica-ranks.html): Replica ranks give each Serve replica a unique rank and world size, with reconfigure hooks and a defined rank assignment lifecycle. - [Replica scheduling](https://docs.ray.io/en/latest/serve/advanced-guides/replica-scheduling.html): Control where Ray Serve places replicas using max_replicas_per_node, placement groups, and node labels, plus downscaling behavior. - [Gang scheduling](https://docs.ray.io/en/latest/serve/advanced-guides/gang-scheduling.html): Co-schedule groups of Serve replicas atomically with gang scheduling, using placement group PACK or SPREAD strategies and gang context. - [Experimental Java API](https://docs.ray.io/en/latest/serve/advanced-guides/managing-java-deployments.html): Experimental Java API for creating, updating, configuring, and scaling Serve deployments, and calling Python deployments from Java. - [Deploy on VM](https://docs.ray.io/en/latest/serve/advanced-guides/deploy-vm.html): Deploy a Ray Serve application to a VM-based cluster with serve deploy, including targeting a remote cluster. - [Run Multiple Applications in Different Containers](https://docs.ray.io/en/latest/serve/advanced-guides/multi-app-container.html): Run Serve applications in separate containers with Podman-backed runtime environments, including compatibility notes and troubleshooting. - [Use Custom Algorithm for Request Routing](https://docs.ray.io/en/latest/serve/advanced-guides/custom-request-router.html): Write a custom Ray Serve request router, from a uniform router to a throughput-aware one, using the request routing utility mixins. - [Use deployment-scoped actors](https://docs.ray.io/en/latest/serve/advanced-guides/deployment-scoped-actors.html): Use deployment-scoped actors to share state across replicas, covering actor recreation, rollout, health, and lifecycle limits. - [Troubleshoot multi-node GPU serving on KubeRay](https://docs.ray.io/en/latest/serve/advanced-guides/multi-node-gpu-troubleshooting.html): Troubleshoot multi-node GPU serving on KubeRay: head pod placement on GPU nodes, AWS OFI plugin issues, and NCCL debugging. - [Architecture](https://docs.ray.io/en/latest/serve/architecture.html): Internals of Ray Serve: Controller, HTTP Proxy, gRPC Proxy, and replica actors; request lifetime from proxy to replica; fault tolerance and recovery mechanisms. - [Examples](https://docs.ray.io/en/latest/serve/examples.html) - [Model composition for recommendation systems](https://docs.ray.io/en/latest/_collections/serve/tutorials/model-composition-recsys/README.html): This tutorial shows you how to build a recommendation system using Ray Serve’s model composition pattern. Model composition breaks complex ML pipelines into independent deployments that you can scale and update separately. - [Model multiplexing with forecasting models](https://docs.ray.io/en/latest/_collections/serve/tutorials/model_multiplexing_forecast/README.html): This tutorial shows you how to efficiently serve multiple forecasting models using Ray Serve’s model multiplexing pattern. Model multiplexing lets you serve dozens or thousands of models from a shared pool of replicas, optimizing cost and resources. - [Serve an Inference Model on AWS NeuronCores Using FastAPI (Experimental)](https://docs.ray.io/en/latest/serve/tutorials/aws-neuron-core-inference.html): Experimental tutorial serving an inference model on AWS NeuronCores with Ray Serve and FastAPI. - [Serve an Inference with Stable Diffusion Model on AWS NeuronCores Using FastAPI](https://docs.ray.io/en/latest/serve/tutorials/aws-neuron-core-inference-stable-diffusion.html): Deploy a precompiled Stable Diffusion XL model on an AWS Inferentia2 instance using Ray Serve and FastAPI. - [Serve a Text Generator with Request Batching](https://docs.ray.io/en/latest/serve/tutorials/batch.html): Deploy a text generator that batches concurrent queries, with CLI and Python deployment options and parallel HTTP querying. - [Scale a Gradio App with Ray Serve](https://docs.ray.io/en/latest/serve/tutorials/gradio-integration.html): Scale an existing Gradio app with Ray Serve using GradioServer, and parallelize multiple models behind one app. - [Serve Llama2-7b/70b on a single or multiple Intel Gaudi Accelerator](https://docs.ray.io/en/latest/serve/tutorials/intel-gaudi-inference.html): Serve Llama 2 7B and 70B on Intel Gaudi HPUs with Ray Serve, on a single accelerator or sharded across several. - [Serve a Java App](https://docs.ray.io/en/latest/serve/tutorials/java.html): Deploy a Java application as a Ray Serve deployment and call it over HTTP, starting from the Maven dependency. - [Building a Real-time Object Detection Service with Ray Serve](https://docs.ray.io/en/latest/serve/tutorials/object-detection.html): Build a real-time object detection service with Ray Serve, covering code structure, deployment, testing, and troubleshooting. - [Serve DeepSeek](https://docs.ray.io/en/latest/serve/tutorials/serve-deepseek.html): Deploy DeepSeek R1 or V3 with Ray Serve LLM, from a quick local deployment to a production KubeRay setup. - [Serve ML Models (Tensorflow, PyTorch, Scikit-Learn, others)](https://docs.ray.io/en/latest/serve/tutorials/serve-ml-models.html): Train models with TensorFlow, PyTorch, or scikit-learn and deploy each of them to Ray Serve. - [Serve a Stable Diffusion Model](https://docs.ray.io/en/latest/serve/tutorials/stable-diffusion.html): Deploy a Stable Diffusion image generation model as a Ray Serve application. - [Serve a Chatbot with Request and Response Streaming](https://docs.ray.io/en/latest/serve/tutorials/streaming.html): Deploy a chatbot that streams responses back to the caller, including WebSocket streaming and batched streaming output. - [Serve a Text Classification Model](https://docs.ray.io/en/latest/serve/tutorials/text-classification.html): Deploy a text classification model as a Ray Serve application. - [Serving models with Triton Server in Ray Serve](https://docs.ray.io/en/latest/serve/tutorials/triton-server-integration.html): Run NVIDIA Triton Server inside a Ray Serve application, building and exporting a model and preparing the model repository. - [Video analysis inference pipeline with Ray Serve](https://docs.ray.io/en/latest/serve/tutorials/video-analysis/README.html): Build a production video analysis pipeline with Ray Serve that processes videos from S3 and extracts embeddings on GPU encoders. ## Ray RLlib - [RLlib: Industry-Grade, Scalable Reinforcement Learning](https://docs.ray.io/en/latest/rllib/index.html): Industry-grade, scalable reinforcement learning with a unified API for single-agent, multi-agent, and offline RL training and deployment. - [Getting Started](https://docs.ray.io/en/latest/rllib/getting-started.html): End-to-end RLlib tutorial: configure and build an Algorithm, train, checkpoint, evaluate, and deploy a trained policy. - [Key concepts](https://docs.ray.io/en/latest/rllib/key-concepts.html): Core RLlib architecture: the Algorithm, AlgorithmConfig, EnvRunner actors for sampling, Learner actors for updates, and RLModule models. - [Environments](https://docs.ray.io/en/latest/rllib/rllib-env.html): How to connect RL environments to RLlib, covering single-agent, multi-agent, hierarchical, and external environment setups with the EnvRunner API. - [Multi-Agent Environments](https://docs.ray.io/en/latest/rllib/multi-agent-envs.html): Write RLlib MultiAgentEnv environments: per-agent observation and action spaces, step and reward dictionaries, turn-based play, and agent grouping. - [Hierarchical Environments](https://docs.ray.io/en/latest/rllib/hierarchical-envs.html): Implement hierarchical reinforcement learning in RLlib as a multi-agent problem, with a top-level policy issuing tasks to lower-level policies. - [External Environments and Applications](https://docs.ray.io/en/latest/rllib/external-envs.html): Connect external simulators and applications to RLlib over the RLlink protocol, covering message types, client-server workflow, and episode exchange. - [AlgorithmConfig API](https://docs.ray.io/en/latest/rllib/algorithm-config.html): Reference for the AlgorithmConfig API: type-safe configuration of training, environment, learner, and framework settings for any RLlib Algorithm. - [Algorithms](https://docs.ray.io/en/latest/rllib/rllib-algorithms.html): Catalog of all RLlib built-in algorithms — PPO, DQN, SAC, APPO, IMPALA, DreamerV3, BC, CQL, IQL, MARWIL — with action-space and multi-GPU support details. - [User Guides](https://docs.ray.io/en/latest/rllib/user-guides.html): RLlib feature guides: callbacks, checkpoints, metrics, connectors, replay buffers, offline RL, RLModule, Learner, fault tolerance, and scaling. - [Advanced Python APIs](https://docs.ray.io/en/latest/rllib/rllib-advanced-api.html): Advanced RLlib control: custom training workflows, curriculum learning, global coordination between actors, custom metrics, and tailored evaluation. - [Callbacks](https://docs.ray.io/en/latest/rllib/rllib-callback.html): Inject code into RLlib experiments with callback lambdas or a stateful RLlibCallback, covering callback events, chaining, and resolution order. - [Checkpointing](https://docs.ray.io/en/latest/rllib/checkpoints.html): Save and restore RLlib Algorithm and subcomponent state through the Checkpointable API, covering directory structure, versions, and partial restores. - [MetricsLogger API](https://docs.ray.io/en/latest/rllib/metrics-logger.html): Log and aggregate metrics across RLlib components with MetricsLogger: scalars, non-scalar data, timers, counters, and throughput measurements. - [Episodes](https://docs.ray.io/en/latest/rllib/single-agent-episode.html): SingleAgentEpisode, RLlib's trajectory data container: construction, getter APIs, numpy'ized episodes, and cut() with lookback buffers. - [ConnectorV2 and ConnectorV2 pipelines](https://docs.ray.io/en/latest/rllib/connector-v2.html): ConnectorV2 pipelines that transform data between environment, module, and learner, covering the three pipeline types and batch construction phases. - [Env-to-module pipelines](https://docs.ray.io/en/latest/rllib/env-to-module-connector.html): Build env-to-module ConnectorV2 pipelines that turn environment observations into RLModule input, with observation preprocessors and custom columns. - [Learner connector pipelines](https://docs.ray.io/en/latest/rllib/learner-connector.html): Build Learner ConnectorV2 pipelines that compile train batches from episodes, with examples for reward shaping and stacking recent observations. - [Replay Buffers](https://docs.ray.io/en/latest/rllib/rllib-replay-buffers.html): Replay buffers in RLlib: sampling and eviction strategies, the ReplayBuffer base class, configuration, and building a custom buffer. - [Working with offline data](https://docs.ray.io/en/latest/rllib/rllib-offline.html): Offline RL in RLlib: record expert data, train on saved experiences, convert tabular data to episode format, and scale offline I/O throughput. - [RL Modules](https://docs.ray.io/en/latest/rllib/rl-modules.html): Define custom neural network logic with RLModule: default MLP, CNN, and LSTM configuration, RLModuleSpecs, and multi-network multi-agent setups. - [Learner (Alpha)](https://docs.ray.io/en/latest/rllib/rllib-learner.html): The Learner API that applies gradient and non-gradient updates to RLModules, with DDP-based distribution, state access, and checkpointing. - [Fault Tolerance And Elastic Training](https://docs.ray.io/en/latest/rllib/rllib-fault-tolerance.html): Fault tolerance and elastic training in RLlib: EnvRunner recovery, environment failures, and how Ray Tune restores interrupted experiments. - [Install RLlib for Development](https://docs.ray.io/en/latest/rllib/rllib-dev.html): Set up RLlib for local development without compiling Ray, plus contribution guidance for algorithms, API decorators, and finding worker memory leaks. - [RLlib scaling guide](https://docs.ray.io/en/latest/rllib/scaling-guide.html): Scale RLlib training by tuning EnvRunner actor count, environments per runner, and Learner actor count for higher sampling and learning throughput. - [Examples](https://docs.ray.io/en/latest/rllib/rllib-examples.html): Runnable RLlib example scripts organized by feature area, with common CLI flags like --num-env-runners and --wandb-key. - [New API stack migration guide](https://docs.ray.io/en/latest/rllib/new-api-stack-migration-guide.html): Migrate from the old RLlib API stack to the new one: mapping ModelV2, RolloutWorker, and Policy to RLModule, Learner, and ConnectorV2. ## More Libraries - [More Ray ML Libraries](https://docs.ray.io/en/latest/ray-more-libs/index.html): Additional Ray integrations: Joblib, multiprocessing, Ray Collective, Dask-on-Ray, RayDP (Spark), Mars-on-Ray, and Modin. - [Distributed Scikit-learn / Joblib](https://docs.ray.io/en/latest/ray-more-libs/joblib.html): Scale scikit-learn across a Ray cluster by registering Ray as a Joblib backend via ray.util.joblib.register_ray. - [Distributed multiprocessing.Pool](https://docs.ray.io/en/latest/ray-more-libs/multiprocessing.html): Scale multiprocessing.Pool programs to a Ray cluster with ray.util.multiprocessing.Pool as a drop-in replacement. - [Ray Collective Communication Lib](https://docs.ray.io/en/latest/ray-more-libs/ray-collective.html): Collective communication primitives (allreduce, broadcast, send/recv) for Ray actors and tasks via ray.util.collective, on NCCL and GLOO. - [Custom Collective Backends](https://docs.ray.io/en/latest/ray-more-libs/ray-collective-custom-backend.html): Create and register a custom collective communication backend for Ray, with driver-managed and worker-managed group initialization. - [Using Dask on Ray](https://docs.ray.io/en/latest/ray-more-libs/dask-on-ray.html): Run Dask DataFrame and array workloads on a Ray cluster via the dask_on_ray scheduler, without a separate Dask cluster. - [RayDaskCallback](https://docs.ray.io/en/latest/ray-more-libs/doc/ray.util.dask.RayDaskCallback.html): Extends Dask’s Callback class with Ray-specific hooks. When instantiating or subclassing this class, both the normal Dask hooks (e.g. pretask, posttask, etc.) and the Ray-specific hooks can be provided. - [ray_active](https://docs.ray.io/en/latest/ray-more-libs/doc/ray.util.dask.RayDaskCallback.ray_active.html) - [_ray_presubmit](https://docs.ray.io/en/latest/ray-more-libs/doc/ray.util.dask.callbacks.RayDaskCallback._ray_presubmit.html): Run before submitting a Ray task. - [_ray_postsubmit](https://docs.ray.io/en/latest/ray-more-libs/doc/ray.util.dask.callbacks.RayDaskCallback._ray_postsubmit.html): Run after submitting a Ray task. - [_ray_pretask](https://docs.ray.io/en/latest/ray-more-libs/doc/ray.util.dask.callbacks.RayDaskCallback._ray_pretask.html): Run before executing a Dask task within a Ray task. - [_ray_posttask](https://docs.ray.io/en/latest/ray-more-libs/doc/ray.util.dask.callbacks.RayDaskCallback._ray_posttask.html): Run after executing a Dask task within a Ray task. - [_ray_postsubmit_all](https://docs.ray.io/en/latest/ray-more-libs/doc/ray.util.dask.callbacks.RayDaskCallback._ray_postsubmit_all.html): Run after Ray submits all tasks. - [_ray_finish](https://docs.ray.io/en/latest/ray-more-libs/doc/ray.util.dask.callbacks.RayDaskCallback._ray_finish.html): Run after Ray finishes executing all Ray tasks and returns the final result. - [Using Spark on Ray (RayDP)](https://docs.ray.io/en/latest/ray-more-libs/raydp.html): Run Apache Spark on Ray via raydp.init_spark, combining PySpark data processing with PyTorch and TensorFlow training in one cluster. - [Using Mars on Ray](https://docs.ray.io/en/latest/ray-more-libs/mars-on-ray.html): Run Mars tensor and dataframe computations on Ray via mars.new_ray_session, scaling NumPy, Pandas, and scikit-learn workloads. - [Using Pandas on Ray (Modin)](https://docs.ray.io/en/latest/ray-more-libs/modin/index.html): Use Modin as a drop-in pandas replacement on Ray: import modin.pandas to parallelize DataFrame operations across a cluster. - [Distributed Data Processing in Data-Juicer](https://docs.ray.io/en/latest/ray-more-libs/data_juicer_distributed_data_processing.html): Process foundation-model datasets with Data-Juicer on Ray: subset splitting, streaming JSON I/O, and MinHash-LSH deduplication at TB scale. ## APIs - [Ray APIs](https://docs.ray.io/en/latest/apis/index.html): Entry point for Ray's API reference, with per-library API documentation for Core, Data, Train, Tune, Serve, and RLlib. - [Ray Train](https://docs.ray.io/en/latest/train/api/api.html): API reference index for Ray Train, covering the PyTorch, Lightning, Transformers, TensorFlow/Keras, XGBoost, and LightGBM trainers and configs. - [Ray Serve](https://docs.ray.io/en/latest/serve/api/index.html): Ray Serve API reference: the Python API for writing and running applications, configs and schemas, request router, CLI, and REST API. ## Ray Clusters - [Ray Clusters Overview](https://docs.ray.io/en/latest/cluster/getting-started.html): Deploy and scale Ray clusters from a laptop to the cloud, with native support for Kubernetes (KubeRay), AWS, GCP, and Azure VMs, plus autoscaling. - [Key Concepts](https://docs.ray.io/en/latest/cluster/key-concepts.html): Core Ray cluster concepts: head and worker nodes, the autoscaler, Ray jobs, the GCS, and namespaces. - [Deploying on Kubernetes](https://docs.ray.io/en/latest/cluster/kubernetes/index.html): Deploy and manage Ray clusters on Kubernetes with the KubeRay operator: RayCluster, RayJob, and RayService CRDs, autoscaling, and GPU support. - [Getting Started with KubeRay](https://docs.ray.io/en/latest/cluster/kubernetes/getting-started.html): Get started with KubeRay, the Kubernetes operator for Ray, and choose between the RayCluster, RayJob, and RayService CRDs. - [KubeRay Operator Installation](https://docs.ray.io/en/latest/cluster/kubernetes/getting-started/kuberay-operator-installation.html): Install the KubeRay operator with Helm or Kustomize and validate the installation on a Kind or existing cluster. - [RayCluster Quickstart](https://docs.ray.io/en/latest/cluster/kubernetes/getting-started/raycluster-quick-start.html): Deploy a RayCluster on Kubernetes, run a job in the head pod or through the job submission SDK, and open the Ray Dashboard. - [RayJob Quickstart](https://docs.ray.io/en/latest/cluster/kubernetes/getting-started/rayjob-quick-start.html): Run a Ray job on Kubernetes with RayJob, which creates a RayCluster on demand and can delete it once the job finishes. - [RayService Quickstart](https://docs.ray.io/en/latest/cluster/kubernetes/getting-started/rayservice-quick-start.html): Serve Ray Serve applications on Kubernetes with the RayService custom resource, deploying two applications as an example. - [RayCronJob Quickstart](https://docs.ray.io/en/latest/cluster/kubernetes/getting-started/raycronjob-quick-start.html): Run Ray jobs on a recurring schedule with the RayCronJob custom resource, including configuration and a worked example. - [User Guides](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides.html): Index of KubeRay user guides covering cluster configuration, autoscaling, GPUs and TPUs, storage, observability, and security. - [Deploy Ray Serve Apps](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/rayservice.html): Deploy Ray Serve applications on Kubernetes with RayService, covering what it provides and a two-application example. - [RayService worker Pods aren’t ready](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/rayservice-no-ray-serve-replica.html): Why a RayService worker pod can stay not-ready with no Serve replica scheduled, and how to verify and update the apps in place. - [RayService high availability](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/rayservice-high-availability.html): Configure RayService high availability with GCS fault tolerance so Serve keeps handling requests when the head pod fails. - [Enable High Throughput on Ray Serve with KubeRay](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/kuberay-serve-high-throughput.html): Enable high throughput mode for Ray Serve on KubeRay, demonstrated by serving Qwen on a GKE GPU node pool. - [RayService Zero-Downtime Incremental Upgrades](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/rayservice-incremental-upgrade.html): Perform zero-downtime incremental RayService upgrades, covering the upgrade process, configuration, and monitoring. - [KubeRay Observability](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/observability.html): Observe KubeRay and Ray on Kubernetes: operator logs, custom resource status and events, the Ray Dashboard, and the State CLI. - [KubeRay upgrade guide](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/upgrade-guide.html): KubeRay and Ray version compatibility, CRD versioning, and how to upgrade a KubeRay installation. - [Managed Kubernetes services](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/k8s-cluster-setup.html): Links for setting up managed Kubernetes clusters with GPU nodes, for the KubeRay examples that need them. - [Start Amazon EKS Cluster with GPUs for KubeRay](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/aws-eks-gpu-cluster.html): Create an Amazon EKS cluster with CPU and GPU node groups ready for KubeRay. - [Start Google Cloud GKE Cluster with GPUs for KubeRay](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/gcp-gke-gpu-cluster.html): Create a GKE cluster with a GPU node pool for KubeRay, including optional manual driver installation. - [Start Google Cloud GKE Cluster with TPUs for KubeRay](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/gcp-gke-tpu-cluster.html): Create a GKE cluster with TPU node pools for KubeRay, including manual install of the TPU webhook. - [Start Azure AKS Cluster with GPUs for KubeRay](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/azure-aks-gpu-cluster.html): Create an Azure AKS cluster with a GPU node pool for KubeRay and fetch its kubeconfig. - [Start an Aliyun ACK cluster with GPUs for KubeRay](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/ack-gpu-cluster.html): Create an Aliyun ACK cluster with GPU node pools for KubeRay, including node labels, taints, and driver upgrades. - [Best Practices for Storage and Dependencies](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/storage.html): Storage and dependency recommendations for Ray on Kubernetes, split between interactive development and production use. - [RayCluster Configuration](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/config.html): Configure a RayCluster on Kubernetes: headGroupSpec and workerGroupSpecs, pod templates, the Ray version, and ray start parameters. - [Scheduling on Kubernetes](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/scheduling.html): How Kubernetes and Ray divide scheduling responsibility when you run Ray on Kubernetes, how KubeRay maps container resources to Ray's logical capacity, and which layer to investigate when a workload doesn't start. - [KubeRay Autoscaling](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/configuring-autoscaling.html): Configure the Ray autoscaler on Kubernetes so a RayCluster adds and removes worker pods as resource demand changes. - [KubeRay In-Place Pod Resizing (IPPR)](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/configuring-ippr.html): Resize Ray worker pod resources in place with KubeRay IPPR, avoiding restarts, with validation rules and resize behavior. - [KubeRay label-based scheduling](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/label-based-scheduling.html): Direct Ray tasks, actors, and placement groups to specific pods with label-based scheduling on Kubernetes. - [GCS fault tolerance in KubeRay](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/kuberay-gcs-ft.html): Make the GCS fault tolerant on KubeRay by backing cluster metadata with Redis, so head pod failure doesn't kill the cluster. - [Tuning Redis for a Persistent Fault Tolerant GCS](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/kuberay-gcs-persistent-ft.html): Tune Redis persistence and backups so a KubeRay cluster's GCS state survives head pod loss and Redis restarts. - [GCS fault tolerance with embedded RocksDB (alpha)](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/kuberay-gcs-rocksdb-ft.html): Alpha GCS fault tolerance using an embedded RocksDB backend instead of Redis, enabled behind an operator feature gate. - [Configuring KubeRay to use Google Cloud Storage Buckets in GKE](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/gke-gcs-bucket.html): Give a RayCluster access to a Google Cloud Storage bucket on GKE using Workload Identity and a Kubernetes service account. - [Persist KubeRay custom resource logs](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/persist-kuberay-custom-resource-logs.html): Persist Pod logs off-cluster with sidecar or DaemonSet collection, using Fluent Bit, Loki, and Grafana. - [Persist KubeRay Operator Logs](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/persist-kuberay-operator-logs.html): Persist KubeRay operator logs to Grafana Loki and view them in a Grafana dashboard. - [Using GPUs](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/gpu.html): Schedule GPU workloads on KubeRay: request GPUs, GPU autoscaling, override Ray GPU capacity, and use taints and node selectors. - [Use TPUs with KubeRay](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/tpu.html): Use TPUs with KubeRay: configure Kubernetes Pods for TPUs, TPU workload scheduling and default labels, and multi-host TPU autoscaling. - [Specify container commands for Ray head/worker Pods](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/pod-command.html): Customize the container command for Ray head and worker pods, running commands before or instead of the generated ray start. - [Helm Chart RBAC](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/helm-chart-rbac.html): Configure the KubeRay Helm chart's RBAC resources to watch all namespaces, only the operator's namespace, or a specific set. - [TLS Authentication](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/tls.html): Enable TLS on Ray's gRPC channels so traffic between client, head, and worker processes is authenticated and encrypted. - [Configure RayCluster to use NetworkPolicies](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/network-policy.html): The KubeRay operator can generate Kubernetes NetworkPolicy resources for each RayCluster. When enabled, the operator creates separate policies for the head pod and worker pods, and removes them when the cluster is deleted. - [(Advanced) Understanding the Ray Autoscaler in the Context of Kubernetes](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/k8s-autoscaler.html): How the Ray autoscaler differs from the Horizontal Pod Autoscaler, and how it interacts with the Kubernetes cluster autoscaler. - [Use kubectl plugin (beta)](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/kubectl-plugin.html): Use the kubectl ray plugin to simplify RayCluster management and RayJob submission, with Krew installation and examples. - [Configure Ray clusters to use token authentication](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/kuberay-auth.html): Enable Ray token authentication on a KubeRay cluster and access it with the Ray CLI and the dashboard. - [Configure Ray clusters to use Kubernetes RBAC authentication](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/kuberay-auth-rbac.html): Enable Ray token authentication backed by Kubernetes RBAC, including external IAM access to the cluster. - [Reducing image pull latency on Kubernetes](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/reduce-image-pull-latency.html): Reduce Ray image pull latency on Kubernetes by preloading images, using private registries, or enabling GKE image streaming. - [Using uv for Python package management in KubeRay](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/uv.html): Use uv for Python package management in a KubeRay cluster and run a Ray script with uv-managed dependencies. - [Use KubeRay dashboard (experimental)](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/kuberay-dashboard.html): Install and use the experimental open-source KubeRay dashboard UI, available from KubeRay v1.4.0. - [Resource isolation with writable cgroups on Google Kubernetes Engine (GKE)](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/resource-isolation-with-writable-cgroups.html): Enable Ray resource isolation on GKE with writable cgroups, reserving system resources and verifying the cgroup hierarchy. - [Ray History Server with KubeRay](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/kuberay-history-server.html): Set up the Ray History Server with KubeRay to access logs, events, and the Ray Dashboard for RayCluster, RayJob, and RayService workloads after they terminate. - [Enable Ray platform events on Kubernetes](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/k8s-events.html): How to enable platform events in the Ray Dashboard on Kubernetes: RBAC permissions, the environment variable that turns the feature on, and how to verify that the dashboard receives events. - [RayJob SidecarSubmitterRestart](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/rayjob-sidecar-submitter-restart.html): SidecarMode runs the RayJob submitter as a container inside the Ray head Pod. Unlike K8sJobMode, SidecarMode does not require pulling a duplicated Ray image and avoids the need for inter-pod communication as the submitter container reaches the head… - [Examples](https://docs.ray.io/en/latest/cluster/kubernetes/examples.html): Example Ray workloads to run on a Kubernetes cluster, spanning training, batch inference, and model serving. - [Train a PyTorch model on Fashion MNIST with CPUs on Kubernetes](https://docs.ray.io/en/latest/cluster/kubernetes/examples/mnist-training-example.html): Train a PyTorch model on Fashion MNIST with CPUs on Kubernetes using a RayJob, end to end. - [Serve a StableDiffusion text-to-image model on Kubernetes](https://docs.ray.io/en/latest/cluster/kubernetes/examples/stable-diffusion-rayservice.html): Serve a Stable Diffusion text-to-image model on a GPU Kubernetes cluster with RayService. - [Serve a Stable Diffusion model on GKE with TPUs](https://docs.ray.io/en/latest/cluster/kubernetes/examples/tpu-serve-stable-diffusion.html): Serve a Stable Diffusion model on GKE TPUs with RayService, from TPU node pool creation to text-to-image prompts. - [Serve a MobileNet image classifier on Kubernetes](https://docs.ray.io/en/latest/cluster/kubernetes/examples/mobilenet-rayservice.html): Serve a MobileNet image classifier on Kubernetes with RayService, from Kind cluster setup to a live classification request. - [Serve a text summarizer on Kubernetes](https://docs.ray.io/en/latest/cluster/kubernetes/examples/text-summarizer-rayservice.html): Serve a text summarization model on a GPU Kubernetes cluster with RayService, including teardown steps. - [RayJob Batch Inference Example](https://docs.ray.io/en/latest/cluster/kubernetes/examples/rayjob-batch-inference-example.html): Run an offline batch inference workload on Kubernetes with RayJob, from GKE cluster creation through job submission. - [Priority Scheduling with RayJob and Kueue](https://docs.ray.io/en/latest/cluster/kubernetes/examples/rayjob-kueue-priority-scheduling.html): Priority-schedule queued RayJob workloads with Kueue so higher-priority jobs move ahead of ones already waiting. - [Gang Scheduling with RayJob and Kueue](https://docs.ray.io/en/latest/cluster/kubernetes/examples/rayjob-kueue-gang-scheduling.html): Gang-schedule RayJob workloads with Kueue on GKE so all pods start together, avoiding partial-allocation deadlock. - [Distributed checkpointing with KubeRay and GCSFuse](https://docs.ray.io/en/latest/cluster/kubernetes/examples/distributed-checkpointing-with-gcsfuse.html): Distributed checkpointing on GKE with KubeRay and GCSFuse: bucket configuration, RayJob deployment, and resuming from a checkpoint. - [Serve a Large Language Model using Ray Serve LLM on Kubernetes](https://docs.ray.io/en/latest/cluster/kubernetes/examples/rayserve-llm-example.html): Serve a large language model with Ray Serve LLM on Kubernetes, including a Hugging Face token Secret and RayService deployment. - [Serve Deepseek R1 using Ray Serve LLM](https://docs.ray.io/en/latest/cluster/kubernetes/examples/rayserve-deepseek-example.html): Serve DeepSeek R1 with Ray Serve LLM on GKE via RayService, from GPU cluster setup to the first request. - [Reinforcement Learning with Human Feedback (RLHF) for LLMs with verl on KubeRay](https://docs.ray.io/en/latest/cluster/kubernetes/examples/verl-post-training.html): Run RLHF post-training for LLMs with verl on KubeRay, demonstrating PPO training on a Ray cluster. - [Deploying Ray Clusters via ArgoCD](https://docs.ray.io/en/latest/cluster/kubernetes/examples/argocd.html): Deploy Ray clusters with ArgoCD, including operator install and the ignoreDifferences settings that stop autoscaling from fighting GitOps sync. - [Sandboxed Code Execution with Ray and Agent Sandbox](https://docs.ray.io/en/latest/cluster/kubernetes/examples/rayjob-agent-sandbox.html): Run sandboxed code execution with KubeRay and Agent Sandbox on GKE with gVisor, covering RBAC and sandbox infrastructure. - [Deploy Ray sandboxes with KubeRay](https://docs.ray.io/en/latest/cluster/kubernetes/examples/ray-sandboxing.html): Deploy Ray Sandboxes on Google Kubernetes Engine (GKE) with KubeRay for kernel-isolated, high-throughput untrusted code execution using gVisor. - [KubeRay Ecosystem](https://docs.ray.io/en/latest/cluster/kubernetes/k8s-ecosystem.html): Index of KubeRay integrations with the Kubernetes ecosystem: ingress, service mesh, batch schedulers, and observability stacks. - [Ingress](https://docs.ray.io/en/latest/cluster/kubernetes/k8s-ecosystem/ingress.html): Expose Ray clusters on Kubernetes through the KubeRay built-in Ingress, AWS ALB ingress on EKS, GKE Ingress, or the GKE Gateway API. - [KubeRay metrics references](https://docs.ray.io/en/latest/cluster/kubernetes/k8s-ecosystem/metrics-references.html): Reference for the metrics KubeRay exposes: controller-runtime metrics plus custom RayCluster, RayService, and RayJob metrics. - [Using Prometheus and Grafana](https://docs.ray.io/en/latest/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.html): Monitor Ray clusters on Kubernetes with Prometheus and Grafana, using PodMonitors for Ray nodes and a ServiceMonitor for KubeRay. - [Profiling with py-spy](https://docs.ray.io/en/latest/cluster/kubernetes/k8s-ecosystem/pyspy.html): Profile Ray workloads on Kubernetes with py-spy, viewing flame graphs in the dashboard from a SYS_PTRACE-capable RayCluster. - [Gang scheduling, queue priority, and GPU sharing for RayClusters using KAI Scheduler](https://docs.ray.io/en/latest/cluster/kubernetes/k8s-ecosystem/kai-scheduler.html): Use NVIDIA KAI Scheduler with RayClusters for hierarchical queues with quotas, gang scheduling, and GPU sharing. - [KubeRay integration with Volcano](https://docs.ray.io/en/latest/cluster/kubernetes/k8s-ecosystem/volcano.html): Integrate KubeRay with Volcano for gang scheduling, job queues, and fair-share policies across RayCluster and RayJob. - [KubeRay integration with Apache YuniKorn](https://docs.ray.io/en/latest/cluster/kubernetes/k8s-ecosystem/yunikorn.html): Integrate KubeRay with Apache YuniKorn for fine-grained multi-tenant resource sharing and gang scheduling. - [Gang scheduling, Priority scheduling, and Autoscaling for KubeRay CRDs with Kueue](https://docs.ray.io/en/latest/cluster/kubernetes/k8s-ecosystem/kueue.html): Integrate KubeRay with Kueue for gang scheduling, priority scheduling, and autoscaling across the KubeRay CRDs. - [mTLS and L7 observability with Istio](https://docs.ray.io/en/latest/cluster/kubernetes/k8s-ecosystem/istio.html): Enable Istio mTLS and L7 traffic observability for a RayCluster, including the headless service Ray requires. - [KubeRay integration with scheduler plugins](https://docs.ray.io/en/latest/cluster/kubernetes/k8s-ecosystem/scheduler-plugins.html): Use the kubernetes-sigs scheduler-plugins with KubeRay for gang scheduling, verifying the RayCluster's Pods and resulting PodGroup. - [KubeRay Benchmarks](https://docs.ray.io/en/latest/cluster/kubernetes/benchmarks.html): Index of KubeRay benchmark results, including operator memory and scalability measurements. - [KubeRay memory and scalability benchmark](https://docs.ray.io/en/latest/cluster/kubernetes/benchmarks/memory-scalability-benchmark.html): KubeRay operator memory and scalability benchmark: setup with Prometheus and Grafana, the experiment procedure, and v0.6.0 results. - [KubeRay Troubleshooting](https://docs.ray.io/en/latest/cluster/kubernetes/troubleshooting.html): Index of KubeRay troubleshooting guides covering RayService and general operator and cluster problems. - [Troubleshooting guide](https://docs.ray.io/en/latest/cluster/kubernetes/troubleshooting/troubleshooting.html): Common KubeRay problems: Ray version mismatches, ARM images on Apple silicon, worker init containers, and cluster domain issues. - [RayService troubleshooting](https://docs.ray.io/en/latest/cluster/kubernetes/troubleshooting/rayservice-troubleshooting.html): Diagnose RayService problems through operator logs, custom resource status, pod logs, the dashboard, and the Ray State CLI. - [API Reference](https://docs.ray.io/en/latest/cluster/kubernetes/references.html): API reference for the KubeRay custom resources, plus KubeRay's API compatibility and stability guarantees. - [KubeRay CRD API reference](https://docs.ray.io/en/latest/cluster/kubernetes/references/api.html): Generated reference for the KubeRay ray.io/v1 custom resource definitions, covering every field of RayCluster, RayCronJob, RayJob, and RayService. - [Deploying on VMs](https://docs.ray.io/en/latest/cluster/vms/index.html): Launch and manage Ray clusters on cloud VMs across AWS, GCP, and Azure, with autoscaling and heterogeneous compute. - [Getting Started](https://docs.ray.io/en/latest/cluster/vms/getting-started.html): Scale a laptop Python application to a cloud VM cluster with the Ray cluster launcher, then tear the cluster down. - [User Guides](https://docs.ray.io/en/latest/cluster/vms/user-guides/index.html): Index of in-depth guides for deploying Ray on cloud VMs and on-premises hardware. - [Launching Ray Clusters on AWS, GCP, Azure, vSphere, On-Prem](https://docs.ray.io/en/latest/cluster/vms/user-guides/launching-clusters/index.html): Index of guides for launching Ray clusters on AWS, GCP, Azure, vSphere, and on-premises hardware. - [Launching Ray Clusters on AWS](https://docs.ray.io/en/latest/cluster/vms/user-guides/launching-clusters/aws.html): Launch a Ray cluster on AWS with the cluster launcher, covering Boto3 setup, EFS, IAM roles, S3 access, and CloudWatch. - [Launching Ray Clusters on GCP](https://docs.ray.io/en/latest/cluster/vms/user-guides/launching-clusters/gcp.html): Launch a Ray cluster on GCP with the cluster launcher, including service account configuration for worker nodes. - [Launching Ray Clusters on Azure](https://docs.ray.io/en/latest/cluster/vms/user-guides/launching-clusters/azure.html): Launch a Ray cluster on Azure with the cluster launcher or from the Azure portal. - [Launching Ray Clusters on vSphere](https://docs.ray.io/en/latest/cluster/vms/user-guides/launching-clusters/vsphere.html): Launch a Ray cluster on vSphere, including the supervisor service and vSAN File Service for persistent storage. - [Launching an On-Premise Cluster](https://docs.ray.io/en/latest/cluster/vms/user-guides/launching-clusters/on-premises.html): Set up a Ray cluster on bare metal or a private cloud, either manually with ray start or through the cluster launcher. - [Best practices for deploying large clusters](https://docs.ray.io/en/latest/cluster/vms/user-guides/large-cluster-best-practices.html): Best practices for large Ray clusters: networking and system configuration, head node sizing, autoscaler tuning, and node choice. - [Configuring Autoscaling](https://docs.ray.io/en/latest/cluster/vms/user-guides/configuring-autoscaling.html): Configure the Ray autoscaler with the cluster launcher: config parameters, upscaling and downscaling speed, and programmatic scaling. - [Log Persistence](https://docs.ray.io/en/latest/cluster/vms/user-guides/logging.html): Persist Ray logs from VM cluster deployments, covering the log directory layout, processing tools, and collection strategies. - [Community-supported Cluster Managers](https://docs.ray.io/en/latest/cluster/vms/user-guides/community/index.html): Index of community-supported cluster managers for Ray, including Slurm, LSF, YARN, and Spark. - [Deploying on YARN](https://docs.ray.io/en/latest/cluster/vms/user-guides/community/yarn.html): Run Ray on YARN with Skein: configuration, dependency packaging, head and worker node commands, and cleanup. - [Deploying on Slurm](https://docs.ray.io/en/latest/cluster/vms/user-guides/community/slurm.html): Run Ray on Slurm: sbatch directives, obtaining the head node IP, starting Ray across allocated nodes, and networking caveats. - [Deploying on LSF](https://docs.ray.io/en/latest/cluster/vms/user-guides/community/lsf.html): Run Ray clusters on an LSF-managed HPC system using the community-supported deployment steps. - [Deploying on Spark Standalone cluster](https://docs.ray.io/en/latest/cluster/vms/user-guides/community/spark.html): Run Ray on a Spark standalone cluster, including long-running clusters and the Ray on Spark APIs. - [Examples](https://docs.ray.io/en/latest/cluster/vms/examples/index.html): Example Ray workloads to try out on a cloud VM cluster, including a distributed XGBoost training run. - [Ray Train XGBoostTrainer on VMs](https://docs.ray.io/en/latest/cluster/vms/examples/ml-example.html): Run a Ray Train XGBoostTrainer workload on an AWS VM cluster, from cluster deploy through job submission and progress. - [API References](https://docs.ray.io/en/latest/cluster/vms/references/index.html): Reference documentation index for running Ray clusters on virtual machines. - [Cluster Launcher Commands](https://docs.ray.io/en/latest/cluster/vms/references/ray-cluster-cli.html): Ray cluster launcher commands: ray up, exec, submit, attach, rsync-up and rsync-down, and dashboard or status monitoring. - [Cluster YAML Configuration Options](https://docs.ray.io/en/latest/cluster/vms/references/ray-cluster-configuration.html): Reference for every field in the Ray cluster launcher YAML: docker, auth, provider, security groups, and vSphere config. - [Collecting and monitoring metrics](https://docs.ray.io/en/latest/cluster/metrics.html): Collect and monitor Ray cluster metrics in Prometheus format, configure Prometheus scraping, and integrate Grafana for system and application-level metrics. - [Configuring and Managing Ray Dashboard](https://docs.ray.io/en/latest/cluster/configure-manage-dashboard.html): Configure Ray Dashboard port, access it from remote clusters via port forwarding or ingress, and integrate Prometheus and Grafana for enhanced monitoring. - [Applications Guide](https://docs.ray.io/en/latest/cluster/running-applications/index.html): Run Ray applications on a cluster using the Ray Jobs API and CLI, and reference autoscaling configuration for production workloads. - [Ray Jobs Overview](https://docs.ray.io/en/latest/cluster/running-applications/job-submission/index.html): Overview of the Ray Jobs API for running applications on an already-deployed cluster, plus interactive execution. - [Quickstart using the Ray Jobs CLI](https://docs.ray.io/en/latest/cluster/running-applications/job-submission/quickstart.html): Submit and monitor a Ray job with the Jobs CLI, including remote clusters, Kubernetes, and dependency management. - [Python SDK Overview](https://docs.ray.io/en/latest/cluster/running-applications/job-submission/sdk.html): Submit Ray jobs programmatically with the Python SDK, covering long-running jobs, dependencies, resources, and TLS verification. - [Python SDK API Reference](https://docs.ray.io/en/latest/cluster/running-applications/job-submission/jobs-package-ref.html): Python SDK reference for Ray Jobs: JobSubmissionClient, JobStatus, JobInfo, JobDetails, JobType, and DriverInfo. - [Ray Jobs CLI API Reference](https://docs.ray.io/en/latest/cluster/running-applications/job-submission/cli.html): CLI reference for Ray Jobs: the submit, status, logs, stop, and list commands of the job submission API. - [Ray Jobs REST API](https://docs.ray.io/en/latest/cluster/running-applications/job-submission/rest.html): The HTTP endpoints behind the Ray Jobs CLI and SDK, with beta OpenAPI documentation, for submitting jobs directly. - [Ray Client](https://docs.ray.io/en/latest/cluster/running-applications/job-submission/ray-client.html): Connect an interactive Python session to a remote Ray cluster with Ray Client, including when to use it and its limitations. - [Programmatic Cluster Scaling](https://docs.ray.io/en/latest/cluster/running-applications/autoscaling/reference.html): Reference for ray.autoscaler.sdk.request_resources, which programmatically requests cluster capacity. - [FAQ](https://docs.ray.io/en/latest/cluster/faq.html): Answers to common Ray cluster questions covering multi-tenancy limitations, node IP address flags, worker connection failures, and cluster networking. - [Ray Cluster Management API](https://docs.ray.io/en/latest/cluster/package-overview.html): Reference index for the Ray cluster management API: CLI commands, job submission package, job submission CLI, and autoscaling API. - [Cluster Management CLI](https://docs.ray.io/en/latest/cluster/cli.html): CLI reference for managing Ray clusters with the cluster launcher: ray up, down, exec, submit, attach, and status. - [Usage Stats Collection](https://docs.ray.io/en/latest/cluster/usage-stats.html): Ray usage-stats collection policy: what data is gathered, how to opt out, and how the reporting mechanism works. ## Monitoring and Debugging - [Monitoring and Debugging](https://docs.ray.io/en/latest/ray-observability/index.html): Monitor and debug Ray applications and clusters with logs, metrics, events, dashboards, and the distributed debugger. - [Ray Dashboard](https://docs.ray.io/en/latest/ray-observability/getting-started.html): Set up and navigate the Ray Dashboard web UI for monitoring cluster state, tracking task and actor performance, and troubleshooting distributed applications. - [Ray Distributed Debugger](https://docs.ray.io/en/latest/ray-observability/ray-distributed-debugger.html): Debug distributed Ray apps with the Ray Distributed Debugger: set breakpoints in remote tasks, attach VS Code, and do post-mortem debugging. - [Key Concepts](https://docs.ray.io/en/latest/ray-observability/key-concepts.html): Key Ray observability concepts: the Dashboard, the States API for querying actors/tasks/objects, Prometheus metrics, and the distributed debugger. - [User Guides](https://docs.ray.io/en/latest/ray-observability/user-guides/index.html): How-to guides for monitoring and debugging Ray apps: state queries, logging, profiling, custom metrics, tracing, and event export. - [Debugging Applications](https://docs.ray.io/en/latest/ray-observability/user-guides/debug-apps/index.html): Entry point for Ray debugging guides covering failures, hangs, memory issues, the interactive debugger, and performance optimization. - [Common Issues](https://docs.ray.io/en/latest/ray-observability/user-guides/debug-apps/general-debugging.html): Common surprises when moving Python code to Ray: environment variables, filename scoping, placement group composition, and outdated function definitions. - [Debugging Memory Issues](https://docs.ray.io/en/latest/ray-observability/user-guides/debug-apps/debug-memory.html): Debug Ray out-of-memory errors: detect OOM, attribute memory to individual tasks and actors, reduce parallelism, and profile per-worker usage. - [Debugging Hangs](https://docs.ray.io/en/latest/ray-observability/user-guides/debug-apps/debug-hangs.html): Find where a Ray application is stuck by viewing worker stack traces in the Ray Dashboard or with the ray stack CLI command. - [Debugging Failures](https://docs.ray.io/en/latest/ray-observability/user-guides/debug-apps/debug-failures.html): Diagnose Ray application failures: the kinds of failures that occur, print and debugger workflows, file-descriptor exhaustion, and memory-related crashes. - [Optimizing Performance](https://docs.ray.io/en/latest/ray-observability/user-guides/debug-apps/optimize-performance.html): Find and fix Ray performance problems: diagnose missing speedup, visualize execution with Ray Timeline, and profile CPU and GPU work. - [Using the Ray Debugger](https://docs.ray.io/en/latest/ray-observability/user-guides/debug-apps/ray-debugging.html): Set breakpoints in Ray tasks and actors and drop into a PDB session, including cluster use, stepping between tasks, and post-mortem debugging. - [Monitoring with the CLI or SDK](https://docs.ray.io/en/latest/ray-observability/user-guides/cli-sdk.html): Monitor cluster and application state with ray status and the Ray State CLI and SDK, covering node status, resource usage, and task and actor queries. - [Configuring Logging](https://docs.ray.io/en/latest/ray-observability/user-guides/configure-logging.html): Configure Ray's logging system: log directory layout, application and system logs, worker-to-driver redirection, and actor log prefixes. - [Profiling](https://docs.ray.io/en/latest/ray-observability/user-guides/profiling.html): Profile Ray applications for CPU, memory, and GPU bottlenecks using py-spy, cProfile, memray, and the PyTorch profiler from the dashboard. - [JAX profiler for Ray on Kubernetes](https://docs.ray.io/en/latest/ray-observability/user-guides/jax-tpu-profiling.html): Profile JAX workloads on Ray TPU workers in Kubernetes: trigger profiling dynamically on a worker, then view the traces in TensorBoard. - [Adding Application-Level Metrics](https://docs.ray.io/en/latest/ray-observability/user-guides/add-app-metrics.html): Define and export custom application metrics from Ray tasks and actors using the Counter, Gauge, and Histogram APIs. - [Tracing](https://docs.ray.io/en/latest/ray-observability/user-guides/ray-tracing.html): Export Ray traces to OpenTelemetry backends such as Jaeger, covering startup hooks, tracer providers, remote span processors, and custom traces. - [Ray Event Export](https://docs.ray.io/en/latest/ray-observability/user-guides/ray-event-export.html): Export structured Ray events for tasks, actors, jobs, and nodes to an HTTP endpoint, with event format reference and reporting configuration. - [Reference](https://docs.ray.io/en/latest/ray-observability/reference/index.html): API and CLI reference for Ray observability: the States API, the ray CLI state subcommands, and the system metrics catalog. - [State API](https://docs.ray.io/en/latest/ray-observability/reference/api.html): Python SDK reference for querying live Ray state: list, get, and summary APIs for tasks, actors, and objects, plus log access and result schemas. - [State CLI](https://docs.ray.io/en/latest/ray-observability/reference/cli.html): CLI reference for inspecting live Ray state with ray list, ray get, ray summary, and ray logs across tasks, actors, objects, and nodes. - [System Metrics](https://docs.ray.io/en/latest/ray-observability/reference/system-metrics.html): Reference table of the system metrics Ray exports for tasks, actors, resources, and hardware utilization, with semantics and consistency guarantees. ## Developer Guides - [Developer guides](https://docs.ray.io/en/latest/ray-contribute/index.html): Entry point to Ray's developer and contributor guides, linking to API stability guarantees, API policy, the contribution workflow, configuration, and architecture whitepapers. Start here to navigate the documentation for developing and contributing to Ray itself. - [API stability](https://docs.ray.io/en/latest/ray-contribute/stability.html): Defines Ray's API stability guarantees and the PublicAPI (alpha, beta, stable), DeveloperAPI, and Deprecated annotations that label public interfaces. Read this to understand the stability to expect from a Ray API or how to annotate one you're adding. - [API policy](https://docs.ray.io/en/latest/ray-contribute/api-policy.html): Ray's policies for documenting, promoting, demoting, and deprecating APIs across exposure levels, including the required annotations and deprecation timelines. Consult this when adding, changing, or removing a public Ray API. - [Getting involved and contributing](https://docs.ray.io/en/latest/ray-contribute/getting-involved.html): The main guide to contributing to Ray, covering the pull request and code review workflow, coding style, the contributor license agreement, and how to find good first issues. Start here to learn how to submit and land a contribution to the Ray project. - [Building Ray from source](https://docs.ray.io/en/latest/ray-contribute/development.html): Step-by-step instructions for building Ray from source, including Python-only fast-loop development, full C++ editable installs, manylinux wheels, and Docker images. Use this to set up a development environment for making and testing changes to Ray. - [CI testing workflow on PRs](https://docs.ray.io/en/latest/ray-contribute/ci.html): Explains the continuous integration workflow on Ray pull requests, including the microcheck default test set, how to add tests to it, and the full suite that runs at merge time. Read this to understand which tests run on your PR and how to trigger more. - [Editing and managing Python dependencies](https://docs.ray.io/en/latest/ray-contribute/dependency-management.html): How Ray's three-layer Python dependency system works, how to add or update a package, and how to diagnose resolution conflicts. - [Contributing to the Ray documentation](https://docs.ray.io/en/latest/ray-contribute/docs.html): How to contribute to the Ray documentation: building the docs locally, the Google developer style guide enforced by Vale, and the conventions for writing and previewing pages. Use this when editing or adding Ray documentation. - [Publishing an example](https://docs.ray.io/en/latest/ray-contribute/publishing-examples.html): How to publish an executable example to the Ray documentation with the recommended template-collections flow: author a template, register it in the docs build, pin its build, and add it to the examples gallery. - [Publishing an example with the in-tree notebook flow (legacy)](https://docs.ray.io/en/latest/ray-contribute/publishing-examples-legacy.html): The legacy in-tree notebook flow for publishing an example to the Ray documentation: commit a notebook under doc/source, wire its table of contents, and register a release test that reruns it in CI. Prefer the template-collections flow for new examples. - [Ray documentation style guide](https://docs.ray.io/en/latest/ray-contribute/writing-style.html): The Ray documentation style and grammar guide: voice, word choice, sentence structure, headings, lists, links, admonitions, and MyST formatting conventions. Read this before writing or editing Ray documentation, whether you're a human contributor or an AI coding agent. - [How to write code snippets](https://docs.ray.io/en/latest/ray-contribute/writing-code-snippets.html): Guide to writing runnable, CI-tested code examples for Ray docs using doctest-style, code-output-style, and literalinclude formats, and to debugging one that fails. Read this when adding code snippets to docstrings or user guides so they keep working for users. - [Testing autoscaling locally](https://docs.ray.io/en/latest/ray-contribute/fake-autoscaler.html): How to run and test Ray's autoscaler locally without a real cluster, using RAY_FAKE_CLUSTER and the fake multi-node provider. Useful for autoscaler development or for debugging applications that depend on autoscaling behavior. - [Tips for testing Ray programs](https://docs.ray.io/en/latest/ray-contribute/testing-tips.html): Practical tips for testing Ray programs, such as fixing the resource quantity with ray.init(num_cpus=...) to avoid flaky, parallelism-dependent tests. Read this when writing reliable tests for code that uses Ray. - [Debugging for Ray developers](https://docs.ray.io/en/latest/ray-contribute/debugging.html): Debugging guide for Ray contributors, covering how to launch Ray processes under gdb, valgrind, and profilers using RAY_{PROCESS_NAME}_{DEBUGGER} environment variables. Use this to debug crashing or misbehaving Ray core processes. - [Profiling for Ray developers](https://docs.ray.io/en/latest/ray-contribute/profiling.html): Profiling guide for Ray contributors, including how to capture stack traces of C++ processes with gdb and analyze Ray's performance. Read this to diagnose high CPU usage, hangs, or bottlenecks in Ray internals. - [Using agents for development](https://docs.ray.io/en/latest/ray-contribute/agent-development.html): How to configure and use AI coding agents like Claude Code on the Ray codebase, including the repository's shared CLAUDE.md instructions, rules, skills, and personal environment setup. Read this to work effectively with AI coding agents when developing Ray. - [Configuring Ray](https://docs.ray.io/en/latest/ray-core/configure.html): How to configure Ray from the Python API (ray.init) and the command line (ray start), including cluster resource overrides and other runtime settings. - [Architecture whitepapers](https://docs.ray.io/en/latest/ray-contribute/whitepaper.html): Links to the Ray 2.0 and 1.0 architecture whitepapers and the Exoshuffle paper for in-depth coverage of Ray's internals, dataplane, scalability, and performance. Visit for authoritative deep-dives into Ray's architecture and design. ## Glossary - [Ray Glossary](https://docs.ray.io/en/latest/ray-references/glossary.html): Alphabetical definitions of key Ray terminology, from tasks, actors, and objects to reinforcement learning terms. - [FAQ](https://docs.ray.io/en/latest/ray-references/faq.html): Index of Ray frequently asked questions, linking the Ray Tune FAQ and where to raise further questions or issues. ## Security - [Security](https://docs.ray.io/en/latest/ray-security/index.html): Security model and best practices for Ray, covering the Dashboard/Jobs/Client trust model, gRPC communication, and token authentication. - [Ray token authentication](https://docs.ray.io/en/latest/ray-security/token-auth.html): Configure token-based authentication for Ray clusters with RAY_AUTH_MODE=token and ray get-auth-token, including token storage and TLS. ## Project Governance - [Governance](https://docs.ray.io/en/latest/ray-governance/index.html): Ray's technical governance: the roles of contributors, committers, and the Technical Steering Committee, and how project decisions are made. - [People](https://docs.ray.io/en/latest/ray-governance/people.html): Directory of Ray project committers and maintainers, listing the people responsible for governance and ongoing development of the Ray open-source project.