Performance Guide¶

django-ray has two layers with different costs:

1. A durable Django task is enqueued, claimed, retried, and recorded through the database.
2. A Ray-native leaf is submitted inside an active task and does not create its own database row.

Most performance decisions come down to choosing the correct durability boundary.

Choose an Execution Model¶

Queues organize durable tasks; they do not replace Ray scheduling. RuntimeEnv packages code; it does not provide tenant security.

Cluster Ray Core uses Ray Client, so the task-manager connection is part of the workload lifetime. A prolonged disconnect can terminate in-flight Ray work even though the outer Django task remains durable and may later be retried. Prefer Ray Job when execution must continue independently of that submitting connection, and keep external effects idempotent in either mode.

Pick Useful Task Granularity¶

There is no universal item-count threshold. Measure these separately:

• database enqueue, claim, and result-write time;
• Ray submission and scheduling time;
• RuntimeEnv cold and warm setup time;
• useful work per leaf;
• serialization and object-transfer time.

If one leaf's useful work is comparable to its submission overhead, batch several items into one leaf. For example, prefer 20 namespace tasks that each fetch several resource types over hundreds of tiny resource tasks when every resource call takes only a few milliseconds.

Increase granularity until:

• leaves run long enough to amortize submission;
• enough leaves remain to occupy available CPUs or I/O concurrency;
• retries do not repeat an unacceptably large batch.

The best batch size sits between those constraints.

Keep Database Traffic at the Outer Boundary¶

A workflow creates one RayTaskExecution for the outer Django task. Internal leaves exchange Ray object references. With the default WORKFLOW_PROGRESS_REPORTING_POLICY="full", they also report events to an in-memory progress actor and django-ray writes a schema-v2 compatibility snapshot of retained actor state when a changed revision is flushed. Each data event is canonical, identity-fenced, and bounded before its Ray call: payloads are at most 16 KiB, complete wire and decoded envelopes are at most 32 KiB, and dependency edges are sent in batches of at most 32. The actor revalidates the envelope and caps retained nodes, edges, recent events, and bytes. Its snapshots also carry a versioned, fixed-shape cost block with saturating counters for logical event calls/bytes received, decoded calls by the fixed event-kind enum, producer-to-actor delivery delay, handler wall/process CPU time, and snapshot-build wall/process CPU time. The block retains no producer identifiers, payloads, labels, errors, metric names, or other application data. Package-default execution uses the durable V1 profile and remains a schema-v2 writer. A terminal snapshot can also carry a fixed-shape aggregate of actor-accepted leaf-invocation producer reports: valid offers, submissions, superseded and locally dropped values, producer-observed acknowledgement outcomes, and terminal-handoff outcomes. It retains neither producer identities nor application values.

WORKFLOW_PROGRESS_FLUSH_SECONDS limits write frequency, but it does not bound the aggregate Ray actor mailbox, queued bytes, or transient snapshot and drain allocations. During one leaf invocation, full reporting retains at most one outstanding application-progress acknowledgement and one canonical latest-value slot. A slow acknowledgement coalesces later application values into that slot; leaf exit makes at most one bounded terminal handoff before its non-coalesced terminal event. When acknowledgements keep up, every accepted update may still be submitted. This is acknowledgement-driven containment, not time-based sampling.

A physical Ray leaf retry or another forked actor handle within the same run can still contribute another independently bounded leaf session. An outer durable-task retry uses a new run identity and actor. Aggregate workflow-wide admission, call/byte limits, and mailbox coalescing therefore remain separate scale work, as does the bounded actor-to-preparer drain. Those gaps prevent treating the hard V1 ceilings as a safe default production profile or offering a truthful sampled policy.

WORKFLOW_PROGRESS_SCHEMA_V3_PILOT=True selects the intentionally smaller schema-v3-pilot-v1 profile for both actor retention and one terminal schema-v3 publication attempt: at most 512 nodes, 2,048 edges, 2 MiB of topology, 1 MiB of detail, and 4 MiB combined, with encoded and decoded byte ceilings. It does not add a schema-v3 database write to every live flush. A rejected or truncated event, invalid snapshot, admission/preparation failure, stale fence, or storage failure refuses the terminal publication while preserving the application result and schema-v2 compatibility snapshots. This strict fail-closed path is a bounded integration pilot, not evidence for hard-V1-scale or arbitrary concurrent workloads.

The bundled testproject enables the pilot so its nested workflow and the guarded local KubeRay gate exercise real topology publication. Benchmark package-default behavior with the setting explicitly disabled unless the pilot itself is the subject of the measurement. The gate also runs terminal-only success and failure separately to prove null legacy progress, one summary-only publication, and no retained detail. Focused unit tests prove the actor is not constructed and progress transport metadata is not sent; those checks do not turn on pilot detail for the terminal-only invocation.

For workloads that need a durable terminal outcome but not live per-node telemetry, select "terminal_only" globally or call WorkflowSignature.with_progress_reporting("terminal_only").run(...). This path creates no progress actor, emits no node or application-progress RPCs, and writes no legacy progress_data. It makes one best-effort fenced schema-v3 summary write on durable success or failure, with the pinned strategy and plan fingerprint, declared plan counts, terminal outcome, and bounded timestamps. Discovered and node-state counts remain zero; detail is OMITTED_BY_POLICY, and no topology or node-detail rows exist. Measure it as one terminal database write rather than as zero observability work.

For workloads where even that summary is unnecessary, set the policy to "disabled" globally or call WorkflowSignature.with_progress_reporting("disabled").run(...) for one invocation. The disabled path creates no progress actor, emits no node or application-progress RPCs, and writes no progress_data; task lifecycle, retry, cancellation, result/error persistence, and monitor heartbeats remain active. Measure before making this a deployment-wide default because node-level live progress is intentionally unavailable.

Terminal-only avoids the current full-mode producer and mailbox cost; it does not make those costs workflow-wide bounded for workflows that still choose full reporting. Aggregate producer/mailbox admission and coalescing across forked handles, the sampled policy that depends on that aggregate bound, bounded actor-to-preparer draining, the still-unavailable network/database and lifetime-resource cost layers, and large-fan-out slow-consumer evidence remain separate #79 work.

The bundled ObservabilityDemoUser makes the three shipped policies directly comparable without becoming a load benchmark. make loadtest-demo runs one tiny nested workflow at a time under full, terminal-only, and disabled reporting, waits for durable completion, and validates the bounded summary contract before continuing. Stable per-policy Locust labels separate enqueue, terminal polling, and summary-read latency. The one-user, one-user-per-second defaults remain intentionally conservative. Each workflow terminal-poll endpoint is limited to a 65,536-byte response, returns a bounded aggregate summary envelope, and guards current inline result and error diagnostics at 16,384 bytes each without resolving external result storage. Its fixed result omission vocabulary is external_result_not_loaded, stored_result_exceeds_poll_limit, malformed_inline_result, encoded_response_limit; errors use stored_error_exceeds_poll_limit or encoded_response_limit. The one-user tour exercises ordinary small-value responses; the focused adapter tests, rather than Locust timings, prove the limit and omission branches. After the five-minute active window, Locust schedules no new scenario and grants the current one a bounded 150-second drain window. That covers the client-bounded enqueue, the longest 120-second poll, the final detail read, and scheduling margin, so an already-enqueued task still reaches terminal and summary validation instead of disappearing from an otherwise green run.

WorkflowShowcaseUser serves a different purpose: it keeps one realistic order-fulfillment graph moving through the Admin and Ray dashboard at a readable pace. It is excluded from default Locust discovery and must be named explicitly. The class fixes the population at one user, submits one successful three-item, full-reporting workflow with work_seconds=0.05, waits for its durable result, and requires its complete 25-node/36-edge publication before repeating. It never submits the deterministic reservation failure or exposes a reporting-policy control. Treat this as a visualization and integration exercise, not a throughput or reporting-cost benchmark; use the counterbalanced command below for policy measurements.

Attribute Live Workflow Reporting Policies¶

The Locust demo proves that the three policy contracts remain usable, but HTTP timings cannot attribute their internal cost. Use the testproject's opt-in command for a counterbalanced, sequential comparison on the guarded local KubeRay stack. Run it only after the exact committed source tree has passed the required cold-Ray local gate:

kubectl --context docker-desktop -n django-ray exec deployment/django-web -- `
  env DJANGO_RAY_RUN_WORKFLOW_REPORTING_BENCHMARK=1 `
  python /app/testproject/manage.py `
  django_ray_benchmark_workflow_reporting `
  --repetitions 6

The three-repetition default runs nine tiny workflows. Each cycle rotates the order of full, terminal_only, and disabled; higher repetition counts must be a multiple of three so they repeat that complete Latin square without introducing concurrent benchmark tasks. Progress is written to stderr and one versioned JSON report is written to stdout, so redirect stdout to an ignored artifact when retaining evidence. The benchmark report schema is version 3. It records the command implementation digest, package and database versions, the one stable workflow-plan fingerprint, exact workload fingerprint, and whether the testproject's workflow-progress schema-v3 pilot was enabled. These are independent schema versions. Checkout-to-deployment source-tree attestation remains the responsibility of the guarded local KubeRay gate and is labeled that way rather than fabricated by the in-pod command.

Primary comparisons use the durable database timestamps for queue wait, outer execution, and end-to-end time. The workload also returns its bounded coordinator wall time and summed leaf work. Client polling duration is retained only as a diagnostic and is excluded from policy aggregates. Each policy aggregate reports the sample count, median, nearest-rank p95, minimum, and maximum; it does not rank a winner, calculate a speedup, or claim that an elapsed-time difference was caused by reporting. Full-mode actor-cost aggregates retain the fixed initialization, ingest, delivery-delay, and snapshot groups. Every numeric field, including per-kind decoded counts and negative-clock samples, has its own distribution. Full-mode producer_progress aggregates do the same for fixed producer counters and terminal-handoff outcomes. Terminal-only and disabled report producer progress as not_applicable.

The fixed cost fields retain their source units in both samples and aggregates:

All producer_progress values use the explicit unit "count":

The benchmark validates the unsaturated reconciliation equations offered = submitted + superseded + locally_dropped and submitted = acknowledged + actor_rejected + ack_failed + pending_acknowledgements. A full sample also requires acknowledged <= actor-accepted application progress <= submitted - actor_rejected. Failed and still-pending acknowledgements remain intentionally uncertain within that range. A pending acknowledgement is producer-local evidence at report time; actor ordering can still process that application-progress call before it processes the report.

For full mode, the report exposes only allowlisted terminal ingress counters: accepted events by kind, rejections by reason, truncation, retained logical bytes/nodes/edges, the actor-authored cost block, and the fixed producer aggregate. The collector's accepted total includes its constructor's one initialized event, so processed_ingest_events subtracts exactly that event. Each accepted producer_report is also an ingress event, but it contains only fixed-cardinality saturating counters. Actor-observed ingress calls and logical wire bytes include malformed and rejected calls that reached the handler. Calls successfully decoded under the exact run fence are counted by the fixed event-kind enum. Delivery delay is the producer event timestamp to actor-handler entry; it includes serialization, transport, scheduling, queueing, and clock effects, so it is neither a pure mailbox-lag measurement nor a network-timing claim. Negative clock samples are counted and excluded from the non-negative delay total.

Handler and snapshot-build wall/process CPU totals cover only work observed through the retained terminal snapshot. Snapshot-call counts include that returned snapshot; later calls and a disable made after it cannot appear. Every counter saturates at the signed 64-bit protocol ceiling, and the benchmark fails instead of aggregating a saturated run. Before recording the evidence, the command reuses the bounded terminal-publication validator even when the schema-v3 pilot is disabled, then checks the exact run identity, terminal success, plan fingerprint, expanded fixture topology, zero ingress rejection/truncation, fixed cost shape and arithmetic, and retained node/edge agreement.

Durable evidence separates reporting-specific storage from the shared task lifecycle:

• reporting storage includes only UTF-8 byte lengths for progress_data, the execution and archived-attempt copies of the bounded summary, plus normalized run, topology, link, and node-detail row/byte counters for the exact run identity;
• shared storage reports only row counts and encoded lengths for task arguments, result, RuntimeEnv envelope, workflow plan/selection, and the archived attempt row;
• raw arguments, result items, progress, summaries, RuntimeEnv content, topology payloads, node identifiers, and payload/node digests are never emitted. The allowlisted implementation, workload, and workflow-plan fingerprints remain available for evidence identity.

WorkflowProgressTopologyManifest.encoded_bytes is already the complete logical topology total, including its linked pages. Likewise, WorkflowProgressRunStorage.detail_encoded_bytes already aggregates its node-detail rows. The report exposes child totals for integrity checking, but those pairs must not be added together. None of these logical protocol sizes represents PostgreSQL table, index, MVCC, statement, latency, or WAL bytes.

Application-progress submissions are now partially attributable through accepted leaf reports, including producer-observed synchronous/acknowledgement failures and still-pending acknowledgements. A terminal latest-value handoff is an application-progress submission and is included. Structural and lifecycle events, producer reports, and coordinator snapshot/disable calls are excluded; a producer report that never reaches the actor is also absent. Actual Ray/network traffic, aggregate mailbox depth or pure queue latency, complete actor-lifetime RSS/process CPU, disable calls after the last snapshot, and PostgreSQL statement/latency/WAL attribution remain unavailable. Canonical event bytes and malformed actor-received bytes are logical wire bytes, not network traffic; actor-handler timings are likewise not relabeled as any of those missing layers. The report distinguishes directly measured, partial, derived, not-applicable, and unavailable scopes instead of estimating them from workflow wall time.

Successful runs retain their bounded task rows by default and include an Admin path for each execution. This makes the policy-specific summaries available for inspection; the full workflow graph is also available when the testproject's schema-v3 pilot is enabled. Pass --cleanup --output-json <new-path> when evidence should be ephemeral. The command first writes the complete report, then deletes only the exact execution primary keys it created and their cascading attempt/progress rows, and finally replaces the artifact atomically with the completed cleanup receipt. Cleanup is intentionally unavailable through the read-only task Admin. A timeout, validation failure, or initial artifact-write failure deliberately preserves the task rows for diagnosis; a cleanup failure leaves the pre-cleanup report at the requested path.

This command is local evidence, not a production SLO or an ordinary public-CI job. It does not enable Compiled Graph, change the compatible full-reporting default, add sampled reporting, or replace the isolated PostgreSQL publication/read/preparation benchmarks below.

This is faster than making every leaf a Django task, but it changes semantics:

• a leaf does not have an independent durable retry record;
• failure fails the outer task;
• retrying the outer task can repeat successful leaves;
• progress is observational, not a recovery log.

Use idempotent leaves and external checkpoints when repeated side effects would be unsafe.

Benchmark Workflow Progress Persistence¶

Use the opt-in workflow-progress benchmark to compare the current complete-row snapshot with bounded alternatives before changing the production persistence contract. The command builds deterministic logical node/payload records and exercises Django plus a disposable PostgreSQL database. It does not import Ray, start a Ray runtime, execute native Compiled Graphs, or add any candidate representation to production.

Install the PostgreSQL dependency set, activate that environment, and point tests.postgres_settings at an isolated PostgreSQL database. Never use a production database: the benchmark creates and removes transient representative rows and measures database-wide effects. Prepare the schema, then run the fixed comparison locally:

export DJANGO_SETTINGS_MODULE=tests.postgres_settings
export DATABASE_NAME=django_ray
export DATABASE_USER=django_ray
export DATABASE_PASSWORD=django_ray
export DATABASE_HOST=127.0.0.1
export DATABASE_PORT=5432

python testproject/manage.py check
python testproject/manage.py migrate --noinput
mkdir -p artifacts
python testproject/manage.py django_ray_benchmark_workflow_progress \
  --nodes 1000 10000 50000 100000 \
  --change-rates 0.01 0.10 1.0 \
  --repetitions 5 \
  --warmups 1 \
  --seed 20260720 \
  --database-deployment local-postgresql \
  --output-json artifacts/workflow-progress-benchmark.json \
  --output-markdown artifacts/workflow-progress-benchmark.md

The nodes values are logical workflow-node counts. A change rate of 0.01 or 0.10 models a sparse publication after 1% or 10% of nodes change; 1.0 models a terminal publication in which every node has changed. Each case retains its first measurement as cold_sample, then discards the requested warm-up measurements before recording its five warm_samples. A fixed seed makes payload content and modeled change distribution reproducible; this command does not construct dependency edges.

The benchmark-only representations have these meanings:

These are comparison models, not supported storage modes. In particular, the external_chunk case does not turn a benchmark artifact into a public storage reference, and the live_only case does not promise that detail survives producer loss.

The versioned JSON has these top-level fields: schema_version, benchmark, environment, configuration, profiles, candidates, cases, and database_evidence. Together they retain the configured matrix and seed, core dependency and platform identity, PostgreSQL identity and schema state, payload profiles, and candidate descriptions. Each cases entry identifies profile, candidate, nodes, change_rate, changed_nodes, and a deterministic workload_fingerprint. Its write_amplification records changed items, typed modeled storage units, estimated database statements, and task, detail, external, and total encoded bytes. External operations are not counted as database statements. cold_sample and warm_samples retain snapshot and serialization measurements, encoded and decoded bytes, parse/redaction work, summary/page/single-node read timings and response bytes, and best-effort process memory evidence.

The same workload_fingerprint is intentionally shared by all seven candidates for a given profile, node count, and change rate. It identifies a comparable workload, not a unique case.

Payload profiles cover short and maximum-size metadata, multibyte UTF-8, secret-bearing values, metric-cardinality limits, compressible content, and seeded incompressible content. database_evidence holds the best-effort PostgreSQL temporary table measurements, including pg_column_size, relation growth, and WAL evidence when the server can provide them. Optional database evidence may be unavailable with an explicit reason without failing the benchmark; an unavailable metric remains distinct from a measured zero.

Treat the JSON file as the source of truth for analysis. It retains environment data, the cold and individual warm samples, full-precision values, and unavailable-metric information needed to recompute comparisons. The Markdown file is a concise, rounded view derived from that JSON. Its first table pools heterogeneous cases into candidate-wide medians across the complete matrix; the sparse-focus table is one specific comparison. Use it to orient a decision and the GitHub Actions job summary, not to replace the raw repetitions or compare runs with different environments.

The environment also records the SHA-256 digest of the benchmark implementation. Manual GitHub Actions runs record their exact GITHUB_SHA; local evidence reports the source revision as unavailable instead of guessing. The configured matrix records that the command touches only Django and its configured database, never Ray or Kubernetes.

Python timings cover construction, serialization, parsing, redaction, and in-process response shaping of bounded representative structures. They exclude ORM queries, database and network round trips, and external-object operations. Write bytes and database statement counts are analytical. The PostgreSQL probe inserts one bounded representative document per candidate to capture JSONB row size, relation growth, and WAL evidence; it does not execute seven production candidate schemas or establish throughput. Real schema/query benchmarks remain required with the implementation.

The ADR-0004 decision uses a committed Windows 11/PostgreSQL 17 summary and its authoritative raw JSON. That local run used a disposable Docker database; the command did not start Ray or access Kubernetes. It therefore does not validate the image or package version in any Kubernetes deployment.

For the modeled uniformly distributed sparse update, immutable 256-item detail pages touched most pages even when only 1% of nodes changed. ADR-0004 therefore retains immutable pages only for normally static topology and selects normalized latest-state rows for mutable node detail. The benchmark compares storage shapes; it does not add that schema or change production persistence.

To collect the reference Ubuntu/PostgreSQL 17 series, open Actions, select Workflow Progress Benchmark, choose the branch or commit under test, and select Run workflow. The manual job runs the same fixed command and uploads both files in the workflow-progress-benchmark-<run>-<attempt> artifact, including any files that exist when the benchmark step fails. It is intentionally absent from pull-request CI.

Do not turn its elapsed-time values into required pull-request thresholds. Shared runner load, CPU scheduling, filesystem cache state, PostgreSQL checkpoints, autovacuum, and fsync behavior add noise that correctness tests cannot normalize. Compare candidates within one run, repeat material decisions on representative infrastructure, and retain the raw artifacts with the ADR evidence.

PostgreSQL byte metrics answer different questions. pg_column_size describes a value or row representation, not the table's complete disk cost. Relation growth can include indexes, TOAST data, free space, and dead tuples, and it need not fall when rows are deleted. WAL deltas vary with server settings, checkpoints, full-page writes, compression, and concurrent activity. Measure on an otherwise idle disposable database, record unavailable WAL evidence explicitly, and compare WAL or relation deltas only between cases collected under the same server configuration.

Benchmark Workflow Progress Preparation¶

ADR-0005's opt-in preparation benchmark has two explicit implementations. The default prototype-composite mode measures the issue-#140 non-production topology/detail prototype. production-topology measures the issue-#141 package path through legacy topology detachment. Each sparse or high-edge scenario runs in a fresh subprocess and removes its private workspace before the parent accepts the result. Neither mode starts Ray, writes Django database rows, accesses Kubernetes, or activates schema-v3 production.

Run the required decision matrix on Linux from an otherwise idle checkout:

mkdir -p artifacts /tmp/django-ray-issue140-benchmark
python scripts/benchmark_workflow_progress_preparation.py \
  --implementation prototype-composite \
  --required-scale \
  --timeout-seconds 7200 \
  --workspace-parent /tmp/django-ray-issue140-benchmark \
  --output artifacts/workflow-progress-preparation.json

Repeat the topology-only production matrix separately:

python scripts/benchmark_workflow_progress_preparation.py \
  --implementation production-topology \
  --required-scale \
  --timeout-seconds 7200 \
  --workspace-parent /tmp/django-ray-issue141-benchmark \
  --output artifacts/workflow-progress-production-topology.json

--required-scale expands to 25,000, 100,000, and 250,000 observed nodes for both a sparse chain and a deterministic high-edge graph. The latter crosses the retained 100,000-edge cap without constructing a quadratic graph. Smaller explicit --nodes values are smoke checks only and cannot satisfy the ADR evidence gate. The explicit two-hour per-case timeout accommodates the deliberately expensive exact high-edge path; it is a watchdog, not a performance target.

Report schema v2 is additive. It retains tracemalloc_peak_bytes and peak_rss_bytes for schema-v1 consumers, adds explicit end_to_end_* aliases, and adds the production-only bounded_phase_* checkpoint fields. Consumers that already accept additive JSON can continue reading the legacy peak fields and should ignore unknown fields. New consumers must check schema_version; the runner rejects a missing or inconsistent required v2 field rather than interpreting a schema-v1 or future-version report as current evidence.

The JSON records the selected implementation, exact source and implementation identity, environment and filesystem characteristics, effective SQLite settings, configured cache/batch/item/spill limits, observed and retained cardinalities, wall and CPU time, Python allocation peaks, explicitly scoped RSS evidence, spill high-water bytes, query plans, and child plus parent-watchdog cleanup outcomes. A separate forced-termination control proves that the parent removes a killed child's workspace. An unavailable RSS metric is null, never zero. Cleanup failure, missing phase evidence, mismatched legacy aliases, and non-monotonic phase/end-to-end peaks fail the run. Unknown extra fields remain valid so the schema can grow additively.

When required_scale is true, validation requires each combination of 25,000, 100,000, and 250,000 observed nodes with sparse and high-edge exactly once. A missing, duplicate, or extra matrix case is not decision evidence even when the individual cases completed.

The committed WSL2 Linux summary and authoritative JSON join the focused canonical and lifecycle tests to satisfy ADR-0005's issue-#140 evidence gate. They are local contract evidence, not a production latency SLO. Compare resident growth only after the retained node, edge, and detail caps are reached, and keep #79's wire, mailbox, and producer allocations separate from this preparation-only measurement.

The production adapter's required WSL2 Linux summary and JSON record all six production-topology cases at the exact issue-#141 implementation revision. Bounded-phase memory plateaued after V1 retention caps while spill grew with observed node and edge identity state. The separate end-to-end fields and legacy-ID counts preserve #142's remaining O(observed) compatibility boundary.

In production-topology reports, detail fields are null, spill-item counts cover only nodes and edges, and legacy_observed_node_ids records the compatibility set materialized after the bounded candidate is sealed. The bounded_phase_* fields are captured after observation, selection, and page construction but before that set is materialized. The legacy peak fields and their end_to_end_* aliases include the later O(observed) compatibility detachment. Compare those two phases separately. Issue #141 removes complete Python node/edge validation collections, but it does not claim that the legacy returned value is O(retained); issue #142 owns that final bound.

Control Fan-Out¶

Do not equate maximum submission count with maximum throughput.

from django_ray.runtime.distributed import parallel_map


def process_one(item_id: int) -> int:
    return item_id * item_id


item_ids = list(range(100))
results = parallel_map(
    process_one,
    item_ids,
    num_cpus=0.25,
    max_concurrency=32,
)

For workflows, set per-leaf ray_options:

from django_ray.workflows import map_step


def process_one(item_id: int) -> int:
    return item_id * item_id


fanout = map_step(
    process_one,
    ray_options={"num_cpus": 0.25},
).with_limits(
    max_concurrency=32,
    max_items=10_000,
)

The map window limits submitted map items and retained result references. Results are collected incrementally but returned in input order. Nested groups can submit multiple physical tasks per item, so include their fixed branch count when sizing the window. Input admission starts only after executor.resolve(): an upstream Ray task still produces its complete list, which is transferred and deserialized into coordinator memory before fan-out begins. Only an iterator already available locally is pulled lazily. Remote or paged input materialization is tracked in GitHub issue #94.

By default, the ordered result payload is still materialized in coordinator memory. An explicitly bounded map can use with_result_buffer() to retain versioned serialized bytes in a resource-accounted Ray actor and forward one direct payload reference to a downstream Ray step. The actor, object store, and final consumer still have O(total output) boundaries. When only a compact summary is needed, use the explicit reduce() ordered fold. Its actor retains one serialized accumulator and at most max_concurrency - 1 out-of-order results, while incorporation-based admission deliberately applies head-of-line backpressure to preserve strict input order. Benchmark the reducer cost and accumulator size as well as map throughput. List-preserving chunk or hierarchical transport remains tracked in GitHub issue #91. Calling map_step() without with_limits() retains the legacy eager behavior.

Use fractional CPUs only when a leaf is mostly waiting or deliberately shares a core. CPU-bound Python or native work should request realistic resources. An oversized concurrency value can increase memory pressure, API throttling, and retries without reducing wall-clock time.

For an API-limited workload, map batches rather than individual tiny requests and keep the client's rate limiter enabled: a concurrency window bounds simultaneous batches, not requests per second. For Kubernetes synchronization, compare namespace-sized batches with (namespace, resource_kind) batches. Choose the smallest batch that still makes Ray scheduling overhead minor, and set max_items to reject accidental discovery explosions. See Bound Dynamic Fan-Out for failure and cleanup semantics.

Avoid passing large repeated values to every leaf. Put shared immutable data in Ray's object store once, or load it in a preceding workflow step and pass references through the graph.

Measure distributed helper preparation¶

parallel_map(), parallel_starmap() and scatter_gather() prepare the next request as a submission slot opens. Strict operations share validated invariant wire fragments; map/starmap reuse one callable digest. This reduces sender-side work before the first submission and avoids a request-sized object for every unscheduled item. It does not stream caller input lists, result lists or Ray argument transport, and leaves still perform full strict-boundary validation.

Measure that boundary from a clean committed checkout on Linux:

uv run python scripts/benchmark_distributed_preparation.py \
  --baseline-ref origin/main --items 100 1000 --window 8 --repetitions 3 \
  --case-timeout 120 --output artifacts/distributed-preparation.json

Both complete src trees are extracted from Git into owned temporary directories. Each baseline/candidate case uses a fresh Python process importing the selected tree, with a 3 GiB address-space cap, CPU-time cap and parent-enforced wall timeout. BLAS thread settings are one. No Ray service, database or application callable runs. Cases are limited to 5,000 items; the default profile compares 100 and 1,000 items. The parent removes its temporary source directories after the run.

The report records revisions, tree identity, environment, first-submission delay, CPU/wall time, traced Python peak bytes, encoding/hash counts and maximum prepared calls ahead of consumption. It verifies identical request bytes and ordered results. Timings include tracemalloc overhead; memory includes helper-created input-reference copies and results, but excludes caller input allocation and Ray. A historical scatter baseline may have no window: that difference is explicit in effective_window. These measurements do not establish end-to-end throughput or remote memory consumption.

The existing manual Workflow Progress Benchmark workflow also accepts target=distributed-preparation and a baseline_ref. That target runs only the sender benchmark on a hosted Linux runner, with a ten-minute job limit and a JSON artifact. The default remains the PostgreSQL workflow-progress benchmark. The new target does not add benchmark work to ordinary PR CI. Real-Ray success, failure cleanup and the applicable cold deployment gate remain separate evidence.

Runtime Environment Cost¶

RuntimeEnv caches are per Ray node. A four-node cold fan-out may prepare the same environment on four nodes before useful work begins. Repeated tasks can then reuse each node's cache.

For predictable latency:

• use a bounded set of named profiles;
• use immutable code archive URIs;
• pin production package versions;
• warm each node before directing latency-sensitive work to it;
• bake system libraries and large common Python dependencies into the Ray image;
• reserve RuntimeEnv for application code and genuinely variable dependencies.

Always compare cold and warm runs. A single first-run measurement mostly benchmarks packaging and installation.

Ray Core, Ray Job, and Compiled Graphs¶

Ray Core is django-ray's low-latency path. Ray Job adds an isolated driver and is a better fit for coarse work.

django-ray workflow signatures currently build ordinary Ray task graphs. They do not use Ray Compiled Graph. Compiled Graph is aimed at repeated execution of a fixed graph from a long-lived Ray driver; django-ray's dynamic maps, durable outer-task lifecycle, and per-run graph metadata have different requirements. It may become useful for a future reusable fixed-DAG execution mode, but it is not a switch that removes RuntimeEnv, database, or first-run costs.

The workflow-plan contract defines how static, dynamic, and fixed-width workloads are classified and which plan, invocation, ownership, and eligibility data a future strategy must use.

The first experimental ownership boundary is deliberately limited to compiling once and invoking repeatedly inside one Ray Core durable task. For a scheduled Kubernetes sync, one generic fixed-width kernel may process several namespaces during that one schedule. The graph is still rebuilt for the next schedule, so this is within-run reuse and has a zero cross-schedule graph-cache hit rate. The current probe covers the direct-ray-core submission transport; the production Ray Client-submitted nested owner has a distinct compatibility identity and remains gated on separate live-cluster lifetime evidence. See ADR-0002 for the process matrix, resource budget, cancellation, and drain requirements. Compatibility evidence must also pin the specific container and immutable deployment/image identity and record explicit shared-memory and Ray object-store configuration. Measurements from a generic host, docker, or unresolved environment cannot be promoted as a native-capability row.

Benchmark the Real Shape¶

The testproject provides:

POST /api/cluster/workflow-benchmark?num_items=20&seconds_per_item=0.25
GET  /api/cluster/workflow-benchmark/{task_id}

POST /api/cluster/runtime-env/benchmark?profile=numpy-2-3&package=numpy&repeats=3
GET  /api/cluster/runtime-env/{task_id}

Run at least:

1. cold environment, cold workers;
2. warm environment with the same profile;
3. one large batch;
4. several leaf batch sizes;
5. Ray Core versus Ray Job when isolation is optional.

Record queue wait, environment setup, leaf runtime, and total duration separately. Optimizing only the function body can miss the dominant cost.

Benchmark Worker Polling¶

Run the polling benchmark against a disposable PostgreSQL database with no production workers consuming the generated benchmark queue:

python manage.py django_ray_benchmark_polling \
  --workers=4 \
  --tasks=100 \
  --idle-seconds=2 \
  --enqueue-interval-seconds=0.05 \
  --base-interval-seconds=0.1 \
  --max-interval-seconds=0.5 \
  --seed=53 \
  --json

The command runs fixed-100-ms and adaptive policies sequentially through the production worker claim loop. Both execute its real SELECT ... FOR UPDATE SKIP LOCKED query on isolated queues. Independent phases report idle claim and total SQL queries per worker-second, peak distinct-worker overlap and a sliding-window overlap ratio, spaced enqueue-to-claim p50/p95, and preloaded-burst claim throughput. The enqueue timestamp is captured before the task row is inserted, so latency includes enqueue database time. Generated rows are validated for exact, unique ownership before deletion. The command refuses non-PostgreSQL databases because SQLite cannot represent the multi-worker locking behavior being measured.

The additive protocol_predicate_evidence block measures the execution-protocol filter separately from those polling-policy phases. It creates at most 256 exact owned rows at the package's active write protocol (currently protocol 1) on a private queue, intercepts the real production priority claim SELECT before execution, and verifies that the measured production variant has the same normalized SQL shape. A control removes only the inclusive protocol-range predicates. Both variants must return the same rows in the same order. One warm-up pair is followed by 12 deterministic AB/BA timing pairs; the report retains each bounded sample, p50/p95, and signed production-minus-control deltas, plus fixed-vocabulary bounded EXPLAIN ANALYZE plan summaries. Raw SQL, parameters, queue/task/row identities, and arbitrary index names are not retained.

Do not turn one shared-runner delta into a latency threshold. Compare repeated artifacts from the same PostgreSQL version and task shape, confirm production_claim_sql_shape_verified=true, inspect both plan shapes, and interpret a positive delta only alongside run-to-run variance. A persistently different or regressed plan is evidence for a focused index investigation; this benchmark does not by itself authorize a model or migration change.

For a scaling series, keep the database and task shape fixed and run --workers=1, 4, and 8. Repeat each case at least five times after a warm-up run. Record PostgreSQL version, host resources, connection-pool settings, database distance, worker count, task count, enqueue interval, both polling intervals, schema version, and seed with every result. Use the same seed when comparing policies; repeat runs still capture scheduler and database variance.

The manually dispatched Polling Benchmark workflow runs a warm-up and five recorded repetitions for each worker count. Launch it from GitHub Actions when polling behavior or its operating environment changes, then download the polling-benchmark-json artifact for exact environment metadata and individual measurements. Normal pull-request CI keeps the PostgreSQL coordination and polling correctness tests but does not run this time-based matrix. Performance varies with shared runner capacity, so the benchmark checks claim integrity, exact protocol-variant row parity, verified production SQL shape, bounded plan summaries, and finite metrics rather than imposing noisy latency or throughput thresholds. Do not substitute SQLite or simulated timings.

The following values are medians from five repetitions after warm-up in GitHub Actions run 29703449242 on 2026-07-19. The environment was PostgreSQL server_version_num=170010, Python 3.12.13, Django 6.0, and schema 0008_raytaskexecution_priority_constraint on a shared Azure Linux runner. Each policy used 100 tasks per phase, a 50 ms enqueue interval, a 2-second idle window, and a 25 ms overlap window.

The single-worker latency phase deliberately offers 20 tasks per second to a worker that claims about 10 per second, so its growing queue produces multi-second latency. Treat that row as saturation behavior, not idle wake-up latency. In this run, adaptive polling cut idle claim queries by roughly two-thirds without reducing burst throughput. Shared-runner values are evidence for comparison and tuning, not service-level objectives.