Why does Django 5 raise “ValueError: chunk_size must be provided when using QuerySet.iterator() after prefetch_related()”?

This error tends to show up right after a Django 5 upgrade. A batch job that has run quietly for years, usually a report, an export, or a data backfill, suddenly fails on its first line of iteration. The code did not change. The Django version did.

ValueError: chunk_size must be provided when using QuerySet.iterator() after prefetch_related().

The message is accurate, but it does not explain why Django cares. In this post I’ll walk through what changed, why Django made this a hard error, and the practical ways to fix it when you’re streaming a large queryset with select_related and Prefetch objects.

What does this ValueError actually mean?

It means you called .iterator() on a queryset that has prefetch_related() lookups attached, and you didn’t pass chunk_size. Django 5.0 and later refuse to guess a value for you.

Here is the smallest code that triggers it:

from myapp.models import Order

qs = Order.objects.select_related("customer").prefetch_related("tags")

for order in qs.iterator():  # ValueError on Django 5.0+
    print(order.customer.name, [t.name for t in order.tags.all()])

Two details are worth noticing:

It has nothing to do with queryset size. Django raises the error as soon as you call iterator(), before a single row is fetched. It fails the same way on 10 rows and on 10 million.
It only involves prefetch_related. select_related is a SQL JOIN that runs inside the main query, so it never triggers this check.

Pro Tip: If you see this error in a stack trace you don’t recognize, check whether a third-party library or a management command is calling iterator() on a queryset you configured. Django’s own dumpdata command hit exactly this after 5.0 and had to be patched to pass a chunk_size.

Why did Django change the behavior?

To understand the change, you need the history of iterator() and prefetching.

QuerySet.iterator() exists to stream results instead of loading everything into memory. It skips the queryset’s result cache and, on supported databases, reads rows in batches. For a long time, it was also incompatible with prefetch_related(). Django simply ignored your prefetch lookups when you used iterator(), which meant each row triggered its own lazy query when you touched a related manager. That is the classic N+1 problem, and it was silent.

The timeline, as documented in the Django release notes and commit history:

Django version	Behavior of `iterator()` with `prefetch_related()`
Before 4.1	Prefetch lookups are silently ignored
4.1	Prefetching is supported if you pass `chunk_size`; omitting it only emits a deprecation warning
5.0 and later	Omitting `chunk_size` raises `ValueError`

The reason for making it strict is about cost. When you combine iterator() with prefetching, Django has to run the prefetch queries once per chunk. Small chunks mean many extra queries. Someone chasing lower memory usage could shrink the batch size without realizing they’ve multiplied the query count. Forcing you to choose chunk_size explicitly makes that trade-off visible instead of hidden behind a default.

How can I reproduce it locally?

You don’t need 200,000 rows. A few thousand is enough to see the full behavior, including the fix.

# models.py
from django.db import models

class Customer(models.Model):
    name = models.CharField(max_length=100)

class Tag(models.Model):
    name = models.CharField(max_length=50)

class Order(models.Model):
    customer = models.ForeignKey(Customer, on_delete=models.CASCADE)
    tags = models.ManyToManyField(Tag, related_name="orders")

Seed the data with bulk_create, including the many-to-many through table, rather than calling .add() in a loop:

# seed.py
from myapp.models import Customer, Order, Tag

customers = Customer.objects.bulk_create(
    [Customer(name=f"customer_{i}") for i in range(100)]
)
tags = Tag.objects.bulk_create([Tag(name=f"tag_{i}") for i in range(10)])
orders = Order.objects.bulk_create(
    [Order(customer=customers[i % 100]) for i in range(10_000)]
)

Through = Order.tags.through
Through.objects.bulk_create(
    [Through(order_id=o.pk, tag_id=t.pk) for o in orders for t in tags[:3]],
    batch_size=5000,
)

Now run the failing loop from the previous section. On Django 5.0 or later you’ll get the ValueError immediately.

Pro Tip: bulk_create only sets primary keys on the returned objects for databases that support RETURNING (PostgreSQL, MariaDB 10.5+, SQLite 3.35+). On other backends, re-query the objects before building the through rows.

How do I fix it?

1. Pass `chunk_size` explicitly

This is the correct fix for most cases. Choose a batch size and keep streaming:

from django.db.models import Prefetch
from myapp.models import Order, Tag

qs = (
    Order.objects
    .select_related("customer")
    .prefetch_related(Prefetch("tags", queryset=Tag.objects.order_by("name")))
    .order_by("id")
)

for order in qs.iterator(chunk_size=2000):
    tag_names = [t.name for t in order.tags.all()]
    # process order.customer.name and tag_names

What happens under the hood:

One main query, with the JOIN for customer, streams rows in batches.
For every chunk of 2,000 orders, Django runs one extra query to fetch their tags.
For 10,000 orders, that is roughly six queries in total. For 250,000 orders, it is about 125 prefetch queries instead of 250,000 lazy ones.

Pro Tip: Always add an explicit order_by() when you stream a large queryset. Without it, row order is not guaranteed, which makes debugging and resuming a failed job much harder.

2. Tune the chunk size for your workload

There is no universal best value. The trade-off is simple:

Larger chunks mean fewer prefetch queries but more objects held in memory at once.
Smaller chunks mean lower memory but more round trips to the database.

Start at 2,000, which is the value Django uses for iterator() by default when no prefetching is involved. Then measure peak memory and query count for your real data. If each row carries large text columns, go smaller. If rows are narrow and the database is far away over the network, go larger.

3. Use keyset pagination for resumable jobs

For long-running exports, I often prefer to avoid iterator() altogether. Keyset pagination gives you normal queryset behavior, works with prefetching without special rules, and lets you resume from the last processed ID if the job dies:

last_id = 0
batch_size = 2000

while True:
    batch = list(
        Order.objects
        .select_related("customer")
        .prefetch_related("tags")
        .filter(id__gt=last_id)
        .order_by("id")[:batch_size]
    )
    if not batch:
        break

    for order in batch:
        ...  # process

    last_id = batch[-1].id

Each loop iteration runs one main query and one prefetch query for that batch. There’s no server-side cursor involved, which matters in the next section.

4. Reconsider whether you need the objects at all

If the loop only counts, sums, or groups, the database can usually do that work faster than Python. For example, counting orders per tag does not need 250,000 model instances:

from django.db.models import Count
from myapp.models import Tag

Tag.objects.annotate(order_count=Count("orders"))

Likewise, values() and values_list() skip model instantiation and cut memory sharply when you only need a few columns.

What about server-side cursors and connection poolers?

This is the part of iterator() that surprises people in production. According to the Django documentation, on PostgreSQL and Oracle iterator() uses server-side cursors, so the database holds the result set and Django pulls it in chunks. On MySQL, the driver loads the entire result set into memory regardless of chunk_size, so iterator() saves you less than you’d expect.

If you run PostgreSQL behind PgBouncer in transaction pooling mode, server-side cursors can break, because the cursor lives on one connection while the pooler may hand your next statement to another. Django provides a setting for this:

# settings.py
DATABASES = {
    "default": {
        # ...
        "DISABLE_SERVER_SIDE_CURSORS": True,
    }
}

With it enabled, iterator() still works, but Django no longer streams from a server-side cursor, so the memory benefit shrinks. In that setup, the keyset pagination pattern above is usually the better choice.

Pro Tip: If memory is the real problem, measure it. Run the job with tracemalloc or watch the process RSS before and after your change. I’ve seen teams “optimize” with iterator() and see no improvement because of the MySQL behavior or a pooler setting.

Should I just call `list(qs)` instead?

You can, and it will work: Django evaluates the queryset, runs the prefetch queries once, and hands you a list. The cost is memory. Every one of those model instances, plus every prefetched related object, lives in RAM at the same time.

For a few thousand rows, that’s fine. For a couple of hundred thousand, it can add up quickly depending on how wide your models are. Use list(qs) when you know the result fits comfortably in memory, and use chunking when you don’t.

FAQ

Which Django versions raise this error?

Django 5.0 and later raise ValueError. In 4.1 and 4.2, the same code emits a deprecation warning. Before 4.1, iterator() ignored prefetch_related() entirely, so the code ran but with N+1 queries.

Does `select_related` trigger the error?

No. select_related adds a JOIN to the main query, so it works with iterator() without any chunk_size. The error only appears when the queryset has prefetch_related() lookups, including custom Prefetch objects.

What `chunk_size` should I use?

Start with 2,000 and measure. Raise it if you see too many prefetch queries, and lower it if memory spikes. The right number depends on row width, network latency, and how many prefetch lookups you have.

Is it safe to drop the prefetch and rely on lazy loading?

It will run, but each row will trigger its own query for the related objects. For a large queryset, that can mean hundreds of thousands of queries. Prefer a chunked prefetch or an aggregate query.

Why does my error say “dictionary changed size during iteration” instead?

That is a different, Python-level error. It means a dictionary was modified while something was looping over it, usually in your own code or a library. It isn’t what Django raises for iterator() and prefetch_related(), so look at the full stack trace to see which code is mutating the dictionary.

Conclusion

The ValueError is not a bug. Django 5 is making a hidden performance trade-off explicit: streaming rows with iterator() and prefetching related objects don’t combine for free, because the prefetch runs once per chunk.

My takeaways:

Pass chunk_size whenever you combine iterator() with prefetch_related(), starting at 2,000 and tuning from there.
Use keyset pagination for long jobs that must be resumable or that run behind a transaction-pooling proxy.
Push work into the database with annotate(), values(), or values_list() when you don’t need full model instances.
Measure memory and query count before and after, so you know the change helped.

Small, explicit choices like these are what keep large Django jobs predictable in production.

Want more practical backend fixes like this one? Keep following SpiritCode.