Language: Go · Versions: Go 1.24 (iter.Seq)
The problem
Using Go 1.23/1.24 range-over-func iterators (iter.Seq[[]byte]), my streaming parser reuses a single buf []byte slice across iterations to avoid allocations, and also spawns a background prefetch goroutine inside the iterator. When the caller exits the for chunk := range Stream() loop early via break or return, two bugs happen:
1. If the caller appended chunk to a slice of slices without copying, every entry aliases the last buffer contents.
2. Worse, if the iterator function recovers or ignores the false return from yield(buf) inside a nested helper, the runtime panics with runtime error: range function continued iterating after exit.
Minimal example
package main
import (
"fmt"
"iter"
)
func Chunks() iter.Seq[[]byte] {
return func(yield func([]byte) bool) {
buf := make([]byte, 4)
for i := range 3 {
buf[0] = byte('A' + i)
// Bug 1:buf is reused; Bug 2: what if a callback calls yield again after false?
if !yield(buf) {
// Cleanup runs here, but what if a deferred flush calls yield(buf)?
defer func() { _ = yield(buf) }()
return
}
}
}
}
func main() {
var saved [][]byte
for c := range Chunks() {
saved = append(saved, c)
break
}
fmt.Println(saved)
}
Expected
Clean termination on break without runtime panic, and safe zero-copy vs. owned buffer semantics.
What happens instead
Calling yield inside defer after yield already returned false crashes the process with runtime error: range function continued iterating after exit, and saved holds a slice aliasing buf.
What are the best practices and API idioms for zero-allocation iter.Seq streaming in Go 1.24 that prevent both buffer aliasing and post-break yield panics?
Both bugs come down to one contract. Once yield returns false, the iterator gets no more calls to yield and returns, and the slice it yielded only belongs to the loop body until that body returns. Here's the shape I'd use.
1. A plain streaming iterator: return when yield says false, flush only on the normal path
func Chunks(r io.Reader, size int) iter.Seq[[]byte] {
return func(yield func([]byte) bool) {
buf := make([]byte, size)
for {
n, err := r.Read(buf)
// buf[:n:n]: if the caller appends, it reallocates
// instead of writing into our spare capacity.
if n > 0 && !yield(buf[:n:n]) {
return // the caller broke out: no yield after this
}
if err != nil {
return
}
}
}
}
A deferred function can close files, cancel goroutines and release buffers, but it can never call yield. If you have a trailing partial chunk to flush, yield it once after the loop on the normal-exit path, as the last call. If the caller broke out early, they've said they don't want it.
2. Ownership: "valid until the body returns", plus an owned adapter
Document the contract the way bufio.Scanner.Bytes does: the slice may be overwritten by the next iteration. Then give callers who keep chunks a one-line opt-in:
func Owned(seq iter.Seq[[]byte]) iter.Seq[[]byte] {
return func(yield func([]byte) bool) {
for b := range seq {
if !yield(bytes.Clone(b)) {
return
}
}
}
}
The hot path stays zero-allocation, and saved = append(saved, c) is only correct under Owned(...) or with an explicit bytes.Clone(c). The [:n:n] cap doesn't stop aliasing. It only stops a caller's append from writing into your buffer.
3. The prefetch goroutine: hand buffers back and forth, and stop the goroutine before returning
With one shared buf, a prefetcher writing into it while the body reads it is a data race, not just aliasing. Use two buffers that change hands through channels, and call yield only on the iterator's own goroutine:
func Prefetch(r io.Reader, size int) iter.Seq[[]byte] {
return func(yield func([]byte) bool) {
free := make(chan []byte, 2)
free <- make([]byte, size)
free <- make([]byte, size)
full := make(chan []byte)
done := make(chan struct{})
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
defer close(full)
for {
var b []byte
select {
case b = <-free:
case <-done:
return
}
n, err := r.Read(b)
if n > 0 {
select {
case full <- b[:n:n]:
case <-done:
return
}
} else {
free <- b
}
if err != nil {
return
}
}
}()
// cleanup only: never yields
defer func() { close(done); wg.Wait() }()
for b := range full {
if !yield(b) {
return
}
free <- b[:cap(b)] // the body is done with it, so recycle it
}
}
}
wg.Wait() means the goroutine can't touch a buffer after the iterator returns. One caveat: it waits for any Read that's in progress. If the source can block forever, make it cancellable (a context-aware reader, or close the underlying conn in the defer). If callers need errors, return iter.Seq2[[]byte, error] instead of dropping them.
About the premise
This is from memory of the Go 1.23/1.24 runtime (runtime/panic.go); I haven't run it.
- Your deferred
yield: it runs while the range function is still returning, after the body already returnedfalse. The checks the compiler builds intoyieldshould panic withrange function continued iteration after function for loop body returned false, not the exact text you quoted. - Calling
yieldafter the wholerangestatement has finished (for example from a stored closure) gives a different message:...after whole loop exit. - Recovering a panic from the loop body and swallowing it gives a third:
range function recovered a loop body panic and did not resume panicking.
Running your snippet on 1.24 would settle which string you actually hit. Also, in your main the panic fires before Println, and with a single break there's only one entry in saved. So the aliasing bug (bug 1) can't show up in that example. You'd see it by ranging without the break: every entry would print the same last contents.