Language: Clojure · Versions: Clojure 1.12.0
The problem
Two classic lazy-sequence traps hit in the same function:
1. Wrapping (map parse-line (line-seq rdr)) inside (with-open [rdr (io/reader path)] ...) returns an unrealized LazySeq after with-open has already closed rdr, throwing IOException: Stream closed when the caller iterates over the result.
2. When I replace map over a chunked sequence ((range 100)) with a side-effecting or expensive function and take only (take 2 ...), Clojure's 32-element chunked sequence batching realizes the first 32 elements instead of 2!
Minimal example
(ns lazy-traps
(:require [clojure.java.io :as io])
(:import [java.io StringReader BufferedReader]))
(defn read-upper [s]
;; Trap 1: lazy-seq escapes with-open!
(with-open [rdr (BufferedReader. (StringReader. s))]
(map #(.toUpperCase ^String %) (line-seq rdr))))
(defn chunk-surprise []
;; Trap 2: (range 100) is a 32-chunked seq!
(take 2 (map (fn [x] (println "evaluating" x) x) (range 100))))
;; (vec (read-upper "hello\nworld")) -> IOException: Stream closed
;; (doall (chunk-surprise)) -> prints "evaluating 0" through "evaluating 31"!
Expected
- Safe resource-scoped streaming without loading the entire file into memory via
mapv/doall. - Strictly demand-driven 1-by-1 evaluation when mapping expensive operations over a sequence.
What happens instead
read-upper closes rdr before the lazy sequence is consumed, and chunk-surprise evaluates 32 elements even though only 2 were requested.
In Clojure 1.12, how do transducers with eduction / transduce (or a self-closing lazy generator) solve both the with-open lifecycle trap and the 32-element chunking over-evaluation?
Short answer: transducers fix trap 2 on their own. They fix trap 1 only if the reduce happens inside the resource's scope. If you return an eduction from inside with-open, it escapes the same way the lazy seq does, because an eduction is a recipe that runs later, not a result.
Trap 1: let the reducible own the reader
(ns lazy-traps
(:require [clojure.java.io :as io]
[clojure.string :as str]))
(defn lines
"A reducible over a file's lines. It opens the reader when reduced
and closes it when the reduce ends: normally, early via `reduced`,
or by an exception."
[path]
(reify clojure.lang.IReduceInit
(reduce [_ f init]
(with-open [^java.io.BufferedReader rdr (io/reader path)]
(loop [acc init]
(if-let [line (.readLine rdr)]
(let [acc (f acc line)]
(if (reduced? acc) @acc (recur acc)))
acc))))))
;; Streams in constant memory: nothing is collected
(transduce (comp (map str/upper-case) (filter seq)) (completing (fn [n _] (inc n))) 0 (lines "big.txt"))
;; Hand the caller a recipe, not an open handle
(defn read-upper [path]
(eduction (map str/upper-case) (lines path)))
(run! println (read-upper "big.txt")) ; opens, streams, closes
(into [] (take 2) (read-upper "big.txt")) ; stops after 2 lines, reader closed
Why this works: with-open now wraps the loop that consumes the lines, not the code that builds them. A transducer's take returns reduced, so the loop exits normally and with-open closes the reader. Each reduce opens a fresh reader, so the eduction can be consumed more than once. Two caveats:
- Reduce it, don't
seqit. Usereduce,transduce,intoorrun!. An eduction'sseqpath goes throughIterable, which thisreifydoesn't implement. That is deliberate: a half-consumed iterator is exactly how a handle leaks. - Always pass an init. I only implemented
IReduceInit, notIReduce, so(reduce f coll)with no init isn't covered.transducesupplies(f)as the init, so it's fine.
If you'd rather use something built in, Clojure 1.12 added stream-reduce!, stream-transduce!, stream-into! and stream-seq! for java.util.stream.Stream:
(with-open [s (java.nio.file.Files/lines (.toPath (io/file path)))]
(stream-transduce! (map str/upper-case) (completing (fn [n _] (inc n))) 0 s))
The scoping rule still applies: everything has to finish inside with-open. Don't return (stream-seq! s) from inside it.
Trap 2: why transducers aren't chunked
(into [] (comp (map (fn [x] (println "evaluating" x) x)) (take 2)) (range 100))
;; evaluating 0
;; evaluating 1
;; => [0 1]
Your premise holds. (range 100) is a LongRange, its seq is chunked in blocks of 32, and lazy map realizes a whole chunk at a time. So doall prints 0 through 31.
The transducer version never calls seq. LongRange implements IReduce, so the reduce feeds one element at a time through map into take. On the second element, take returns reduced and the reduce stops, so f runs exactly twice. eduction behaves the same way. sequence with an xform pulls through an iterator, not chunks, so it's demand-driven too. But it's lazy, so it brings trap 1 back if a resource is involved.
Two side notes:
line-seqisn't chunked. It builds onelazy-seq/consper line. Your two traps come from different sources, and onlyrange(or vectors) chunk.- If you must keep a lazy seq over a chunked source, the usual "unchunk" helper rebuilds it one element at a time with
(lazy-seq (when-let [s (seq xs)] (cons (first s) (unchunk (rest s))))). It works, but reducing is the cleaner way to get strict one-at-a-time evaluation.
What I checked: I'm reasoning from Clojure's source and the 1.12 changelog: LongRange's chunk size and its IReduce, the reduced behaviour of take, and the stream-* functions added in 1.12. I haven't run these snippets. To confirm the cleanup, wrap the reader in a proxy that prints on .close, then try (into [] (take 2) (read-upper path)) and a reducing fn that throws. You should see one close each time.