Language: OCaml · Versions: OCaml 5.2.1 (Multicore Effects)
The problem
I'm implementing a generator/iterator in OCaml 5.2 using deep algebraic effect handlers (Effect.Deep.match_with). Inside the generator body, I open a file or acquire a resource using Fun.protect ~finally:(fun () -> close ()). When the consumer reads only the first yielded value and stops—dropping the captured continuation k : (unit, unit) Effect.Deep.continuation without resuming it—the ~finally block inside the suspended fiber is never executed, leaking the file descriptor AND leaking the underlying C fiber stack!
Minimal example
open Effect
open Effect.Deep
type _ Effect.t += Yield : int -> unit Effect.t
let producer () =
Fun.protect
~finally:(fun () -> print_endline "CLEANUP: resource released")
(fun () ->
perform (Yield 1);
perform (Yield 2))
let () =
match_with producer ()
{ retc = (fun () -> ());
exnc = raise;
effc = fun (type a) (eff : a Effect.t) ->
match eff with
| Yield v -> Some (fun (k : (a, _) continuation) ->
Printf.printf "Got %d, stopping early without resuming k!\n" v
(* Bug: dropping `k` here skips Fun.protect ~finally and leaks the fiber! *))
| _ -> None
}
Expected
Got 1, stopping early without resuming k! followed by CLEANUP: resource released and deallocation of the fiber stack.
What happens instead
Only Got 1, stopping early without resuming k! is printed! Because OCaml 5.2 continuations are one-shot and own an allocated fiber stack, every k must be linear: either resumed via continue k () or explicitly unwound via discontinue k Exit! Dropping k on the floor skips all Fun.protect / try ... with handlers on the fiber stack and leaks memory.
What is the idiomatic OCaml 5.2 design pattern to ensure a suspended continuation k wrapped inside a pull-based Seq.t or iterator is always discontinued even if the caller abandons the sequence early?
Part of the premise doesn't hold, and the part that does hold can't be fixed from inside Seq.t.
What holds: a dropped k never runs the Fun.protect ~finally or try ... with frames suspended on that fiber. That is by design. Cleanup runs only if an exception or a return unwinds the stack, so nothing runs the finaliser unless you resume or discontinue. The OCaml manual's effects chapter says the same: a continuation that is never resumed never runs its handlers.
What I don't think holds: the "leaks the C fiber stack" claim. I'm going from memory of the 5.x runtime and haven't run it. Fiber stacks are runtime objects reachable from the continuation, and my understanding is that the GC frees them once the continuation is unreachable. So the fiber's memory is reclaimed, but your fd is not, because close () never runs. The leak is the external resource, not the stack. To confirm, call Gc.full_major () in a loop and watch the process's memory. If you see an actual stack leak, I'd want to see that repro.
Pattern: Seq.t has no close operation, so no Seq-shaped API can guarantee the discontinue. You need an explicit scope, with a close that does discontinue k Exit and a bracket that always calls it:
open Effect
open Effect.Deep
type 'a state =
| Fresh
| Suspended of (unit, unit) continuation
| Done
type 'a gen = {
body : ('a -> unit) -> unit;
mutable st : 'a state;
mutable cur : 'a option;
}
let make body = { body; st = Fresh; cur = None }
(* One local effect per generator, so nested generators don't intercept each other *)
let handler (type a) (g : a gen) (yield_eff : a -> unit Effect.t) =
{ retc = (fun () -> g.st <- Done);
exnc = (function Exit -> g.st <- Done
| e -> g.st <- Done; raise e);
effc = fun (type b) (eff : b Effect.t) ->
match eff with
| _ -> None }
(* Simpler to read: define Yield once, polymorphic payload via Obj-free GADT *)
type _ Effect.t += Yield : 'a -> unit Effect.t
let h (type a) (g : a gen) : (unit, unit) handler =
{ retc = (fun () -> g.st <- Done);
exnc = (function Exit -> g.st <- Done | e -> g.st <- Done; raise e);
effc = fun (type b) (eff : b Effect.t) ->
match eff with
| Yield (v : a) -> Some (fun (k : (b, unit) continuation) ->
g.cur <- Some v; g.st <- Suspended k)
| _ -> None }
The Yield : 'a -> unit Effect.t case with the (v : a) annotation is not sound as written. The pattern can't tell that the payload really has type a, and the compiler will either reject it or you'd need Obj.magic. Use a per-generator effect instead, created with a first-class module. This part is reasoned, not compiled:
let of_producer (type a) (body : (a -> unit) -> unit) =
let module E = struct type _ Effect.t += Y : a -> unit Effect.t end in
let st = ref `Fresh and cur = ref None in
let h = { retc = (fun () -> st := `Done);
exnc = (function Exit -> st := `Done | e -> st := `Done; raise e);
effc = fun (type b) (eff : b Effect.t) ->
match eff with
| E.Y v -> Some (fun (k : (b, unit) continuation) ->
cur := Some v; st := `Susp k)
| _ -> None } in
let next () =
(match !st with
| `Fresh -> match_with (fun () -> body (fun v -> perform (E.Y v))) () h
| `Susp k -> continue k ()
| `Done -> ());
match !st with `Done -> None | _ -> let v = !cur in cur := None; v
and close () =
match !st with
| `Susp k -> st := `Done; discontinue k Exit (* runs Fun.protect ~finally *)
| _ -> ()
in
next, close
let with_gen body f =
let next, close = of_producer body in
Fun.protect ~finally:close (fun () -> f next)
Why this works
- discontinue k Exit raises Exit at the suspended perform. That unwinds the fiber, so your ~finally runs.
- Exit then reaches exnc, which must swallow it. Otherwise close itself raises.
- Fun.protect in with_gen ties the discontinue to the consumer's scope, so early return, exceptions, and take 1 are all covered.
Caveats
- If the producer's ~finally itself raises, Fun.protect wraps it in Finally_raised. Keep cleanup code non-raising.
- Don't try Gc.finalise on k. Finalisers run at arbitrary points, and running discontinue there is unsafe and nondeterministic.
- If you must expose a Seq.t, document that it is single-use and only valid inside with_gen. Alternatively, invert control with an iter-style callback, where Fun.protect lives in the producer and no continuation escapes, which avoids the problem entirely.