I didn't have to rewrite my iterator, just use it. I believe you are pointing out ways to make it easier to rewrite a naive recursive solution so that it can be re-implemented as an iterator instead.

By "Try to go the other way" I meant (at least in part), try to use jdporter's callback interface (as-is) in a situation where an iterator is needed.

Though, I don't even understand what you linked to yet so I still consider what I did to go from iterator to callback to be "trivial" compared to what you are proposing as "not too difficult". (:

- tye        

I will agree that there's a distinct difference between "trivial" and "not too difficult" that applies to converting between iterators and recursive solutions. However, jd and I found it easier to write recursive solutions, so there's a trade-off in difficulty: It might be easier en toto to come up with a recursive one and then convert it to an iterator.

Not even understanding jdporter's algorithm, I converted it to be iterative (or perhaps more precisely, lazily evaluated). I made the derange function print only the first 15 results, for convenient testing of large inputs.

sub _derange_iter {
  my ($cb, $todo, @v) = @_;
  @$todo or return do { $cb->( @v ); sub {} }; # this line was wrong b
+efore
  my %seen;
  @seen{@v} = ();
  my ( $range, @todo ) = @$todo;
  my @sub_iter = map {
    my $my_ = $_;
    sub { _derange_iter ( $cb, \@todo, @v, $my_ ) }
  } grep { ! exists $seen{$_} } @$range;
  return sub {} unless (@sub_iter);
  # Grab and unwrap an iterator from the list
  my $iter = (shift @sub_iter)->();
  return sub {
      my $rval;
      $iter = (shift @sub_iter)->()
        until ($rval = $iter->() or @sub_iter == 0);
      return $rval;
  }
}

sub derange(&@)
{
    my $cb = shift;
    my $iter = _derange_iter( $cb,
        [ map { my $x = $_; [ grep { $_ ne $x } @_ ] } @_ ] );
    for (1..15) {
      $iter->();
    }
}

derange { print "@_\n" } @ARGV;
[download]

---

Caution: Contents may have been coded under pressure.

Most programming languages support collections, represented by an in-memory data structure, a file, a database, or a generating function. A programming language system gives us typically one of the two interfaces to systematically access elements of a collection. One traversal API is based on enumerators -- e.g., for-each, map, filter higher-order procedures -- of which the most general is fold. The second approach relies on streams, a.k.a. cursors, lazy lists. Generators such as the ones in Icon, Ruby and Python are a hybrid approach.

It is well-known that given a cursor interface to a collection, we can implement an enumerator. It is less appreciated that given an enumerator interface, we can always derive a cursor -- in an automatic way. We demonstrate that generic procedure for languages with and without first-class continuations.

Now that cursors and enumerators are inter-convertible, an implementor of a collection has a choice: which of the two interfaces to implement natively? We argue that he should offer the enumerator interface as the native one. The paper elaborates that enumerators are superior: in efficiency; in ease of programming; in more predictable resource utilization and avoidance of resource leaks. We present a design of the overall optimal collection traversal interface, which is based on a left-fold-like combinator with premature termination. The design has been implemented and tested in practice.

Only having read the two abstracts, it seems to me that the authors rely on a Scheme-like language that has continuations or at least coroutines. With coroutines, it's trivial to convert between an "enumerator" (callback) and a "stream" (list/iterator). Without a coroutine mechanism, it's not as easy. As you seem to have read the paper, can you maybe post a link to the actual paper or, even better, give an application of the automatic way discussed there in Perl?

Using the Coro.pm module, it's quite easy to have generators and to convert between enumerator and iterator, but Coro has the disadvantage of relying on a GPLed library and it doesn't (immediately) work on Win32. Without Coro, one has to manage the stack oneself and/or to create a large buffer for all values passed by the enumerator from what I know. But maybe the paper shows a technique I don't know (yet).

Update: I had another look at Coro, and it isn't under the GPL. I also found that half-support for Win32/MSVC is there, now. Yay! ;)

From enumerators to cursors: turning the left fold inside out...

The mechanical inversion procedure presented in * had a catch: it relies on shift/reset (or call/cc plus a mutable cell, which is the same thing). How can we do such an inversion in Haskell? We can introduce a right fold enumerator, which is more amenable to such transformations. Or we can use a continuation monad and emulate shift/reset. The present article demonstrates the third approach: a non-recursive left-fold. We argue that such a left fold is the best interface for a collection. Indeed, given the non-recursive left-fold we can:

• instantiate it into the ordinary left fold
• instantiate in into a stream

If we turn two enumerators into streams, we can *safely* interleave these streams.

We should point out that the relation between the left fold, the non-recursive left fold and the stream is deep. The ordinary, recursive left fold is the fix point of the non-recursive one. On the other hand, the instantiation of the non-recursive left fold as a stream, as we shall see, effectively captures a continuation in a monadic action. We see once again that call/cc and Y are indeed two sides of the same coin **.

The rest of the article demonstrates the inversion procedure. The procedure is generic, as evidenced by its polymorphic type. We illustrate the technique on an example of a file considered a collection of characters. Haskell provides a stream interface to that collection: hGetChar. We implement a left fold enumerator. We then turn that enumerator back to a stream: we implement a function 'myhgetchar' _only_ in terms of the left fold enumerator. The approach is general and uses no monadic heavy lifting.