Python iterators considered harmful

... subclassing built-in types considered harmful?  Or generator-properties considered harmful?  Or self-iterables (iterables which are their own iterators) considered harmful?

Mixing up generators with iterators considered harmful.  Also, mixing property decorators with generator functions considered stupid.

property of type iterator does not make any sense to me

try this:

class Even(object):
  def __init__(self, list):
    self.list = list

  def __iter__(self):
    for val in self.list:
      if val % 2 == 0: yield val

class Odd(object):
  def __init__(self, list):
    self.list = list

  def __iter__(self):
    for val in self.list:
      if val % 2 != 0: yield val

class Numbers(list):
  even = property(lambda self: Even(self))
  odd = property(lambda self: Odd(self))

seriously, instantiating a finite generator and then wondering that it can be exhausted?

You should read about descriptors.

Yes, it is essential to know the difference between iterables and iterators in Python. Iterators are stateful objects - they know how far through their sequence they are. Once they reach the end (if they have one), that's it. Iterables (or, more accurately, reiterables) instead are able to create iterators on demand. Since each iteration operation on a reiterable object implicitly creates a new iterator, they automatically start again from the beginning.

When all you have is an iterator, and you need to guarantee reiterability, then you need to either store the values in a list, or else use an appropriate tool (such as itertools.tee or a size limited deque) to retain access to the earlier values that you need.

Your "gotcha" is just a natural consequence of the stateful nature of iterators. You can get exactly the same effect by doing the following:

with open("a_file") as f:
   for line in f:
     print "A line from the file: %s" % line
   for line in f:
     print "This will never be printed"

The reason being, of course, that the first loop moved the read pointer to the end of the file, so you need to do an f.seek() to get back to the start (or somewhere else in the file) before iterating again will produce any output.

However, the huge advantage of iterators (and generators in particular) is precisely this point - because each value is kept around for only as long as it is needed, they can save vast amounts of memory compared to actual container objects. And, of course, some iterators produce infinite sequences that simply *cannot* be created as a finite list:

def evens(start=2):
  if start <= 0 or start % 2 != 0:
    raise ValueError("Start value %d is not positive and even" % start)
  i = start
  while 1:
    yield i
    i += 2


def odds(start=1):
  if start <= 0 or start % 2 != 1:
    raise ValueError("Start value %d is not positive and odd!" % start)
  i = start
  while 1:
    yield i
    i += 2

Nick Coghlan

Thanks a lot