Iteration, generators & comprehensions

A forloop in Python is not magic over lists — it's a tiny protocol anyone can join. The loop calls iter(x) (which calls x.__iter__()) to get an iterator, then calls next(it) (__next__) repeatedly until the iterator raises StopIteration. That's the whole contract:

# "for item in x:" is really:
it = iter(x)              # x.__iter__()
while True:
    try:
        item = next(it)   # it.__next__()
    except StopIteration:
        break
    ...                   # loop body

Lists, dicts, files, database cursors, network streams — anything that implements those two methods works in a for loop, in sum(), in unpacking, everywhere. That uniformity is why Python code composes so well.

Generators are the cheapest way to implement that protocol: a function containing yield doesn't run when called — it returns a paused computation. Each next() resumes it until the next yield, then freezes it again, locals intact. The killer property: it holds O(1) state instead of O(n) results.

sum(x * x for x in range(10**9))   # runs fine: one x at a time in memory

sum([x * x for x in range(10**9)]) # builds a billion-element list first
                                   # (tens of GB) — likely killed by the OS

Same arithmetic, same answer — one holds a single number at a time, the other materializes a billion before adding the first pair.

Comprehensions are the idiomatic transform layer: [f(x) for x in xs if p(x)] (list), plus dict, set, and generator-expression forms. A comprehension beats map/filterwhen there's any expression logic — it reads left-to-right as “what, from where, under what condition”. But it should become a plain loop the moment you need side effects, multiple statements, or error handling per item. A comprehension that mutates things or spans four lines is a loop wearing a disguise.

prices = {row.ticker: row.close for row in rows if row.close > 0}   # good

[log.write(r) for r in rows]   # bad: side effects — write a for loop

The senior-level payoff is pipelines: chained generators are streaming. Each stage pulls one item from the stage before it, so a 1M-row price file flows through parse → filter → aggregate with one row in flight and nothing materialized:

def load_prices(path):
    with open(path) as f:
        rows    = csv.DictReader(f)                        # streams lines
        parsed  = (parse_row(r) for r in rows)             # one at a time
        valid   = (p for p in parsed if p.price > 0)
        yield from valid

total = sum(p.price * p.qty for p in load_prices("fills.csv"))
# constant memory whether the file has 1k rows or 100M

The itertools module is the toolbox for these pipelines: islice (take/skip without indexing), chain (concatenate streams), groupby (run-length grouping of sorted input), and accumulate — which is exactly a running prefix sum, the same structure we use for O(1) range queries in Prefix sums.

One trap to respect: generators are one-shot. Once exhausted, iterating again silently yields nothing — no error, just an empty second pass. If two consumers need the data, either re-create the generator or materialize it once with list(). See it happen below.