Lua Programming by EezyTutorials

Lua - Iterators

Back to top ↑

Overview

Estimated time: 30–35 minutes

Iterators provide a powerful way to traverse collections and generate sequences of values. Lua offers built-in iterators like pairs and ipairs, and allows you to create custom iterators for specialized iteration patterns. This tutorial covers all aspects of iterator usage and creation.

Learning Objectives

  • Master built-in iterators: pairs, ipairs, and their use cases
  • Understand the generic for loop and how iterators work
  • Create custom stateless and stateful iterators
  • Explore advanced iterator patterns and techniques
  • Apply iterators for data processing and generation

Prerequisites

  • Understanding of tables, arrays, and functions
  • Knowledge of loops and closures
  • Familiarity with function parameters and return values

Built-in Iterators

Lua provides several built-in iterators for common iteration patterns:

ipairs() - Array Iterator

-- ipairs() iterates over consecutive integer keys starting from 1
local fruits = {"apple", "banana", "orange", "grape"}

print("Using ipairs() for arrays:")
for index, value in ipairs(fruits) do
    print(index .. ":", value)
end

-- ipairs() stops at the first nil value
local sparse_array = {"a", "b", nil, "d", "e"}
print("\nSparse array with ipairs():")
for i, v in ipairs(sparse_array) do
    print(i .. ":", v)  -- Only prints indices 1 and 2
end

-- Compare with numeric for loop
print("\nSame sparse array with numeric loop:")
for i = 1, #sparse_array do
    print(i .. ":", sparse_array[i] or "nil")
end

-- ipairs() with empty table
local empty = {}
print("\nEmpty table with ipairs():")
for i, v in ipairs(empty) do
    print(i, v)  -- Nothing printed
end
print("No iterations performed")

Expected Output:

Using ipairs() for arrays:
1:	apple
2:	banana
3:	orange
4:	grape

Sparse array with ipairs():
1:	a
2:	b

Same sparse array with numeric loop:
1:	a
2:	b
3:	nil
4:	d
5:	e

Empty table with ipairs():
No iterations performed

pairs() - Table Iterator

-- pairs() iterates over all key-value pairs in a table
local student = {
    name = "Alice",
    age = 20,
    grade = "A",
    subjects = {"Math", "Physics", "Chemistry"}
}

print("Using pairs() for general tables:")
for key, value in pairs(student) do
    if type(value) == "table" then
        print(key .. ":", table.concat(value, ", "))
    else
        print(key .. ":", value)
    end
end

-- pairs() includes both array and hash parts
local mixed_table = {
    "first",   -- index 1
    "second",  -- index 2
    name = "Mixed Table",
    count = 42
}

print("\nMixed table with pairs():")
for key, value in pairs(mixed_table) do
    print(type(key) .. " key '" .. tostring(key) .. "':", value)
end

-- pairs() vs ipairs() comparison
print("\nSame mixed table with ipairs():")
for i, v in ipairs(mixed_table) do
    print(i .. ":", v)
end

-- Order is not guaranteed with pairs()
local hash_table = {b = 2, a = 1, d = 4, c = 3}
print("\nHash table iteration (order may vary):")
for key, value in pairs(hash_table) do
    print(key .. ":", value)
end

Expected Output:

Using pairs() for general tables:
subjects:	Math, Physics, Chemistry
name:	Alice
age:	20
grade:	A

Mixed table with pairs():
number key '1':	first
number key '2':	second
string key 'name':	Mixed Table
string key 'count':	42

Same mixed table with ipairs():
1:	first
2:	second

Hash table iteration (order may vary):
a:	1
b:	2
c:	3
d:	4

How Iterators Work

Understanding the iterator protocol and generic for loop:

-- The generic for loop calls the iterator function repeatedly
-- for var1, var2, ... in iterator_function do

-- Manual iteration equivalent to: for k, v in pairs(table) do
local function manual_pairs_iteration(t)
    local iterator_func, state, initial_key = pairs(t)
    local key, value = iterator_func(state, initial_key)
    
    while key ~= nil do
        print("Manual iteration - " .. tostring(key) .. ":", tostring(value))
        key, value = iterator_func(state, key)
    end
end

local test_table = {a = 1, b = 2, c = 3}
print("Understanding iterator protocol:")
manual_pairs_iteration(test_table)

-- Iterator function anatomy
print("\nIterator function details:")
local iter_func, state, initial = pairs(test_table)
print("Iterator function:", type(iter_func))
print("State:", type(state))
print("Initial key:", tostring(initial))

-- First call
local first_key, first_value = iter_func(state, initial)
print("First call result:", tostring(first_key), tostring(first_value))

-- Second call (using first key as input)
local second_key, second_value = iter_func(state, first_key)
print("Second call result:", tostring(second_key), tostring(second_value))

-- How ipairs works internally
print("\nipairs internal structure:")
local arr = {"x", "y", "z"}
local ipairs_func, ipairs_state, ipairs_initial = ipairs(arr)
print("ipairs state (the table):", type(ipairs_state))
print("ipairs initial value:", ipairs_initial)

-- ipairs returns (index, value) pairs
for i = 1, 3 do
    local index, value = ipairs_func(ipairs_state, i - 1)
    if index then
        print("ipairs call " .. i .. ":", index, value)
    end
end

Expected Output:

Understanding iterator protocol:
Manual iteration - a:	1
Manual iteration - b:	2
Manual iteration - c:	3

Iterator function details:
Iterator function:	function
State:	table
Initial key:	nil
First call result:	a	1
Second call result:	b	2

ipairs internal structure:
ipairs state (the table):	table
ipairs initial value:	0
ipairs call 1:	1	x
ipairs call 2:	2	y
ipairs call 3:	3	z

Custom Stateless Iterators

Create custom iterators that don't maintain internal state:

-- Custom stateless iterators
local custom_iterators = {}

-- Iterator for even numbers in a range
function custom_iterators.even_numbers(max)
    return function(state, current)
        current = current + 2
        if current <= max then
            return current
        end
    end, nil, 0  -- iterator_func, state, initial_value
end

-- Iterator for array elements in reverse order
function custom_iterators.reverse_ipairs(array)
    return function(arr, index)
        index = index - 1
        if index >= 1 then
            return index, arr[index]
        end
    end, array, #array + 1
end

-- Iterator for characters in a string
function custom_iterators.string_chars(str)
    return function(s, i)
        i = i + 1
        if i <= #s then
            return i, string.sub(s, i, i)
        end
    end, str, 0
end

-- Iterator for words in a string
function custom_iterators.words(str)
    local function word_iterator(s, pos)
        local start_pos, end_pos, word = string.find(s, "(%S+)", pos)
        if start_pos then
            return end_pos + 1, word
        end
    end
    return word_iterator, str, 1
end

-- Iterator for file lines (simulated)
function custom_iterators.lines(content)
    local function line_iterator(text, pos)
        if pos <= #text then
            local line_end = string.find(text, "\n", pos) or (#text + 1)
            local line = string.sub(text, pos, line_end - 1)
            return line_end + 1, line
        end
    end
    return line_iterator, content, 1
end

-- Test custom iterators
print("Custom stateless iterators:")

print("\nEven numbers up to 12:")
for number in custom_iterators.even_numbers(12) do
    print("Even:", number)
end

local colors = {"red", "green", "blue", "yellow"}
print("\nReverse iteration of colors:")
for index, color in custom_iterators.reverse_ipairs(colors) do
    print(index .. ":", color)
end

print("\nCharacters in 'Hello':")
for pos, char in custom_iterators.string_chars("Hello") do
    print("Position " .. pos .. ":", char)
end

print("\nWords in sentence:")
local sentence = "Lua is a powerful scripting language"
for pos, word in custom_iterators.words(sentence) do
    print("Word:", word)
end

print("\nLines in text:")
local text = "Line 1\nLine 2\nLine 3\nLine 4"
for pos, line in custom_iterators.lines(text) do
    print("Line:", line)
end

Expected Output:

Custom stateless iterators:

Even numbers up to 12:
Even:	2
Even:	4
Even:	6
Even:	8
Even:	10
Even:	12

Reverse iteration of colors:
4:	yellow
3:	blue
2:	green
1:	red

Characters in 'Hello':
Position 1:	H
Position 2:	e
Position 3:	l
Position 4:	l
Position 5:	o

Words in sentence:
Word:	Lua
Word:	is
Word:	a
Word:	powerful
Word:	scripting
Word:	language

Lines in text:
Line:	Line 1
Line:	Line 2
Line:	Line 3
Line:	Line 4

Custom Stateful Iterators

Create iterators that maintain their own state using closures:

-- Stateful iterators using closures
local stateful_iterators = {}

-- Fibonacci sequence iterator
function stateful_iterators.fibonacci()
    local a, b = 0, 1
    return function()
        local result = a
        a, b = b, a + b
        return result
    end
end

-- Counter iterator with step
function stateful_iterators.counter(start, stop, step)
    start = start or 1
    step = step or 1
    local current = start - step
    
    return function()
        current = current + step
        if not stop or current <= stop then
            return current
        end
    end
end

-- Random number generator iterator
function stateful_iterators.random_numbers(count, min, max)
    min = min or 0
    max = max or 1
    local generated = 0
    
    return function()
        if generated < count then
            generated = generated + 1
            return min + (max - min) * math.random()
        end
    end
end

-- Permutation iterator (simple)
function stateful_iterators.permutations(elements)
    local n = #elements
    local indices = {}
    for i = 1, n do indices[i] = i end
    local first = true
    
    return function()
        if first then
            first = false
            local result = {}
            for i = 1, n do
                result[i] = elements[indices[i]]
            end
            return result
        end
        
        -- Generate next permutation (simplified)
        -- This is a basic example - full permutation generation is more complex
        return nil
    end
end

-- Batch iterator - groups elements into batches
function stateful_iterators.batches(array, batch_size)
    local index = 1
    local length = #array
    
    return function()
        if index <= length then
            local batch = {}
            for i = index, math.min(index + batch_size - 1, length) do
                table.insert(batch, array[i])
            end
            index = index + batch_size
            return batch
        end
    end
end

-- Filter iterator - yields only elements matching predicate
function stateful_iterators.filter(array, predicate)
    local index = 1
    local length = #array
    
    return function()
        while index <= length do
            local value = array[index]
            index = index + 1
            if predicate(value) then
                return value
            end
        end
    end
end

-- Test stateful iterators
print("Stateful iterators:")

print("\nFirst 8 Fibonacci numbers:")
local fib = stateful_iterators.fibonacci()
for i = 1, 8 do
    print("F(" .. i .. "):", fib())
end

print("\nCounter from 5 to 15 with step 2:")
local count = stateful_iterators.counter(5, 15, 2)
local num = count()
while num do
    print("Count:", num)
    num = count()
end

math.randomseed(42)  -- For consistent output
print("\n5 random numbers between 10 and 20:")
local rand = stateful_iterators.random_numbers(5, 10, 20)
local random_num = rand()
while random_num do
    print("Random:", string.format("%.2f", random_num))
    random_num = rand()
end

print("\nBatches of 3 from array:")
local data = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
local batch_iter = stateful_iterators.batches(data, 3)
local batch = batch_iter()
local batch_num = 1
while batch do
    print("Batch " .. batch_num .. ":", table.concat(batch, ", "))
    batch = batch_iter()
    batch_num = batch_num + 1
end

print("\nFiltered even numbers:")
local numbers = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
local even_filter = stateful_iterators.filter(numbers, function(x) return x % 2 == 0 end)
local even_num = even_filter()
while even_num do
    print("Even:", even_num)
    even_num = even_filter()
end

Expected Output:

Stateful iterators:

First 8 Fibonacci numbers:
F(1):	0
F(2):	1
F(3):	1
F(4):	2
F(5):	3
F(6):	5
F(7):	8
F(8):	13

Counter from 5 to 15 with step 2:
Count:	5
Count:	7
Count:	9
Count:	11
Count:	13
Count:	15

5 random numbers between 10 and 20:
Random:	16.40
Random:	18.98
Random:	11.59
Random:	17.84
Random:	15.56

Batches of 3 from array:
Batch 1:	1, 2, 3
Batch 2:	4, 5, 6
Batch 3:	7, 8, 9
Batch 4:	10

Filtered even numbers:
Even:	2
Even:	4
Even:	6
Even:	8
Even:	10

Advanced Iterator Patterns

Combining and chaining iterators for complex data processing:

-- Advanced iterator patterns
local advanced_iterators = {}

-- Chain multiple iterators together
function advanced_iterators.chain(...)
    local iterators = {...}
    local current_iter_index = 1
    local current_iter = iterators[current_iter_index]
    
    return function()
        while current_iter do
            local value = current_iter()
            if value ~= nil then
                return value
            else
                current_iter_index = current_iter_index + 1
                current_iter = iterators[current_iter_index]
            end
        end
    end
end

-- Take first n items from iterator
function advanced_iterators.take(iterator, n)
    local count = 0
    return function()
        if count < n then
            count = count + 1
            return iterator()
        end
    end
end

-- Skip first n items from iterator
function advanced_iterators.skip(iterator, n)
    for i = 1, n do
        iterator()  -- Consume n items
    end
    return iterator
end

-- Map function over iterator values
function advanced_iterators.map(iterator, func)
    return function()
        local value = iterator()
        if value ~= nil then
            return func(value)
        end
    end
end

-- Zip two iterators together
function advanced_iterators.zip(iter1, iter2)
    return function()
        local val1, val2 = iter1(), iter2()
        if val1 ~= nil and val2 ~= nil then
            return val1, val2
        end
    end
end

-- Enumerate iterator (add index)
function advanced_iterators.enumerate(iterator)
    local index = 0
    return function()
        local value = iterator()
        if value ~= nil then
            index = index + 1
            return index, value
        end
    end
end

-- Create iterator from array
function advanced_iterators.from_array(array)
    local index = 0
    return function()
        index = index + 1
        return array[index]
    end
end

-- Test advanced patterns
print("Advanced iterator patterns:")

-- Create some basic iterators
local numbers1 = advanced_iterators.from_array({1, 2, 3})
local numbers2 = advanced_iterators.from_array({4, 5, 6})
local numbers3 = advanced_iterators.from_array({7, 8, 9})

-- Chain iterators
print("\nChained iterators:")
local chained = advanced_iterators.chain(numbers1, numbers2, numbers3)
local val = chained()
while val do
    print("Chained:", val)
    val = chained()
end

-- Take and skip
print("\nFibonacci - skip 3, take 5:")
local fib = stateful_iterators.fibonacci()
local skipped_fib = advanced_iterators.skip(fib, 3)
local limited_fib = advanced_iterators.take(skipped_fib, 5)
local fib_val = limited_fib()
while fib_val do
    print("Fibonacci:", fib_val)
    fib_val = limited_fib()
end

-- Map transformation
print("\nSquared counter:")
local counter = stateful_iterators.counter(1, 5)
local squared = advanced_iterators.map(counter, function(x) return x * x end)
local sq_val = squared()
while sq_val do
    print("Squared:", sq_val)
    sq_val = squared()
end

-- Zip iterators
print("\nZipped iterators:")
local letters = advanced_iterators.from_array({"a", "b", "c", "d"})
local nums = stateful_iterators.counter(1, 4)
local zipped = advanced_iterators.zip(letters, nums)
local letter, number = zipped()
while letter and number do
    print("Zipped:", letter, number)
    letter, number = zipped()
end

-- Enumerate
print("\nEnumerated colors:")
local colors = advanced_iterators.from_array({"red", "green", "blue"})
local enumerated = advanced_iterators.enumerate(colors)
local idx, color = enumerated()
while idx and color do
    print("Index " .. idx .. ":", color)
    idx, color = enumerated()
end

Expected Output:

Advanced iterator patterns:

Chained iterators:
Chained:	1
Chained:	2
Chained:	3
Chained:	4
Chained:	5
Chained:	6
Chained:	7
Chained:	8
Chained:	9

Fibonacci - skip 3, take 5:
Fibonacci:	2
Fibonacci:	3
Fibonacci:	5
Fibonacci:	8
Fibonacci:	13

Squared counter:
Squared:	1
Squared:	4
Squared:	9
Squared:	16
Squared:	25

Zipped iterators:
Zipped:	a	1
Zipped:	b	2
Zipped:	c	3
Zipped:	d	4

Enumerated colors:
Index 1:	red
Index 2:	green
Index 3:	blue

Practical Iterator Applications

Real-world examples of iterator usage:

-- Practical iterator applications
local practical = {}

-- File processing iterator (simulated)
function practical.process_csv(csv_content)
    local lines = {}
    for line in string.gmatch(csv_content, "[^\n]+") do
        table.insert(lines, line)
    end
    
    local index = 0
    return function()
        index = index + 1
        local line = lines[index]
        if line then
            local fields = {}
            for field in string.gmatch(line, "([^,]+)") do
                table.insert(fields, field)
            end
            return fields
        end
    end
end

-- Log entry iterator
function practical.parse_log(log_content)
    local function log_iterator(content, pos)
        local line_start, line_end = string.find(content, "[^\n]*", pos)
        if line_start then
            local line = string.sub(content, line_start, line_end)
            if line ~= "" then
                local timestamp, level, message = string.match(line, "^(%S+)%s+(%w+):%s+(.+)$")
                if timestamp then
                    return line_end + 2, {
                        timestamp = timestamp,
                        level = level,
                        message = message
                    }
                end
            end
            return line_end + 2, nil
        end
    end
    return log_iterator, log_content, 1
end

-- Data validation iterator
function practical.validate_data(data, validators)
    local index = 0
    return function()
        index = index + 1
        local item = data[index]
        if item then
            local errors = {}
            for field, validator in pairs(validators) do
                if not validator(item[field]) then
                    table.insert(errors, field)
                end
            end
            return item, #errors == 0, errors
        end
    end
end

-- Pagination iterator
function practical.paginate(data, page_size)
    local total_pages = math.ceil(#data / page_size)
    local current_page = 0
    
    return function()
        current_page = current_page + 1
        if current_page <= total_pages then
            local start_idx = (current_page - 1) * page_size + 1
            local end_idx = math.min(current_page * page_size, #data)
            local page_data = {}
            
            for i = start_idx, end_idx do
                table.insert(page_data, data[i])
            end
            
            return {
                page = current_page,
                total_pages = total_pages,
                data = page_data
            }
        end
    end
end

-- Test practical applications
print("Practical iterator applications:")

-- CSV processing
print("\nCSV Processing:")
local csv_data = "name,age,city\nAlice,25,New York\nBob,30,San Francisco\nCharlie,22,Chicago"
local csv_iter = practical.process_csv(csv_data)
local csv_row = csv_iter()
while csv_row do
    print("CSV Row:", table.concat(csv_row, " | "))
    csv_row = csv_iter()
end

-- Log parsing
print("\nLog Parsing:")
local log_data = [[2024-01-15T10:30:45 ERROR: Database connection failed
2024-01-15T10:30:46 INFO: Retrying connection
2024-01-15T10:30:47 WARN: Connection timeout increased]]

for pos, entry in practical.parse_log(log_data) do
    if entry then
        print(string.format("Log: %s [%s] %s", entry.timestamp, entry.level, entry.message))
    end
end

-- Data validation
print("\nData Validation:")
local user_data = {
    {name = "Alice", email = "[email protected]", age = 25},
    {name = "", email = "bob@invalid", age = 30},
    {name = "Charlie", email = "[email protected]", age = -5}
}

local validators = {
    name = function(name) return name and #name > 0 end,
    email = function(email) return email and string.find(email, "@.*%.") end,
    age = function(age) return age and age > 0 and age < 150 end
}

local validator_iter = practical.validate_data(user_data, validators)
local user, is_valid, errors = validator_iter()
while user do
    local status = is_valid and "VALID" or "INVALID (" .. table.concat(errors, ", ") .. ")"
    print("User " .. user.name .. ": " .. status)
    user, is_valid, errors = validator_iter()
end

-- Pagination
print("\nPagination:")
local items = {}
for i = 1, 23 do
    table.insert(items, "Item " .. i)
end

local page_iter = practical.paginate(items, 5)
local page = page_iter()
while page do
    print("Page " .. page.page .. "/" .. page.total_pages .. ":")
    for _, item in ipairs(page.data) do
        print("  " .. item)
    end
    if page.page < 3 then  -- Only show first 3 pages
        page = page_iter()
    else
        print("  ... (showing only first 3 pages)")
        break
    end
end

Expected Output:

Practical iterator applications:

CSV Processing:
CSV Row:	name | age | city
CSV Row:	Alice | 25 | New York
CSV Row:	Bob | 30 | San Francisco
CSV Row:	Charlie | 22 | Chicago

Log Parsing:
Log:	2024-01-15T10:30:45 [ERROR] Database connection failed
Log:	2024-01-15T10:30:46 [INFO] Retrying connection
Log:	2024-01-15T10:30:47 [WARN] Connection timeout increased

Data Validation:
User Alice: VALID
User : INVALID (name, email)
User Charlie: INVALID (age)

Pagination:
Page 1/5:
  Item 1
  Item 2
  Item 3
  Item 4
  Item 5
Page 2/5:
  Item 6
  Item 7
  Item 8
  Item 9
  Item 10
Page 3/5:
  Item 11
  Item 12
  Item 13
  Item 14
  Item 15
  ... (showing only first 3 pages)

Common Pitfalls

  • State management: Be careful with stateful iterators and side effects
  • Iterator exhaustion: Once consumed, some iterators cannot be reused
  • Memory leaks: Closures in iterators can hold references to large objects
  • Nil handling: Iterator protocol uses nil to signal end of iteration
  • Performance: Complex iterators may have overhead compared to simple loops

Checks for Understanding

  1. What's the difference between pairs() and ipairs()?
  2. How many values does an iterator function return?
  3. What signals the end of iteration in Lua?
  4. What are the three components returned by iterator constructors?
  5. How do you create a stateful iterator?
Show answers
  1. pairs() iterates over all table elements; ipairs() only over consecutive integer keys starting from 1
  2. Usually 1-3 values: the next key/index and associated value(s)
  3. When the iterator function returns nil for the first value
  4. Iterator function, state (usually a table), and initial key/index
  5. Use closures to capture and maintain state variables outside the iterator function

Next Steps

Now that you understand iterators and iteration patterns, you're ready to explore modules and learn how to organize and structure larger Lua applications.

« Previous: Lua - Arrays | Next: Lua - Modules »