Regular Expressions (Regex)

Master pattern matching and text processing with regular expressions for data validation, parsing, and advanced text manipulation.

What Are Regular Expressions?

A regular expression (regex) is a sequence of characters that defines a search pattern. Think of it as a mini-language for describing text patterns.

What You'll Learn

Regular expressions (regex) are powerful patterns for matching and manipulating text. They're essential for validation, parsing log files, extracting data, and text processing tasks.

What Are Regular Expressions?

A regular expression (regex) is a sequence of characters that defines a search pattern. Think of it as a mini-language for describing text patterns.

Common Use Cases

Validating email addresses, phone numbers, URLs
Extracting data from log files or text documents
Search and replace with complex patterns
Parsing structured text formats
Web scraping and data extraction

Python's re Module

Python's built-in re module provides regex functionality:

re.search() - Find pattern anywhere
re.match() - Match from start
re.findall() - Find all matches
re.sub() - Search and replace

Getting Started with the re Module

Basic Pattern Matching

Let's start with simple pattern matching using re.search(). It returns a match object if the pattern is found, or None if not found.

import re

# Search for a pattern in text
text = "The quick brown fox jumps over the lazy dog"
pattern = "fox"

match = re.search(pattern, text)
if match:
    print(f"Found '{pattern}' at position {match.start()}")
    print(f"Match: {match.group()}")
else:
    print("Not found")

# Output:
# Found 'fox' at position 16
# Match: fox

Common re Functions

re.search() - Find pattern anywhere in the string

import re

text = "Email: contact@example.com"

# search() looks anywhere in the string
match = re.search(r'\w+@\w+\.com', text)
if match:
    print(match.group())
    # contact@example.com

re.match() - Match only at the beginning of the string

# match() only checks the start
text = "Hello world"

match1 = re.match(r'Hello', text)
print(match1.group() if match1 else "No match")
# Hello

match2 = re.match(r'world', text)
print(match2.group() if match2 else "No match")
# No match (world is not at the start)

re.findall() - Find all non-overlapping matches

# findall() returns a list of all matches
text = "Call me at 555-1234 or 555-5678"

numbers = re.findall(r'\d{3}-\d{4}', text)
print(numbers)
# ['555-1234', '555-5678']

re.sub() - Search and replace

# sub() replaces matches with a replacement string
text = "Contact us at info@old-domain.com"

new_text = re.sub(r'old-domain', 'new-domain', text)
print(new_text)
# Contact us at info@new-domain.com

Raw Strings: Notice the r prefix before regex patterns (e.g., r'\d+'). This creates a raw string where backslashes are treated literally, making regex patterns easier to write.

Basic Regex Patterns

Metacharacters

Metacharacters are special characters with special meanings in regex. They are the building blocks of regex patterns.

Character	Meaning	Example
`.`	Any character (except newline)	`a.c` matches "abc", "a5c", "a@c"
`^`	Start of string	`^Hello` matches "Hello world"
`$`	End of string	`end$` matches "The end"
`*`	0 or more repetitions	`ab*c` matches "ac", "abc", "abbc"
`+`	1 or more repetitions	`ab+c` matches "abc", "abbc" (not "ac")
`?`	0 or 1 repetition (optional)	`colou?r` matches "color", "colour"
`\|`	OR operator	`cat\|dog` matches "cat" or "dog"
`\`	Escape special character	`\.` matches literal "."

import re

# . matches any character
print(re.search(r'c.t', 'cat').group())     # cat
print(re.search(r'c.t', 'cut').group())     # cut
print(re.search(r'c.t', 'c@t').group())     # c@t

# ^ matches start of string
print(re.search(r'^Hello', 'Hello world'))  # Match object
print(re.search(r'^Hello', 'Say Hello'))    # None

# $ matches end of string
print(re.search(r'end$', 'The end'))        # Match object
print(re.search(r'end$', 'end of story'))   # None

Character Classes

Character classes match any one character from a set of characters. They're defined using square brackets [ ].

# [abc] - matches any single character a, b, or c
text = "The cat sat on the mat"
matches = re.findall(r'[cms]at', text)
print(matches)
# ['cat', 'sat', 'mat']

# [a-z] - matches any lowercase letter
text = "abc123XYZ"
letters = re.findall(r'[a-z]', text)
print(letters)
# ['a', 'b', 'c']

# [0-9] - matches any digit
text = "Room 123, Floor 4"
digits = re.findall(r'[0-9]', text)
print(digits)
# ['1', '2', '3', '4']

# [^abc] - matches any character EXCEPT a, b, or c (negation)
text = "abc123"
not_abc = re.findall(r'[^abc]', text)
print(not_abc)
# ['1', '2', '3']

Special Sequences

Special sequences are shortcuts for common character classes. They make patterns more readable.

Sequence	Equivalent	Matches
`\d`	`[0-9]`	Any digit
`\D`	`[^0-9]`	Any non-digit
`\w`	`[a-zA-Z0-9_]`	Any word character
`\W`	`[^a-zA-Z0-9_]`	Any non-word character
`\s`	`[ \t\n\r\f\v]`	Any whitespace
`\S`	`[^ \t\n\r\f\v]`	Any non-whitespace

# \d matches digits
text = "Order #12345 costs $99"
numbers = re.findall(r'\d+', text)
print(numbers)
# ['12345', '99']

# \w matches word characters
text = "user_name123"
words = re.findall(r'\w+', text)
print(words)
# ['user_name123']

# \s matches whitespace
text = "Hello   World\tPython\n"
spaces = re.findall(r'\s+', text)
print(spaces)
# ['   ', '\t', '\n']

Quantifiers

Quantifiers specify how many times a pattern should match. They give you precise control over repetition.

Quantifier	Meaning	Example
`{ n }`	Exactly n times	`\d{3}` matches "123" (exactly 3 digits)
`{ n, }`	n or more times	`\d{2,}` matches "12", "123", "1234"...
`{ n,m }`	Between n and m times	`\d{2,4}` matches "12", "123", "1234"
`*`	0 or more (same as {0,})	`a*` matches "", "a", "aa", "aaa"...
`+`	1 or more (same as {1,})	`a+` matches "a", "aa", "aaa"...
`?`	0 or 1 (same as {0,1})	`a?` matches "" or "a"

# Exactly 3 digits
phone = "Call 555-1234"
match = re.search(r'\d{3}', phone)
print(match.group())
# 555

# 2 to 4 digits
numbers = "1 12 123 1234 12345"
matches = re.findall(r'\b\d{2,4}\b', numbers)
print(matches)
# ['12', '123', '1234']

# Practical example: Phone number pattern
phone_pattern = r'\d{3}-\d{3}-\d{4}'

text = "My number is 555-123-4567"
match = re.search(phone_pattern, text)
print(match.group())
# 555-123-4567

# Flexible phone pattern (with optional area code)
flexible_pattern = r'(\d{3}-)?\d{3}-\d{4}'

print(re.search(flexible_pattern, '555-123-4567').group())
# 555-123-4567

print(re.search(flexible_pattern, '123-4567').group())
# 123-4567

Greedy vs Non-Greedy Matching

By default, quantifiers are greedy - they match as much as possible. Add ? after a quantifier to make it non-greedy (match as little as possible).

html = "<div>First</div><div>Second</div>"

# Greedy: matches as much as possible
greedy = re.search(r'<div>.*</div>', html)
print(greedy.group())
# <div>First</div><div>Second</div>

# Non-greedy: matches as little as possible
non_greedy = re.search(r'<div>.*?</div>', html)
print(non_greedy.group())
# <div>First</div>

Performance Tip: Non-greedy matching (*?, +?) can be slower than greedy matching. Use it only when necessary.

Groups and Capturing

Groups allow you to extract specific parts of a match and apply quantifiers to multiple characters together.

Capturing Groups

Use parentheses ( ) to create capturing groups. Groups let you extract specific parts of a match.

# Extract parts of a date
date_pattern = r'(\d{4})-(\d{2})-(\d{2})'
text = "Date: 2024-03-15"

match = re.search(date_pattern, text)
if match:
    print(f"Full match: {match.group(0)}")    # 2024-03-15
    print(f"Year: {match.group(1)}")          # 2024
    print(f"Month: {match.group(2)}")         # 03
    print(f"Day: {match.group(3)}")           # 15

    # Or use groups() to get all at once
    year, month, day = match.groups()
    print(f"{year}/{month}/{day}")            # 2024/03/15

# Extract email components
email_pattern = r'([\w.]+)@([\w.]+)\.([a-z]{2,})'
email = "contact@example.com"

match = re.search(email_pattern, email)
if match:
    username = match.group(1)
    domain = match.group(2)
    tld = match.group(3)
    print(f"Username: {username}")   # contact
    print(f"Domain: {domain}")       # example
    print(f"TLD: {tld}")             # com

Named Groups

Named groups make patterns more readable. Use (?P<name>...) syntax.

# Named groups for better readability
phone_pattern = r'(?P<area>\d{3})-(?P<prefix>\d{3})-(?P<line>\d{4})'
phone = "555-123-4567"

match = re.search(phone_pattern, phone)
if match:
    print(f"Area code: {match.group('area')}")      # 555
    print(f"Prefix: {match.group('prefix')}")       # 123
    print(f"Line: {match.group('line')}")           # 4567

    # Or get all named groups as a dictionary
    print(match.groupdict())
    # {'area': '555', 'prefix': '123', 'line': '4567'}

Non-Capturing Groups

Use (?:...) when you need grouping but don't want to capture the content. This saves memory and makes group numbering easier.

# Non-capturing group (?:...)
# Useful when you need grouping for quantifiers but don't need to capture

url_pattern = r'https?://(?:www\.)?([\w.]+)'

urls = ["http://example.com", "https://www.google.com"]

for url in urls:
    match = re.search(url_pattern, url)
    if match:
        # Group 1 is the domain (www. part is not captured)
        print(f"Domain: {match.group(1)}")

# Output:
# Domain: example.com
# Domain: google.com

Backreferences

Backreferences let you match the same text that was previously matched by a capturing group. Use \1, \2, etc., or \g<name> for named groups.

# Find repeated words
text = "The the cat sat on the the mat"

# \1 refers to whatever group 1 matched
repeated = re.findall(r'\b(\w+)\s+\1\b', text)
print(repeated)
# ['the']

# Find duplicated HTML tags
html = "<b>bold</b> <i>italic</i> <b>more bold</b>"
pattern = r'<(\w+)>.*?</\1>'

tags = re.findall(pattern, html)
print(tags)
# ['b', 'i', 'b']

Lookahead and Lookbehind

Lookahead and lookbehind assertions check if a pattern exists without including it in the match. They're called "zero-width" assertions because they don't consume characters.

Assertion	Syntax	Meaning
Positive Lookahead	`(?=...)`	Matches if ... is ahead
Negative Lookahead	`(?!...)`	Matches if ... is NOT ahead
Positive Lookbehind	`(?<=...)`	Matches if ... is behind
Negative Lookbehind	`(?<!...)`	Matches if ... is NOT behind

# Positive lookahead: Match word followed by a number
text = "item1 item2 other"

# Match 'item' only if followed by a digit
matches = re.findall(r'item(?=\d)', text)
print(matches)
# ['item', 'item']  (matched 'item' before 1 and 2)

# Negative lookahead: Match 'item' NOT followed by a digit
text2 = "item1 item2 item other"
matches = re.findall(r'item(?!\d)', text2)
print(matches)
# ['item']  (only standalone 'item', not 'item1' or 'item2')

# Positive lookbehind: Match numbers preceded by '$'
text = "Price: $50, Quantity: 10"

# Match digits only if preceded by $
prices = re.findall(r'(?<=\$)\d+', text)
print(prices)
# ['50']

# Extract values with currency symbol
values = re.findall(r'\$\d+', text)
print(values)
# ['$50']  (includes the $)

# Practical: Password validation
# At least 8 chars, must contain uppercase, lowercase, and digit

password_pattern = r'^(?=.*[a-z])(?=.*[A-Z])(?=.*\d).{8,}$'

passwords = [
    "weakpass",        # No uppercase, no digit
    "WeakPass",        # No digit
    "StrongPass1",     # Valid!
    "SHORT1"           # Too short
]

for pwd in passwords:
    if re.match(password_pattern, pwd):
        print(f"{pwd}: Valid")
    else:
        print(f"{pwd}: Invalid")

# Output:
# weakpass: Invalid
# WeakPass: Invalid
# StrongPass1: Valid
# SHORT1: Invalid

Regex Flags

Flags modify how regex patterns are interpreted. They're passed as the third argument to regex functions.

Flag	Constant	Description
`re.IGNORECASE`	`re.I`	Case-insensitive matching
`re.MULTILINE`	`re.M`	^ and $ match start/end of each line
`re.DOTALL`	`re.S`	. matches newlines too
`re.VERBOSE`	`re.X`	Allow comments and whitespace in pattern

# re.IGNORECASE - Case-insensitive matching
text = "Python is AWESOME"

match = re.search(r'python', text, re.IGNORECASE)
print(match.group())
# Python


# re.MULTILINE - ^ and $ match each line
text = """First line
Second line
Third line"""

# Without MULTILINE: only matches first line
matches = re.findall(r'^\w+', text)
print(matches)
# ['First']

# With MULTILINE: matches start of each line
matches = re.findall(r'^\w+', text, re.MULTILINE)
print(matches)
# ['First', 'Second', 'Third']

# re.VERBOSE - Add comments to complex patterns
# Makes complex regex more readable

email_pattern = re.compile(r"""
    ^                      # Start of string
    ([\w.+-]+)             # Username (group 1)
    @                      # @ symbol
    ([\w.-]+)              # Domain name (group 2)
    \.                     # Dot
    ([a-z]{2,})            # TLD (group 3)
    $                      # End of string
""", re.VERBOSE | re.IGNORECASE)

email = "user.name+tag@example.com"
match = email_pattern.search(email)

if match:
    print(f"Username: {match.group(1)}")
    print(f"Domain: {match.group(2)}")
    print(f"TLD: {match.group(3)}")

# Output:
# Username: user.name+tag
# Domain: example
# TLD: com

# Combine multiple flags with | (bitwise OR)
pattern = r'hello'
text = "HELLO\nWORLD"

# Case-insensitive AND multiline
match = re.search(pattern, text, re.IGNORECASE | re.MULTILINE)
print(match.group())
# HELLO

Practical Examples

Example 1: Email Validation

import re

def validate_email(email):
    """
    Validate email address format
    """
    # Pattern breakdown:
    # ^[\w.+-]+ - Start with word chars, dots, plus, or hyphen
    # @ - Literal @ symbol
    # [\w.-]+ - Domain with word chars, dots, or hyphens
    # \. - Literal dot
    # [a-z]{2,} - TLD (2+ lowercase letters)

    pattern = r'^[\w.+-]+@[\w.-]+\.[a-z]{2,}$'

    return re.match(pattern, email, re.IGNORECASE) is not None


# Test emails
emails = [
    "user@example.com",           # Valid
    "user.name@example.co.uk",    # Valid
    "user+tag@example.com",       # Valid
    "invalid.email",              # Invalid - no @
    "@example.com",               # Invalid - no username
    "user@.com",                  # Invalid - no domain
]

for email in emails:
    status = "✓ Valid" if validate_email(email) else "✗ Invalid"
    print(f"{email:30} {status}")

Example 2: Phone Number Formatting

def format_phone_number(phone):
    """
    Extract and format phone numbers from various formats
    """
    # Remove all non-digits
    digits = re.sub(r'\D', '', phone)

    # Check if we have 10 or 11 digits
    if len(digits) == 10:
        # Format as (XXX) XXX-XXXX
        return re.sub(r'(\d{3})(\d{3})(\d{4})', r'(\1) \2-\3', digits)
    elif len(digits) == 11 and digits[0] == '1':
        # Format as 1 (XXX) XXX-XXXX
        return re.sub(r'1(\d{3})(\d{3})(\d{4})', r'1 (\1) \2-\3', digits)
    else:
        return None


# Test various phone formats
phones = [
    "5551234567",
    "555-123-4567",
    "(555) 123-4567",
    "1-555-123-4567",
    "555.123.4567",
]

for phone in phones:
    formatted = format_phone_number(phone)
    print(f"{phone:20} -> {formatted}")

Example 3: URL Parsing

def parse_url(url):
    """
    Extract components from a URL
    """
    pattern = r'''
        ^
        (?P<protocol>https?://)?      # Optional protocol
        (?P<subdomain>[\w.-]+\.)?    # Optional subdomain
        (?P<domain>[\w-]+)            # Domain name
        (?P<tld>\.[a-z]{2,})          # TLD
        (?P<port>:\d+)?              # Optional port
        (?P<path>/[^?#]*)?            # Optional path
        (?P<query>\?[^#]*)?           # Optional query string
        (?P<fragment>#.*)?            # Optional fragment
        $
    '''

    match = re.match(pattern, url, re.VERBOSE | re.IGNORECASE)

    if match:
        return match.groupdict()
    return None


# Test URLs
urls = [
    "https://www.example.com/path/to/page?id=123#section",
    "http://subdomain.example.co.uk:8080/api/users",
    "example.com/about",
]

for url in urls:
    print(f"\nURL: {url}")
    parts = parse_url(url)
    if parts:
        for key, value in parts.items():
            if value:
                print(f"  {key:12} {value}")

Example 4: Log File Parsing

def parse_log_entry(log_line):
    """
    Parse Apache/Nginx style log entries
    Format: IP - - [timestamp] "method path protocol" status size
    """
    pattern = r'''
        ^
        (?P<ip>[\d.]+)\s+            # IP address
        -\s+-\s+                      # Ignore fields
        \[(?P<timestamp>[^\]]+)\]\s+ # Timestamp in brackets
        "(?P<method>\w+)\s+           # HTTP method
        (?P<path>\S+)\s+              # Request path
        (?P<protocol>[^"]+)"\s+       # Protocol
        (?P<status>\d{3})\s+         # Status code
        (?P<size>\d+|-)               # Response size
    '''

    match = re.match(pattern, log_line, re.VERBOSE)

    if match:
        return match.groupdict()
    return None


# Sample log entry
log = '192.168.1.1 - - [15/Mar/2024:10:30:45 +0000] "GET /api/users HTTP/1.1" 200 1234'

parsed = parse_log_entry(log)
if parsed:
    print("Parsed log entry:")
    for key, value in parsed.items():
        print(f"  {key:12} {value}")

# Output:
# Parsed log entry:
#   ip           192.168.1.1
#   timestamp    15/Mar/2024:10:30:45 +0000
#   method       GET
#   path         /api/users
#   protocol     HTTP/1.1
#   status       200
#   size         1234

Example 5: Data Extraction from Text

def extract_prices(text):
    """
    Extract all prices from text in various formats
    """
    # Match: $X.XX, $XXX, $X,XXX.XX
    pattern = r'\$([\d,]+\.?\d{0,2})'

    matches = re.findall(pattern, text)

    # Convert to float (remove commas)
    prices = [float(price.replace(',', '')) for price in matches]

    return prices


text = """
Special offers:
- Widget: $19.99
- Gadget: $1,299
- Premium Bundle: $2,499.00
- Discount: -$50.00
"""

prices = extract_prices(text)
print("Found prices:", prices)
# Found prices: [19.99, 1299.0, 2499.0, 50.0]

total = sum(prices)
print(f"Total: ${total:.2f}")
# Total: $3867.99

Best Practices

When to Use Regex vs String Methods

Use Regex When

Pattern matching complex formats (emails, URLs, phone numbers)
Validating input against a pattern
Extracting structured data from unstructured text
Search and replace with patterns
Parsing log files or similar formatted text

Use String Methods When

Simple exact string matching (in, ==)
Case conversion (.lower(), .upper())
Simple splits (.split())
Prefix/suffix checks (.startswith(), .endswith())
Simple replacements (.replace())

Performance Tips

# ✓ GOOD: Compile patterns used multiple times
email_pattern = re.compile(r'[\w.+-]+@[\w.-]+\.[a-z]{2,}', re.IGNORECASE)

emails = ["user1@example.com", "user2@example.com", "user3@example.com"]
for email in emails:
    if email_pattern.match(email):
        print(f"Valid: {email}")


# ✗ BAD: Re-compiling the same pattern repeatedly
for email in emails:
    if re.match(r'[\w.+-]+@[\w.-]+\.[a-z]{2,}', email, re.IGNORECASE):
        print(f"Valid: {email}")

# ✓ GOOD: Be as specific as possible
# Matches exactly 3 digits
pattern = r'\d{3}'

# ✗ BAD: Overly broad patterns
# Matches any number of digits (slower, less precise)
pattern = r'\d+'

Making Regex Readable

# Use re.VERBOSE for complex patterns
# Add comments to explain each part

url_pattern = re.compile(r"""
    ^                          # Start of string
    (https?://)?               # Optional protocol
    (www\.)?                   # Optional www subdomain
    ([a-z0-9]([a-z0-9-]*[a-z0-9])?)  # Domain name
    (\.[a-z]{2,})+             # TLD (one or more)
    (/.*)?                     # Optional path
    $                          # End of string
""", re.VERBOSE | re.IGNORECASE)

# Much more readable than:
# ^(https?://)?(www\.)?([a-z0-9]([a-z0-9-]*[a-z0-9])?)(.[a-z]{2,})+(/.*)?$

Testing Regex: Always test your regex patterns with multiple test cases, including edge cases. Use online tools like regex101.com or pythex.org for interactive testing.

Key Takeaways

Regular expressions are powerful tools for pattern matching and text processing
Use raw strings (r'...') for regex patterns to avoid escaping issues
Metacharacters like . * + ? ^ $ | \ have special meanings
Character classes [...] and special sequences \d \w \s simplify common patterns
Quantifiers {n,m} * + ? control repetition
Capturing groups (...) extract specific parts of matches
Named groups (?P<name>...) make patterns more readable
Lookahead/lookbehind assertions check context without consuming characters
Compile patterns with re.compile() when using them multiple times
Use re.VERBOSE flag to add comments and make complex patterns readable
Choose wisely: Use regex for complex patterns, string methods for simple operations

What's Next?

You've mastered regular expressions! Now let's learn how to build professional command-line tools that users love to interact with.

Command-Line Interfaces - Build interactive CLI tools with argparse and click
Argument Parsing - Handle user input, flags, and options professionally
User Experience - Create helpful error messages and beautiful terminal output

Back to "Error Handling"

Python IntermediateLesson 14

Next: "Command-Line Interfaces (CLI)"