Build a strong Python foundation with practical examples for data engineering work. Learn to manipulate data structures, control program flow, write reusable functions, handle errors, and process files and JSON payloads.
1. Variables, Data Types, and Operators
Variables are names that refer to objects. Python is dynamically typed, so you don't declare a variable's type in advance. The type belongs to the object, and a variable can later refer to an object of another type.
name = "Asha" # str
age = 29 # int
salary = 85000.50 # float
is_engineer = True # bool
middle_name = None # NoneType
print(type(name)) # <class 'str'>
print(type(age)) # <class 'int'>Common data types
| Type | Example | Typical use |
|---|---|---|
int | 42 | Counts, IDs where numeric arithmetic is intended |
float | 3.14 | Approximate numeric values |
str | "data" | Text |
bool | True | Logical conditions |
NoneType | None | Missing or absent value |
Operators
# Arithmetic
print(10 + 3) # 13
print(10 / 3) # 3.333...
print(10 // 3) # 3: floor division
print(10 % 3) # 1: remainder
print(2 ** 3) # 8: exponentiation
# Comparison and logical operators
print(5 > 3) # True
print(5 == 5 and 2 < 4) # True
print(not False) # True
# Membership and identity
print("py" in "python") # True
x = None
print(x is None) # True== to compare values and is to test object identity. For checking a missing value, value is None is the standard pattern.Type conversion
count = int("25")
price = float("19.99")
code = str(1234)
enabled = bool(1)
# Be careful: bool("False") is True because
# every non-empty string is truthy.
print(bool("")) # False
print(bool("False")) # True2. Lists, Tuples, Sets, and Dictionaries
Python collections store multiple values, but differ in ordering, mutability, and uniqueness.
| Collection | Ordered? | Mutable? | Duplicates? | Typical use |
|---|---|---|---|---|
List [] | Yes | Yes | Yes | Sequence of records or values |
Tuple () | Yes | No | Yes | Fixed group of values |
Set {...} | No guaranteed iteration order | Yes | No | Unique values and membership tests |
Dictionary {key: value} | Insertion-ordered | Yes | Keys must be unique | Lookups by key |
Lists
numbers = [10, 20, 30, 20]
numbers.append(40)
numbers[0] = 5
print(numbers) # [5, 20, 30, 20, 40]
print(numbers[1:3]) # [20, 30]
print(numbers[-1]) # 40Tuples
coordinates = (12.9, 77.6)
latitude, longitude = coordinates
print(latitude, longitude)
# A one-element tuple needs a trailing comma
single_item = (42,)Sets
source_a = {"A", "B", "C"}
source_b = {"B", "C", "D"}
print(source_a | source_b) # union
print(source_a & source_b) # intersection
print(source_a - source_b) # values only in source_a
unique_ids = set([101, 102, 101, 103])
print(unique_ids) # {101, 102, 103}Dictionaries
customer = {
"customer_id": 101,
"name": "Asha",
"country": "India"
}
print(customer["name"])
print(customer.get("email")) # None if key is absent
customer["active"] = True
for key, value in customer.items():
print(key, value)customer["email"] raises KeyError if the key is absent, while customer.get("email") returns None by default. Use a distinct default if a missing key must be distinguished from a stored None.3. Loops and Conditional Statements
Use if, elif, and else to branch, and use for or while loops to repeat work.
row_count = 120
if row_count == 0:
status = "empty"
elif row_count < 100:
status = "small"
else:
status = "large"
print(status) # largeFor loops and enumerate
tables = ["customers", "orders", "products"]
for table in tables:
print(f"Processing {table}")
for index, table in enumerate(tables, start=1):
print(index, table)While loops and control flow
retries = 0
max_retries = 3
while retries < max_retries:
print("Attempt", retries + 1)
retries += 1
for number in range(1, 6):
if number == 3:
continue # skip this iteration
if number == 5:
break # stop the loop
print(number)4. Functions, Lambda, and Comprehensions
Functions package reusable logic and make code easier to test. Prefer explicit parameters and return values over hidden side effects.
Defining functions
def calculate_net_amount(amount, discount=0):
"""Return the amount after applying a discount."""
return amount - discount
print(calculate_net_amount(100, 15)) # 85
print(calculate_net_amount(100)) # 100Positional and keyword arguments
def describe_job(name, region="asia-south1"):
return f"{name} runs in {region}"
print(describe_job("daily_orders"))
print(describe_job(name="customer_load", region="europe-west3"))Lambda functions
A lambda is a small anonymous function, useful for short expressions passed to other functions.
records = [
{"name": "Asha", "amount": 120},
{"name": "Ravi", "amount": 90},
{"name": "Mina", "amount": 150}
]
sorted_records = sorted(records, key=lambda row: row["amount"])
print(sorted_records)List, set, and dictionary comprehensions
values = [1, 2, 3, 4, 5, 6]
squares = [x * x for x in values]
evens = [x for x in values if x % 2 == 0]
unique_lengths = {len(name) for name in ["sql", "python", "beam"]}
lookup = {x: x * x for x in values}
print(squares)
print(evens)
print(lookup)Generator expressions
# Generates values lazily instead of building a full list.
total = sum(int(value) for value in ["10", "20", "30"])
print(total) # 60A generator expression can reduce memory use for large inputs, but it is usually consumed only once.
5. Exception Handling
Exceptions represent errors or unusual conditions. Handle errors you can recover from, add useful context, and avoid silently swallowing failures.
raw_value = "not-a-number"
try:
value = int(raw_value)
except ValueError as exc:
print(f"Invalid integer: {raw_value!r}")
# In a real pipeline, log context or route the row
else:
print("Parsed:", value)
finally:
print("Parsing attempt finished")try: code that may raise an exception.except: handles a matching exception type.else: runs only if thetryblock completes without an exception.finally: runs whether or not an exception occurred, commonly for cleanup.
Raise a meaningful exception
def validate_amount(amount):
if amount < 0:
raise ValueError("amount must not be negative")
return amount
try:
validate_amount(-10)
except ValueError as exc:
print("Validation failed:", exc)ValueError, KeyError, or FileNotFoundError. Avoid a bare except: because it can hide programming errors and interrupts.6. File Handling and JSON Processing
Use with open(...) so files are closed automatically, including when an exception occurs. Specify an encoding when working with text files.
Read and write text files
# Read an entire text file
with open("input.txt", "r", encoding="utf-8") as file:
content = file.read()
# Read a large file line by line
with open("events.log", "r", encoding="utf-8") as file:
for line in file:
line = line.rstrip("\\n")
if line:
print(line)
# Write a text file (overwrites existing contents)
with open("output.txt", "w", encoding="utf-8") as file:
file.write("Pipeline completed\\n")Process JSON
JSON is commonly used for API responses, configuration files, and event payloads. Python's built-in json module converts between JSON text and Python objects.
import json
payload = {
"event_id": 501,
"event_type": "order_created",
"amount": 1299.50,
"tags": ["web", "priority"]
}
# Python object -> JSON string
json_text = json.dumps(payload, indent=2)
print(json_text)
# JSON string -> Python object
parsed = json.loads(json_text)
print(parsed["event_type"])
print(parsed["tags"][0])Read and write JSON files
import json
# Read a JSON document
with open("config.json", "r", encoding="utf-8") as file:
config = json.load(file)
# Write a JSON document
with open("result.json", "w", encoding="utf-8") as file:
json.dump(config, file, indent=2)Process a JSONL file safely
import json
with open("events.jsonl", "r", encoding="utf-8") as file:
for line_number, line in enumerate(file, start=1):
if not line.strip():
continue
try:
event = json.loads(line)
except json.JSONDecodeError as exc:
print(f"Bad JSON on line {line_number}: {exc}")
continue
event_id = event.get("event_id")
print(event_id)In production, malformed records should be logged with enough context and sent to a quarantine or dead-letter path when required. Don't log secrets or sensitive payload fields.
Interview Questions to Practice
1. What is the difference between a list and a tuple?
Both preserve order and allow duplicates, but lists are mutable while tuples are immutable. Use lists for collections that change and tuples for fixed groupings of values.
2. What is the difference between a set and a dictionary?
A set stores unique elements. A dictionary maps unique, hashable keys to values and preserves insertion order in modern Python.
3. What is the difference between == and is?
== checks value equality. is checks whether two references point to the same object. Use is None to test for None.
4. When would you use a lambda or a comprehension?
A lambda is useful for a short function expression, such as a sorting key. Comprehensions concisely build collections from an iterable. Use named functions or regular loops when the logic becomes complex.
5. Why use a context manager for file handling?
The with statement ensures the file is closed when the block exits, including when an exception occurs. It makes resource handling safer and clearer.
6. What is the difference between json.load and json.loads?
json.load(file) reads JSON from a file-like object. json.loads(text) parses a JSON string.
7. How would you process a very large JSONL file?
Read it line by line, parse each non-empty line, validate its structure, and write results incrementally. Add malformed-record handling, observability, and retry-safe output rather than loading the entire file into memory.
Chapter Summary
- Python variables reference dynamically typed objects; understand common data types and truthiness.
- Choose lists, tuples, sets, and dictionaries based on ordering, mutability, uniqueness, and lookup needs.
- Use loops and conditionals to control processing flow.
- Write reusable functions; use lambdas and comprehensions for small, clear transformations.
- Handle expected exceptions specifically and preserve useful debugging context.
- Use context managers for files and the
jsonmodule for JSON data; stream JSONL records for large files.
Note: Examples use Python 3. Adapt file paths, validation rules, logging, and error handling to your runtime and production requirements.
Comments
Post a Comment