Pandas for Machine Learning: 10 Essential Features You Need to Know

Most time in machine learning isn't spent building models — it's spent preparing the data that goes into them. Pandas is the industry-standard Python library for exactly that: loading, cleaning, transforming, and inspecting datasets.

To keep things grounded, every example below works with a single sample dataset built using the code below.

import pandas as pd
import numpy as np

data = {
    'StudentID': [1, 2, 3, 4, 5, 6],
    'Name': ['Aarav', 'Priya', 'Rohan', 'Ananya', 'Vikram', 'Neha'],
    'Age': [20, 21, np.nan, 22, 20, 21],
    'City': ['Delhi', 'Mumbai', 'Delhi', 'Bangalore', np.nan, 'Mumbai'],
    'Score': [85, 92, 78, np.nan, 88, 95],
    'Passed': [1, 1, 0, 1, 1, 1]
}

df = pd.DataFrame(data)
print(df)

1. DataFrames and Series: The Core Data Structures

Everything in pandas is built around two structures: the Series, a single labeled column of data, and the DataFrame, a full table made up of multiple Series. Together they let you store and manipulate data the way a spreadsheet would, but with the speed and programmability of Python.

# A single column is a Series
ages = df['Age']
print(type(ages))      # Series

# The full table is a DataFrame
print(type(df))        # DataFrame

Almost every ML workflow starts by loading data into a DataFrame, since libraries like Scikit-learn are built to work directly with this format.

2. Reading Data from Multiple Sources

Real datasets rarely arrive as neatly typed-out dictionaries. They usually live in CSV files, Excel sheets, JSON files, or databases. Pandas makes loading any of these formats simple, often in a single line.

df_csv = pd.read_csv('students.csv')
df_excel = pd.read_excel('students.xlsx')
df_json = pd.read_json('students.json')

Being able to pull data from almost any source this easily is one of the reasons pandas is the first tool reached for in any ML project, no matter how the raw data shows up.

3. Data Inspection and Exploration

Before cleaning or modeling anything, you need to understand what you're actually working with — how many rows and columns you have, what data types each column holds, and whether anything looks off. Pandas has a handful of methods built specifically for this.

print(df.head())        # first 5 rows
print(df.shape)         # (rows, columns)
print(df.info())        # data types and non-null counts
print(df.describe())    # statistical summary of numeric columns

describe() is especially useful for ML: it instantly shows the mean, standard deviation, and range of your numeric features, which helps you spot outliers or scaling issues before they reach your model.

4. Handling Missing Values

Missing data is one of the most common problems in real-world datasets, and most machine learning algorithms can't handle NaN values directly. Pandas gives you full control over finding and fixing them.

# Check for missing values in each column
print(df.isnull().sum())

# Fill missing numeric values with the column mean
df['Age'] = df['Age'].fillna(df['Age'].mean())
df['Score'] = df['Score'].fillna(df['Score'].mean())

# Fill missing categorical values with the most frequent value
df['City'] = df['City'].fillna(df['City'].mode()[0])

# Alternatively, you could drop rows with missing values instead:
# df = df.dropna()

How you handle missing values — filling them with a statistic versus removing them entirely — can directly affect your model's accuracy, so it's worth giving this step real thought rather than treating it as a formality.

5. Selecting and Filtering Data

Once your data is clean, you'll often need to pull out specific rows, columns, or subsets that meet certain conditions — for example, separating your input features from your target variable, or filtering out rows that don't meet a quality threshold.

# Select specific columns
features = df[['Age', 'City', 'Score']]
target = df['Passed']

# Label-based selection
print(df.loc[0:2, ['Name', 'Score']])

# Position-based selection
print(df.iloc[0:2, 1:3])

# Boolean filtering
high_scorers = df[df['Score'] > 85]
print(high_scorers)

This kind of slicing is exactly how you'll separate your features (X) from your target label (y) before passing data into a model.

6. GroupBy and Aggregation

groupby() lets you split your data into groups based on a category, run a calculation on each group, and combine the results — invaluable for spotting patterns before you ever start modeling.

city_avg_score = df.groupby('City')['Score'].mean()
print(city_avg_score)

summary = df.groupby('City').agg({
    'Score': 'mean',
    'Age': 'mean',
    'Passed': 'sum'
})
print(summary)

Grouped summaries like this often reveal patterns — say, one city's students consistently scoring higher — that can directly inform your feature engineering decisions later on.

7. Merging, Joining, and Concatenating DataFrames

ML datasets are often split across multiple files or tables — student scores in one file, attendance records in another, for instance. Pandas gives you several ways to bring them back together.

attendance = pd.DataFrame({
    'StudentID': [1, 2, 3, 4, 5, 6],
    'AttendancePercent': [92, 88, 75, 95, 80, 99]
})

merged_df = pd.merge(df, attendance, on='StudentID')
print(merged_df)

merge() works much like a SQL join, while concat() is handy for stacking datasets that share the same structure — for example, combining training data collected across different months.

8. Applying Functions for Feature Engineering

Feature engineering — creating new, more useful columns out of existing ones — is often what separates an average model from a genuinely good one. Pandas' apply() and map() methods let you transform data using your own custom logic.

# Create a new feature using a custom function
def score_category(score):
    if score >= 90:
        return 'Excellent'
    elif score >= 80:
        return 'Good'
    else:
        return 'Average'

merged_df['ScoreCategory'] = merged_df['Score'].apply(score_category)

# A quick one-line transformation using a lambda
merged_df['ScorePerAttendance'] = merged_df.apply(
    lambda row: row['Score'] / row['AttendancePercent'], axis=1
)

print(merged_df[['Name', 'Score', 'ScoreCategory', 'ScorePerAttendance']])

This is exactly how raw columns get turned into the kind of meaningful features that actually move the needle on model performance.

9. Encoding Categorical Data

Machine learning models work with numbers, not text, so categorical columns like City need to be converted into a numeric form before training. Pandas makes this straightforward with one-hot encoding.

encoded_df = pd.get_dummies(merged_df, columns=['City'], drop_first=True)
print(encoded_df.head())

get_dummies() creates a separate binary column for each category — a technique called one-hot encoding — and it's one of the most common preprocessing steps before feeding data into almost any model.

10. Sorting and Removing Duplicates

Sorting helps you quickly spot your top or bottom performers, while catching duplicate rows stops your model from accidentally learning from repeated, biased samples.

# Sort by score, highest first
print(merged_df.sort_values(by='Score', ascending=False))

# Check for and remove duplicate rows
print(merged_df.duplicated().sum())
merged_df = merged_df.drop_duplicates()

Try It Yourself

Quick Reference Cheat Sheet

Conclusion

Pandas isn't a machine learning library itself, but it's the foundation almost every ML project is built on. From reading raw files to engineering the final features that go into a model, the ten features covered here — DataFrames, file I/O, inspection tools, missing value handling, selection, GroupBy, merging, apply functions, encoding, and sorting — cover the vast majority of what you'll actually use in day-to-day ML work.

Get comfortable with these, and reading real ML project code will stop feeling like decoding a foreign language.