Tutorial 2: NumPy and Pandas

Goal¶

In the previous notebook, we learned the core programming concepts in Python, including variables, lists, dictionaries, loops, and functions.

In this notebook, we’ll introduce three of the most widely used Python libraries for scientific computing:

• NumPy for working with numerical arrays and matrices.

• Pandas for organizing and analyzing tabular data.

By the end of this notebook, you should be able to:

• create and work with NumPy arrays,

• understand rows and columns,

• organize data using Pandas DataFrames,

• perform simple calculations on your data,

These two libraries form the foundation of many neuroscience and data science workflows, and you’ll use them extensively throughout the rest of this guide.

Importing Python Packages¶

Before we can use a package, we first need to import it.

Think of importing a package as telling Python:

“I’d like to use the tools provided by this package in my program.”

At the beginning of almost every data science project, you’ll commonly see:

Notice that we import each package using a shorter alias:

• np for NumPy

• pd for Pandas

• plt for Matplotlib

These abbreviations are widely used by the Python community, so you’ll see them in almost every tutorial and research project.

Try running the cell above. If it runs without any errors, you’re ready to continue.

If you receive an error such as:

ModuleNotFoundError: No module named 'numpy'

it usually means that the package hasn’t been installed in your current Python environment.

First, make sure you’ve selected the correct Jupyter kernel. If the package still isn’t available, check the package’s official installation instructions or install it using Conda. Instructions to do this can be found in the “Conda, Packages, and Jupyter Notebook” tutorial.

Why NumPy?¶

So far, we’ve stored collections of values using Python lists.

For example:

ages = [23, 25, 27, 29]

Lists are extremely useful, and you’ll continue using them throughout this guide.

However, in neuroscience we often work with much larger numerical datasets.

For example:

• hundreds of participant ages,

• thousands of brain regions,

• millions of voxel intensities,

• or time series containing hundreds of measurements for every brain region.

For these types of numerical data, Python provides NumPy.

NumPy introduces a data structure called an array, which is specifically designed for fast and efficient numerical computation.

Throughout neuroimaging, you’ll frequently encounter NumPy arrays representing:

• participant measurements,

• brain region time series,

• functional connectivity matrices,

• and images stored as large multidimensional arrays.

Let’s start with the simplest example: a one-dimensional NumPy array.

One-Dimensional NumPy Arrays¶

A NumPy array is similar to a Python list, except that it is specifically designed for storing and working with numerical data efficiently.

To create a NumPy array, we use the np.array() function.

For example, suppose we have the ages of five participants.

Notice that the code looks very similar to creating a Python list.

The main difference is that the data are now stored as a NumPy array, giving us access to many useful mathematical operations.

Like Python lists, NumPy arrays also use zero-based indexing.

Let’s breakdown the syntax. From the inside out:

• Square brackets [] create a Python list containing the numbers.

• np.array(...) is a NumPy function that converts the Python list into a NumPy array.

• The parentheses () are used because we’re calling a function (np.array).

• Finally, the resulting NumPy array is stored in the variable ages.

Once we create Numpy arrays, we can also quickly calculate simple summary statistics.

Notice how we didn’t have to write our own function to calculate the average or the maximum value. NumPy already provides many useful functions for working with numerical data.

More formally, NumPy arrays come with many built-in functions, called methods, that perform common calculations.

For example:

ages.mean()

The dot (.) tells Python:

“Use the mean() method that belongs to the NumPy array called volume.”

The mean() method calculates the average of all values stored in the array.

NumPy documentation has a list of all the methods you can use. For instance, numpy.mean (or np.mean in short) can be found here: https://numpy.org/doc/2.2/reference/generated/numpy.mean.html

Similarly, you can find documentation for numpy.max, numpy.min, numpy.sum, etc.

These methods save us from having to write our own code to calculate the average, maximum, minimum, or sum.

Try it yourself¶

• Create a new array consisting of participant’s hippocampal brain volume in mm^3 (e.g., 3450, 4566, 3213, 4332, 3334)

• Print the average participant’s age and hippocampal volume.

Two-Dimensional NumPy Arrays¶

Many scientific datasets are naturally organized as rows and columns.

For example, suppose we’ve measured the activity of three brain regions over five time points.

                Time →

           T1   T2   T3   T4   T5
Region 1    •    •    •    •    •
Region 2    •    •    •    •    •
Region 3    •    •    •    •    •

We can represent this as a two-dimensional NumPy array. Let’s try to represent this in code.

Here:

• each row represents one brain region,

• each column represents one time point.

One of the first things we often want to know is the shape of an array.

We can use the numpy.shape method tells us how many rows and columns it contains.

(3, 5) means that the 2D-array contains 3 rows and 5 columns.

We can also use indexing to access specific rows and columns.

To make sense of the above results, let’s remind ourselves what the rows and columns are.

Unlike a one-dimensional array, a two-dimensional array has both rows and columns.

When indexing a 2D array, we specify two positions:

array[row, column]

The first position selects the row, and the second position selects the column.

For example:

timeseries[0, :]

means:

• 0 → select the first row.

• : → select all columns.

So this retrieves the entire first row (all time points for the first brain region).

Similarly,

timeseries[:, 0]

means:

• : → select all rows.

• 0 → select the first column.

So this retrieves the first time point across all brain regions.

Finally,

timeseries[1, 2]

means:

• 1 → select the second row.

• 2 → select the third column.

This returns a single value at that row and column.

Remember that Python starts counting from 0, so:

• Row 0 = first row

• Row 1 = second row

• Column 0 = first column

• Column 2 = third column

Now, let’s try to change one value from the second row and fouth column in the 2D array timeseries.

First, let’s see what is the original value of the 2nd row and 4th column.

Let’s assign timeseries[1, 3] a new value (0.99).

Now, let’s print timeseries[1, 3] again. It should be 0.99 instead of 0.31.

Try it yourself¶

• Print the third brain region.

• Print the fourth time point across all brain regions.

• Print the value from the second brain region at the fifth time point.

• Change the value of the third region at the second timepoint to 0.02.

Calculating the Mean Along Different Axes¶

Earlier, we calculated the mean of a one-dimensional array:

volume.mean()

For a two-dimensional array, we have an extra choice to make:

Which direction should NumPy calculate the mean?

This is controlled by the axis argument.

Suppose our data look like this:

If we calculate:

NumPy combines the values down each column.

                Time →

           T1     T2     T3     T4     T5
Region 1  0.10   0.20   0.15   0.18   0.22
Region 2  0.30   0.35   0.28   0.31   0.29
Region 3  0.42   0.39   0.45   0.47   0.44
-----------------------------------------------
Mean      0.27   0.31   0.29   0.32   0.32

Each column is averaged separately, producing one mean for each time point.

If we instead calculate:

NumPy combines the values across each row.

                Time →

           T1     T2     T3     T4     T5      Mean
Region 1  0.10   0.20   0.15   0.18   0.22   → 0.17
Region 2  0.30   0.35   0.28   0.31   0.29   → 0.31
Region 3  0.42   0.39   0.45   0.47   0.44   → 0.43

Each row is averaged separately, producing one mean for each brain region.

In summary:

• axis=0 → combine values vertically, which leads to one result per column.

• axis=1 → combine values horizontally, which leads to one result per row.

The same idea applies to many other NumPy functions, including:

Only the calculation changes. The meaning of axis stays the same.

Pandas: Working with Tables of Data¶

Why do we need Pandas?¶

NumPy is great for numerical arrays, especially when we want to work with rows and columns of numbers.

But in real neuroscience projects, our data are often more than just numbers. We may have:

• participant IDs

• ages

• diagnoses

• brain measurements

• neuron types

• neurotransmitters

• connection strengths

This is where Pandas becomes useful.

Pandas is designed for working with tables of data, called DataFrames. A DataFrame is similar to a spreadsheet or a table in a paper: each row usually represents one object, and each column represents one variable.

For example, in the LEMON dataset, a DataFrame might contain demographic information, brain measures, and other participant-level variables. In a fly connectome dataset, a DataFrame might contain neuron type, neurotransmitter, and connection strength.

In this notebook, we will start with a simple example: a table containing information about participants.

Recall that we have already imported pandas

import pandas as pd

Creating a DataFrame¶

Let’s create a DataFrame containing information about each participant.

We will include:

• subject_id

• age

• drug_use

• left_hippocampal_volume

• right_hippocampal_volume

A DataFrame can be thought of as a dictionary of columns.

To create a DataFrame, we pass a Python dictionary to pd.DataFrame(). Each key (e.g., "subject_id") in the dictionary becomes a column name, and each value (e.g., ["S045", "S049", "S051", "S063", "S071"]) is a list containing all the entries for that column.

For example, in the code below:

• "subject_id" becomes one column.

• "age" becomes another column.

• "drug_use" becomes another column.

• Each list contains the values that will appear in that column.

So, let’s break down the syntax:

• The curly braces {} create a Python dictionary, where

  • Each key becomes a column name.

  • Each value is a Python list containing the values for that column.

• pd.DataFrame(...) is a Pandas function that converts this dictionary into a DataFrame.

Inspecting a DataFrame¶

One of the first things we usually want to do is inspect the table.

Pandas gives us several useful ways to do this.

head() shows the first few rows of the DataFrame.

This is useful when your table is large and you just want a quick preview.

shape tells us how many rows and columns are in the DataFrame.

In this case, the result means:

• 20 rows

• 5 columns

columns shows the names of the variables stored in the DataFrame.

Accessing Columns¶

To access a single column, use square brackets with the column name.

A column is a Pandas Series, which is like a single vector of values.

This is useful because we can now perform calculations on that column.

Basic Statistics¶

Because age and hippocampal volume are numeric columns, we can calculate summary statistics such as the mean.

Filtering Rows¶

One of the most useful features of Pandas is the ability to filter a DataFrame.

Filtering allows us to select only the rows that satisfy a particular condition.

Suppose we want to find participants who are younger than 30 years old.

The first step is to ask Pandas a simple question:

Is each participant younger than 30?

We can write:

Pandas checks the age of every participant, one row at a time, and returns either True or False, as you can see from the above.

A value of True means that the participant is younger than 30, while False means they are not.

Next, we use these True/False values to select the rows we want:

You can think of this as saying:

“From participant_data, keep only the rows where participant_data["age"] < 30 is True.”

The result is a new DataFrame containing only participants younger than 30.

Notice the structure:

DataFrame[condition]

The condition is evaluated for every row, and only the rows where the condition is True are returned.

Similarly, we can filter participants based on other variables.

For example, suppose we only want participants who report nicotine use.

Try it yourself¶

Write code to display:

• participants aged 40 or older,

• participants whose drug use is "cocaine",

• participants younger than 35.

Adding a New Column¶

Another common task is creating a new variable from existing information.

Suppose we want to classify participants into age groups:

• Young Adult: age < 30

• Middle Aged: age 30–49

• Older Adult: age ≥ 50

We’ll create a new column called age_group.

One way to do this is to:

1. Create an empty Python list.

2. Loop through the age column one participant at a time.

3. Decide which age group each participant belongs to using an if statement.

4. Add the age group to our list. (We can use .append() method used for list: https://www.w3schools.com/python/ref_list_append.asp)

5. Once the loop finishes, assign the completed list as a new column in the DataFrame.

Although there are more advanced ways to do this in Pandas, this approach allows us to combine many of the Python concepts we’ve already learned, including loops, conditionals, and lists.

Notice that we first create an empty Python list.

Then, for each participant’s age, we determine the appropriate age group and add it to the list using append().

Finally, we assign this list as a new column in the DataFrame.

The DataFrame now contains an additional column.

Creating new columns like this is one of the most common data-processing tasks in neuroscience and data science.

Rather than modifying the original data manually, we can generate new variables automatically from existing ones.

Try it yourself¶

Let’s say a neuroscientist wants to see whether there’s an asymmetry between the left and the right hippocampus.

Create a new column called hemisphere_difference that contains:

left_hippocampal_volume - right_hippocampal_volume

Hint:

Unlike the previous example, you don’t need a loop. Try subtracting one column from another directly.

Selecting Multiple Columns¶

Earlier, we selected a single column using:

participant_data["age"]

We can also select multiple columns by passing a list of column names.

Sorting Data¶

Suppose we want to see participants from youngest to oldest.

We can sort the DataFrame by a particular column.

Combining Conditions¶

Sometimes we want multiple conditions.

For example,

participants who

• are older than 35

• AND report nicotine use.

We use & symbol to represent AND.

what about participants who

• are EITHER older than 35

• ORreport nicotine use.

We use | symbol to represent “OR”.

groupby(): Summarizing Data by Group¶

Sometimes we do not want to look at each participant one by one. Instead, we want to answer a group-level question.

For example, suppose we want to ask:

Do participants in different drug-use groups have different average left hippocampal volumes?

To answer that, we need to first separate the DataFrame into groups based on drug_use, and then calculate a summary statistic within each group.

That is exactly what groupby() does.

You can think of groupby() as:

1. split the DataFrame into groups,

2. apply a calculation to each group,

3. combine the results into a summary table.

This is very useful in neuroscience because we often want to compare one group against another, such as healthy controls versus patients, or different diagnostic categories, or different brain region types.

What if you want to know the average left hippocampal volume of young participants who are addicted heroin? What about older adults who are healthy?

To answer this question, we can have two groups: drug use and age group.

Similar to the code above, we simply need to put the two groups as a list into .groupby such as groupby(["group1", "group2"])

In this example:

• groupby("drug_use") splits the table into one group for each drug-use category.

• ["left_hippocampal_volume"] selects the column we want to summarize.

• .mean() calculates the average left hippocampal volume for each group.

Instead of manually filtering one group at a time, Pandas does the work for us.

Try it yourself¶

• Find the mean age for each drug-use group.

• Find the mean asymmetry (difference between left and right hippocampal volume) for drug-use group. (Note, you may have to create a new column that calculate the volume difference)

• Try grouping by a different column if your DataFrame has one.

loc: Selecting Rows and Columns by Label¶

Sometimes we want to select specific rows and columns from a DataFrame.

For example, we may want to keep only participants who use cocaine, and only show their subject ID and age.

This is where loc is useful.

Think of loc as:

“select rows and columns by label”

The general form is:

df.loc[rows, columns]

This lets us say exactly which rows we want and exactly which columns we want.

Here:

• participant_data["drug_use"] == "cocaine" selects only the rows where drug use is cocaine.

• ["subject_id", "age", "drug_use"] selects only those three columns.

So loc is very useful when we want a clean subset of our data.

We can also use : if we want every row or column.

So, for the code above, participant_data["drug_use"] == "cocaine" (which appeared before comma ,) means we select only the ROWS where drug use is cocaine.

The : after the comma means we want to show every column (i.e., no subsetting).

Try it yourself¶

• Use loc to show only participants older than 40.

• Use loc to show only the subject_id and left_hippocampal_volume columns.

• Use loc to show the subject_id and age and age_group of participants with nicotine use.

merge(): Combining Information from Different Tables¶

In real research projects, all of our data are rarely stored in a single table.

For example:

• one table may contain participant demographics,

• another may contain brain measurements,

• another may contain behavioral scores.

Although these tables contain different information, they often describe the same participants.

To perform analyses, we usually want to combine all of this information into a single DataFrame.

This is exactly what merge() allows us to do.

For instance, if we want to merge two tables belonging to the same participants, we find rows that refer to the same participant, then combine the information from both tables into one larger table.

To do this, Pandas needs a column that uniquely identifies each participant, such as subject_id.

For example, if both tables contain:

subject_id

Pandas matches rows with the same subject ID and joins the remaining information together.

Let’s see some examples. Say we have demographics table and brain data imported from our excel sheet belonging to the same participants (see subject_id).

Now each table contains different information about the same participants.

We can combine them using merge():

Here:

• demographics is the first table,

• brain_data is the second table,

• on="subject_id" tells Pandas to match rows using the subject_id column.

The result is a single table that contains all the information together.

Try it yourself¶

Change one subject ID only in one of the tables and see what happens. (i.e., try merging two tables that do not have exactly the same subject IDs.)