Lab 2: Visualization and Simpson’s Paradox¶

Goals:

• Load and inspect real admissions data

• Compare overall and department-level admission rates by gender

• Visualize Simpson’s Paradox

Dataset Source: The dataset used in this notebook is a cleaned version of the original UC Berkeley 1973 graduate admissions data, processed by the University of Illinois at Urbana-Champaign.

References:

• Freedman, Pisani, and Purves. “Sex Bias in Graduate Admissions.” Statistics (1978) Chapter 2. pp. 17-20.

• Bickel, Hammel, and O’Connell. “Sex Bias in Graduate Admissions: Data from Berkeley” Science vol. 187 (1975) pp. 398-404. DOI

• Wikipedia

[Discussion] Data Talk¶

[Discussion] Visualization Talk¶

Introduction to Jupyter Notebooks¶

[Tutorial] Python the Calculator¶

The two building blocks of Python code are expressions and statements. An expression is a piece of code that

• is self-contained, meaning it would make sense to write it on a line by itself, and

• usually evaluates to a value.

The below expression evaluates to 3:

One important type of expression is the call expression. A call expression begins with the name of a function and is followed by the argument(s) of that function in parentheses. The function returns some value, based on its arguments. For example the function abs returns the absolute value of its argument. The below call expression therefore also evaluates to 3:

A statement is a whole line of code. Some statements are just expressions, like the ones above.

Other statements make something happen rather than having a value. For example, an assignment statement assigns a value to a name.

A good way to think about this is that we’re evaluating the right-hand side of the equals sign and assigning it to the left-hand side. Here are some assignment statements:

height = 1.3
the_number_five = abs(-5)
absolute_height_difference = abs(height - 1.688)

An important idea in programming is that large, interesting things can be built by combining many simple, uninteresting things. The key to understanding a complicated piece of code is breaking it down into its simple components.

Part 1: Table Transformations¶

[Tutorial] Load the data¶

Load in the file below by running the cell below.

We can transform this table in many ways by performing an operation on its rows. Let’s say we wanted all the applicants who were admitted. We can use the table method where:

[Tutorial] where and num_rows¶

Now let’s dive into the details a bit more. where takes 2 arguments:

1. The name of a column. where finds rows where that column’s values meet some criterion.

2. A predicate that describes the criterion that the column needs to meet.

The expression above found all rows whose "Admission" value matched "Admitted". where returns a table that’s a copy of the original table, but with only the rows that meet the given predicate.

To count the number of rows in a table, we can use num_rows. The table property num_rows returns the number of rows in a table. (A “property” is just a method that doesn’t need to be called by adding parentheses.)

Note: The terms “method” and “function” are technically not the same thing, but for the purposes of this course, we will use them interchangeably.

The expression below does many things. Try talking it through with a neighbor, then expand the explanation below.

Test your understanding! What do you think the “double” where does below?

Now that we’ve built these values, we can use them in mathematical expressions.

Question 1: Compute Admit Rate¶

Fill in the ... with the given named values

• total_f

• total_m

• admit_f, and

• admit_m

to compute the 1973 admission rates (admit rates, in %) for

• women (admit_rate_f) and

• men (admit_rate_m).

NOTE: In Python, / is division, and * is multiplication. How might we compute percentages (between 0 and 100)?

Note 2: If you are encountering an error, raise your hand and we will come by to help.

[Tutorial] .pivot (Berkeley admissions)¶

Let’s practice one classical table transformation. The pivot method creates a two-way table showing the intersection of two columns.

pivot(columns, rows, values=None, collect=None)

• columns: unique values become the new column headers

• rows: unique values become the row labels

• if you do not pass values and collect, each cell is a count of rows

Example: For this dataset,

• columns='Admission' gives columns like Admitted and Rejected

• rows='Gender' gives rows for F and M

So running the below cell builds a table admit_vs_reject that answers:

For each gender, how many applications were admitted vs. rejected?

Question 2: Build count_by_dept¶

Using the cal_data table loaded earlier and the table method from the section above, complete the cell below to create a table named count_by_dept that shows the number (count) of applicants by Major and Gender. Your count_by_dept table should look like the following:

IMPORTANT NOTE: Use the correct column names as strings. The string of the column names that you want to pass into as the arguments must have the exact match in capitalization, spelling, spaces, etc.

Question 3: Build admit_by_dept¶

Recall what we said earlier:

Since the value of the expression cal_data.where("Admission", "Admitted") is itself a table, you can just call a table method on it.

Now, we know a second table method: pivot...!

Put together what you know about where and pivot to use cal_data to build the admit_by_dept table that shows the number (count) of admits by major and gender. Your admit_by_dept table should look like the following:

We will reuse your newly created count_by_dept and admit_by_dept tables later in Part 3’s visualizations with bar charts. For now, you are done with this section, so give yourself a pat on the back!

For those curious, check our the full list of table transformations available in the datascience package.

We’ll circle back for some discussion before we revisit the next part.

Part 3: Bar Charts¶

The “proper pooling” section of the Simpson’s paradox visualization encodes four variables:

a. Total admits (and rejects) by gender b. Admits (and rejects) by department c. Total admit rate by gender d. Admit rate by department

These can all be visualized with bar charts (one per variable). Let’s tackle (a) and (c) first.

[Tutorial].barh (Berkeley admissions)¶

Use .barh to make a horizontal bar chart or .bar to make a vertical bar chart from a table with categories.

(horizontal) table.barh(category_column, optional_numeric_columns)
(vertical) table.bar(category_column, optional_numeric_columns)

• first argument: the category labels (y-axis)

• if no numeric columns are specified, barh plots all numeric columns

(a) [Tutorial] Plot Total Admits by Gender¶

Earlier, we built the admit_vs_reject table for you:

We can use the admit_vs_reject table to plot a (horizontal) bar chart with one horizontal bar for each gender, disaggregated by accepted vs rejected counts.

• category column: Gender

• numeric columns: Admitted, Rejected

(b) Question 4: Plot Acceptance Counts by Department¶

Earlier, you built the admit_by_dept table:

Use the admit_by_dept table to plot the below bar chart.

[Tutorial] Arrays and Tables¶

How can we add columns to tables? In the datascience library, we can make new columns by creating an array, which is a list of values, in order. Run the cell below and notice that admit_rate is an array with two elements.

An array is not a table, because it doesn’t have labeled columns.

We can add new columns to tables by calling with_columns on that table. The with_columns method requires two arguments:

1. The name of the first column as a string

2. An array of values to put in the first column

3. (optional) The name of the second column as a string

4. (optional) An array of values to put in the second column

5. (optional) etc.

For example, the admit_vs_reject_rate table below adds one column to admit_vs_reject; this column is labeled "Admit Rate" and has the values in the admit_rate array.

(c) [Tutorial] Plot Acceptance Rates by Gender¶

We can then use our newly created admit_vs_reject_rate to plot a bar chart of the total admit rates, not the counts as before.

[Tutorial] Array Arithmetic¶

Now we are going to do something really cool: array arithmetic. This is when we use arrays themselves in expressions. When we do so, arithmetic is applied to each element individually.

The cell below illustrates how to use arrays to convert multiple temperatures in Celsius (in the highs array) to Farenheit:

Below, Table() returns an “empty” table, from which we can add new columns, provided that the number of elements in each column are equal.

Above, we showed how to make columns out of arrays to add to existing tables or make new tables. But what about going the other way?

The column method takes a column label and returns a copy of the values of that column as an array (which, again, is a list of the values in that column, in order). For example, the below cell gets the “M” column in count_by_dept and stores it in m_by_dept:

Since the column method returns arrays, we can now perform array arithmetic “on columns.” The below cell adds the number of male and female applicants by department and stores it in total_by_dept:

Before we introduce the next question, here is a refresher on two tables you have malready built:

Question 5: Compute Admit Rate by Department¶

Use the two tables count_by_dept and admit_by_dept with the column method to compute admit_f_by_dept_rate and admit_m_by_dept_rate to the admission percentages (between 0 and 100) of female and males to each department, respectively.

If all goes well, then running the cell below should correctly produce admit_by_dept_rate, a table of rates that match what you observed in the Simpson’s website demo.

(d) Question 6: Plot Admit Rate by Department¶

Finally, in the cell below, choose one of the many many tables that we have built over the entire notebook to plot the admit rate by department.

Submission¶

Make sure you have run all cells in your notebook in order before running the cell below, so that all images/graphs appear in the output. The cell below will generate a zip file for you to submit. Please save before exporting!