02_tidy_data1.qmd

# Tidy data analysis I

The [`tidyverse`](https://www.tidyverse.org/) is a suite of packages that streamline data analysis in R. After installing the `tidyverse` with `install.packages("tidyverse")` (see the previous module), you can load it with:

```{r}
library(tidyverse)
```

::: callout-tip
Upon loading, the `tidyverse` prints a message like the one above. Notice that multiple packages (the constituent elements of the "suite") are actually loaded. For instance, `dplyr` and `tidyr` help with data wrangling and transformation, while `ggplot2` allows us to draw plots. In most cases, one just loads the `tidyverse` and forgets about these details, as the constituent packages work together nicely.
:::

Throughout this module, we will use `tidyverse` functions to load, wrangle, and visualize real data.

## Loading data

Throughout this module we will work with a dataset of senators during the Trump presidency, which was adapted from [FiveThirtyEight (2021)](https://projects.fivethirtyeight.com/congress-trump-score/).

We have stored the dataset in .csv format under the `data/` subfolder. Loading it into R is simple (notice that we need to assign it to an object):

```{r}
trump_scores <- read_csv("data/trump_scores_538.csv")
```

```{r}
trump_scores
```

Let's review the dataset's columns:

-   `bioguide`: A unique ID for each politician, from the Congress Bioguide.
-   `last_name`
-   `state`
-   `party`
-   `num_votes`: Number of votes for which data was available.
-   `agree`: Proportion (0-1) of votes in which the senator voted in agreement with Trump.
-   `agree_pred`: Predicted proportion of vote agreement, calculated using Trump's margin (see next variable).
-   `margin_trump`: Margin of victory (percentage points) of Trump in the senator's state.

We can inspect our data by using the interface above. An alternative is to run the command `View(trump_scores)` or click on the object in RStudio's environment panel (in the top-right section).

Do you have any questions about the data?

By the way, the `tidyverse` works amazingly with *tidy data*. If you can get your data to this format (and we will see ways to do this), your life will be much easier:

::: {layout-nrow="2"}
![](images/tidy_data.jpg)

![Source: Illustrations from the [Openscapes](https://www.openscapes.org/) blog [*Tidy Data for reproducibility, efficiency, and collaboration*](https://www.openscapes.org/blog/2020/10/12/tidy-data/) by Julia Lowndes and Allison Horst.](images/tidy_data2.jpg)
:::

## Wrangling data with `dplyr`

We often need to modify data to conduct our analyses, e.g., creating columns, filtering rows, etc. In the `tidyverse`, these operations are conducted with multiple *verbs*, which we will review now.

### Selecting columns

We can select specific columns in our dataset with the `select()` function. All `dplyr` wrangling verbs take a data frame as their first argument---in this case, the columns we want to select are the other arguments.

```{r}
select(trump_scores, last_name, party)
```

This is a good moment to talk about "pipes." Notice how the code below produces the same output as the one above, but with a slightly different syntax. Pipes (`|>`) "kick" the object on the left of the pipe to the first argument of the function on the right. One can read pipes as "then," so the code below can be read as "take `trump_scores`, then select the columns `last_name` and `party`." Pipes are very useful to *chain multiple operations*, as we will see in a moment.

```{r}
trump_scores |> 
  select(last_name, party)
```

::: callout-tip
You can insert a pipe with the `Cmd/Ctrl + Shift + M` shortcut. If you have not changed the default RStudio settings, an "old" pipe (`%>%`) might appear. While most of the functionality is the same, the `|>` "new" pipes are more readable and don't need any extra packages (to use `%>%` you need the `tidyverse` or one of its packages). You can change this RStudio option in `Tools > Global Options > Code > Use native pipe operator`. Make sure to check the other suggested settings in our [Setup module](/00_setup.html)!
:::

Going back to selecting columns, you can select ranges:

```{r}
trump_scores |> 
  select(bioguide:party)
```

You can also **de**select columns using a minus sign:

```{r}
trump_scores |> 
  select(-last_name)
```

And use a few helper functions, like `matches()`:

```{r}
trump_scores |> 
  select(last_name, matches("agree"))
```

Or `everything()`, which we usually use to reorder columns:

```{r}
trump_scores |> 
  select(last_name, everything())
```

::: callout-tip
Notice that all these commands have not edited our existent objects---they have just printed the requested outputs to the screen. In order to modify objects, you need to use the assignment operator (`<-`). For example:

```{r}
trump_scores_reduced <- trump_scores |> 
  select(last_name, matches("agree"))
```

```{r}
trump_scores_reduced
```
:::

::: callout-note
#### Exercise

Select the variables `last_name`, `party`, `num_votes`, and `agree` from the data frame. Your code:
:::

### Renaming columns

We can use the `rename()` function to rename columns, with the syntax `new_name = old_name`. For example:

```{r}
trump_scores |> 
  rename(prop_agree = agree, prop_agree_pred = agree_pred)
```

This is a good occasion to show how pipes allow us to chain operations. How do we read the following code out loud? (Remember that pipes are read as "then").

```{r}
trump_scores |> 
  select(last_name, matches("agree")) |> 
  rename(prop_agree = agree, prop_agree_pred = agree_pred)
```

### Creating columns

It is common to want to create columns, based on existing ones. We can use `mutate()` to do so. For example, we could want our main variables of interest in terms of percentages instead of proportions:

```{r}
trump_scores |> 
  select(last_name, agree, agree_pred) |> # select just for clarity
  mutate(pct_agree = 100 * agree,
         pct_agree_pred = 100 * agree_pred)
```

We can also use multiple columns for creating a new one. For example, let's retrieve the total *number* of votes in which the senator agreed with Trump:

```{r}
trump_scores |> 
  select(last_name, num_votes, agree) |> # select just for clarity
  mutate(num_votes_agree = num_votes * agree)
```

### Filtering rows

Another common operation is to filter rows based on logical conditions. We can do so with the `filter()` function. For example, we can filter to only get Democrats:

```{r}
trump_scores |> 
  filter(party == "D")
```

Notice that `==` here is a *logical operator*, read as "is equal to." So our full chain of operations says the following: take `trump_scores`, then filter it to get rows where party is equal to "D".

There are other logical operators:

| Logical operator | Meaning                       |
|------------------|-------------------------------|
| `==`             | "is equal to"                 |
| `!=`             | "is not equal to"             |
| `>`              | "is greater than"             |
| `<`              | "is less than"                |
| `>=`             | "is greater than or equal to" |
| `<=`             | "is less than or equal to"    |
| `%in%`           | "is contained in"             |
| `&`              | "and" (intersection)          |
| `|`              | "or" (union)                  |

Let's see a couple of other examples.

```{r}
trump_scores |> 
  filter(agree > 0.5)
```

```{r}
trump_scores |> 
  filter(state %in% c("CA", "TX"))
```

```{r}
trump_scores |> 
  filter(state == "WV" & party == "D")
```

::: callout-note
#### Exercise

1.  Add a new column to the data frame, called `diff_agree`, which subtracts `agree` and `agree_pred`. How would you create `abs_diff_agree`, defined as the absolute value of `diff_agree`? Your code:

2.  Filter the data frame to only get senators for which we have information on fewer than (or equal to) five votes. Your code:

3.  Filter the data frame to only get Democrats who agreed with Trump in at least 30% of votes. Your code:
:::

### Ordering rows

The `arrange()` function allows us to order rows according to values. For example, let's order based on the `agree` variable:

```{r}
trump_scores |> 
  arrange(agree)
```

Maybe we only want senators with more than a few data points. Remember that we can chain operations:

```{r}
trump_scores |> 
  filter(num_votes >= 10) |> 
  arrange(agree)
```

By default, `arrange()` uses increasing order (like `sort()`). To use decreasing order, add a minus sign:

```{r}
trump_scores |> 
  filter(num_votes >= 10) |> 
  arrange(-agree)
```

You can also order rows by more than one variable. What this does is to order by the first variable, and resolve any ties by ordering by the second variable (and so forth if you have more than two ordering variables). For example, let's first order our data frame by party, and then within party order by agreement with Trump:

```{r}
trump_scores |> 
  filter(num_votes >= 10) |> 
  arrange(party, agree)
```

::: callout-note
#### Exercise

Arrange the data by `diff_agree`, the difference between agreement and predicted agreement with Trump. (You should have code on how to create this variable from the last exercise). Your code:
:::

### Summarizing data

`dplyr` makes summarizing data a breeze using the `summarize()` function:

```{r}
trump_scores |> 
  summarize(mean_agree = mean(agree),
            mean_agree_pred = mean(agree_pred))
```

To make summaries, we can use any function that takes a vector and returns one value. Another example:

```{r}
trump_scores |> 
  filter(num_votes >= 5) |> # to filter out senators with few data points
  summarize(max_agree = max(agree),
            min_agree = min(agree))
```

*Grouped summaries* allow us to disaggregate summaries according to other variables (usually categorical):

```{r}
trump_scores |> 
  filter(num_votes >= 5) |> # to filter out senators with few data points
  summarize(mean_agree = mean(agree),
            max_agree = max(agree),
            min_agree = min(agree),
            .by = party) # to group by party
```

::: callout-note
#### Exercise

For each party, find the maximum absolute difference in agreement with Trump (the `abs_diff_agree` variable from before).
:::

### Overview

| Function               | Purpose                      |
|------------------------|------------------------------|
| `select()`             | Select columns               |
| `rename()`             | Rename columns               |
| `mutate()`             | Creating columns             |
| `filter()`             | Filtering rows               |
| `arrange()`            | Ordering rows                |
| `summarize()`          | Summarizing data             |
| `summarize(…, .by = )` | Summarizing data (by groups) |

## Visualizing data with `ggplot2`

`ggplot2` is the package in charge of data visualization in the `tidyverse`. It is extremely flexible and allows us to draw bar plots, box plots, histograms, scatter plots, and many other types of plots (see [examples at R Charts](https://r-charts.com/ggplot2/)).

Throughout this module we will use a subset of our data frame, which only includes senators with more than a few data points:

```{r}
trump_scores_ss <- trump_scores |> 
  filter(num_votes >= 10)
```

The `ggplot2` syntax provides a unifying interface (the "grammar of graphics" or "gg") for drawing all different types of plots. One draws plots by adding different "layers," and the core code always includes the following:

-   A `ggplot()` command with a `data =` argument specifying a data frame and a `mapping = aes()` argument specifying "aesthetic mappings," i.e., how we want to use the columns in the data frame in the plot (for example, in the x-axis, as color, etc.).

-   "geoms," such as `geom_bar()` or `geom_point()`, specifying what to draw on the plot.

So *all* `ggplot2` commands will have at least three elements: data, aesthetic mappings, and geoms.

### Univariate plots: categorical

Let's see an example of a bar plot with a categorical variable:

```{r}
ggplot(data = trump_scores_ss, mapping = aes(x = party)) +
  geom_bar()
```

::: callout-tip
As with any other function, we can drop the argument names if we specify the argument values in order. This is common in `ggplot2` code:

```{r}
ggplot(trump_scores_ss, aes(x = party)) +
  geom_bar()
```
:::

Notice how `geom_bar()` automatically computes the number of observations in each category for us. Sometimes we want to use numbers in our data frame as part of a bar plot. Here we can use the `geom_col()` geom specifying both `x` and `y` aesthetic mappings, in which is sometimes called a "column plot:"

```{r}
ggplot(trump_scores_ss |> filter(state == "ME"),
       aes(x = last_name, y = agree)) +
  geom_col()
```

::: callout-note
## Exercise

Draw a column plot with the agreement with Trump of Bernie Sanders and Ted Cruz. What happens if you use `last_name` as the `y` aesthetic mapping and `agree` in the `x` aesthetic mapping? Your code:
:::

A common use of `geom_col()` is to create "ranking plots." For example, who are the senators with highest agreement with Trump? We can start with something like this:

```{r}
ggplot(trump_scores_ss,
       aes(x = agree, y = last_name)) +
  geom_col()
```

We might want to (1) select the top 10 observations and (2) order the bars according to the `agree` values. We can do these operations with `slice_max()` and `fct_reorder()`, as shown below:

```{r}
ggplot(trump_scores_ss |> slice_max(agree, n = 10),
       aes(x = agree, y = fct_reorder(last_name, agree))) +
  geom_col()
```

We can also plot the senators with the *lowest* agreement with Trump using `slice_min()` and `fct_reorder()` with a minus sign in the ordering variable:

```{r}
ggplot(trump_scores_ss |> slice_min(agree, n = 10),
       aes(x = agree, y = fct_reorder(last_name, -agree))) +
  geom_col()
```

### Univariate plots: numerical

We can draw a histogram with `geom_histogram()`:

```{r}
ggplot(trump_scores_ss, aes(x = agree)) +
  geom_histogram()
```

Notice the warning message above. It's telling us that, by default, `geom_histogram()` will draw 30 bins. Sometimes we want to modify this behavior. The following code has some common options for `geom_histogram()` and their explanations:

```{r}
ggplot(trump_scores_ss, aes(x = agree)) +
  geom_histogram(binwidth = 0.05,   # draw bins every 0.05 jumps in x
                 boundary = 0,      # don't shift bins to integers
                 closed   = "left") # close bins on the left
```

Sometimes we want to manually alter a scale. This is accomplished with the `scale_*()` family of `ggplot2` functions. Here we use the `scale_x_continuous()` function to make the x-axis go from 0 to 1:

```{r}
ggplot(trump_scores_ss, aes(x = agree)) +
  geom_histogram(binwidth = 0.05, boundary = 0, closed   = "left") +   
  scale_x_continuous(limits = c(0, 1))
```

Adding the `fill` aesthetic mapping to a histogram will divide it according to a categorical variable. This is actually a bivariate plot!

```{r}
ggplot(trump_scores_ss, aes(x = agree, fill = party)) +
  geom_histogram(binwidth = 0.05, boundary = 0, closed   = "left") +   
  scale_x_continuous(limits = c(0, 1)) +
  # change default colors:
  scale_fill_manual(values = c("D" = "blue", "R" = "red"))
```

### Bivariate plots

Another common bivariate plot for categorical and numerical variables is the grouped box plot:

```{r}
ggplot(trump_scores_ss, aes(x = agree, y = party)) +
  geom_boxplot() +
  scale_x_continuous(limits = c(0, 1)) # same change as before
```

For bivariate plots of numerical variables, scatter plots are made with `geom_point()`:

```{r}
ggplot(trump_scores_ss, aes(x = margin_trump, y = agree)) +
  geom_point()
```

We can add the `color` aesthetic mapping to add a third variable:

```{r}
ggplot(trump_scores_ss, aes(x = margin_trump, y = agree, color = party)) +
  geom_point() +
  scale_color_manual(values = c("D" = "blue", "R" = "red"))
```

Let's finish our plot with the `labs()` function, which allows us to add labels to our aesthetic mappings, as well as titles and notes:

```{r}
ggplot(trump_scores, aes(x = margin_trump, y = agree, color = party)) +
  geom_point() +
  scale_color_manual(values = c("D" = "blue", "R" = "red")) +
  labs(x = "Trump margin in the senator's state (p.p.)",
       y = "Votes in agreement with Trump (prop.)",
       color = "Party",
       title = "Relationship between Trump margins and senators' votes",
       caption = "Data source: FiveThirtyEight (2021)")
```

We will review a few more customization options, including text labels and facets, in a subsequent module.