Tidy Data In R

Contents

1. Introduction
2. What is tidy data
3. What is messy data
4. Install tidyr
   1. Loading the tidyr package with library()
5. How to tidy data in R - Example

Introduction

All manuals on tidying data start off with these two quotes:

“Happy families are all alike; every unhappy family is unhappy in its own way.” – Leo Tolstoy

“Tidy datasets are all alike, but every messy dataset is messy in its own way.” – Hadley Wickham

Tidy data is a specific organization of data that allows it to be used and manipulated easily for the purpose of data analysis. As we have learned from the introductory quotes, tidy data is uniform in its organization no matter its origin and size. And in contrast, data is always messy in a specific and unique way, making tidying messy data is a unique process in every case. For this reason, tidying data cannot be learned in a single guide, but is best learned from real world experience. The messier the datasets you have, you will become better at tidying data. Nevertheless, having a solid grasp on the basics of data tidying will help you becoming better at this process. Luckily, R has a wide array of packages, the wider tidyverse, and more specifically tidyr that provides powerful tools for tidying data.

What is tidy data?

Any dataset can be described as essentially a collection of values. Each value has a variable and an observation. Variables contain values that measure the same attribute, for example the sepal widths in the iris dataset. All values that are a result of measuring different attributes (i.e. variables) of a single entity are grouped into observations, for example one row in the iris dataset corresponds to all values derived from measuring a single plant.

Here on the iris dataset we can see the basic rules of tidy data. Namely, every column is a variable, every row an observation and every cell of the dataframe a single value.

What is messy data?

As mentioned before, every messy dataset is unique in its own way. However there are some most common themes in messy datasets that should be mentioned. These are:

1. Column headers are values, instead of being variable names
2. Two or more variables stored in a single column
3. Variables are stored in both rows and columns

These issues will be addressed individually in subsequent chapters of this guide.

Install tidyr

Before using the tidyr package it is necessary to install it, since it is not a part of basic R installation. The tidyr package is hosted in the CRAN repository. Any package from the CRAN repository can be installed using the function install.packages().

To install using the following command...

install.packages("tidyr")

Alternatively, we can install the tidyverse which is a collection of packages (which includes tidyr).

install.packages("tidyverse")

Tip:

This is a useful chunk of code to make sure all the packages you want to use are installed if they are not already installed.

Loading the tidyr package with library()

Now that we have the tidyr package installed, we need to load it or the tidyverse in to R workspace. For this purpose we can use the library() function.

How to tidy data in R - Example

Consider the following dataset, relig_income. Would you consider it tidy? At the first look, the dataset (see below) looks clean and it should be easy to manipulate it? But if you look closely, the header row is not right. However looks like the first column in the dataset has the headers. Let us try to convert this wide format to a long format using the pivot_longer function.

With -religion, we imply: Pivot (convert to long) all columns except religion. The names_to argument takes a string as the name of a new column. The values_to argument takes a string as the name of a new column that will contain values. This dataframe is now tidy since it has each column represents a variable and each row represents an observation. The opposite function of pivot_longer() is pivot_wider(), converting long dataframe format to wide.

The same two functions pivot_longer() and pivot_wider() can be used to tidy datasets that have multiple variables in a single column, let us take a look at the World Health Organization data on TB infections. This table has a row for each year and strain of tuberculosis (in 2nd and 3rd column). The remaining columns state the number of cases for a given demographic. For example, m1524 corresponds to males between 15 and 24 years old, and f1524 are females between age 15 and 24. As you can see each of these columns has two variables: sex and age.

So in order to tidy this dataset, we need to perform two tasks. Change all the columns with the number of infections to long format

Let us separate the male and female from the rest of the column name such as new_sp_m_2534. This way the sex and age will be separated into different columns.

Now we should break the sex_and_age column into separate columns. The column sex_and_age holds both the values of the sex and age of each observation. In tidy data they should be in separate columns. Conveniently we can use the separate() function from tidyr.

Now we have a dataset that we can call tidy. However there are still some rows that have NAs. To drop rows with 'NA' in our dataset, we can use the drop_na() function. It will simply drop all rows that have NAs in from our dataframe.

A good way to check if your dataframe has NAs, is with the complete.cases function.