Cumulative Distribution Plots for Frequency Data in R

R has some great tools for generating and plotting cumulative distribution functions. However, they are suited for raw data, not when the data is summarized in frequency counts. However, reducing to frequency counts is often necessary when processing data at the scale of tens of gigabytes or more. Here I describe a convenient two-liner in R to plot CDFs in R based on aggregated frequency data.

For example, suppose you want to analyze the number of times people exercise in a month. One option would be to work with a data table consisting of the number of active days for each person.

library(dplyr)
library(ggplot2)
running_log <- data.frame(person_id=1:10000, 
                          num_active_days=rpois(10000, lambda=15)) %>% 
  transmute(num_active_days=ifelse(num_active_days<=30, num_active_days, 30))
tbl_df(running_log)
## Source: local data frame [10,000 x 1]
## 
##    num_active_days
## 1               14
## 2               12
## 3               13
## 4               12
## 5               17
## 6               18
## 7               16
## 8               18
## 9               16
## 10              11
## ..             ...

CDFs from raw data

Using built-in function ecdf, it is easy to generate the cumulative distribution function for such data.

cdf_fun <- ecdf(running_log$num_active_days)
plot(cdf_fun, xlab="Number of active days", 
     ylab="Number of people",main="")

For those interested, ggplot provides a function stat_ecdf, which makes this process even simpler.

ggplot(running_log, aes(x=num_active_days)) + stat_ecdf()

CDFs with processed, frequency data

Let us now compress our data by precomputing the frequency of people active on each day.

frequency_counts <- running_log %>% group_by(num_active_days) %>%
  summarize(num_people=n())
frequency_counts
## Source: local data frame [28 x 2]
## 
##    num_active_days num_people
## 1                2          1
## 2                4          4
## 3                5         18
## 4                6         55
## 5                7        103
## 6                8        188
## 7                9        328
## 8               10        522
## 9               11        650
## 10              12        847
## ..             ...        ...

Note how this is a 30x2 table at worst, instead of 10,000x2 entries that we started with. Calculating the empirical cumulative distribution involves sorting the data and using cumsum to calculate cumulative sums, then plot using the step function as before.

cumulative_frequencies <- frequency_counts %>% 
  arrange(num_active_days) %>% 
  mutate(cum_frequency=cumsum(num_people))
cumulative_frequencies
## Source: local data frame [28 x 3]
## 
##    num_active_days num_people cum_frequency
## 1                2          1             1
## 2                4          4             5
## 3                5         18            23
## 4                6         55            78
## 5                7        103           181
## 6                8        188           369
## 7                9        328           697
## 8               10        522          1219
## 9               11        650          1869
## 10              12        847          2716
## ..             ...        ...           ...
ggplot(cumulative_frequencies, aes(x=num_active_days, y=cum_frequency)) +
 geom_step() + xlab("Number of active days") + ylab("Number of people")

Bonus: plotting multiple CDFs together

Now, suppose you wanted to break down your analysis by the type of activity. Let us assume that there are two types of activities: running and cycling.

cycling_log <- data.frame(person_id=1:10000, 
                          num_active_days=rpois(10000, lambda=15)) %>% 
  transmute(num_active_days=ifelse(num_active_days<=30, num_active_days, 30))
activity_log <- rbind(running_log %>% mutate(activity_type="running"), 
                      cycling_log %>% mutate(activity_type="cycling")) 
frequency_counts = activity_log %>%
  group_by(num_active_days, activity_type) %>%
  summarize(num_people=n())
frequency_counts
## Source: local data frame [56 x 3]
## Groups: num_active_days
## 
##    num_active_days activity_type num_people
## 1                2       running          1
## 2                3       cycling          1
## 3                4       cycling          8
## 4                4       running          4
## 5                5       cycling         22
## 6                5       running         18
## 7                6       cycling         60
## 8                6       running         55
## 9                7       cycling        100
## 10               7       running        103
## ..             ...           ...        ...

The same code can be used to generate the CDF, with an additional group_by.

cumulative_frequencies <- frequency_counts %>% group_by(activity_type) %>% 
  mutate(cum_frequency=cumsum(num_people))
cumulative_frequencies
## Source: local data frame [56 x 4]
## Groups: activity_type
## 
##    num_active_days activity_type num_people cum_frequency
## 1                2       running          1             1
## 2                3       cycling          1             1
## 3                4       cycling          8             9
## 4                4       running          4             5
## 5                5       cycling         22            31
## 6                5       running         18            23
## 7                6       cycling         60            91
## 8                6       running         55            78
## 9                7       cycling        100           191
## 10               7       running        103           181
## ..             ...           ...        ...           ...
ggplot(cumulative_frequencies, aes(x=num_active_days, y=cum_frequency)) + 
  geom_step() + facet_wrap(~activity_type) +
  xlab("Number of active days") + ylab("Number of people")