Hypothesis Testing

STAT 20: Introduction to Probability and Statistics

Agenda

• Concept Questions
• Problem Set: Is Yawning Contagious
• Appendix: Hypothesis Testing with infer

Pick a Card Activity

Goals: 1. Identify a situation where people naturally reason like a hypothesis test 2. Map the parts of that situation to the formalism of a hypothesis test 3. Provide some intuition about why .05 is used as the standard threshold

Take two decks of cards and mix them so that you have a single deck that is all of one color, say: black. Bring a prize to class and ham it up that the winner of this simple game will win the prize. All they have to do is pick a red card.

Shuffle the deck, say that’s its fair, then select a random student to be the first contestant. Fan the cards out for them and have them pick a card. When the lose, fein disappointment / surprise, replace the card and move on to the next student. Do this several times. You’ll start hearing groans that the game is rigged. At the point where the groans become clearly articulated (usually around 4 or 5 draws), stop the game, admit to the scam, then go to the board.

Walk through how students started with a previous notion that the deck was fair (null hypothesis: p(red) = .5). As they observed data, they assessed the degree of consistency with that notion by computing the probability of the data. Calculate a table of the binomial probabilities of one black, two blacks, three blacks, on the board. With 4 blacks, the probability is 1/2^4 = .0625 and with 5 blacks its 1/2^5 = .03125. Draw a line between these two probability and identify it as the alpha = .05 threshold that is generally used for “calling bullshit on the null”.

Concept Questions

Which of the following statements below represents claims that correspond to a null hypothesis (as opposed to an alternative hypothesis)?

Hint: try to write them using parameters (statements about means / proportions / etc)

A. King cheetahs on average run the same speed as standard spotted cheetahs.

B. For a particular student, the probability of correctly answering a 5-option multiple choice test is larger than 0.2 (i.e., better than guessing).

C. The mean length of African elephant tusks has changed over the last 100 years.

D. The risk of facial clefts is equal for babies born to mothers who take folic acid supplements compared with those from mothers who do not.

E. Mean birth weight of newborns is dependent on caffeine intake during pregnancy.

F. The probability of getting in a car accident is the same if using a cell phone than if not using a cell phone.

01:00

Correct answers: A, D, F.

Most of these they haven’t seen examples of, so it will be worth laying out the null hypotheses.

We want to understand whether blood thinners are helpful or harmful. We’ll consider both of these possibilities using a two-sided hypothesis test.

Null: Blood thinners do not have an overall survival effect, i.e., the survival proportions are the same in each group.

Alternative: Blood thinners have an impact on survival, either positive or negative, but not zero.

What is your guess at the p-value?

01:00

.3 is probably the best answer but .2 is ok too.

A pharmaceutical company developed a new treatment for eczema and performed a hypothesis test to see if it worked better than the company’s old treatment. The P-value for the test was \(10\%\). Which of the following statements are true?

A. The probability that the null hypothesis is false is \(10\%\).

B. The probability that the null hypothesis is false is \(90\%\).

C. The P-value of about \(10\%\) was computed assuming that the null hypothesis was true.

D. The new drug is significantly better than the old.

E. The alternative hypothesis is 10 times more likely the null.

01:00

Problem Set: Is Yawning Contagious?

20:00

Hypothesis Testing with infer

Appendix

Example: Class Survey

Question: Do beach lovers prefer the warm seasons more than mountain lovers?

Load packages and data.

library(tidyverse)
library(stat20data)

Create new column with just two levels and drop NAs.

class_survey <- class_survey %>%
  mutate(warm_fav = season%in% c("Summer", "Fall")) %>%
  drop_na(beach_or_mtns, warm_fav)

Visualizing the data

What sort of visualization can we use to see the association between these two variables?

class_survey %>%
  select(beach_or_mtns, warm_fav)

# A tibble: 619 × 2
   beach_or_mtns    warm_fav
   <chr>            <lgl>   
 1 At the beach     TRUE    
 2 At the beach     TRUE    
 3 In the mountains FALSE   
 4 At the beach     TRUE    
 5 In the mountains FALSE   
 6 At the beach     TRUE    
 7 In the mountains FALSE   
 8 At the beach     FALSE   
 9 At the beach     FALSE   
10 At the beach     TRUE    
# ℹ 609 more rows

ggplot(class_survey, aes(x = beach_or_mtns,
                         fill = warm_fav)) +
  geom_bar(position = "fill")

Answering with a statistic

Question: Do beach lovers prefer the warm seasons more than mountain lovers?

library(infer)
obs_stat <- class_survey %>%
  specify(response = warm_fav,
          explanatory = beach_or_mtns,
          success = "TRUE") %>%
  calculate(stat = "diff in props")
obs_stat

Response: warm_fav (factor)
Explanatory: beach_or_mtns (factor)
# A tibble: 1 × 1
   stat
  <dbl>
1 0.231

We see the difference is non-zero, but could that just be a product of this particular small sample of data that we have?

Hypothesis Test Pipeline

class_survey %>%
  specify(response = warm_fav,  
          explanatory = beach_or_mtns,  
          success = "TRUE")  

Response: warm_fav (factor)
Explanatory: beach_or_mtns (factor)
# A tibble: 619 × 2
   warm_fav beach_or_mtns   
   <fct>    <fct>           
 1 TRUE     At the beach    
 2 TRUE     At the beach    
 3 FALSE    In the mountains
 4 TRUE     At the beach    
 5 FALSE    In the mountains
 6 TRUE     At the beach    
 7 FALSE    In the mountains
 8 FALSE    At the beach    
 9 FALSE    At the beach    
10 TRUE     At the beach    
# ℹ 609 more rows

Hypothesis Test Pipeline

class_survey %>%
  specify(response = warm_fav,
          explanatory = beach_or_mtns,
          success = "TRUE") %>%
  hypothesize(null = "independence")  

Response: warm_fav (factor)
Explanatory: beach_or_mtns (factor)
Null Hypothesis: independence
# A tibble: 619 × 2
   warm_fav beach_or_mtns   
   <fct>    <fct>           
 1 TRUE     At the beach    
 2 TRUE     At the beach    
 3 FALSE    In the mountains
 4 TRUE     At the beach    
 5 FALSE    In the mountains
 6 TRUE     At the beach    
 7 FALSE    In the mountains
 8 FALSE    At the beach    
 9 FALSE    At the beach    
10 TRUE     At the beach    
# ℹ 609 more rows

Hypothesis Test Pipeline

class_survey %>%
  specify(response = warm_fav,
          explanatory = beach_or_mtns,
          success = "TRUE") %>%
  hypothesize(null = "independence") %>%
  generate(reps = 1,
           type = "permute")

Response: warm_fav (factor)
Explanatory: beach_or_mtns (factor)
Null Hypothesis: independence
# A tibble: 619 × 3
# Groups:   replicate [1]
   warm_fav beach_or_mtns    replicate
   <fct>    <fct>                <int>
 1 TRUE     At the beach             1
 2 TRUE     At the beach             1
 3 FALSE    In the mountains         1
 4 FALSE    At the beach             1
 5 TRUE     In the mountains         1
 6 FALSE    At the beach             1
 7 FALSE    In the mountains         1
 8 TRUE     At the beach             1
 9 TRUE     At the beach             1
10 TRUE     At the beach             1
# ℹ 609 more rows

Hypothesis Test Pipeline

class_survey %>%
  specify(response = warm_fav,
          explanatory = beach_or_mtns,
          success = "TRUE") %>%
  hypothesize(null = "independence") %>%
  generate(reps = 1,
           type = "permute") # a second shuffle

Response: warm_fav (factor)
Explanatory: beach_or_mtns (factor)
Null Hypothesis: independence
# A tibble: 619 × 3
# Groups:   replicate [1]
   warm_fav beach_or_mtns    replicate
   <fct>    <fct>                <int>
 1 FALSE    At the beach             1
 2 TRUE     At the beach             1
 3 FALSE    In the mountains         1
 4 TRUE     At the beach             1
 5 FALSE    In the mountains         1
 6 FALSE    At the beach             1
 7 TRUE     In the mountains         1
 8 TRUE     At the beach             1
 9 TRUE     At the beach             1
10 FALSE    At the beach             1
# ℹ 609 more rows

Hypothesis Test Pipeline

class_survey %>%
  specify(response = warm_fav,
          explanatory = beach_or_mtns,
          success = "TRUE") %>%
  hypothesize(null = "independence") %>%
  generate(reps = 1,
           type = "permute") # a third shuffle

Response: warm_fav (factor)
Explanatory: beach_or_mtns (factor)
Null Hypothesis: independence
# A tibble: 619 × 3
# Groups:   replicate [1]
   warm_fav beach_or_mtns    replicate
   <fct>    <fct>                <int>
 1 TRUE     At the beach             1
 2 TRUE     At the beach             1
 3 FALSE    In the mountains         1
 4 TRUE     At the beach             1
 5 FALSE    In the mountains         1
 6 FALSE    At the beach             1
 7 TRUE     In the mountains         1
 8 TRUE     At the beach             1
 9 FALSE    At the beach             1
10 FALSE    At the beach             1
# ℹ 609 more rows

Hypothesis Test Pipeline

class_survey %>%
  specify(response = warm_fav,
          explanatory = beach_or_mtns,
          success = "TRUE") %>%
  hypothesize(null = "independence") %>%
  generate(reps = 500,  
           type = "permute")

Response: warm_fav (factor)
Explanatory: beach_or_mtns (factor)
Null Hypothesis: independence
# A tibble: 309,500 × 3
# Groups:   replicate [500]
   warm_fav beach_or_mtns    replicate
   <fct>    <fct>                <int>
 1 FALSE    At the beach             1
 2 FALSE    At the beach             1
 3 TRUE     In the mountains         1
 4 TRUE     At the beach             1
 5 TRUE     In the mountains         1
 6 FALSE    At the beach             1
 7 FALSE    In the mountains         1
 8 TRUE     At the beach             1
 9 FALSE    At the beach             1
10 TRUE     At the beach             1
# ℹ 309,490 more rows

Hypothesis Test Pipeline

class_survey %>%
  specify(response = warm_fav,
          explanatory = beach_or_mtns,
          success = "TRUE") %>%
  hypothesize(null = "independence") %>%
  generate(reps = 500,
           type = "permute") %>%
  calculate(stat = "diff in props")  

Response: warm_fav (factor)
Explanatory: beach_or_mtns (factor)
Null Hypothesis: independence
# A tibble: 500 × 2
   replicate     stat
       <int>    <dbl>
 1         1 -0.0245 
 2         2  0.00394
 3         3 -0.0174 
 4         4 -0.0316 
 5         5 -0.0103 
 6         6  0.0111 
 7         7 -0.0103 
 8         8 -0.0103 
 9         9  0.0537 
10        10  0.00394
# ℹ 490 more rows

Hypothesis Test Pipeline

class_survey %>%
  specify(response = warm_fav,
          explanatory = beach_or_mtns,
          success = "TRUE") %>%
  hypothesize(null = "independence") %>%
  generate(reps = 500,
           type = "permute") %>%
  calculate(stat = "diff in props") %>%
  visualize()  

Hypothesis Test Pipeline

class_survey %>%
  specify(response = warm_fav,
          explanatory = beach_or_mtns,
          success = "TRUE") %>%
  hypothesize(null = "independence") %>%
  generate(reps = 500,
           type = "permute") %>%
  calculate(stat = "diff in props") %>%
  visualize() +
  shade_pvalue(obs_stat = obs_stat,  
               direction = "both")  

Hypothesis Test Pipeline

class_survey %>%
  specify(response = warm_fav,
          explanatory = beach_or_mtns,
          success = "TRUE") %>%
  hypothesize(null = "independence") %>%
  generate(reps = 500,
           type = "permute") %>%
  calculate(stat = "diff in props") %>%
  get_p_value(obs_stat = obs_stat,  
               direction = "both")  

# A tibble: 1 × 1
  p_value
    <dbl>
1       0