(R) VIF() and COR()

Revisiting multiple regression for the purpose of establishing a greater understanding of the topic, and additionally, to foster a more efficient application of the regression model; in this entry we will review functions and statistical concepts which were overlooked in the prior article pertaining to the subject matter.

Variance Influence Factor or VIF():

What is VIF()? According to Wikipedia, it is defined as such: “The variance inflation factor (VIF) quantifies the severity of multicollinearity in an ordinary least squares regression analysis.”

To define VIF() in layman’s terms, The Variance Influence Factor is a method for weighing each variable’s coefficient of determination (R-Squared), against all other variables within a multiple regression equation. It’s really as simple as that, but none of this will make sense without an example.

Example:

Consider the following numerical sets:

w <- c(23, 42, 55, 16, 24, 27, 24, 15, 23, 85)
x <- c(27, 34, 22, 30, 17, 32, 25, 34, 46, 37)
y <- c(70, 80, 73, 77, 60, 93, 85, 72, 90, 85)
z <- c(13, 22, 18, 30, 15, 17, 20, 11, 20, 25)

Let’s utilize these values to create a multiple regression model:

lm.multiregressw <- (lm(w ~ x + y + z))

Now let’s take a look at that model:

summary(lm.multiregressw)

Call:
lm(formula = w ~ x + y + z)

Residuals:
      Min      1Q       Median 3Q    Max
-29.701 -11.020 -4.462 3.465 44.108

Coefficients:
           Estimate Std. Error t value Pr(>|t|)
(Intercept) -1.40034 68.57307 -0.020 0.984
x            -0.08981 1.41549 -0.063 0.951
y             0.19770 1.20528 0.164 0.875
z            1.15241 1.60560 0.718 0.500

Residual standard error: 25.18 on 6 degrees of freedom
Multiple R-squared: 0.1109, Adjusted R-squared: -0.3336
F-statistic: 0.2495 on 3 and 6 DF, p-value: 0.8591

Given the Multiple R-squared value of this output, we can assume that this model is pretty awful. Regardless of such, our model could be effected by a phenomenon known as multiple collinearity. What this means, is that the variables could be interacting with each other, and thus, disrupting our model from providing accurate results. To prevent this from happening, we can measure the VIF() for each independent variable within the equation as it pertains to each other independent variable. If we were to do this manually, the code would resemble:

lm.multiregressx <- (lm(x ~ y + z )) # .4782 #

lm.multiregressy <- (lm(y ~ x + z )) # .5249 #

lm.multiregressz <- (lm(z ~ y + x )) # .1488 #

With each output, we would note the Multiple R-Squared value. I have provided those values to the right of each code line above. To then individually calculate the VIF(), we would utilize the following code for each R-Squared variable.

1 / (1 - .4282) # 1.916 #

1 / (1 - .5249) # 2.105 #

1 / (1 - .1488) # 1.175 #

This produces the values to the right of each code line. These values are the VIF() or Variance Influence Factor.

An easier way to derive the VIF() is to install the R package, “car”. Once “car” is installed, you can execute the following command:

vif(lm.multiregressw)

Which provides the output:

x y z
1.916437 2.104645 1.174747 

Typically, most data scientists consider values of VIF() which are either over 5 or 10 (depending on sensitivity), to indicate that a variable ought to be removed. If you do plan on removing a variable from your model for such reasons, remove one variable at a time, as the removal of a single variable will effect subsequent VIF() measurements.

(Pearson) Coefficient of Correlation or cor():

Another tool at your disposal is the function cor(). Cor() allows the user to derive the coefficient of correlation ( r ) from numerical sets. For example:

x <- c(27, 34, 22, 30, 17, 32, 25, 34, 46, 37)
y <- c(70, 80, 73, 77, 60, 93, 85, 72, 90, 85)

cor(x,y)

[1] 0.6914018

As exciting as this seems, there is actually a better use for this function. For example, if you wanted to derive the coefficient of correlation as one variable as it pertains to others, you could use the following code lines to build a correlation matrix.

# Set values: #

w <- c(23, 42, 55, 16, 24, 27, 24, 15, 23, 85)
x <- c(27, 34, 22, 30, 17, 32, 25, 34, 46, 37)
y <- c(70, 80, 73, 77, 60, 93, 85, 72, 90, 85)
z <- c(13, 22, 18, 30, 15, 17, 20, 11, 20, 25)

# Create a data frame #

dframe <- data.frame(w,x,y,z)

# Create a correlation matrix #

correlationmatrix <- cor(dframe)

The output will resemble:

              w               x                 y             z
w 1.0000000 0.1057911 0.1836205 0.3261470
x 0.1057911 1.0000000 0.6914018 0.2546853
y 0.1836205 0.6914018 1.0000000 0.3853427
z 0.3261470 0.2546853 0.3853427 1.0000000

In the next article, we will again review the F-Statistic in preparation for a discussion pertaining to the concept of ANOVA. Until then, hang in there statistics fans!

(R) Confidence Interval of The Mean (T)

Below is an example which illustrates a statistical concept:

# A new machine is purchased by a pastry shop which will be utilized to fill the insides of various pastries. A test run is commissioned in which the following proportions of jelly are inserted into each pastry. We are tasked with finding the 90% confidence interval for the mean filling amount from this process. #

s <- c(.28,.25,.22,.20,.33,.20)

w <- sd(s) / sqrt(length(s))

w

[1] 0.02092314 

degrees <- length(s) - 1

degrees

 [1] 5 

t <- abs(qt(0.05, degrees))

t 

[1] 2.015048 

m <- (t * w)

m

[1] 0.04216114

mean(s) + c(-m, m)

[1] 0.2045055 0.2888278 

# We can state, with 90% confidence, that the machine will insert into pastries proportions of filling between .205 and .289. In the above execise, a new concept is introduced in step 4. The abs() function, in conjunction with the qt() function, allows you to find Critical-T values through the utilization of R without having to refer to a textbook. A few examples as to how this function can be utilized are listed below. #

# T VALUES (Confidence / Degrees of Freedom ) #
abs(qt(0.25, 40)) # 75% confidence, 40 degrees of freedom, 1 sided (same as qt(0.75, 40)) #
abs(qt(0.01, 40)) # 99% confidence, 40 degrees of freedom, 1 sided (same as qt(0.99, 40)) #
abs(qt(0.01/2, 40)) # 99% confidence, 40 degrees of freedom, 2 sided #

I hope that you found this abbreviated article to be interesting and helpful. Until next time, I'll see you later Data Heads!

(R) T-Tests and The T-Statistics

Welcome to the 60th article of Confessions of a Data Scientist. In this article we will be reviewing an incredibly important statistical topic: The T-Statistic.

The T-Statistic has many usages, but before we can begin to discuss these methods, I will briefly explain the conceptual significance of the subject matter, and when it is appropriate to utilize the T-Statistic.

The T-Statistic was created by a man named William Sealy Gosset. This concept was created by Gosset while he was working as a chemist at The Guinness Brewing Company. “Student”, was the anonymous name that was chosen by Gosset, which he utilized when publishing his findings. The reason for this anonymity, is that Guinness forbade its chemists from publishing their research. Gosset originally devised the T-Statistic and the concept of the “T-Test” as methods to measure the quality of stout. *

When to utilize The T-Statistic:

1. When the population standard deviation is unknown. (This is almost always the case).
2. When the current sample size is less than or equal to 30.

It is important that I mention now, that a common misconception is that the t-distribution assumes normality. This is not the case. Though, the t-distribution does begin to more closely resemble a normal distribution as the degrees of freedom approach 100. **

There are many uses for the t-distribution, some of which will be covered in future articles. However, the most commonly applied methods are: The One Sample T-Test, The Two Sample T-Test, and The Paired Test.

One Sample T-Test

This test is utilized to compare a sample mean to a specific value, it is used when the dependent variable is measured at the interval or ratio level.

To apply this test, you will need to define the following variables:

The size of the sample (N).
The applicable confidence interval, and the corresponding alpha.
The standard deviation of the sample (s).
The mean of the sample (M).
The mean of the population (mu).

To demonstrate the application of this method, I will provide two examples.

Example 1:

A factory employee believes that the cakes produced within his factory are being manufactured with excess amounts of corn syrup, thus altering the taste. 10 cakes were sampled from the most recent batch and tested for corn syrup composition. Typically, each cake should comprise of 20% corn syrup. Utilizing a 95 % confidence interval, can we assume that the new batch of cakes contain more than a 20% proportion of corn syrup?

The levels of the samples were:

.27, .31, .27, .34, .40, .29, .37, .14, .30, .20

Our hypothesis test will be:

H0: u = .2
HA: u > .2

With our hypothesis created, we can assess that mu = .2, and that this test will be right tailed.

We do do not to derive the additional variables, as R will perform that task for us. However, we must first input the sample data into R:

N <- c(.27, .31, .27, .34, .40, .29, .37, .14, .30, .20)

Now R will automate the rest of the equations.

t.test(N, alternative = "greater", mu = .2, conf.level = 0.95)

# " alternative = " Specifies the type of test that R will perform. "greater" indicates a right tailed test. "left" indicates a left tailed test."two.sided" indicates a two tailed test.  #

Which produces the output:

One Sample t-test

data: N
t = 3.6713, df = 9, p-value = 0.002572
alternative hypothesis: true mean is greater than 0.2
95 percent confidence interval:
0.244562 Inf
sample estimates:
mean of x
0.289

t = The t-test statistic, which derived by hand would be: the sample mean (M), minus the population mean (mu), divided by the standard deviation (S) divided by the square root of the size of the sample (N).

df = This value is utilized in conjunction with the confidence interval to define critical-t. It is derived by subtracting 1 from the value of N.

p-value = This is the probability that, given the parameters of the test, that we will obtain a value equal or greater than the t-region defined within our model.

##% confidence interval = This describes the numerical values which exists on each side of the t-distribution, which between both, contains ##% of the values of the model.

sample estimates = This value is the mean of the sample(s) utilized within the function.

With this output we can conclude:

With a p-value of .002572 (.002572 < .05), and a corresponding t-value of 3.6713, we can conclude that, at a 95% confidence interval, that the cakes are being produce contain an excess amount of corn syrup.

Example 2:

A new type of gasoline has been synthesized by a company that specializes in green energy solutions. The same fuel type, when synthesized traditionally, typically provides coup class vehicles with a mean of 25.6 mpg. Company statisticians were provided with a test sample of 8 cars which utilized the new fuel type. This sample of vehicles utilized the fuel at a mean rate of 23.2 mpg, with a sample standard deviation of 5 mpg. Can it be assumed, at a 95% confidence interval, that vehicle performance significantly differs when the new fuel type is utilized?

Our hypothesis will be:

H0: u = 25.6
HA: u NE 25.6

With our hypothesis created, we can assess that mu = 25.6, and that this test will be two tailed.
In this case we will need to create a random sample within R that adheres to the following parameters.

N = 8 Sample Size
S = 5 Sample Standard Deviation
M = 23.2 Sample Mean

This can be achieved with the following code block:

sampsize <- 8
standarddev <- 5
meanval <- 23.2

N <- rnorm(sampsize)
N <- standarddev*(N-mean(N))/sd(N)+meanval

With our “N” variable defined, we can now let R do the rest of the work:

t.test(N, alternative = "two.sided", mu = 25.6, conf.level = 0.95)

Which produces the output:

One Sample t-test

data: N
t = -1.3576, df = 7, p-value = 0.2167
alternative hypothesis: true mean is not equal to 25.6
95 percent confidence interval:
19.0199 27.3801
sample estimates:
mean of x
23.2

With this output we can conclude:

With a p-value of 0.2167 (.2167 > .05), and a corresponding t-value of -1.3576, we cannot conclude that, at a 95% confidence interval, that vehicle performance differs depending on utilization of fuel type.

Two Sample T-Test

This test is utilized if you randomly sample different sets of items from two separate control groups.

Example:

A scientist creates a chemical which he believes changes the temperature of water. He applies this chemical to water and takes the following measurements:

70, 74, 76, 72, 75, 74, 71, 71

He then measures temperature in samples which the chemical was not applied.

74, 75, 73, 76, 74, 77, 78, 75

Can the scientist conclude, with a 95% confidence interval, that his chemical is in some way altering the temperature of the water?

For this, we will use the code:

N1 <- c(70, 74, 76, 72, 75, 74, 71, 71)

N2 <- c(74, 75, 73, 76, 74, 77, 78, 75)

t.test(N2, N1, alternative = "two.sided", var.equal = TRUE, conf.level = 0.95)

Which produces the output:

Two Sample t-test

data:  N2 and N1

t = 2.4558, df = 14, p-value = 0.02773

alternative hypothesis: true difference in means is not equal to 0

95 percent confidence interval:

 0.3007929 4.4492071

sample estimates:

mean of x mean of y 

   75.250    72.875 

# Note: In this case, the 95 percent confidence interval is measuring the difference of the mean values of the samples. #

# An additional option is available when running a two sample t-test, The Welch Two Sample T-Test. To utilize this option while performing a t-test, the "var.equal = TRUE" must be changed to "var.equal = FALSE". The output produced from a Welch Two Sample t-test is slightly more robust and accounts for differing sample sizes. #

From this output we can conclude:

With a p-value of 0.02773 (.0.02773 < .05), and a corresponding t-value of 2.4558, we can state that, at a 95% confidence interval, that the scientist's chemical is altering the temperature of the water.

Paired T-Test

This test is utilized if you are sampling the same set twice, once for each variable.

Example:

A watch manufacturer believes that by changing to a new battery supplier, that the watches that are shipped which include an initial battery, will maintain longer lifespan. To test this theory, twelve watches are tested for duration of lifespan with the original battery.

The same twelve watches are then re-rested for duration with the new battery.

Can the watch manufacturer conclude, that the new battery increases the duration of lifespan for the manufactured watches? (We will assume an alpha value of .05).

For this, we will utilize the code:

N1 <- c(376, 293, 210, 264, 297, 380, 398, 303, 324, 368, 382, 309)
N2 <- c(337, 341, 316, 351, 371, 440, 312, 416, 445, 354, 444, 326)

t.test(N2, N1, alternative = "greater", paired=TRUE, conf.level = 0.95 )

This produces the output:

Paired t-test

data: N2 and N1
t = 2.4581, df = 11, p-value = 0.01589
alternative hypothesis: true difference in means is greater than 0
95 percent confidence interval:
12.32551 Inf
sample estimates:
mean of the differences
45.75

From this output we can state:

With a p-value of 0.01589 (0.01589 < .05), and a corresponding t-value of 2.4581, we can conclude that, at a 95% confidence interval, that the new battery increases the duration of lifespan for the manufactured watches.

That's all for now. Stay tuned, data heads!

* https://en.wikipedia.org/wiki/Student%27s_t-test

** For more information as to why this is the case: 

https://www.researchgate.net/post/Can_I_use_a_t-test_that_assumes_that_my_data_fit_a_normal_distribution_in_this_case_Or_should_I_use_a_non-parametric_test_Mann_Whitney2

(R) The Distribution of Sample Means

Below are two examples which illustrate a statistical concept:

The Distribution of Sample Means:

The average mortgage for new families in Texas in $ 1300, with a standard deviation of $ 800. If 100 new families are surveyed, what is the probability that the mean mortgage payment will exceed $ 1500?

800/sqrt(100)

[1] 80

(1500-1300) / 80

[1] 2.5

pnorm(2.5, lower.tail=FALSE)

[1] 0.006209665

There is a 0.62% chance of this occurring.

An American football team typically scores 17 points per game, with a standard deviation of 3 points. In a sample of 16 games, what is the probability that the team scored an average of between 16 - 19 points?

3/sqrt(16)

[1] 0.75

(16 - 19) / 0.75

[1] -4

1 - pnorm(-4, lower.tail = TRUE) * 2

[1] 0.9999367

There is a 99.99% chance of this occurring.

(R) Differences of Two Proportions

Below are two exercises which illustrate statistical concepts.

Confidence interval estimate for the difference of two proportions:

# Disable Scientific Notation in R Output #

options(scipen = 999)

Suppose that 85% of a sample of 100 factory workers, employed on a day shift, express positive job satisfaction, while 70% of 80 factory workers, who are employed on the night shift, share similar sentimentality. Establish a 90% confidence interval estimate for the difference.

# n1 = 100 #
# n2 = 80 #
# p1 = .85 #
# p2 = .70 #

sqrt((.85*.15/100) + (.70*.30/80))

[1] 0.06244998

z <- qnorm(.05, lower.tail = FALSE) * 0.06244998

.85 - .70

[1] 0.15

.15 + c(-z,z)

[1] 0.04727892 0.25272108

We can be 90% certain that the proportion of workers of day shift is between .05 (5%) and .25 (25%) higher than those on the day shift.

Hypothesis test for the difference of two proportions:

A pollster took a survey of 1300 individuals, the results of such indicated that 600 were in favor of candidate A. A second survey, taken weeks later, showed that 500 individuals out of 1500 voters were now in favor with candidate A. At a 10% significant level, is there evidence that the candidate's popularity has decreased.

# H0: p1 - p2 = 0 #
# Ha: p1 - p2 > 0 #
# Alpha = .10 #
# p1 = # 600 / 1300 # = 0.4615385 #
# p2 = # 500 / 1500 # = 0.3333333 #
# p = # (600 + 500) / (1300 + 1500) # = 0.3928571
# SigmaD = # sqrt(0.3928571 * (1 - 0.3928571) * (1/1300 + 1/1500)) # =  0.01850651 #

0.4615385 - 0.3333333

[1] 0.1282052

(0.1282052 - 0) / 0.01850651

[1] 6.927573

pnorm(6.927573, lower.tail = FALSE)

[1] 0.000000000002140608

Due to 0.000000000002140608 < .10 (Alpha), we can conclude that candidate A's popularity has decreased.

Hypothesis test for the difference of two proportions (automation):

Additionally, I have created a code block below which automates the process for the hypothesis testing for two means. To utilize this code, simply input the number of affirmative responders as the first two variables, and input the total individuals surveyed for the subsequent variables. The variable "finalres", will be the p-value of the output. Multiply this value by 2 if the test is two tailed.

survey1 <- # Survey 1 Variable #
survey2 <- # Survey 2 Variable #

total1 <- # Total Survey 1 Participants #
total2 <- # Total Survey 2 Participants #

p1 <- survey1/total1
p2 <- survey2/total2
p3 <- (survey1 + survey2) / (total1 + total2)
sigmad <- sqrt(p3 * (1 - p3) * (1/total1 + 1/total2))

vara <- p1 - p2
varb <- vara / sigmad

res1 <- pnorm(varb, lower.tail = FALSE)
res2 <- pnorm(varb, lower.tail = TRUE)
finalres <- ifelse(res1 > res2, res2, res1)

finalres # Multiply by 2 if utilizing a two tailed test #

(R) Sample Size and Margin of Error

If you are in the business of creating surveys, there are two important statistical concepts that you should be somewhat familiar with.

The first concept is known as margin of error. Margin of error can be defined as a value or as a percentage. What the margin of error illustrates, is the assumed variation that will occur within a statistical inference. For example, if a survey was taken of 100 individuals, and 60 of those individuals answered positively to a survey prompt, with a margin of error value of 5, the true value which is being inferred from the survey could alternate in either direction by the value of 5. Which would mean that the true value could range anywhere from 55-66.

Let's explore this concept in the following examples:

A pollster wants to infer, from surveying a population, the likelihood of a certain candidate winning a local election. The population of the voting public is 1000 individuals, and the pollster surveys 500 members of the voting body. From his survey, he has determined that 60% of those individuals are voting for candidate A. Assuming a confidence interval of .95, and a population percentage of 50%, what will the margin of error be for this survey?

####################################################

popsize <- 1000 # N #
samplesize <- 500 # n #
confidencelevel <- .95
populationpercentage <- .50 # p #

####################################################

con2 <- (1 - confidencelevel) / 2

MOE <- qnorm(con2, lower.tail = FALSE) * sqrt(populationpercentage * (1 - populationpercentage)) / sqrt((popsize - 1) * samplesize/(popsize - samplesize))

MOE

[1] 0.03100526

The margin of error is 3.10%.

Therefore, the pollster can conclude, with 95% confidence, that 60% of the population will vote for candidate A, with a margin of error of 3.10%. (The true value could fluctuate between 56.90% and 63.10%.

The next concept that we will review is sample size. This is essentially how many people need to take your survey for it be statistically significant. In this case, we will be solving for that value, and we will be assuming a population percentage of 50%.

The same pollster wants to survey another group of individuals to determine what type of milk that they would most prefer to drink. The pollster already knows his population size (1200), and has decided on a confidence level (95%) and a margin of error (5%). To meet this predetermined criteria, how many individuals should the pollster survey?

##################################

popsize <- 1200 # N #
confidencelevel <- .95
marginoferror <- .05 # MOE #
populationpercentage <- .5 # p #

##################################

con2 <- (1 - confidencelevel) / 2

a <- qnorm(con2, lower.tail = FALSE)^2 * populationpercentage * (1 - populationpercentage) / (marginoferror)^2

b <- 1 + qnorm(con2, lower.tail = FALSE)^2 * populationpercentage * (1 - populationpercentage) / ((marginoferror)^2 * popsize)

a/b

[1] 290.9928

The pollster should survey 291 individuals.

Below are some links that provide simplified online calculators, just in case you are not using R. Also provided, within those links, are more detailed definitions of the concepts illustrated above.

https://www.surveysystem.com/sscalc.htm

https://www.surveymonkey.com/mp/margin-of-error-calculator/

https://www.surveymonkey.com/mp/sample-size-calculator/

(R) Hypothesis Tests of Proportions

In the past two articles, we discussed various topics related to proportions. In this article, we will continue to explore the various aspects pertaining to the subject of proportions. Specifically, this entry will discuss hypothesis tests as they are applicable to proportion data.

In testing data, we are faced with choosing between two separate hypotheses.

Those hypothesis are:

The Null Hypothesis - Which is a statement that is assumed to be true. This is the assertion that you will be seeking to disprove through the application of statistical methods.

The Alternative Hypothesis - This statement is the antithesis of the Null Hypothesis.

A rough example of stated hypothesis might resemble something like:

The Null Hypothesis - It is raining outside.

The Alternative Hypothesis - It is not raining outside.

This brings us to type errors, which cannot exist without hypothesis tests. There are two types of hypothesis errors.

The error types are:

Type I - Mistakenly rejecting a true Null Hypothesis.

Type II - Mistakenly failing to reject a false Null Hypothesis.

So let's try applying this knowledge to a few example problems.

###################

# A local university claims that only 20% of its incoming freshman class attend new student orientation. A statistician who is employed by the university believes that the real percentage is higher. He plans to ask 100 new students if they plan on attending the orientation. He will inform the admissions department if 30 or more students plan on attending. What is the probability of committing a Type I Error? #

H0:p = .20 (Null Hypothesis)
Ha:p > .20 (Alternative Hypothesis)

sqrt(.2 * .8/ 100)

[1] 0.04


(.3 - .2)/0.04

[1] 2.5

pnorm(2.5,lower.tail=FALSE)

[1] 0.006209665

Probability of Type I Error: .62 percent.

###################

# A radio manufacturer claims that 85% of the radios assembled from the latest batch are defective. A quality assurance representative believes that the number is lower and wishes to test at a 5% significance level. What is the conclusion if 90 of 125 radios are defective? #

H0:p = .85 (Null Hypothesis)
Ha:p < .85 (Alternative Hypothesis)

sqrt(.15 * .85/ 100)

[1] 0.03570714

# p = #
90/125

[1] 0.72

(.72 - .85)/0.03570714

[1] -3.640728

pnorm(-3.640728,lower.tail=FALSE)

[1] 0.9998641

Since 0.9998641 > .05, there is not sufficient evidence to reject the null hypothesis at the 5% significance level. The quality assurance representative should not challenge the manufacturer's claim.

###################

# A research group conducting a nutritional survey, decides to interview 500 households to test the hypothesis that 30% of the households eat dinner after 8:00 PM. Should the hypothesis be rejected at the 5% significance level if 200 families respond that they do indeed dine after 8:00 PM? #

H0:p = .30 (Null Hypothesis)
Ha:p NE .30 (Alternative Hypothesis)
Alpha = .05

sqrt(.30 * .70/ 100)

[1] 0.04582576

#p = #
200/500

[1] 0.4

(.4 - .3)/0.04582576

[1] 2.182179

pnorm(2.182179, lower.tail = FALSE) * 2

[1] 0.02909632

Since 0.02909632 < .05, there is sufficient evidence to reject the null hypothesis.

###################

# A local university has enrolled 10% of all eligible students from an adjacent neighborhood. The office of administration plans a survey of 2000 houses. If less 7% indicate that they are interested in potential enrollment, it will be concluded that the market share has dropped. What is the probability of a Type I Error? #

H0:p = .10 (Null Hypothesis)
Ha:p < .10 (Alternative Hypothesis)

sqrt(.10 * .90/ 100)

[1] 0.03

(.07 - .10)/0.03

[1] -1

pnorm(-1, lower.tail = TRUE)

[1] 0.1586553

Probability of Type I Error: 15.87 percent.

# What is the probability of a Type II Error if university enrollment is 8%. Meaning, what is the probability that we will fail to reject the 10% null hypothesis? #

(.07 - .08)/0.03

[1] -0.3333333
pnorm(-0.3333333, lower.tail = FALSE)

[1] 0.6305586

Probability of Type II Error: 63.05 percent.

In the next article, we will continue to investigate the hypothesis testing procedure, stay tuned Data Heads!

INTCK (Duration Measurement) (SAS)

I recently had to re-visit a SAS project that I had previously completed. This particular project required that I create an additional variable within an existing data set, this variable was to store the difference of days which occurred between two pre-existing date values. In running the code, it occurred to me that I had not previously discussed the function necessary to obtain such a result. Therefore, this will be the topic of todays article.

The SAS function, INTCK, serves as a way of determining a selected duration of time which has elapsed between two SAS variables.

The form of the function is as follows:

INTCK(‘<measured duration>’ , <DATEA>, <DATEB>);

For example, if you wanted to measure the days that occurred between variable DATEA and DATEB, the code would resemble:

INTCK(‘day’ , DATEA, DATEB);

You will need to store the output, so we will create a new variable, DATEC, to do so. The code would then resemble:

DATEC = INTCK(‘day’ , DATEA, DATEB);

This function can evaluate many different measurements of time, however, I will only mention those that are most commonly utilized.

DATEC = INTCK(‘day’ , DATEA, DATEB); /* Days */

DATEC = INTCK(‘week’ , DATEA, DATEB); /* Weeks */

DATEC = INTCK(‘month’ , DATEA, DATEB); /* Months */

DATEC = INTCK(‘year’ , DATEA, DATEB); /* Years */

DATEC = INTCK(‘qtr’ , DATEA, DATEB); /* Quarters */

/*****************************************************************************/

!IMPORTANT NOTE! - INTCK() only returns INTEGERS! Meaning, that if a week and a half elapsed, and <INTCK(‘week’, x , y)> was being utilized, then the result would be 1, as only 1 WHOLE week has lapsed in duration.

In the next article, I will return to discussing the R coding language, specifically, hypothesis tests and error types.

(R) The Confidence Interval Estimate of Proportions

In this article, we will again be focusing on sample data. Specifically, what today's exercises will be demonstrating, is the process necessary to determine whether we can say that a proportion lies within a certain interval.

Example 1:

If 60% of a sample of 120 individuals leaving a diner claim to have spent over $12 for lunch, determine a 99% confidence interval estimate for the proportion of patrons who spent over $12.

sqrt(.4 * .6/ 100) 

[1] 0.04898979

z <- qnorm(.005, lower.tail=FALSE)  * 0.04898979
# .005 is due to our test containing 2 tails #

.60 + c(-z, z) 

[1] 0.4738107 0.7261893

Conclusion: We are 99% certain that the proportion of diner patrons spending over $12 for lunch is between 0.4738107 (47.34%) and 0.7261893 (72.62%).

Example 2:

In random sample of lightbulbs being produced by a factory, 20 out of 300 were found to be shattered during the shipping process. Establish a 95% confidence interval estimate that accounts for these damages.

p = 20/300 

[1] 0.06666667

sqrt(.066 * .934 / 100)

[1] 0.02482821

z <- qnorm(.025, lower.tail=FALSE)  * 0.02482821

0.06666667 + c(-z,z)

[1] 0.01800427 0.11532907

Therefore, we can be 95% certain that the proportion of light bulbs damaged during the shipping process is between 0.01800427 (1.80% and 11.53%).

Furthermore, if we wished, we could apply these ratios to a total shipment to create an estimation.

If 1000 light bulbs shipped, we can be 95% confident that between 18 and 115.3 light bulbs are damaged within the shipment.

In the next article, we will be discussing hypothesis tests. Until then, stay tuned Data Monkeys!

(R) Distributions of Sample Proportions

Let's suppose that we have been presented with sample data from a larger collection of population data; or, let's suppose that we have been presented with incomplete information pertaining to a larger population.

If we were tasked to reach various conclusions based on such data, how would we structure our models? This article sets to answer these questions. To begin this study, we will review a series of example problems.

Example 1:

The military has instituted a new training regime in order to screen candidates for a newly formed battalion. Due to the specialization of this unit, candidates are vetted through exercises which screen through the utilization of extremely rigorous physical routines. Presently, only 60% of candidates who have attempted the regime, have successfully passed. If 100 new candidates volunteer for the unit, what is the probability that more than 70% of those candidates will pass the physical?

# Disable Scientific Notation in R Output #

options(scipen = 999)

# Find The Standard Deviation of The Sample #

Standard Deviation = Square Root of: (x)(1-x) / 100

sqrt(.4 * .6/ 100) 

[1] 0.04898979

# Find the Z-Score #

(.7 - .6)/0.04898979 

[1] 2.041242

Probability of Z-Score 2.041242 = .4793
(Check Z-Table)

Finally, conclude as to whether the probability of the sample exceeds 70%
(One tailed test)

.50 - .4793

[1] 0.0207

In R, the following code can be used to expedite the process:

sqrt(.4 * .6/ 100) 

[1] 0.04898979

pnorm(q=.7, mean=.6, sd=0.04898979 , lower.tail=FALSE) 

[1] 0.02061341

So, we can conclude, that if 100 new candidates volunteer for the unit, there is only a 2.06% chance that more than 70% of those candidates will pass the physical.

The process really is that simple.

In the next article we will review confidence interval estimate of proportions.