Here we will trick a statistical analysis with what is known as a p-value optimization.
We will test if clapping your boots three time before leave your house in the morning will increase your average salary. Try it, it doesnt.
In [19]:
# set up some random data.
income = rnorm(1000,mean=5000,sd=200) # your sample. Everybody must clap boots every day for 3 years, then you record salary. You know that average salary of this population is 5000.
In [29]:
#now lets do a T test. Your hypothesis is that the average salary is larger than the known average 5000.
for (i in seq(1,989,by=1))
{
partialdata=income[i:(i+9)]
T=(mean(partialdata)-5000)/sd(partialdata)*sqrt(10)
if (T>qt(p=0.95,df=9))
{break}
}
i
T
31
2.86334177746778
There you go, clap your boots to become rich! Where do we cheat? Incomplete data! We have performed i+10 experiments, but only analyze 10 datapoints, dropping those before i. Note that we did not have to "pick" specific values, like low incomes, that did not suit us. We did blindly, without bias, take 10 random datapoints. We let statistical fluctuations do the dirty work.
Typical setting to look out for: "Student A does not show effect" "Probably student A was not skilled" "Let student B have a go". "Student B is also not skilled" "Lets see if student C can do it"...
In [30]:
#now lets do a proper T test. Your hypothesis is that the average salary is larger than the known average 5000.
for (i in seq(1,989,by=1))
{
partialdata=income[1:(i+9)]
T=(mean(partialdata)-5000)/sd(partialdata)*sqrt(10)
if (T>qt(p=0.95,df=9))
{break}
}
i
T
989
0.00240801927382591
The dirty variation: The Chocolate Diet
It is obvious that omitting datapoints from a study, irrespective of whether you pick specific points or choose some at random, is dubious. How about dropping factors?
We take 1000 people from the same population, sort them into 100 groups a 10 people each, and subject them to totally nonesense diets. "Liquid butter diet", "Travel to the moon diet", "Chicken nuggets diet", etc. None of these actually work.
In [65]:
# set up some random data. The value denotes the weight loss/gain after 6 weeks of diet.
n=10 #group size
m=100 # number of groups
weightdata = matrix( rnorm(n*m,mean=0,sd=3), n, m) # weightloss of participants. mean=0, so all 100 diets are useless.
In [66]:
summary(weightdata)
V1 V2 V3 V4
Min. :-2.7387 Min. :-6.2774 Min. :-4.5216 Min. :-2.4288
1st Qu.:-1.7837 1st Qu.:-0.2651 1st Qu.:-0.6638 1st Qu.:-0.1889
Median : 0.2215 Median : 2.0853 Median : 1.6549 Median : 0.9696
Mean : 0.2599 Mean : 1.1211 Mean : 1.1930 Mean : 1.0904
3rd Qu.: 1.2789 3rd Qu.: 3.4290 3rd Qu.: 2.6253 3rd Qu.: 2.1775
Max. : 6.0157 Max. : 4.5204 Max. : 6.3859 Max. : 4.4411
V5 V6 V7 V8
Min. :-3.6839 Min. :-3.6010 Min. :-5.257 Min. :-3.9968
1st Qu.:-0.8926 1st Qu.:-2.9950 1st Qu.:-3.309 1st Qu.:-2.4232
Median : 0.2585 Median : 0.7410 Median :-2.115 Median :-0.2503
Mean :-0.2065 Mean :-0.2975 Mean :-1.351 Mean : 0.1509
3rd Qu.: 1.1194 3rd Qu.: 1.5913 3rd Qu.: 1.387 3rd Qu.: 2.6697
Max. : 1.8276 Max. : 3.1081 Max. : 2.876 Max. : 4.7618
V9 V10 V11 V12
Min. :-3.6981 Min. :-8.5235 Min. :-6.127 Min. :-7.3124
1st Qu.:-1.6022 1st Qu.:-2.8714 1st Qu.:-2.587 1st Qu.:-1.4340
Median :-0.9391 Median : 0.6387 Median : 1.322 Median : 1.2521
Mean :-0.5562 Mean : 0.6502 Mean :-0.144 Mean : 0.6589
3rd Qu.: 0.5766 3rd Qu.: 4.3568 3rd Qu.: 2.635 3rd Qu.: 3.5662
Max. : 4.0115 Max. : 7.9055 Max. : 3.583 Max. : 4.9850
V13 V14 V15 V16
Min. :-5.2386 Min. :-3.60503 Min. :-4.5492 Min. :-3.1401
1st Qu.:-3.2376 1st Qu.:-1.29229 1st Qu.:-1.0744 1st Qu.:-1.8784
Median : 0.2321 Median :-0.02803 Median : 0.3035 Median : 1.2445
Mean :-0.8010 Mean : 0.52617 Mean : 0.8219 Mean : 0.5374
3rd Qu.: 1.3161 3rd Qu.: 2.60564 3rd Qu.: 3.1928 3rd Qu.: 2.6841
Max. : 2.6894 Max. : 4.36617 Max. : 6.4502 Max. : 3.7110
V17 V18 V19 V20
Min. :-4.7366 Min. :-2.82933 Min. :-6.6116 Min. :-4.27837
1st Qu.:-1.9053 1st Qu.:-1.33012 1st Qu.:-2.2258 1st Qu.:-2.26918
Median :-0.9012 Median :-0.09478 Median :-1.0821 Median :-0.02806
Mean :-0.3886 Mean : 1.01402 Mean :-1.2373 Mean : 0.32870
3rd Qu.: 1.1414 3rd Qu.: 0.69461 3rd Qu.: 0.6779 3rd Qu.: 3.48848
Max. : 5.8054 Max. : 9.85899 Max. : 1.2416 Max. : 4.79003
V21 V22 V23 V24
Min. :-3.889556 Min. :-8.0861 Min. :-7.4525 Min. :-6.7814
1st Qu.:-2.506045 1st Qu.:-3.1595 1st Qu.:-2.7016 1st Qu.:-2.9653
Median :-0.627048 Median :-1.6304 Median :-2.0253 Median : 0.2354
Mean :-0.005771 Mean :-1.9243 Mean :-1.6573 Mean :-0.1857
3rd Qu.: 1.044636 3rd Qu.:-0.6009 3rd Qu.: 0.6433 3rd Qu.: 1.7496
Max. : 6.382749 Max. : 3.3835 Max. : 3.4973 Max. : 6.6333
V25 V26 V27 V28
Min. :-5.2645 Min. :-4.1320 Min. :-5.4829 Min. :-7.2746
1st Qu.:-2.9971 1st Qu.:-2.4753 1st Qu.:-4.3276 1st Qu.:-3.9064
Median :-0.8133 Median :-0.8886 Median :-3.2116 Median :-2.0478
Mean :-1.0575 Mean :-0.2999 Mean :-2.3287 Mean :-1.7122
3rd Qu.: 0.5644 3rd Qu.: 1.4955 3rd Qu.:-0.4598 3rd Qu.: 0.8398
Max. : 4.8862 Max. : 5.8128 Max. : 2.9291 Max. : 3.9566
V29 V30 V31 V32
Min. :-4.0293 Min. :-6.13637 Min. :-5.4326 Min. :-4.7377
1st Qu.:-1.3605 1st Qu.:-1.23132 1st Qu.:-2.0664 1st Qu.:-0.5838
Median : 0.2867 Median :-1.04713 Median :-0.9520 Median : 0.3090
Mean : 0.1865 Mean : 0.08589 Mean :-0.4567 Mean : 0.4762
3rd Qu.: 2.0280 3rd Qu.: 3.42855 3rd Qu.: 0.8119 3rd Qu.: 2.5849
Max. : 3.9142 Max. : 4.48216 Max. : 5.0003 Max. : 3.6401
V33 V34 V35 V36
Min. :-1.3908 Min. :-6.11141 Min. :-1.2187 Min. :-3.5014
1st Qu.:-0.3587 1st Qu.:-2.26360 1st Qu.:-0.6871 1st Qu.:-2.4399
Median : 0.6135 Median :-0.09133 Median : 0.4042 Median :-0.1805
Mean : 1.5321 Mean :-0.28936 Mean : 0.8160 Mean :-0.4881
3rd Qu.: 3.1194 3rd Qu.: 2.21535 3rd Qu.: 2.1187 3rd Qu.: 1.0160
Max. : 7.3491 Max. : 5.03239 Max. : 3.6831 Max. : 3.2615
V37 V38 V39 V40
Min. :-3.8037 Min. :-5.8418 Min. :-4.8735 Min. :-4.59629
1st Qu.:-2.6220 1st Qu.:-2.5992 1st Qu.:-3.2764 1st Qu.:-2.06800
Median :-0.6491 Median : 0.4256 Median :-1.1163 Median :-0.16480
Mean :-0.4337 Mean :-0.7668 Mean :-1.3830 Mean :-0.04249
3rd Qu.: 0.7986 3rd Qu.: 1.0016 3rd Qu.: 0.6823 3rd Qu.: 1.27648
Max. : 3.8737 Max. : 1.8474 Max. : 1.4900 Max. : 6.46894
V41 V42 V43 V44
Min. :-3.0495 Min. :-5.2623 Min. :-2.8562 Min. :-2.85473
1st Qu.:-0.6384 1st Qu.:-1.4195 1st Qu.:-1.1437 1st Qu.:-0.99668
Median : 0.8009 Median :-0.4979 Median : 0.8957 Median : 0.04019
Mean : 0.6714 Mean : 0.7669 Mean : 0.7331 Mean : 0.20297
3rd Qu.: 2.3859 3rd Qu.: 3.8956 3rd Qu.: 2.2222 3rd Qu.: 1.17033
Max. : 3.4813 Max. : 7.0075 Max. : 4.6657 Max. : 4.07790
V45 V46 V47 V48
Min. :-5.38055 Min. :-3.4603 Min. :-6.7713 Min. :-2.2898
1st Qu.:-0.92784 1st Qu.:-0.6142 1st Qu.:-3.3013 1st Qu.:-1.4995
Median :-0.21939 Median : 0.5084 Median :-1.9644 Median : 0.3583
Mean :-0.07288 Mean : 0.0765 Mean :-1.8958 Mean : 0.7756
3rd Qu.: 1.70221 3rd Qu.: 1.3192 3rd Qu.:-0.1547 3rd Qu.: 0.8926
Max. : 5.52106 Max. : 2.8123 Max. : 2.9755 Max. : 8.4801
V49 V50 V51 V52
Min. :-3.6087 Min. :-3.8625 Min. :-8.3220 Min. :-3.2985
1st Qu.:-0.8305 1st Qu.:-3.3829 1st Qu.:-1.9049 1st Qu.:-2.9366
Median : 1.3502 Median :-2.9301 Median : 0.4876 Median :-0.5306
Mean : 0.6117 Mean :-1.6878 Mean :-0.3437 Mean :-0.5468
3rd Qu.: 2.1119 3rd Qu.:-0.2984 3rd Qu.: 2.5113 3rd Qu.: 0.8550
Max. : 2.7773 Max. : 2.9689 Max. : 4.4144 Max. : 4.4288
V53 V54 V55 V56
Min. :-3.4745 Min. :-4.96658 Min. :-4.2060 Min. :-5.6279
1st Qu.:-1.0855 1st Qu.:-1.97070 1st Qu.:-2.1646 1st Qu.:-1.6898
Median : 0.5801 Median : 0.04154 Median :-0.8205 Median : 0.6932
Mean : 0.8464 Mean :-0.10137 Mean :-0.7876 Mean : 0.3396
3rd Qu.: 2.8242 3rd Qu.: 1.82526 3rd Qu.:-0.2133 3rd Qu.: 2.4876
Max. : 5.4924 Max. : 3.20107 Max. : 4.3878 Max. : 5.9250
V57 V58 V59 V60
Min. :-3.4964 Min. :-5.2207 Min. :-3.9131 Min. :-2.0504
1st Qu.:-0.3855 1st Qu.:-2.1553 1st Qu.: 0.3406 1st Qu.:-0.5297
Median : 0.8869 Median : 0.1165 Median : 1.5290 Median : 1.0673
Mean : 0.6363 Mean :-0.3584 Mean : 1.4126 Mean : 0.6702
3rd Qu.: 1.7703 3rd Qu.: 0.6783 3rd Qu.: 2.8064 3rd Qu.: 1.8851
Max. : 3.7388 Max. : 4.0330 Max. : 5.9108 Max. : 2.9805
V61 V62 V63 V64
Min. :-7.7404 Min. :-4.999 Min. :-4.5805 Min. :-4.0908
1st Qu.:-1.7868 1st Qu.:-1.169 1st Qu.:-3.7782 1st Qu.:-0.3189
Median :-0.1115 Median : 1.212 Median :-1.2213 Median : 1.5848
Mean :-0.2392 Mean : 1.210 Mean :-0.7975 Mean : 1.5143
3rd Qu.: 2.4393 3rd Qu.: 3.943 3rd Qu.: 1.1804 3rd Qu.: 3.9967
Max. : 4.1072 Max. : 8.360 Max. : 5.4101 Max. : 6.0471
V65 V66 V67 V68
Min. :-3.593 Min. :-5.5280 Min. :-6.6670 Min. :-4.4258
1st Qu.:-2.959 1st Qu.:-2.7325 1st Qu.:-3.5371 1st Qu.:-3.6897
Median :-2.030 Median : 0.9414 Median : 0.8520 Median :-2.8846
Mean :-1.296 Mean : 0.1612 Mean :-0.4244 Mean :-1.5048
3rd Qu.:-1.004 3rd Qu.: 2.9022 3rd Qu.: 1.7432 3rd Qu.: 0.9953
Max. : 2.962 Max. : 4.4893 Max. : 4.6650 Max. : 3.1263
V69 V70 V71 V72
Min. :-4.1867 Min. :-3.5920 Min. :-1.5778 Min. :-6.2671
1st Qu.:-2.9221 1st Qu.:-1.7924 1st Qu.:-0.6966 1st Qu.:-1.4800
Median : 0.4474 Median :-0.2807 Median : 0.4413 Median :-0.4585
Mean :-0.2254 Mean :-0.1259 Mean : 0.9255 Mean :-0.5278
3rd Qu.: 2.1030 3rd Qu.: 1.8812 3rd Qu.: 1.9306 3rd Qu.: 1.2729
Max. : 3.2509 Max. : 3.1321 Max. : 5.0731 Max. : 3.7592
V73 V74 V75 V76
Min. :-2.8741 Min. :-6.063 Min. :-2.437 Min. :-6.36781
1st Qu.:-0.2621 1st Qu.:-2.462 1st Qu.:-1.362 1st Qu.:-1.44698
Median : 0.3135 Median :-1.479 Median : 1.118 Median :-0.22778
Mean : 0.7916 Mean :-1.101 Mean : 1.030 Mean : 0.08612
3rd Qu.: 1.1646 3rd Qu.: 1.749 3rd Qu.: 3.551 3rd Qu.: 2.51234
Max. : 7.2964 Max. : 2.853 Max. : 4.281 Max. : 4.38210
V77 V78 V79 V80
Min. :-2.9275 Min. :-5.9778 Min. :-3.52059 Min. :-5.1360
1st Qu.:-1.8053 1st Qu.:-3.1982 1st Qu.:-1.73778 1st Qu.:-3.3350
Median : 0.7098 Median :-1.9262 Median :-0.26206 Median :-1.0066
Mean : 0.3157 Mean :-1.5544 Mean :-0.04367 Mean :-1.4568
3rd Qu.: 1.7679 3rd Qu.: 0.3854 3rd Qu.: 1.56273 3rd Qu.: 0.5765
Max. : 4.0622 Max. : 3.3637 Max. : 4.30285 Max. : 2.0140
V81 V82 V83 V84
Min. :-4.1083 Min. :-1.7479 Min. :-6.2640 Min. :0.4621
1st Qu.:-2.5350 1st Qu.:-0.9795 1st Qu.:-1.5113 1st Qu.:1.0495
Median :-1.2026 Median : 1.3971 Median : 0.7994 Median :1.4419
Mean :-0.9683 Mean : 1.4375 Mean : 0.3501 Mean :1.8670
3rd Qu.: 0.4388 3rd Qu.: 3.6563 3rd Qu.: 3.1489 3rd Qu.:2.8223
Max. : 3.5350 Max. : 5.4576 Max. : 5.7092 Max. :3.8786
V85 V86 V87 V88
Min. :-2.8525 Min. :-1.2197 Min. :-5.00529 Min. :-5.4783
1st Qu.:-0.9390 1st Qu.:-0.1384 1st Qu.:-3.15741 1st Qu.:-0.4943
Median :-0.4581 Median : 1.1440 Median :-0.08258 Median : 1.4565
Mean : 0.8771 Mean : 0.8451 Mean :-0.65594 Mean : 0.8884
3rd Qu.: 3.2803 3rd Qu.: 1.8053 3rd Qu.: 1.74454 3rd Qu.: 2.6112
Max. : 5.3565 Max. : 2.6130 Max. : 2.57776 Max. : 6.0365
V89 V90 V91 V92
Min. :-3.59884 Min. :-4.6714 Min. :-3.3265 Min. :-3.9736
1st Qu.:-1.09594 1st Qu.:-1.6123 1st Qu.:-1.4255 1st Qu.: 0.6773
Median :-0.06314 Median : 0.2407 Median :-0.2765 Median : 2.2629
Mean : 0.18418 Mean : 0.5685 Mean : 0.4017 Mean : 1.5662
3rd Qu.: 1.82922 3rd Qu.: 3.1508 3rd Qu.: 2.4910 3rd Qu.: 2.6754
Max. : 3.65400 Max. : 6.1124 Max. : 5.8832 Max. : 6.1553
V93 V94 V95 V96
Min. :-2.47096 Min. :-6.3451 Min. :-4.3624 Min. :-3.5323
1st Qu.:-1.73569 1st Qu.:-1.6137 1st Qu.:-0.4261 1st Qu.:-1.1714
Median :-0.71519 Median :-1.3399 Median : 1.3717 Median :-0.3008
Mean :-0.08103 Mean :-0.9288 Mean : 0.4653 Mean : 0.4759
3rd Qu.: 0.66572 3rd Qu.: 0.4041 3rd Qu.: 1.9404 3rd Qu.: 2.2224
Max. : 4.24817 Max. : 4.8719 Max. : 3.1000 Max. : 6.0299
V97 V98 V99 V100
Min. :-3.84499 Min. :-4.6715 Min. :-5.4650 Min. :-5.4981
1st Qu.:-1.29470 1st Qu.:-1.3319 1st Qu.:-2.6876 1st Qu.:-3.5600
Median : 0.06828 Median :-0.9314 Median :-0.4928 Median :-2.5573
Mean : 0.32726 Mean :-0.7450 Mean :-0.9623 Mean :-2.0001
3rd Qu.: 0.84174 3rd Qu.: 0.1483 3rd Qu.: 0.8145 3rd Qu.:-0.5725
Max. : 5.92879 Max. : 3.0226 Max. : 2.5577 Max. : 2.2577
In [81]:
successfuldiets=vector()
for (i in seq(1,100))
{
mypvalue=t.test(weightdata[,i],mu=0,alternative="less")$p.value
if (mypvalue<0.05)
{successfuldiets=c(successfuldiets,i)}
}
In [82]:
successfuldiets
- 22
- 27
- 39
- 47
- 50
- 100
In [ ]: