A recent request by a researcher to help generate participants got me thinking about the rise of psychological research over the internet. The web can be tantalising for psychological researchers – access to millions upon millions of participants, reaching beyond the standard undergraduate pool to specialised groups, cheap online implementation and the resulting data already in electronic form. All these are acknowledged advantages but how many investigators are aware of some of the major pitfalls of online research? Birnbaum (2004) examines some of the problems researchers will face.
- Multiple submissions – This is a commonly cited worry for online research, but, in practice, is not considered a problem by Birnbaum (2004). Examining large datasets, he has found little or no evidence of multiple submissions. Standard practices to avoid this include simply asking participants to avoid multiple submissions, scanning IP addresses or other identifiers for multiple instances and removing incentives.
- Dropouts in Between-Subjects Research – High dropout rats can easily skew the results of between-subjects research. Perhaps this is partly because there’s no social pressure to finish when people are sat on their own, as there is in a lab. One technique to combat this is providing a ‘high-hurdle’ to entry to get rid of the quitters early.
- Sampling Bias and Stratified Analysis – Finding a representative sample is extremely important in many types of research and the sample reached online cannot be automatically considered representative. This can be countered by using a stratified analysis – looking at the results within, for example, age or education ranges. If these sub-samples reveal similar trends, the overall message of the data is easier to support.
- Response Bias – It’s even possible for researchers to be unaware of the different ways people are biased to respond to types of input devices, like check-boxes or drop-down menus. Each has its own characteristics which need to be taken into account.
Some researchers have looked at the differences found when lab results are compared with those obtained over the internet. There’s quite a variation in their conclusions (Reips, 2001) – some have obtained the same results, some worse and some even claim that internet data is better (Birnbaum, 2004 reports Online Social Sciences as the source for this).
Overall, internet research can be useful, like any research method, as long as its advantages and disadvantages are understood.
Birnbaum (2004) Human Research and Data Collection via the Internet (Abstract)Reips (2001) The Web Experimental Psychology Lab: Five years of data collection on the Internet (Abstract | PDF)
