Studying programmer behaviour at scale: a case study using Amazon Mechanical Turk

Jason T. Jacques; Per Ola Kristensson

doi:10.1145/3464432.3464436

Studying programmer behaviour at scale: a case study using Amazon Mechanical Turk

Jason T. Jacques, Per Ola Kristensson

School of Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

3 Citations (Scopus)

17 Downloads (Pure)

Abstract

Developing and maintaining a correct and consistent model of how code will be executed is an ongoing challenge for software developers. However, validating the tools and techniques we develop to aid programmers can be a challenge plagued by small sample sizes, high costs, or poor generalisability. This paper serves as a case study using a web-based crowdsourcing approach to study programmer behaviour at scale. We demonstrate this method to create controlled coding experiments at modest cost, highlight the efficacy of this approach with objective validation, and comment on notable findings from our prototype experiment into one of the most ubiquitous, yet understudied, features of modern software development environments: syntax highlighting.

Original language	English
Title of host publication	Programming '21
Subtitle of host publication	companion proceedings of the 5^th International conference on the art, science, and engineering of programming
Editors	Luke Church, Shigeru Chiba, Elisa Gonzalez Boix
Publisher	ACM
Pages	36-48
Number of pages	13
ISBN (Print)	9781450389860
DOIs	https://doi.org/10.1145/3464432.3464436
Publication status	Published - 22 Mar 2021
Event	5th International Conference on the Art, Science, and Engineering of Programming, Programming 2021 - Virtual, Online, United Kingdom Duration: 22 Mar 2021 → 26 Mar 2021

Conference

Conference	5th International Conference on the Art, Science, and Engineering of Programming, Programming 2021
Country/Territory	United Kingdom
City	Virtual, Online
Period	22/03/21 → 26/03/21

Keywords

Behaviour
Crowdsourcing
Programming

Access to Document

10.1145/3464432.3464436

Jacques_2021_Studying-programmer_AAM
Copyright © 2021 Copyright held by the owner/author(s). This work has been made available online in accordance with publisher policies or with permission. Permission for further reuse of this content should be sought from the publisher or the rights holder. This is the author created accepted manuscript following peer review and may differ slightly from the final published version. The final published version of this work is available at https://doi.org/10.1145/3464432.3464436.
Accepted author manuscript, 832 KB

https://www.repository.cam.ac.uk/handle/1810/322069

Cite this

@inproceedings{332557810458462caf30089ceb9716c5,

title = "Studying programmer behaviour at scale: a case study using Amazon Mechanical Turk",

abstract = "Developing and maintaining a correct and consistent model of how code will be executed is an ongoing challenge for software developers. However, validating the tools and techniques we develop to aid programmers can be a challenge plagued by small sample sizes, high costs, or poor generalisability. This paper serves as a case study using a web-based crowdsourcing approach to study programmer behaviour at scale. We demonstrate this method to create controlled coding experiments at modest cost, highlight the efficacy of this approach with objective validation, and comment on notable findings from our prototype experiment into one of the most ubiquitous, yet understudied, features of modern software development environments: syntax highlighting. ",

keywords = "Behaviour, Crowdsourcing, Programming",

author = "Jacques, {Jason T.} and Kristensson, {Per Ola}",

note = "Funding: This work was funded by an EPSRC studentship and EPSRC grant EP/R004471/1. Data available at https://doi.org/10.17863/CAM.66593. ; 5th International Conference on the Art, Science, and Engineering of Programming, Programming 2021 ; Conference date: 22-03-2021 Through 26-03-2021",

year = "2021",

month = mar,

day = "22",

doi = "10.1145/3464432.3464436",

language = "English",

isbn = "9781450389860",

pages = "36--48",

editor = "Luke Church and Shigeru Chiba and Boix, {Elisa Gonzalez}",

booktitle = "Programming '21",

publisher = "ACM",

address = "United States",

}

Jacques, JT & Kristensson, PO 2021, Studying programmer behaviour at scale: a case study using Amazon Mechanical Turk. in L Church, S Chiba & EG Boix (eds), Programming '21: companion proceedings of the 5^th International conference on the art, science, and engineering of programming. ACM, pp. 36-48, 5th International Conference on the Art, Science, and Engineering of Programming, Programming 2021, Virtual, Online, United Kingdom, 22/03/21. https://doi.org/10.1145/3464432.3464436

Studying programmer behaviour at scale: a case study using Amazon Mechanical Turk. / Jacques, Jason T.; Kristensson, Per Ola.
Programming '21: companion proceedings of the 5^th International conference on the art, science, and engineering of programming. ed. / Luke Church; Shigeru Chiba; Elisa Gonzalez Boix. ACM, 2021. p. 36-48.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Studying programmer behaviour at scale

T2 - 5th International Conference on the Art, Science, and Engineering of Programming, Programming 2021

AU - Jacques, Jason T.

AU - Kristensson, Per Ola

N1 - Funding: This work was funded by an EPSRC studentship and EPSRC grant EP/R004471/1. Data available at https://doi.org/10.17863/CAM.66593.

PY - 2021/3/22

Y1 - 2021/3/22

N2 - Developing and maintaining a correct and consistent model of how code will be executed is an ongoing challenge for software developers. However, validating the tools and techniques we develop to aid programmers can be a challenge plagued by small sample sizes, high costs, or poor generalisability. This paper serves as a case study using a web-based crowdsourcing approach to study programmer behaviour at scale. We demonstrate this method to create controlled coding experiments at modest cost, highlight the efficacy of this approach with objective validation, and comment on notable findings from our prototype experiment into one of the most ubiquitous, yet understudied, features of modern software development environments: syntax highlighting.

AB - Developing and maintaining a correct and consistent model of how code will be executed is an ongoing challenge for software developers. However, validating the tools and techniques we develop to aid programmers can be a challenge plagued by small sample sizes, high costs, or poor generalisability. This paper serves as a case study using a web-based crowdsourcing approach to study programmer behaviour at scale. We demonstrate this method to create controlled coding experiments at modest cost, highlight the efficacy of this approach with objective validation, and comment on notable findings from our prototype experiment into one of the most ubiquitous, yet understudied, features of modern software development environments: syntax highlighting.

KW - Behaviour

KW - Crowdsourcing

KW - Programming

UR - https://dl.acm.org/doi/proceedings/10.1145/3464432

U2 - 10.1145/3464432.3464436

DO - 10.1145/3464432.3464436

M3 - Conference contribution

AN - SCOPUS:85113847225

SN - 9781450389860

SP - 36

EP - 48

BT - Programming '21

A2 - Church, Luke

A2 - Chiba, Shigeru

A2 - Boix, Elisa Gonzalez

PB - ACM

Y2 - 22 March 2021 through 26 March 2021

ER -

Studying programmer behaviour at scale: a case study using Amazon Mechanical Turk

Abstract

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Research data supporting "Studying Programmer Behaviour at Scale: A Case Study Using Amazon Mechanical Turk"

Cite this

Studying programmer behaviour at scale: a case study using Amazon Mechanical Turk

Abstract

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Datasets

Research data supporting "Studying Programmer Behaviour at Scale: A Case Study Using Amazon Mechanical Turk"

Cite this