SpeCam: sensing surface color and material with the front-facing camera of mobile device

Hui Shyong Yeo; Juyoung Lee; Andrea Bianchi; David Harris-Birtill; Aaron John Quigley

doi:10.1145/3098279.3098541

SpeCam: sensing surface color and material with the front-facing camera of mobile device

Hui Shyong Yeo, Juyoung Lee, Andrea Bianchi, David Harris-Birtill, Aaron John Quigley

School of Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

SpeCam is a lightweight surface color and material sensing approach for mobile devices which only uses the front-facing camera and the display as a multi-spectral light source. We leverage the natural use of mobile devices (placing it face-down) to detect the material underneath and therefore infer the location or placement of the device. SpeCam can then be used to support discreet micro-interactions to avoid the numerous distractions that users daily face with today's mobile devices. Our two-parts study shows that SpeCam can i) recognize colors in the HSB space with 10 degrees apart near the 3 dominant colors and 4 degrees otherwise and ii) 30 types of surface materials with 99% accuracy. These findings are further supported by a spectroscopy study. Finally, we suggest a series of applications based on simple mobile micro-interactions suitable for using the phone when placed face-down.

Original language	English
Title of host publication	Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '17)
Place of Publication	New York, NY
Publisher	ACM
Pages	1-9
Number of pages	9
ISBN (Print)	9781450350754
DOIs	https://doi.org/10.1145/3098279.3098541
Publication status	Published - 4 Sept 2017
Event	19th International Conference on Human-Computer Interaction with Mobile Devices and Services - Aula der Wissenschaft – Hall of Science, Vienna, Austria Duration: 4 Sept 2017 → 7 Sept 2017 Conference number: 19 https://mobilehci.acm.org/2017/index.html

Conference

Conference	19th International Conference on Human-Computer Interaction with Mobile Devices and Services
Abbreviated title	MobileHCI
Country/Territory	Austria
City	Vienna
Period	4/09/17 → 7/09/17
Internet address	https://mobilehci.acm.org/2017/index.html

Keywords

Surface detection
Material detection
Discreet interaction

Access to Document

10.1145/3098279.3098541

Quigley_2017_HCIMDS_SpeCam_AAM
Copyright © 2017, ACM. This work has been made available online in accordance with the publisher’s policies. This is the author created, accepted version manuscript following peer review and may differ slightly from the final published version. The final published version of this work is available at https://doi.org/10.1145/3098279.3098541
Accepted author manuscript, 15 MB

David Cameron Christopher Harris-Birtill
- dcchbst-andrews.acuk
- School of Computer Science - Senior Lecturer
- Centre for Research into Ecological & Environmental Modelling
Person: Academic

SpeCam: sensing surface color and material with the front-facing camera of mobile device (dataset)
Yeo, H. S. (Creator), Lee, J. (Creator), Bianchi, A. (Creator), Harris-Birtill, D. C. C. (Creator) & Quigley, A. J. (Creator), GitHub, 25 May 2017
https://github.com/tcboy88/SpeCam/tree/master/Materials%20Phone
Dataset

Cite this

@inproceedings{c2655f98d0fc42c5a38883f8e03318be,

title = "SpeCam: sensing surface color and material with the front-facing camera of mobile device",

abstract = "SpeCam is a lightweight surface color and material sensing approach for mobile devices which only uses the front-facing camera and the display as a multi-spectral light source. We leverage the natural use of mobile devices (placing it face-down) to detect the material underneath and therefore infer the location or placement of the device. SpeCam can then be used to support discreet micro-interactions to avoid the numerous distractions that users daily face with today's mobile devices. Our two-parts study shows that SpeCam can i) recognize colors in the HSB space with 10 degrees apart near the 3 dominant colors and 4 degrees otherwise and ii) 30 types of surface materials with 99% accuracy. These findings are further supported by a spectroscopy study. Finally, we suggest a series of applications based on simple mobile micro-interactions suitable for using the phone when placed face-down.",

keywords = "Surface detection, Material detection, Discreet interaction",

author = "Yeo, {Hui Shyong} and Juyoung Lee and Andrea Bianchi and David Harris-Birtill and Quigley, {Aaron John}",

year = "2017",

month = sep,

day = "4",

doi = "10.1145/3098279.3098541",

language = "English",

isbn = "9781450350754",

pages = "1--9",

booktitle = "Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '17)",

publisher = "ACM",

address = "United States",

note = "19th International Conference on Human-Computer Interaction with Mobile Devices and Services, MobileHCI ; Conference date: 04-09-2017 Through 07-09-2017",

url = "https://mobilehci.acm.org/2017/index.html",

}

Yeo, HS, Lee, J, Bianchi, A, Harris-Birtill, D & Quigley, AJ 2017, SpeCam: sensing surface color and material with the front-facing camera of mobile device. in Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '17)., 25, ACM, New York, NY, pp. 1-9, 19th International Conference on Human-Computer Interaction with Mobile Devices and Services, Vienna, Austria, 4/09/17. https://doi.org/10.1145/3098279.3098541

SpeCam: sensing surface color and material with the front-facing camera of mobile device. / Yeo, Hui Shyong; Lee, Juyoung; Bianchi, Andrea et al.
Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '17). New York, NY: ACM, 2017. p. 1-9 25.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - SpeCam: sensing surface color and material with the front-facing camera of mobile device

AU - Yeo, Hui Shyong

AU - Lee, Juyoung

AU - Bianchi, Andrea

AU - Harris-Birtill, David

AU - Quigley, Aaron John

N1 - Conference code: 19

PY - 2017/9/4

Y1 - 2017/9/4

N2 - SpeCam is a lightweight surface color and material sensing approach for mobile devices which only uses the front-facing camera and the display as a multi-spectral light source. We leverage the natural use of mobile devices (placing it face-down) to detect the material underneath and therefore infer the location or placement of the device. SpeCam can then be used to support discreet micro-interactions to avoid the numerous distractions that users daily face with today's mobile devices. Our two-parts study shows that SpeCam can i) recognize colors in the HSB space with 10 degrees apart near the 3 dominant colors and 4 degrees otherwise and ii) 30 types of surface materials with 99% accuracy. These findings are further supported by a spectroscopy study. Finally, we suggest a series of applications based on simple mobile micro-interactions suitable for using the phone when placed face-down.

AB - SpeCam is a lightweight surface color and material sensing approach for mobile devices which only uses the front-facing camera and the display as a multi-spectral light source. We leverage the natural use of mobile devices (placing it face-down) to detect the material underneath and therefore infer the location or placement of the device. SpeCam can then be used to support discreet micro-interactions to avoid the numerous distractions that users daily face with today's mobile devices. Our two-parts study shows that SpeCam can i) recognize colors in the HSB space with 10 degrees apart near the 3 dominant colors and 4 degrees otherwise and ii) 30 types of surface materials with 99% accuracy. These findings are further supported by a spectroscopy study. Finally, we suggest a series of applications based on simple mobile micro-interactions suitable for using the phone when placed face-down.

KW - Surface detection

KW - Material detection

KW - Discreet interaction

U2 - 10.1145/3098279.3098541

DO - 10.1145/3098279.3098541

M3 - Conference contribution

SN - 9781450350754

SP - 1

EP - 9

BT - Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '17)

PB - ACM

CY - New York, NY

T2 - 19th International Conference on Human-Computer Interaction with Mobile Devices and Services

Y2 - 4 September 2017 through 7 September 2017

ER -

SpeCam: sensing surface color and material with the front-facing camera of mobile device

Abstract

Conference

Keywords

Access to Document

Fingerprint

Profiles

David Cameron Christopher Harris-Birtill

Datasets

SpeCam: sensing surface color and material with the front-facing camera of mobile device (dataset)

Cite this