Project Desciption
Overview
Imagine reading a text that knows you are looking at it – imagine reading a text that adapts to how you read it and supports you as you read. How we read today has changed little since the invention of printing. However, new technologies can make it possible to identify individual reading patterns and adjust the reading material according to the reader, making reading easier and more accessible also for reading impaired to improve comprehension. This needs a robust set of machine learning methods that integrate and classify features from complex, noisy, multimodal data of biological origin in real-time.
We apply an interdisciplinary approach of data- and vision science, typographic design and software engineering to develop new robust methods to extract features of reading from real-time measures of eye-movements, pupil dilation and e.g., posture, allowing us to adopt and assess the impact of individualized typographic settings and support the changing needs of the user.
Fig 1: An overview of the system. The user interacts with a reading device, controlled by a reading app. Simultaneously an eye tracker records gaze points, pupil size, and posture, from which additional features (saccades, fixations, and regressions) are extracted. A classifier inspects this stream as well as signals from the reader (originating from the stimuli and from user-generated input) and generates intervention signals from this and a trained dataset. (From ( Citation: Hansen, Bækgaard & al., 2023 Hansen, J., Bækgaard, P., Valgeirsdottir, D. & Beier, S. (2023). Universal design of gaze interactive applications for people with special needs. ) ).
Background
More than 216 m people have moderate to severe vision impairment ( Citation: Bourne, Flaxman & al., 2017 Bourne, R., Flaxman, S., Braithwaite, T., Cicinelli, M., Das, A., Jonas, J., Keeffe, J., Kempen, J., Leasher, J., Limburg, H. & (2017). Magnitude, temporal trends, and projections of the global prevalence of blindness and distance and near vision impairment: A systematic review and meta-analysis. The Lancet Global Health, 5(9). e888–e897. ) and many are challenged readers ( Citation: Legge, Rubin & al., 1985 Legge, G., Rubin, G., Pelli, D. & Schleske, M. (1985). Psychophysics of reading—II. Low vision. Vision research, 25(2). 253–265. ) . However, reading has changed little since the invention of printing, although new technology makes it possible to identify individual reading patterns and personalize the presentation of text, making reading easier and text more accessible and comprehensible.
Now, imagine reading a text that knows you are looking at it and adapts to how you read, actively supporting your style of reading while you read.
We will develop a robust set of automated methods that
- integrate,
- classify,
- interpret and
- respond
to complex multimodal observations of user behavior in real-time.
This requires interdisciplinary work across data science, software engineering, typography design, psychology and vision science with the aim to extract user behavior data in real time by tracking fixations, eye movements, pupil changes, user interaction, time, text progression, reading patterns and more.
This allows us to characterize individual reading style, to identify typographic preferences and to adapt the presentation of text, in order to enable a new level of adaptive support for readers with impaired vision or cognition.
The project extends a recently initiated strategic collaboration between the Technical University of Denmark (DTU) and the Royal Danish Academy (RDA) that explores synergies between art, science, and technology to enrich society.
It involves pre-graduate and post-graduate students at both institutions and provides a vital element of research-based training. The developed tools have generic qualities, which subsequently can be repurposed for the assistance of individuals with other types of (reading) disabilities than the specific vision disorders included in this protocol.
The majority of the description above is courtesy the original funding application authored by Sofie Beier and Per Bækgaard.
Additional Reading
You can read more about the project and related work here:
Bibliography
- Bourne, Flaxman, Braithwaite, Cicinelli, Das, Jonas, Keeffe, Kempen, Leasher, Limburg & (2017)
- Bourne, R., Flaxman, S., Braithwaite, T., Cicinelli, M., Das, A., Jonas, J., Keeffe, J., Kempen, J., Leasher, J., Limburg, H. & (2017). Magnitude, temporal trends, and projections of the global prevalence of blindness and distance and near vision impairment: A systematic review and meta-analysis. The Lancet Global Health, 5(9). e888–e897.
- Hansen, Bækgaard, Valgeirsdottir & Beier (2023)
- Hansen, J., Bækgaard, P., Valgeirsdottir, D. & Beier, S. (2023). Universal design of gaze interactive applications for people with special needs.
- Legge, Rubin, Pelli & Schleske (1985)
- Legge, G., Rubin, G., Pelli, D. & Schleske, M. (1985). Psychophysics of reading—II. Low vision. Vision research, 25(2). 253–265.