Naturalistic Digital Behavior Predicts Cognitive Abilities

Individual differences in digital behavior have been extensively studied [9, 14, 39, 54]. A series of studies (Table 1) indicated that individuals exhibit varying behaviors in digital environments. For instance, prior works investigated how users processed information on the Web pages, such as how long the users dwell on a Web page [38], how many tabs the users opened, and how long these tabs were kept open [22]. Studies by Dubroy and Balakrishnan [22] revealed that a specific group of users switched tabs and changed pages more often than others. Similarly, the duration for which users spend on a page or Website, also known as dwelling time, has been used to reveal underlying cognitive processing in language comprehension and decision making [38]. Navalpakkam and Churchill [38] found that dwelling time and mouse tracking can help measure user attention on Web pages. However, their studies did not consider cognitive abilities associated with behavioral differences. Another line of research has focused on predicting the Web page the users would open next [11, 42]. They found that users with similar behavior patterns belong to the same group and would visit the same Web pages [42]. Although their studies were limited in context, they found that various behavioral patterns could emerge across user groups.

The way information is presented on a user interface has also been found to affect users’ perception and behavior [8, 9]. Chin and Fu [9] examined how users with lower cognitive performance perceived information presented in health information Websites. They asked users to interact with two user interfaces having different information organizations—one with a higher amount of text focused on health symptoms, and the other with more general information about body parts and less text. They found that lower cognitive performing users, who have less knowledge about health symptoms, tended to spend more time examining individual Web page’s information and made fewer clicks on links within the Web page. On the other hand, higher cognitive performing users spent less time on individual Web pages and made more clicks on links within the page. This was likely due to cognitive differences between individuals, which can impact how users perceive and interact with information in user interface [9, 62]. Zhou et al. [73] also found that lower cognitive performing users struggle with understanding the meaning behind graphic icons, leading to difficulties in using the computer. Furthermore, visual search behavior has been found to be associated with cognition and individual personality traits [19, 40]. For instance, introverts completed visual search tasks longer than extroverts [19]. Additionally, previous studies have suggested that the speed with which individuals understand information, as captured by rapid eye gazes during text skimming, is influenced by cognitive ability [23, 37].

Differences have also been shown in input behavior across individuals [15]. Aula et al. [2] found the expertise level of computer use impacted the speed of typing a search query in the query box. Expert users would formulate queries and evaluate the search results more quickly than novice users. Imbir et al. [28] examined individual differences in writing behavior and such behavioral data was used to infer the user’s emotion during interaction with a word processor [28].

Overall, digital behaviors, including visual information perception, information processing, and typing speed, are well-established factors and have been considered broadly for research studying individual differences [17, 25, 35, 55]. Despite extensive research, to our knowledge, there has been no work on predicting cognitive abilities from naturalistic interaction data. Most previous studies have been based on lab-based experimentation, and results and findings remained hypothetical approximations of real-life digital behavior. For instance, in lab-based studies, users were asked to engage in a simulated work task (e.g., interact with visualizations [7] or perform a search task [10]). Task difficulties were often determined before the study so individual differences could occur [10]. Our research contributes further evidence on the impact of cognitive abilities to users’ online behavior by capturing real-life data through continuous 24/7 monitoring of users’ digital activities in natural settings. To characterize naturalistic digital behavior, we extract features from monitoring data, including cumulative text, cumulative entropy, tab change rate, number of page visits, number of site visits, and typing speed. We consider these general behavioral features as they do not depend on any specific task but on the direct behavior of the users in the digital world. Furthermore, previous studies have typically relied upon data from adults having normal cognitive abilities [17, 22]. In this article, we analyzed the digital behaviors of a diverse sample of users, including both adults with reduced cognitive ability and those with normal cognitive ability.

The role of cognition in technology usage has suggested that behavioral differences are linked to users’ cognitive abilities [13, 54, 58, 71] (see Table 1). However, it is unclear whether cognitive abilities are directly linked to digital behavior in the wild. Researchers have studied technology usage and cognitive abilities in specific tasks. Findings from an early study [58] suggested that cognition is an important predictor of technology usage. For instance, increased usage of video games and navigation applications was associated with higher cognitive abilities. In another study, adults with reduced cognitive ability were found to perceive technology as more difficult to use than those without cognitive deficits [47]. However, most studies relied primarily on surveys of older adults and did not consider real-world behavior. A few studies, however, analyzed the actual usage data of the users [16, 33, 51]. Karanam and van Oostendorp [33] considered searching behavior and found that users with higher cognitive abilities completed tasks faster. Crabb and Hanson [16] analyzed the users’ browsing histories and suggested that individuals with higher processing capacity tended to browse more Web pages than their lower-cognitive-ability counterparts. Smith and Chaparro [51] focused on input behavior and found that adults having reduced cognitive ability wrote slower than those without cognitive decline. However, users were instructed to work on an information task in a single uninterrupted lab session, e.g., usually an hour. In real life, users are unlikely to concentrate on a search task for such a long session, making user behavior in the lab different from that in the wild. In addition, those studies mainly focused on the impact of age-related differences in searching and browsing but did not examine how the user’s cognitive ability is associated with different digital behavioral patterns. Age differences have been shown to affect psychomotor and processing speed in Web browsing [10] and restrain user behavior in general [31]. Growing evidence has also been reported on variance in visual scanning being dependent on the cognitive properties of each individual [14, 36].

Working memory is another cognitive ability that has been found to be associated with browsing behavior [10]. Peterson et al. [41] showed how working memory imposed significant constraints on how people work on interactive tasks. Working memory has also been considered in the design of interaction in adaptive systems [30], and information-intensive tasks are performed with strategies that are compliant with individuals’ particular memory limits [54].

Moreover, selective attention ability has been found to affect people’s process of the information they visually attend to [10]. Specifically, users having higher cognitive ability have been shown to perform better in tasks requiring divided attention, such as web and folder navigation tasks [9]. The structure in which the Websites are organized may affect the amount of attention required for the navigation process. The deeper in the Website hierarchy the target page is located, the more attention is needed for the navigation process. Chin and Fu [9] suggested that recommendation systems could minimize such processing demands and better support users having reduced cognitive ability. However, previous research has not focused on predicting cognitive ability, and studies have been conducted in limited contexts, such as navigation tasks within a health information system [9].

Another cognitive ability affecting digital behavior is fluid intelligence. Trewin et al. [56] explored individual differences due to fluid intelligence in information search tasks. The results suggest that individuals with higher fluid intelligence performed computer activities more intensively than those with weaker fluid intelligence.

Research has also aimed beyond finding associations between cognition and behavior by building predictive models to estimate cognitive ability. Predicting cognitive ability has previously been found to be important for personalization for a wide array of services in technology usage [18] or office productivity [26], and in avoiding motor vehicle accidents [70]. Gordon et al. [25] found that users with lower levels of cognitive ability used fewer smartphone apps, and sent few messages, and such behavioral information can be used to predict the cognitive test performance of the users. However, their work focused on a limited task context, such as classifying whether a user would fall into the cognitively young or cognitively old groups but did not directly predict cognitive abilities. Conati et al. [13] considered eye-tracking data to predict the user’s cognitive ability while interacting with visualization. However, their work was conducted in the laboratory and relied solely on eye-tracking data.

Here, we focus on understanding whether the user’s cognitive abilities can be associated with differences in digital behaviors and examine whether it is possible to predict cognitive ability variables from data recorded from real-world naturalistic digital behavior. Although there is already solid evidence that cognitive abilities are related to several behavioral factors [25], most studies examining digital behavior have focused on only a few of these factors in isolation and in controlled laboratory studies. Our unique contribution is to assess the relationships among many cognitive abilities (psychomotor speed, processing speed, selective attention, working memory, and fluid intelligence) with behavioral factors as they occur in the naturalistic in-the-wild behavior of a diverse sample of users. Moreover, we study whether naturalistic behavioral data is predictive of cognitive abilities. That is, whether it is possible to record implicit behavioral logs and use that data to draw reliable predictions on different cognitive abilities.

We used a digital activity monitoring and screen recording system to record all user behaviors on the computer continuously. The system recorded screen frames at two-second intervals, and OS log information associated with screen frames, including the titles of active windows, the names of active applications, the Uniform Resource Locators (URLs) of Web pages on active applications, and timestamps. In addition, the system also collected keystrokes and mouse behaviors, including clicks and scrolls and associated timestamps. The digital activity monitoring and screen monitoring system was developed for Microsoft Windows OS. We used Desktop App UI to implement the monitoring system. The system performed monitoring functions: saving active windows as images and collecting the aforementioned OS log information. In addition, a stopping function was also implemented to allow the users to stop the monitoring.

To ensure our sample of participants reflects variance in cognitive ability, we recruited participants with varying ages as age is often associated with cognitive differences. Overall, 20 individuals aged 19–71 participated in our study (Table 2). Ten reported themselves as males, ten as females, and none reported themselves as other gender. Their mean age was 40.4, SD 18.3. It should be noted that we did not use age grouping in the study, but the purpose was to have representative samples of varying aged users. This design choice allowed us to acquire participants with a wide range of cognitive abilities and observe behavior differences among them.

The participants were recruited via a posting that was distributed to the University of Helsinki’s mailing lists and alumni communities. A questionnaire (see Appendix A.1) was attached to the recruiting message to collect background information on potential candidates. Only respondents who used laptops as their main devices for everyday digital activities and had a higher education background were considered eligible for the study. Another eligibility criterion for participating in the study was that respondents were required to have sufficient skills and experience in using various software. Respondents had to answer nine open-ended questions assessing their theoretical and practical knowledge in the context of the usage of computers and software applications. Respondents were also asked to work on a software installation task. They were asked to follow a guide for installing a specific software application on their computers. Only respondents who answered all questions correctly and managed to install the software were qualified for the study. All participants had laptops of recent models and had Windows 10 OS installed on their laptops. Such criteria of eligibility would help reduce the potential confounding effects of other factors on the overall results of the experiment. For instance, users with lower levels of education and computer knowledge might have less interest in using their computers. This could affect their computer use in frequency, intensity, and duration. They might not use their computer as much, or for as long, as other users with a higher level of education and computer knowledge, which would have become a factor in the experiment.

Participants were all knowledge workers. They were University researchers, faculty members, students, and city administration workers. Out of the 20 participants, six participants had a graduate degree as their highest level of education, 12 participants had an undergraduate degree, and two participants were in undergraduate programs. All participants had advanced computer skills, were experienced computer users, and their working language was English. All participants were healthy, right-handed, and with normal or corrected-to-normal vision.

The participants were informed of our privacy guidelines prior to joining the experiment. They were told that the monitoring of behavioral data was stored on their computers during the monitoring phase. Afterward, the data would be transferred to a secured server and used only for research purposes. After the experiment, the participants were compensated with 150 Euros. A consent form was obtained from the participants regarding the data usage, privacy, and experiment procedure.

Upon agreeing to participate in the experiment, the digital activity monitoring and screen recording system was installed on each participant’s laptop and set to run continuously in stealth mode for 14 days. Participants were explained that the monitoring system was automatically launched whenever the laptop was turned on. The whole session including receiving instructions and installing the monitoring system took approximately 15 minutes. The participants were advised to use their laptops as usual and to avoid stopping the monitoring unless necessary during the monitoring phase. After 14 days, the participants visited our lab, and the digital activity monitoring system was uninstalled from their laptops. The monitoring data was then transferred to our secured system. The whole process took approximately 10 minutes. The research was approved by the ethical committee of the University of Helsinki and complied with the declaration for managing data obtained from human participants.

Table 3 presents the summary statistics of digital activity data. Overall, 1,442,447 (\(M=75,918,\text{SD}=75,570\)) screen frames were recorded from the participants for a duration of 14 days. Participants spent an average of 38.13 (\(\text{SD}=24.41\)) hours over two weeks using the computer.

We examined the data gathered from the participants to ensure that it was approximately balanced across participants with respect to the amount of usage by the hour of the day and by application type. The amount of usage (time spent using the computer) was considered “screen time” during which there were input and output signals from the keyboard, mouse, and information changes on the screen. We computed average screen time by hourly rate. Furthermore, we extracted the average share in percentage for individual applications and categorized them into appropriate categories (Appendix A.3) based on the coding structure from the prior work [35].

Figure 2(a) and 2(b) present participants’ average screen time and average share in percentage for various applications. Participants had balanced screen time and actively used their computers between 8–11 and 13–17. Participants mostly spent time using Web browsers with an average screen time of approximately 80%.

Table 4 presents digital behavior factors used in the study. Three behavioral factors were studied: information perception, information processing, and input behavior. Information perception refers to how much information a user perceives by measuring the amount of cumulative text and the number of cumulative entropy users consumed. Information processing refers to how much information a user processes by measuring the number of tab changes, page visits, and site visits. These measures were normalized to be on an hourly rate as participants may spend a varying amount of time on their computers. Lastly, input behavior describes how fast a user provides input to a computer system by key presses. We measured input behavior by computing the average typing speed of texts entered in the Omnibox (address bars and search boxes). Behavioral factors were extracted from the digital activity data of 20 participants and are described in greater detail below:

Figure 3 illustrates an example of how digital behaviors were extracted from 24/7 monitoring data. In the figure, the user engaged in learning activities for Web design. To compute cumulative text, screenshots were first converted into texts. The number of unique words appearing in the screenshots was considered cumulative text (341 unique words in this example). Then, screenshots were converted from color to grayscale to compute cumulative entropy. The entropy was computed by summing the proportion of pixels with each grayscale color weighted by its logarithm (the cumulative entropy of five screenshots in the example is 185). In the example, there were three Web browser tabs used, and the user browsed three distinct Web pages in this example. Two Websites were opened. The user opened the Google search interface and entered a query, for which the average typing speed was 242 milliseconds per character.

Table 5 presents the set of measures used to quantify cognitive ability. We focused on these cognitive abilities because they have been shown to influence user behavior in prior research [10, 13, 25, 33, 50]. All participants performed a battery of standardized tasks that measured their cognitive abilities, including psychomotor speed, processing speed, selective attention, and working memory. Pavlovia platform¹ and PsychoPy framework² were used for the implementation of the cognitive testing tasks (see Appendix A.2 for details of these tasks). The cognitive measures and tasks are described as follows:

To avoid possible confounds in relation to fatigue, the cognitive testing session was conducted in the morning on a different day from the digital activity monitoring with a 5 minute break following each test. In addition, after the third test, there was a longer break in which participants were allowed to take as much time as they needed. There were instructions and practice trials prior to each test (except the intelligence test). Participants did not know which measures the tests involved and were asked to do the tests as best as they could following the instructions. Participants performed the tests in a randomized order, approximately 5–7 minutes per test. A cognitive testing session was conducted after the monitoring so that participants were unaware that their digital behavior would be associated with their cognitive abilities. Participants received no feedback or knowledge of results on these tests throughout the cognitive testing session but were debriefed upon exiting the study.

Regression analyses and path analyses were conducted to reveal associations and causal dependencies among behavioral factors and cognitive abilities. Shapiro–Wilk tests were conducted to determine whether or not the behavioral data followed a normal distribution.

Linear regression was used with measures of behavioral factors as dependent variables and measures of cognitive abilities as independent variables. All \(p\) values were adjusted using Bonferroni correction.

Path analysis was conducted to test causal relationships for the three behavioral factors using structural equation modeling [57]. In the model, the five cognitive measures were set as the independent variables. Measures of the behavioral factors were set as the dependent variables. Then behavioral factors: information perception, information processing, and input behavior were added as mediating variables in the model.

The summary statistics of cognitive measures and behavioral measures are shown in Table 6. The results show that cognitive ability varies across individuals and large variances are observed for most cognitive measures indicating a recording of data from a diverse pool of participants.

Information Perception. Table 7 and Figure 4 show the results for the associations between cognitive abilities and the users’ information perception behavior. We found substantial effects of psychomotor speed and fluid intelligence on the amount of cumulative text consumed by the users. Regression analyses reveal an association between psychomotor speed and cumulative text rate with a coefficient \(F(1,18)=19.42\) and an effect size with \(R^{2}=0.533\) (\(p\,{\lt}\,0.001\)). Similarly, the results show a significant linear relationship between fluid intelligence and cumulative text per hour with an effect size (\(F(1,18)=13.21\), \(R^{2}=0.43\), \(p=0.002\)). The Shapiro-Wilk test indicated that cumulative text followed a normal distribution (\(W=0.95,p=0.39\)).

Figure 4(a) also indicates an association between the amount of cumulative text consumed per hour and the measures of each of the two cognitive abilities (psychomotor speed and fluid intelligence). Generally, regression tests revealed that individuals who scored higher in psychomotor speed (higher in the speed of thinking and decision-making) and higher in fluid intelligence were able to scan texts on the screen more quickly than those who scored lower.

Furthermore, we examined the effects of cognitive abilities on the amount of cumulative entropy, as shown in Figure 4(b). Significant associations between the amount of cumulative entropy per hour and five cognitive abilities were not observed.

Information Processing. Table 8 and Figure 5 show the associations between cognitive abilities and the user’s information processing behavior. For tab-changing behavior, three cognitive abilities: processing speed, selective attention, and fluid intelligence, were associated with the tab-change rate with substantial effect sizes. Regression analysis revealed an association between processing speed and tab change rate with a coefficient \(F(1,18)=8.604\) and an effect size \(R^{2}=0.34\) (\(p=0.0093\)). Similarly, \(F(1,18)=8.61\) and \(R^{2}=0.34\) (\(p=0.0092\)) were found for selective attention ability; and \(F(1,18)=6.28\) and \(R^{2}=0.35\) (\(p=0.023\)) were found for fluid intelligence. The Shapiro-Wilk test revealed that the tab changing rate followed a normal distribution (\(W=0.91,p=0.25\)).

Figure 5(a) presents an illustration of the regression analysis for the tab change rate against measures of the five cognitive abilities. Linear relationships between the number of tabs that changed over time and the three cognitive abilities were found: processing speed, selective attention, and fluid intelligence. Intuitively, this result indicates that users with lower processing speed, lower attention ability, and lower fluid intelligence spent more time on individual tabs than their counterparts who obtained higher scores on these cognitive tests.

Figure 5(b) shows the effects of five cognitive abilities on the user’s page visit behavior. Associations were found between processing speed and page visit rate (\(F(1,18)=5.98\), \(R^{2}=0.26\), \(p=0.026\)) and found to be normally distributed with \(W=0.97,p=0.87\) for page visit rate. The results indicate that users with higher processing speed visited Web pages more often than users with lower processing speed. In contrast, users with lower processing speed tended to spend more time dwelling on the pages they accessed.

Figure 5(c) shows the effects of five cognitive abilities on the site visit behavior. Interestingly, we found that only selective attention ability was associated with this behavior with a coefficient \(F(1,18)=5.18\) and an effect size \(R^{2}=0.23\) (\(p=0.036\)). The Shapiro-Wilk test also revealed that site visit behavioral measure was normally distributed with \(W=0.97,p=0.81\). The results suggest that individuals with better selective attention could focus on relevant information better while moving between Websites.

Input Behavior. Table 9 and Figure 6 present the results of regression analyses between cognitive abilities and Omnibox typing speed. We found an effect of working memory on how fast the users type in the Omnibox (address bar and search box). The effect was substantial with a coefficient \(F(1,18)=5.033\) and \(R^{2}=0.23\) (\(p=0.038\)). The Shapiro-Wilk test indicated that input behavior data was normally distributed with \(W=0.91\) and \(p=0.21\). The results suggest that people who scored lower in the working memory test would take longer to type in the Omnibox. The reason is likely due to the need to recall a Website URL or the content of a query, which may depend on a user’s working memory.

Causal Effects. The linear regression analyses revealed how cognitive differences were associated with multiple behavioral factors. We considered structural equation modeling for path analysis to further understand the causal dependencies of digital behaviors on cognitive abilities. Because prior works have shown possible relations between cognitive abilities [12] that may explain the direct and indirect effect on user behavior, we also report results using a path model that accounts for correlation and causality between the abilities. Additionally, the path model also revealed associations among behavioral factors, such as information processing, information perception, and input behavior.

Figure 7 presents the results of structural equation models with each node showing a coefficient for each link. Nodes with links in between indicate significant dependencies with \(p\,{\lt}\,0.05\), and the arrows indicate causal directions. In contrast, nodes without links in between indicate non-significant relations.

The structural equation models confirmed the results of regression analyses. Selective attention was associated with the amount of information processed by the users and also caused individual differences in this behavior—-with a coefficient of 7.73 (\(p=0.009\)). This suggests that higher selective attention causes a better ability to process information from multiple sources, including increased tab change rate and the number of page and site visits per hour. For example, for users with higher selective attention abilities, the cost of switching between tabs and Web pages (the ability to maintain focus on important information while changing tabs) is much lower than for those with lower attention spans. We also found that processing speed and fluid cognitive abilities indirectly affected how users processed information.

Fluid intelligence has a coefficient of 0.17 (\(p=0.03\)), meaning that higher intelligence causes moderately better perception. Users with higher fluid intelligence were faster at scanning textual and visual information than those with lower fluid intelligence. On the other hand, users with lower fluid intelligence could compensate for their lower cognitive ability with a more focused approach. For instance, they could slowly examine the on-screen content so as not to miss any important information, and carefully decide to change the tab so that each interaction would lead to useful information to maximize the information gained per tab unit. The path models confirmed a relationship between information processing behavior and information perception behavior with a coefficient of 0.44 (\(p=0.024\)). In addition, psychomotor speed has a coefficient of 0.51 (\(p=0.004\)) with fluid intelligence, indicating that psychomotor speed may have an indirect effect on the user’s information perception ability.

The results also showed a direct path between working memory and typing speed confirming that faster typing speed was caused by better working memory of individuals. A negative coefficient of \(-0.29\) (\(p=0.035\)) was found for the causal dependence of input behavior on working memory. Consequently, individuals with lower working memory capacity would type more slowly than those with larger working memory capacity. This result must be interpreted in context. The typing speed was measured from typing in the address bar and search boxes, which have a particular requirement of working memory access. Therefore, the result may not generalize to typing or writing more generally.

Furthermore, the path model revealed an association between information perception and information processing with a coefficient of 0.44 (\(p=0.024\)). This result indicates that the amount of information that a user processed was influenced by the number of pages the user opened. However, there were no associations among input behavior information perception, and information processing.

We considered both linear and non-linear approaches for the prediction. Two prediction models were chosen: a Random Forest model and a Multiple Linear Regression model. These models have also been previously shown to be effective in similar tasks [13, 25]. Behavioral variables were used to construct a model for predicting cognitive ability.

The models were trained with leave-one-user-out cross-validation, and the data were split into a training set and a test set accordingly. In each iteration, the model was trained using data from 19 participants and predicted the cognitive measures of the left-out participant, such that the test set always contains an unseen user. More specifically, the input for training the models consisted of feature vectors of digital behaviors (cumulative text, cumulative entropy, tab change rate, page visit, site visit, and Omnibox typing speed) and the corresponding cognitive test scores of 19 participants. The model output was the prediction of individual cognitive test scores of the left-out participant—as the experiment followed a cross-validation setup. The performance of the models was then evaluated using Root Mean Squared Error (RMSE) for all cognitive measures. RMSE is a commonly used measure in the field of machine learning and data science [1, 20]. The RMSE was calculated by taking the difference between the predicted values and the actual values. The results were then averaged and the square root of the average was taken. This metric indicates how close the predicted values were to the actual values. Lower values indicate better performance.

For Random Forest models, the hyper-parameter max tree depth and the number of trees were determined by 10-fold cross-validation using the training set within each test user separately. To set the hyper-parameter of Multiple Linear Regression models, we enabled fit intercept and disabled normalization. This ensured that data leakage could be fully avoided as the model had no information about the user’s data used for evaluation.

The models were compared with two control models: (1) prediction using a permutation technique and (2) random guessing/prediction. For the permutation technique, we randomly permuted all features (behavior factors) and targets (cognitive measures) on the training set so that any possible relation between features and targets, except their distribution, was destroyed when constructing the model. Thus, this training procedure retains the distribution of the data but should not carry any predictive information. We used these permuted data to fit the Random Forest model for prediction. For each LOO iteration, the permutation process was repeated 1,000 times to get a vector of 1,000 values of RMSE, and the mean of this vector was used as the output value. For random guessing/prediction, we generated a random value for a measure of cognitive ability. We also repeated the random guessing process 1,000 times. We used paired t-tests with Bonferroni correction to test for significant differences between the models.

Figure 8 shows the results for cognitive ability prediction. The Random Forest model that leveraged all available features produced the lowest RMSE in all cognitive abilities. Pairwise comparisons indicate significant differences in RMSE between the Random Forest model and the control models (permutation and random guessing/prediction), with all \(p\) values \(\,{\lt}\,\)0.007.

The Multiple Linear Regression model also produced small RMSEs compared to the control models. Significant differences were found between the Multiple Linear Regression model and the two control models (permutation and random guessing/prediction) for psychomotor speed and selective attention, with \(p\) values \(\,{\lt}\,\)0.007. However, using Multiple Linear Regression to predict processing speed and working memory did not produce an RMSE better than the permutation models.

The permutation models also show relatively low error rates because the cognitive ability measures seem to have a natural range. In other words, cognitive measures do not vary between 0 and a maximum score but are on a more limited range reflecting empirical lower-bound and upper-bound cognitive abilities of healthy users. However, all results show significant differences between the permutation and prediction models trained with non-permutated data. In summary, the results suggest that cognitive abilities can be predicted reliably based on behavioral patterns with low error rates, and significantly outperforming permutation and random models.

Figure 9 presents the importance of behavioral features and their contributions in the Random Forest models. The results indicate that cumulative text was the most important feature for predicting psychomotor speed, processing speed, and fluid intelligence. Information processing behavior comprising tab change rate, page visit, and site visit were the most important features for predicting selective attention. Lastly, typing speed was among the top features for predicting working memory. The results of feature importance are consistent with regression analysis and path model analysis that working memory, selective attention, and fluid intelligence affect users’ digital behavior.

The results show how cognitive abilities can explain differences in digital behaviors and how the use of behavioral patterns predicts cognitive abilities. In this section, we distill the findings and answer the RQs defined earlier.

RQ1. How Is Cognitive Ability Associated with Naturalistic Digital Behaviors? The results of regression analyses and structural equation modeling indicate that different measures of cognitive ability are associated with features of individuals’ digital behavior. We found that higher fluid intelligence was associated with and caused faster information processing. More specifically, people having higher fluid intelligence processed information, such as scanned text, faster than those who scored lower. In addition, our results suggested an indirect effect of psycho-motor speed on fluid intelligence. Higher selective attention was associated and had a causal relationship with the amount of information processed by individuals. Participants with higher selective attention changed tabs more frequently. This finding is congruent with findings from previous research that linked cognitive abilities, and multitasking [44]. On the other hand, individuals who scored lower in selective attention spent more time examining information on individual tabs. Findings in recent work [13] revealed that lower perceptual speed negatively affected the ability to perceive visual information. Thus, our results suggest an extension to the previous findings by determining that selective attention could also affect the user’s information perception. We also found that higher working memory capacity was associated and had a causal relationship with typing speed. Participants with higher working memory capacity typed faster than participants with lower working memory capacity.

RQ2. Can We Predict Cognitive Abilities from Naturalistic Digital Behaviors? We found that predictive models utilizing all behavioral features significantly outperformed control models for all five cognitive abilities. The Random Forest model yielded the best performance with the smallest error rates compared to the other models. The results indicate that behavioral and interaction data are predictive of users’ cognitive abilities and applicable to naturalistic data obtained in the wild. This indicates that naturalistic behavioral data can be utilized to estimate users’ cognitive ability. We also found that using a combination of behavioral features can be beneficial for predicting cognitive abilities. Different cognitive abilities can be predicted using a different set of behavioral cues. For example, working memory can be predicted using a combination of typing speed and tab change rate. Similarly, fluid intelligence can be predicted using a combination of the frequency of page visits and the amount of perceived cumulative text.

We found that the cognitive abilities of individuals have a direct association with their daily digital activities. While a recent work shows that differences in computer usage may be primarily attributable to differences in user preference [22], our analyses revealed that many behavioral differences could be largely caused by cognitive ability. Our findings suggested that digital behavior could be used as a passive preliminary detection tool for cognitive ability and even impairment.

The results of our prediction strongly indicated the feasibility of modeling and predicting cognitive abilities. In particular, a simple model, such as Random Forest yielded low error rates for predicting cognitive abilities. Currently, most cognitive assessment is done in clinical trials when a person visits a physician, but our study revealed it may be possible to detect cognitive decline directly from digital behavior sources, opening new opportunities for researchers in cognitive sciences and possibly even clinical practice.

A generalizable, predictive model can also save time and cost for user interface designers or researchers by offering them prior insights into individuals’ backgrounds and interaction difficulties before testing with the users. Although our results provide initial guidance on how to build compelling user models depending on the target cognitive abilities, there is a need to examine whether these findings would apply to other modeling tasks beyond the ones analyzed here.

Detection of cognitive abilities may also benefit users when coupled with an adaptive system [69]. Various designs and techniques may be chosen to adapt to individual users’ cognitive abilities. In particular, adaptive systems and services can benefit people affected by the high intensity of information presentation and the requirement of memory recall for input. For instance, users with reduced ability to maintain focus on relevant pieces of information (low selective attention) could benefit from customized user interface layouts that prioritize essential information while removing other distractions or extraneous details. For users who have low fluid intelligence, because those people have difficulty figuring out rules operating user interfaces; less dense text designs featuring more visual metaphors and navigation aids could be implemented. For users with reduced working memory who had slow typing speed, interfaces could be designed with simplified navigation-based menu options in which the information could be accessed through shortcuts rather than manually typing it in. This approach could lower the cognitive load in retrieving the information.

The error rates of the predictive models were low, and we believe the prediction accuracy is sufficient for transferring these results to practical systems. Nonetheless, given that there are still small errors in predicting the user’s cognitive ability, researchers should consider that the benefit of supporting users who need help might still outweigh the possible hindrance caused by unwarranted adaptation delivered to a minority of the users. However, further experiments should investigate the findings in the scope of real-world adaptive systems to enable useful interventions and reduce the possible intrusiveness of unwarranted adaptation. For instance, personalization could be suggested to the users via a gentle prompt message, in a mixed-initiative fashion, so that users could still ignore or decline the adaptations if they do not perceive them as useful.

Sample Size. The sample of participants was relatively small and consisted primarily of knowledge workers. Nevertheless, it was carefully designed with the assumption that knowledge workers would use computers more often than other groups and that the varying ages of participants would likely account for cognitive differences among them. However, results were found to be significant even after correcting for multiple comparisons in a very conservative way (Bonferroni). Moreover, the predictive models also performed better than random prediction, which indicated that data was sufficient to reveal the suggested associations and to demonstrate that prediction was possible. In addition, our sample is not fully representative of the computer-using population, and our results should be interpreted in this context. Future work could consider a large-scale data collection campaign spanning different user types and target populations to provide further evidence on computer usage patterns.

Behavioral Recordings Source. We focused on behavioral signals recorded on a personal computer and did not consider data from other devices, such as smartphones. More and more diverse data could complement the present findings and possibly improve the prediction results. Similar to interaction with mobile devices [25], digital behavior in computer systems also depends on many contextual factors, which could become additional confounding factors for data analysis. For instance, the data collected did not account for individual device quality. Because our experiments were conducted in the wild, device factors, such as screen size, speed, and ergonomics could potentially impact how participants used their computers. Nevertheless, we have already taken an active step toward this limitation by involving participants in completing a prescreening questionnaire, reporting that they have been using computers for daily work and leisure activities and were familiar and comfortable with their devices. Therefore, it is unlikely that these factors would have affected the significance of our findings.

In addition, digital behavior captured for this research occurred within the naturally occurring context of their work and home activities. It is difficult to ascertain the full scope of influences on digital behavior in such an environment. Despite this factor, we believe that data captured from the real-world environment contributed to the strength of our findings. Our studies can better explore how individuals understand, use, and appropriate technologies in real-world situations. The possibility of linking such behavior with cognitive ability could open up a new direction for understanding the cognitive aspects of HCL. For instance, predicting cognitive ability could benefit personalized adaptive systems, such as designing cognition-aware technologies focusing on ways to minimize cognitive effort for interface interaction by reducing the complexity of user interfaces, such as adjusting the amount of information presented to users having shorter span of attention or recommending browsing histories [29, 64, 65] to aid users having reduced working memory.

Behavioral Factors. Our analyses only concerned a limited number of behavioral factors since they were used in previous research, e.g., [19, 22, 34, 63, 66, 72]. We accounted for different tasks and habits by normalizing the measures per hour so that measures would be comparable and independent of temporal and task variability. However, future work could consider other metrics of digital behaviors, such as re-visitation patterns, sequential patterns of application use, or contextual factors from other devices (e.g., mobile). Furthermore, behavioral metrics used in prior works were mostly Web browser-specific [8, 9, 13, 17]. In this study, we also considered other behavioral factors that were independent of specific application sources, including cumulative text and entropy. Future research could investigate how the models could benefit from non-browser behavioral data to predict the cognitive abilities of users who do not use the Web frequently.

In addition, only Shannon theory [49] was utilized for the calculation of cumulative entropy; future work could consider color entropy and luminance entropy of an image [72]. Cumulative text measure only accounts for new text appearing on the screen. However, participants might not necessarily focus on all the words in the text, or participants would just change tabs without reading the content. However, prior research has considered these behavior signals as useful indicators of differences in information processing and perception across different demographic populations [68, 72]. They are easy to obtain because current logging systems in many modern web applications are able to capture these types of signals, while sophisticated approaches, such as web-based eye tracking are costly, require calibration, and sometimes encounter certain problems with external variables, such as varying lighting and device configurations, as well as privacy issues. Our research focused on accessible metrics, such as the amount of text presented on the screen and the findings show that these indicators are sufficient for revealing meaningful relationships. We are also aware that these measures do not perfectly capture the user’s information processing activity. To address the limitations of the measures, in addition to cumulative text and entropy, we also considered several moderating factors, such as user engagement and interaction. For instance, we have used a variety of measures involved with reading, such as time spent on Web pages and typing behavior, to better understand the degree to which users process and use information from Web pages. Our research has shown that the combination of these measures and moderating variables offers considerable insight into how people process and perceive Web pages connecting to their cognitive ability. However, further research is required to better understand the accuracy of our measurements compared to the accuracy of alternative measures, such as eye tracking, scrolls, and clicks.

Some measures assigned to behavioral factors may not entirely represent the characteristics of those factors but fractions of them. For instance, the Omnibox input may be limited but is still indicative of more general input behavior from the user. Furthermore, modern browsers often autocomplete the URLs and limit the number of characters that the user has to input. This limits the extent of input behavior that Omnibox inputs can represent. However, it was also the aim of this research to investigate how such commonly used measures could reveal cognitive ability. In addition, those behavioral signals might be affected by the user’s physical environment, such as sitting position and ambient noise that could affect the speed of typing queries or URLs. It might take days or weeks of monitoring for the model to learn from this behavior signal to successfully predict cognitive abilities, while it would take less data (e.g., an hour or so of monitoring) when a user works on the computer in a lab-based controlled environment, where the user is provided with a proper chair and a quiet condition. This is also an important implication as we demonstrated that a larger portion of data was necessary to obtain significant results. However, the duration of the experiment, of course, was determined in the design phase and it may be possible that shorter monitoring would also be enough. Future studies could examine how the predictive accuracy of cognitive abilities would be affected with respect to the length of the monitoring phase. This is to improve user privacy and reduce the amount of data collected.

Data Collection. Users could turn off the data collection software anytime during the study. This feature was necessary for ethical data collection as participants consented with the option to opt out of the study at any time. It also allowed for a potential bias in the collected data, as participants could disable data collection on sensitive Websites. However, we observed that the screen monitoring was operational well over 95% of the time and was turned off on average less than 30 minutes per participant. Consequently, we believe that the data used for the analysis represents naturalistic digital behavior. Furthermore, because age has been found to be associated with cognitive ability, future studies could also explore the potential for predicting age from digital behavior.

The ability to detect cognitive abilities from simple computer usage offers a powerful tool for understanding users and adapting computation to make information more accessible for users with varying cognitive abilities. At the same time, the finding that simple behavioral data can predict cognitive ability raises severe privacy concerns.

Our results indicate that as relatively simple behavioral measures could allow us to predict cognitive abilities, we should be careful about the kind of data we allow service providers to collect via Web browsers or other applications. Cognitive profiles can be used to derive the user’s identity and cognitive ability with little or no consent from users. This means that data-sharing policies should be put in place that restrict the use of user behavioral data for predicting cognitive ability. Users should be given control over the data they are willing to give out for analysis, such as control of what goes into the user model, what information from their model is available to different services, and how the model is managed and maintained.

We followed strict guidelines and policies also in the research reported here. As monitoring data can reveal the user’s cognitive ability to unauthorized third-party entities, we were well aware of such implications and carefully managed users’ data with encryption, controlling access, and building models so that private data was not available even for the researchers themselves. Our study procedure and consent forms were reviewed and ethically accepted before data collection. All analyses were conducted offline, and data was not exposed to undesired third parties.