For product cognition testing, an automated reporting system based on R.

1
Report Automation for Cognitive Tests
How we developed an automated reporting system
using R for a cognitive tests product
Distribution of QuickTap Distract median reaction
time
On QuickTap Distract, a task involving processing
speed, James Lomass median reactiontime is 538.5
msec, which is less than the median reaction time
of 653 msec. This median reaction time falls into
81th percentile of all participants.
QuickTap is a test that measures reaction time
when a user clicks after a visual stimuli is
presented
2
Mental health researchers and practitioners
administer cognitive tests to understand the
mental capabilities of people. Cognitive tests
assess different cognitive domains such as
Attention, Executive Function, Psychomotor Speed,
Memory, and Emotional Social Cognition. Some
applications of cognitive tests include
Understand how external factors affect a persons
cognitive ability Determine within reason if a
person has any mental impairments Assessing a
candidates cognitive ability during the hiring
process Measure traits that are related to
mathematics learning
A cognitive test report for a participant helps
us gain essential insights into their cognition.
The report can act as a tool to inform necessary
actions. Data collection of cognitive tests
through mobile technologies provides an efficient
way to generate such reports. We had the
opportunity to work with one such products data.
The goal was to prepare a real-time automated
reporting system. A researcher could generate
reports for the participants part of the research
study to compare their performance.
3
High-quality data is an essential prerequisite to
performing any data analysis to gain insights. We
performed different data quality checks to ensure
the data validation for the required data
analysis.
Accuracy The data should reflect real-world
scenarios. We used the product and analyzed the
log data for our interactions with the product.
Completeness The data should contain all the
required values. We checked if there were any
null values in the data.
Validity The data should follow some rules, and
the values should be within specified ranges. We
collaborated with the product team to validate
the data. Uniqueness There should be no
duplicate records or columns in the data. We
removed multiple columns that had redundant
information. Timeliness The data should be
available when required. The product team ensured
that data was available in real-time.
Consistency The data should be consistent
across product versions. We cleaned the data from
old versions of the product.
4
While performing the data quality checks, we also
came across some challenges. Multiple teams
developed the cognitive tests at different times.
There was no standard data format or data logging
guideline in place. We faced issues like
redundant variables, various metrics for the same
variable, values out of range, incomplete data,
etc. We collaborated with the product team to
prepare a data dictionary for the existing data
based on the data profiling. We designed a data
guideline by eliminating redundant variables,
specifying ranges and data format for each
variable. We asked the product team to use the
data logging guideline to make the required
changes to log data. Meanwhile, we cleaned the
old data and ensured it was consistent with the
latest version.
5
After validating the data, we performed
Exploratory Data Analysis (EDA) for each
cognitive test. We discovered patterns and tested
our hypotheses by using summary statistics and
visualizations. We then collaborated with the
product designers to create metrics and
visualizations for the report. Here are a few
metrics that we reported on
Error Rate Percentage of attempts with the
incorrect answer. It was an essential metric in
tests involving inhibitory control- the ability
to resist interference from distracting stimuli.
Reaction Time Time taken to complete a task.
We used this metric in tests involving
psychomotor speed- the ability to detect and
respond to rapid changes. Percentage Score
Percentage of attempts with the correct answer.
We used this metric in tests involving inhibitory
control, working memory, and social cognition.
Percentile Score We also had the percentile
score for each metric to know the participants
placement in the distribution.
6
For each cognitive test, we prepared a
visualization showing the distribution of scores
for all users and highlighted the median score by
a dashed line and the users score with a thick
brown colored line. We summarized the users
performance on each task with a text describing
their score, the median score, and where they lie
in the distribution using percentile score.
On QuickTap Distract, a task involving processing
speed, James Lomass median reactiontime is 538.5
msec, which is less than the median reaction time
of 653 msec. This median reaction time falls into
81th percentile of all participants.
QuickTap is a test that measures reaction time
when a user clicks after a visual stimuli is
presented
On Copy Kat, a task involving working memory,
James Lomas received a score of 10, which is more
than the median score of 8. This score falls into
the 66th percentile of all participants
CopyKat is a test that involves working memory.
The user is shown a sequence of colored tiles,
and thenthe user is asked to copy the sequence.
7
Color Trick is a test that involves inhibitition
control. The user asked to match the color of the
word with its meaning
On Color Trick, a task involving inhibition
control, James Lomas received a score of 86.7,
which is less than the median score of 90. This
score falls into the 49th percentile of all
participants.
Memory game is a test that involves working
memory. The user is shown a screen to memorize
and then asked a question
On Memory Game, a task involving working memory,
James Lomas received a score of 85.7, which is
more than the median score of 85. This score
falls into the 50th percentile of all
participants.
8
Once we finalized the metrics, visualizations,
and report format, we built an API using Plumber
(an R Package) to get details about the user. To
prepare data for the report, we wrote an R script
to fetch data for the user and transform it for
the next step. We used R Markdown and created
child R Markdown scripts for each cognitive test
and a master R Markdown script to combine
multiple child R Markdown scripts. Based on the
users tests, the master R Markdown would call
different child R Markdown and connect them to
generate the final report in PDF format. We then
packaged it in a docker container to be deployed.
9
We developed an automated reporting system using
R and integrated it into the product. We did a
pilot with a doctor, got positive feedback, and
then rolled it to other clients. Their reports
helped them understand their patients cognitive
abilities and track their progress over sessions.
By the end of the project, we validated the data
and provided a data logging guideline to the
product team and a data dictionary for future
reference