Creating and Evaluating a Shorter, Lower Literacy Patient Experience CAHPS on an Accessible Digital Platform, Tickit® 


Authors: Courtney Lyles PhD, 1 Lina Tieu MPH, 1 Erin Curtis, 2 Alicia Hobbs, 3 Sandy Whitehouse MD, 4 Urmimala Sarkar MD MPH

1. Division of General Internal Medicine at Zuckerberg San Francisco General Hospital and Trauma Center, Center for Vulnerable Populations, University of California, San Francisco 2. San Francisco Department of Public Health 3. University of British Columbia Faculty of Medicine 4. University of California, Berkeley School of Public Health


• Healthcare policy supports inclusion of patient experience ratings in healthcare quality measurement & reporting. 
• Consumer Assessment of Healthcare Providers and Systems (CAHPS) survey is gold standard for collecting patient perceptions and experiences with care. 
• Traditional mailed CAHPS surveys have low response rates (~30%) 
• Racial/ethnic minorities and older adults are less likely to respond. 
• Mobile technology can be used to capture perceptions of patient experience at the point of care. 


PDF Download

Presented at: IHI National Forum Orlando Florida Dec 13th 2017, eHealth conference 2018 May 27th-31st Vancouver



• Design and implement tablet-based CAHPS administration in a safety net setting. 
• Conduct formative qualitative work to create shorter, lower literacy patient experience items. 
• Explore broader concepts of & preferences for reporting patient experience among vulnerable patients. 

Girl point at device.png


   Figure 1. Tablet Survey Interface   Meets WG2 standards for visual disability 

 Figure 1. Tablet Survey Interface
Meets WG2 standards for visual disability 

Survey Adaptation & Design
• Partnered with start-up company Tickit® Health to create tablet-based survey interface that was visually attractive, 
• icon-based, and simple. 
• Adapted Clinician and Group CAHPS (CG-CAHPS) survey to lower literacy levels of questions and shorten survey length. 

Patient Feedback & Validation
• Elicited feedback from 3 existing patient advisory councils to refine survey questions, interface, and workflow for administration. 
• Conducted 25 in-depth interviews with patients to: Elicit perspectives about care experiences and preferences for reporting feedback to providers/clinics. 
• Validate a tablet-based survey compared to the standard paper-based CG-CAHPS (patients received both surveys in randomized order). 
• Collect feedback about the interface and content of a tablet-based survey to inform future iterations. 

• Analyzed readability and concordance of answers between paper vs. tablet-based questions. Deductive (informed by interview guide) and inductive (open) coding. 


Table 1. patient characteristics, n=25 N(%)
Mean Age 53

    Black or African American
    Asian or Pacific Islander
    White or Caucasian

5 (20%)
13 (52%)
5 (20%)
5 (20%)

8 (32%)
13 (52%)
    Weekly Every 2-3 Weeks
    Monthly or Less

7 (28%)
5 (20%)
3 (12%)
9 (36%)
Smarphones Use (Freq.)
    Monthly or less

18 (72%)
1 (4%)
6 (24%)
Highest Education
    Some college or more
    High School degree
    Less than high school

12 (48%)
6 (24%)
6 (24%)

 Preferences for Survey Administration
• Timing of data collection – Value of collecting feedback at the point of care to avoid mail delay, occupy time in waiting room. 

“It’s better in the clinic because you’re in the same environment…it makes you think about all the questions.” 

• Prefer tablet-based survey administration – Novelty, color, interactivity, “fun” factor, familiarity with mobile platforms. 

“People are used to playing with their phones and it’s just kind of – it’s more familiar.” 

Usability of Tablet-Based Survey
• High ease of use – Tablet quick, easy, and convenient to use. 

“This (tablet) is quite easy to do, because everything is easy to read. You just literally have to point your finger at the answer.” 

• Few technical barriers – Difficulty with key-in answers, lack of flexibility in answering questions. 

“I like to type. I like to write period, but if I’m thinking in terms of someone who’s never written, never used a tablet, this would be very intimidating.” 

Importance of Reporting Patient Experience
• Survey not capturing positive feedback/experiences – Value of reporting good experiences to recognize staff and providers. 

 “I like to thank them [clinic staff]….I’m grateful. They should know that it’s appreciated.” 

• Survey capturing the need for clinic improvement, but not personal negative experiences – Value of reporting bad experiences to get resolution or drive changes in quality of care. 

“It’s important to communicate with the clinic and let them know what we need and what is left out.” 

Perceptions of Survey Capture of True Experience
• Quality of provider communication, staff respect, and access to care were prioritized concepts. 

“For example, they might ask you, how do you feel about your doctor’s care from one to 10? Well, if you haven’t been with that doctor that long, you can’t really gauge how you like them.” 

• Need for open-ended reporting format. 

Table 2. example of adaptation Original question (paper) Low-literacy adapted question (tablet) Readability (f-k grade) Answer concordance
Health Care Utilization In the last 6 months, how many times did you visit this provider to get care for yourself? In the last 6 months: How many times did you go to this clinic to get care? 6.5 to 4.2 70.8%
Timely Access to Care In the last 6 months, when you contacted this provider’s office to get an appointment for care you needed right away, how often did you get an appointment as soon as you needed? In the last 6 months: How often were you able to get an appointment when you needed it? 13.7 to 6.5 83.4%
Provider Rating Using any number from 0 to 10, where 0 is the worst provider possible and 10 is the best provider possible, what number would you use to rate this provider? What number would you use to rate your doctor? Use any number from 0 to 10, where 0 is the worst provider possible and 10 is the best provider possible. 14.2 to 7.5 87.5%
Care coordination In the last 6 months, when this provider ordered a blood test, x-ray, or other test for you, how often did someone from this provider’s office follow up to give you those results? In the last 6 months: How often were you able to get your test results if you wanted them? 13.3 to 6.1 41.7%
In the last 6 months, how often did you and someone from this provider’s office talk about all the prescription medicines you were taking? In the last 6 months: How often did you talk to someone from the clinic about your medicine? 11.4 to 7.8 69.6%
Office Staff In the last 6 months, how often did clerks and receptionists at this provider’s office treat you with courtesy and respect? In the last 6 months: How often did the staff at the front desk treat you with respect? 10.0 to 4.5 91.7%

Iterations to Tablet-Based Survey
• Reduced total number of items from 31 to 17 while preserving core domains of patient experience. 
• Adapted CG-CAHPS survey from 7th to 5th grade reading level (Flesch-Kincaid test)
• Added open-ended questions to capture patient-directed perspectives of care. 

Conclusions & implications

• A majority of patients served in safety net healthcare settings are interested in using tablets to provide timely feedback to their clinic. 
• Engaging patients in the design process produced a tablet-based survey with high usability and appropriate content. 
• Clinics can integrate quality improvement & patient experience efforts with enhanced data collection. 
• Current CAHPS questionnaires capture some core concepts of diverse patient care experience, but not the full range of responses. 

Federal policy should support improved content and format for collecting patient experience data from diverse populations. 


This project was supported by the UCSF CTSI-Strategic Opportunities Support (SOS) Program. Dr. Lyles was supported by a career development award from AHRQ (R00HS022408). 
Drs. Sarkar and Lyles receive support from AHRQ (R24HS022047). 



University of California, San Francisco Centre for Vulnerable Populations