Cognitive Debriefing: A Challenging but Rewarding Project

No matter how short or long, any questionnaire developed in one part of the world to be used in another part may carry linguistic, cultural, and conceptual features that present challenges when translated into another language.

Eight years ago, when a large language services company providing translations to the pharmaceutical industry first contacted me to work on a cognitive debriefing project, I had a vague idea of what the job entailed. The project manager explained that the assignment would involve collaborating with other linguists to translate a short patient questionnaire from English into Vietnamese, but the major task would include testing the translation on five patients in Vietnam with a medical condition. Impressed by the high rate the company offered, I accepted the assignment after a brief training session.

I’ve learned a lot about this area since that first job. Today, I receive requests to conduct cognitive debriefings from companies almost every month and derive a portion of my income from these assignments. So, how can you take advantage of this specialized market? What skills are required? Let me share my experiences working on cognitive debriefing projects, both as a recruiter and interviewer, along with tips for success.

What Is Cognitive Debriefing?

Cognitive debriefing, sometimes referred to as pilot testing, is part of a linguistic validation process generally used to evaluate clinical research instruments designed to gather information from respondents in multinational trials.1 These instruments can include clinical outcomes assessments, such as patient-reported outcomes, clinician-reported outcomes, observer-reported outcomes, or similar questionnaires.

Basically, cognitive debriefing is the process by which a questionnaire or other assessment is tested among representatives of the target population and target language group. Ideally, this takes place during face-to-face interviews. The purpose is to verify that the translations of all items on the questionnaire will be understood in the same way across target populations and regions. The U.S. Food and Drug Administration requires cognitive debriefing to show content validity for all submitted translations.

For instance, suppose an American company develops a questionnaire for adult patients with Hepatitis B consisting of 10 questions with multiple response options. The questionnaire originally targets American patients who live in the U.S. and speak American English. Now suppose the researchers involved in the study want to use the same questionnaire in Vietnam to collect data from Vietnamese patients with the same medical condition. This will require the questionnaire to be translated into Vietnamese. Since patients in the U.S. and those in Vietnam don’t share the same language, culture, or even certain concepts, conducting a cognitive debriefing will help the researchers bridge that gap to ensure that all respondents have an equal experience.

Patient questionnaires vary in length. The average length of the source text can be 300 words, although I was once asked to work with a 4,000-word questionnaire and another time to debrief only 66 words. No matter how short or long, any questionnaire developed in one part of the world to be used in another part may carry linguistic, cultural, and conceptual features that present challenges when translated into another language.

Prior to the cognitive debriefing phase, companies always invest much time and energy into translating a questionnaire. The translation process usually involves two forward translators and two back translators. Drafts are often exchanged between translators, project managers, and developers before a final, harmonized version is approved. However, no one can be sure this final translation will be understood by the target audience without first testing it on them.

Bridging Cultural Barriers

One major purpose of cognitive debriefing is to determine if the target population would find certain words, phrases, or concepts inappropriate or confusing. For example, one question in English from a questionnaire I worked on for patients in hospice care read: “Do you believe in God or someone or something greater than yourself?” During the translation phase, a translator, aware that a majority of people in Vietnam are Buddhist, translated it as: “Quý vi. có tin vào Đức Phâ.t, hay người nào đó, hay điều gì đó lớn hơn bản thân quý vị không?” (“Do you believe in Buddha or someone or something greater than yourself?”). However, during the cognitive debriefing phase, I realized that some patients who were Christian didn’t find this relevant. We ended up changing it to: “Quý vị có tin vào Đức Phật, Đức Chúa, hay một đấng tối cao, hay người nào đó, hay điều gì đó lớn hơn bản thân quý vị không?” (“Do you believe in Buddha or God or a supreme being or someone or something greater than yourself?”).

Another example is the term “block, which appeared in the following question: “Can you walk more than one block?” The meaning is clear in English, but the literal translation of it into Vietnamese can be problematic. A cognitive debriefing revealed that while the literal translation of “block” as “tòa nhà” was understood by Vietnamese patients living in an urban setting, it was not understood by others living in rural areas where street blocks are generally nonexistent. For this, a culturally appropriate translation could be: “Quý vị có thể đi được hơn 100 mét không?” (“Can you walk more than 100 meters?”).

In many cases, even a translation commonly used in mass media can offend a certain group. For example, the word “questionnaire” or “survey” can be translated into Vietnamese as “Bản Điều Tra” (“probing text”). This translation is not wrong, but through a cognitive debriefing we learned that many patients found this translation offensive. Why? Since the word “Bản Điều Tra” can imply an interrogation, people with a medical condition would generally react negatively because they would consider this type of questioning an invasion of their privacy.

Bridging Linguistic Barriers

For patient questionnaires, ensuring that a term in the source text will be understood the same way in the target text is extremely important. A mistranslation that leads to a misunderstanding of a term could result in medical complications or the rejection of an entire clinical research project.

I find translating adverbs of frequency or adverbs of intensity from English into Vietnamese most challenging. For example, response options such as Not at all, A little bit, Somewhat, Quite a bit, and Very much for the question “In the last seven days, how much did mouth or throat sores interfere with your usual or daily activities?” are difficult to translate. There are Vietnamese equivalents for these words, but there’s a great deal of disagreement among translators since adverbs of frequency or intensity in Vietnamese can vary from their English counterparts. For instance, the translated term “Khá nhiều” in Vietnamese can mean either “Somewhat” or “Quite a bit.”

In this case, a cognitive debriefing is very helpful as it allows the patient’s voice to come through in the translated content and ensures that the correct terminology is used. Some companies encourage the cognitive interviewer to use a visual analog scale to assess how well the target group comprehends the translated response options. For example, a visual scale ranging from 0 to 100%, in which 0 represents Not at all and 100 represents Very much, would enable the respondent to better express the value (e.g., intensity) to each option by pointing at a place on the scale. Using a card ranking game is another useful strategy. A respondent is asked to arrange the selected response options in an order that makes sense to them (e.g., Not at all at the top of the page, Very much at the bottom, and other terms in a continuum).

Recruiting Participants

Before cognitive debriefing can begin, you or your project manager must recruit in-country for both investigators and respondents, which can prove challenging. Some projects require the recruitment of five healthy adult patients, which is generally easy to do. Other projects are more difficult because they may require five patients with a specific diagnosed condition. For example, I’ve had to find five patients with gastroenteritis, and a recent project required me to recruit 10 patients with blood cancer, ideally chronic myeloid leukemia.

As a general rule, the selected participants within the target countries should be diverse in terms education level, gender, and age. In most cases, the primary selection criterion is to find patients with the required medical condition being studied.

The project manager usually allows up to four weeks for the recruitment process. I typically need to report the status of the recruitment on a weekly basis. I’m encouraged to use chat rooms, message boards, networking sites, national support groups, patient associations, professional websites, hospitals, doctors, and private practices as resources to find participants. Treatment doctors who know the patient well also serve as good contacts, but they’re often busy and sometimes hard to contact. In recent years, I’ve found that cancer support groups and clubs (e.g., Gilda’s Club) are excellent resources for potential recruitment, and there’s one in most big cities. The club’s chair can be the first point of contact, as they often have a good relationship with members. Members usually trust the chair and can be open to being interviewed if the chair encourages them.

From my experience, successful recruitment also depends on how well the company communicates to potential participants. One company I’ve worked with prepares excellent recruiting forms, including patient consent and privacy notice forms that clearly state the project’s purpose, the specific things participants are expected to do, tangible benefits and potential risks, and how privacy will be protected. All forms should be written in plain language and be translated into the target language.

Companies have different policies regarding payment for participants. As a recruiter, I’ve learned that companies who are willing to reimburse participants, doctors, medical personnel, or the heads of patient associations for their efforts have higher rates of success compared to those who don’t. If I receive a request for cognitive debriefing from companies that don’t at least mention payment for the patients, I ask the project manager if there’s the possibility for compensation. Though many patients don’t need it, offering a small sum is something the company can do to express appreciation for their participation. I also tell the project manager I’m uncomfortable contacting patients with a chronic disease to ask them to spend 30-45 minutes for an interview without payment. In many cases, the project manager will talk to the project lead and perhaps spend a portion of the project budget to compensate participants.

Conducting Cognitive Interviews

Companies normally provide the recruiter and interviewer with some training for cognitive debriefing. Before conducting real interviews with selected patients, companies usually ask me to complete a practice interview and record the data on a data collection form (DCF) to be sent back to the project manager. This will enable the project manager to confirm that I’m on the right track and approve the DCF before I proceed to interview the selected patients.

There are many good tips for cognitive debriefing interviewing available online.2 The general principle is that interviewers need to do everything possible to ensure that participants feel comfortable before and during the interview.

As I mentioned previously, face-to-face interviews are ideal for cognitive debriefing. However, during the pandemic face-to-face interviews have proven difficult, so interviews can be done via phone or remote video conferencing. Remote interviews can allow the interviewer to conduct the job without being in the country where the participants live. This has been good for me. When I was stuck outside my home country due to the pandemic, I was still able to call to set up interviews with people there. For virtual interviews, the interviewer needs to make sure participants receive the questionnaire, ideally in printed form, to use during the interview. If the questionnaire is online, participants are encouraged to open it using a laptop or an iPad rather than a smartphone. (I did have problems with participants who weren’t able to see the questionnaire well because of the tiny screen on their smartphones.) Additionally, for virtual interviews, visual analog scales or card ranking games originally designed for use during face-to-face meetings may need to be redesigned or customized for online use.

From my experience, patients often tend to misunderstand the purpose of the interview. Despite the availability of consent forms and other materials that clearly explain the purpose, many patients still assume they are being asked to provide responses to each question based on their own medical condition. To avoid this misunderstanding, I always tell the patient at the beginning that the purpose of the interview is not to answer the questions but to test if they can understand everything—every sentence, question, and response option in the questionnaire.

Some questions during an interview can also be sensitive and can make patients feel embarrassed, shocked, or upset. When dealing with sensitive questions, I often tell participants: “I know you might not want to explain this, but this doesn’t ask about your own symptoms or conditions. The questionnaire designer wants to know if you understand this word, how you say this in your own words, and there’s no right or wrong answer.”

Coping with Vicarious Trauma

Though not all questionnaires involve working with people who suffer from a chronic or a terminal disease, many do target those groups. As a result, interviewers many experience vicarious trauma, burnout, compassion fatigue, or stress from working with people who suffer long-term or untreatable diseases. Several years ago, I contacted a patient who agreed to be interviewed when the translated questionnaire was ready. When the time came to schedule the interview, I learned that he had passed away. On another occasion, I asked a young patient after an interview if he happened to know anyone with the same medical condition who would be willing to be interviewed. Sadly, he replied that all the people he knew with that medical condition had died.

For those situations, especially when I first started conducting interviews, I had to work very hard to calm myself, and it usually took me a few days or even a week to get back to normal after hearing a tragic story. I’ve since learned that talking with friends who have similar jobs, such as a nurse working in an oncology center or a medical interpreter who helps patients in a hospital, can help because they understand better than anyone what my job entails. Sometimes I still need to hide myself in some corner and cry, but I’ve also found that taking a walk in a quiet neighborhood or sitting in a Buddhist temple for meditation or prayer, and even listening to music can help me get back to normal. I’ve also learned that declining a particularly challenging assignment or avoiding conducting too many cognitive interviews with chronic patients within a short period also helps.

But there are also many positive aspects associated with conducting cognitive debriefings with patients suffering from long-term illnesses. Talking with people who are in a less fortunate situation can make me appreciate my well-being, my life, and the well-being of my family and friends much more, and that I shouldn’t take them for granted. Also, many questionnaires are developed to collect data that will be used to improve certain treatment strategies or therapies, which in turn will help improve the quality of the mental and/or physical health of a certain group of patients. Ensuring that patients understand the translated questionnaire is a small contribution I can make to help ease the pain in this world.

Final Note

Since COVID-19 hit the world, freelance translators might be experiencing unstable workloads. While some can stay with their regular clients and do the work they’ve always done, others need to diversify the jobs they take on to maintain an income. One thing I find interesting in the language industry is there are always more opportunities for jobs than translation per se. Cognitive debriefing may be complex and challenging, but it’s a job that involves using language, culture, and communication skills, so any experienced translator will already have the basic skills to perform the job well.

Acknowledgement: I dedicate this in memory of my brother-in-law who helped me with many practice interviews for cognitive debriefings. He passed away on the day this article was accepted for publication.

Notes
  1. “Cognitive Debriefing Explained, https://bit.ly/cognitive-debriefing.
  2. Richards, Anna. “7 Ways to Master Linguistic Cognitive Debriefing Interviews of Patient-Reported Outcomes,” https://bit.ly/Richards-interviews.

Pham Hoa Hiep has worked as a translator, editor (English>Vietnamese), and translator trainer for over 15 years in New Zealand, the U.S., Australia, and Vietnam. For the past 10 years, he has mainly translated and edited life sciences documents with an emphasis on health plans, insurance providers, health care, clinical trials, and research studies. He has a BA in English from the University of Hue, an MA in applied linguistics from the University of Massachusetts-Boston, and an EdD in language education from the University of Melbourne. Besides working as a freelancer, he has taught courses in applied translation studies at many universities. He has published widely in the fields of translation, applied linguistics, and English language teaching. hiepsuu@gmail.com

1 Responses to "Cognitive Debriefing: A Challenging but Rewarding Project"

  1. Georganne Weller says:

    Excellent article, thanks so much for making us aware of this extremely interesting avenue of cross-cultural collaboration for interpreters and translators.

Leave a Comment

Your email address will not be published. Required fields are marked *

The ATA Chronicle © 2021 All rights reserved.