Full Terms & Conditions of access and use can be found at https://www.tandfonline.com/action/journalInformation?journalCode=ujhe20
American Journal of Health Education
ISSN: 1932-5037 (Print) 2168-3751 (Online) Journal homepage: https://www.tandfonline.com/loi/ujhe20
Conducting a Formative Evaluation of an Intervention Promoting Mammography Screening in an American Indian Community: The Native Women’s Health Project
Eleni L. Tolma, Julie A. Stoner, Cara Thomas, Kimberly Engelman, Ji Li, Aleksandar Dichkov & Norma Neely
To cite this article: Eleni L. Tolma, Julie A. Stoner, Cara Thomas, Kimberly Engelman, Ji Li, Aleksandar Dichkov & Norma Neely (2019) Conducting a Formative Evaluation of an Intervention Promoting Mammography Screening in an American Indian Community: The Native Women’s Health Project, American Journal of Health Education, 50:1, 52-65, DOI: 10.1080/19325037.2018.1552216
To link to this article: https://doi.org/10.1080/19325037.2018.1552216
Published online: 31 Dec 2018.
Submit your article to this journal
Article views: 14
View Crossmark datahttps://www.tandfonline.com/action/journalInformation?journalCode=ujhe20https://www.tandfonline.com/loi/ujhe20https://www.tandfonline.com/action/showCitFormats?doi=10.1080/19325037.2018.1552216https://doi.org/10.1080/19325037.2018.1552216https://www.tandfonline.com/action/authorSubmission?journalCode=ujhe20&show=instructionshttps://www.tandfonline.com/action/authorSubmission?journalCode=ujhe20&show=instructionshttp://crossmark.crossref.org/dialog/?doi=10.1080/19325037.2018.1552216&domain=pdf&date_stamp=2018-12-31http://crossmark.crossref.org/dialog/?doi=10.1080/19325037.2018.1552216&domain=pdf&date_stamp=2018-12-31
Conducting a Formative Evaluation of an Intervention Promoting Mammography Screening in an American Indian Community: The Native Women’s Health Project Eleni L. Tolmaa, Julie A. Stonerb, Cara Thomasb, Kimberly Engelmanc, Ji Lib, Aleksandar Dichkovb, and Norma Neelyd
aKuwait University Health Sciences Center; bUniversity of Oklahoma Health Sciences Center; cUniversity of Kansas School of Medicine; dUniversity of Oklahoma
ABSTRACT Background: Breast cancer is an important public health issue among American Indian/Alaska Native (AI/AN) women. However, there are very few published studies describing the evaluation of breast health promotion programs among AI/AN women. Purpose: To describe the formative evaluation of a multicomponent intervention to promote mammography screening in an AI community in rural Oklahoma. Methods: A comprehensive process evaluation plan with emphasis on context, reach, dose received, dose delivered, and fidelity was developed. Data collection included mixed research methodology and impact was assessed via one group pre/post research design. Data analysis consisted of descriptive statistical analysis and content analysis. The study utilized a community-based participatory research (CBPR) approach. Results: Process evaluation revealed a relatively high reach within the priority population for both components (clinic and community) and a moderate implementation. Focus group research showed that participants were overall satisfied with program implementation. The intervention was feasible to implement in real-world settings. Translation to Health Education Practice: Community-based evaluation of breast health promotion programs among AI communities is challenging, because one has to balance methodological rigor with practical constraints. An evaluation plan, mixed methods, and a collaborative approach are useful tools in conducting the evaluation.
ARTICLE HISTORY Received 11 June 2018 Accepted 10 September 2018
Breast cancer continues to be a major public health issue among American Indian/Alaska Native (AI/AN) women in the United States.1 AI/AN women are more likely to be diagnosed with late-stage breast cancer2 and with a more aggressive form of breast cancer.3 In Oklahoma, where the study took place, the incidence rate of breast cancer among AI/AN women was 140.5/ 100 000 compared to 121.5/100 000 among non- Hispanic white women for 2007 to 2009.4 Based on a more recent study, AI/AN and white women in Oklahoma had similar 5-year observed survival rates (79.4% and 80.2%, respectively), however, within the 40- to 49-year-old group, the survival rates were sig- nificantly lower among AI/ANs (P < .0001; 70% and 63.8%, respectively).5 Moreover, among AI/AN women, the mammography screening rate is low, 55% in 20176
and 53% in 2015,7 which was in fact the lowest among all racial/ethnic groups.
Despite the intense efforts to promote mammogra- phy screening in Oklahoma, there are still geographical disparities among AI communities in terms of breast cancer mortality, incidence, and stage diagnosis.8 One way to eliminate disparities is by building local health promotion programs that address the specific needs of the communities they serve.
Literature reveals a limited number of published articles on community-based breast health promotion interventions within the AI/AN community.9–17
Among these programs, only a few11,16,17 describe in detail the evaluation of the interventions.
A study by English et al11 evaluated a multicomponent intervention through tribal participatory approach. All women who participated in the study got a mammogram. The study produced community-related outcomes, such as the development of a partnership between the tribe and external partners, development of local policy initiatives to address cancer in the
CONTACT Eleni L. Tolma [email protected] Department of Social and Behavioral Sciences, Faculty of Public Health, Kuwait University Health Sciences Center, P.O. Box 24923, Safat 13110, Kuwait. Color versions of one or more of the figures in the article can be found online at www.tandfonline.com/ujhe.
AMERICAN JOURNAL OF HEALTH EDUCATION 2019, VOL. 50, NO. 1, 52–65 https://doi.org/10.1080/19325037.2018.1552216
community, and reinforcement of public health skills among community health providers.
Strickland and colleagues16 conducted a feasibility study of a multicomponent intervention that included an educational program, recommendation by a tribal medical provider to get screening mammography, and a follow-up reinforcement with the use of phone calls, cards, and home visits. The focus of the study was mostly on the quality of the implementation of the intervention and included the use of tracking logs and interviews, a survey administration, and field observations. The study showed that the intervention was feasible.
Finally, a study by Friederichs-Fitzwater and colleagues17 evaluated an educational workshop that focused on the enhancement of knowledge, attitudes, and intentions to change lifestyle and screening behaviors. The evaluation design used was a one-group pretest–posttest. The evaluation showed that the participants at the end of the intervention had increased knowledge about breast cancer and more positive attitudes toward mammography.
One can observe from the above studies that none incorporated an advanced evaluation design. This is not surprising. Historically, the use of an experimental design assessing the impact of an intervention among AI/AN communities is generally discouraged for ethical and practical reasons.18 There are also challenges in doing comparative research among tribal communities because of varying tribal histories, locations, resources, and sizes. Nevertheless, despite these challenges, eva- luators recently were able to incorporate more complex evaluation designs besides one group pre and post in the evaluation of programs related to the prevention of alcohol and drug abuse,19 promotion of human papil- lomavirus vaccination,20 and prevention of obesity.21
Research also shows that there is a lack of evidence- based behavioral studies conducted in real-world settings.22 Evidence-based recommendations for beha- vioral interventions are the result of highly controlled efficacy trials, which, although they have high internal validity, have low generalizability. In other words, the interventions might not be applicable in real-world settings. Feasibility studies are useful in helping to fill this important gap in the research literature.23 To ensure that feasibility studies reflect the reality of a community, it is essential that practitioners and com- munity members be involved in meaningful ways in the design and implementation of study.23 One way to do that is by using community-based participatory research (CBPR).24
In summary, there are very few published evaluation studies of evidence-based behavioral interventions pro- moting mammography screening among AI/AN women, even though in recent years there is more
emphasis on the evaluation of health promotion pro- grams among AI communities. More important, there are even fewer studies that have incorporated a holistic approach to evaluation that includes process evaluation and impact evaluation with emphasis on different out- comes besides knowledge.11,16
This study attempted to fill this research gap. The article describes the formative evaluation of a culturally appro- priate intervention promoting mammography screening called the Native Women’s Health Project (NWHP).15
Culturally appropriate interventions are those interven- tions that emerge from the community’s traditions and values and capitalize on strengths and resources of the community.18 In formative evaluation, there is a heavy emphasis on process evaluation while assessing early impact of the intervention.25 Therefore, the study’s research questions focused first on the intervention’s fea- sibility, “Can the intervention work?” and, second, on its immediate impact; that is, “Does it work in terms of promoting mammography screening” and, if yes, “Why”?
For tribal programs to be successful and effective, they need to be evaluated within their own context.18
Information derived from the evaluation can help pro- gram planners find out what worked or did not work in terms of program implementation, revise accordingly, improve, and re-implement. Therefore, the purpose of this article is to use the evaluation of the NWHP as a case study to showcase how an innovative, new pro- gram on mammography screening can be evaluated by using a comprehensive approach and through CBPR.24
Planning of an evaluation is equally important as its implementation. Evaluation planning in this study included the development of the theory of change (Figure 1), a logic model, the use of a community steering committee (CSC), and the development of a process evaluation plan.18
The theory of change was based on the theory of planned behavior26,27 and was the outcome of years of formative research in the community.28,29 According to the theory of change, AI women who communicate with their medical provider on mammography screen- ing, participate in a related discussion group, and receive a bracelet as a congratulatory gift are more likely to intend to or decide to get a mammogram and subsequently get a mammogram.
The CSC provided the structure through which community members were actively involved and guided
THE NATIVE WOMEN’S HEALTH PROJECT 53
the research process, so that the investigators assured that the process was responsive and respectful to the communities.30 The CSC consisted of laypeople in the community, breast cancer survivors, tribal clinic repre- sentatives, and representatives of other local coalitions and other state organizations.
The process evaluation plan depicted how each com- ponent of the intervention would be evaluated and by whom. The evaluation research design consisted of a one- group pretest–posttest design. Upon entry in the study, the research participants completed a baseline evaluation survey and they received the post-evaluation survey after 3 to 5 months’ post-baseline measurement. Formative evaluation was carried out by incorporating a mixed research methodology and by using CBPR.24
Description of the intervention
The NWHP15 took place at a tribal clinic and its sur- rounding Native American community south east of Oklahoma City. This 4-year study is a continuation of two preliminary studies28,29 that led to the development of the intervention. During the first study,28 we identi- fied the salient beliefs regarding screening mammogra- phy among AI/AN women and during the second study29 we measured the strength of each belief through
a cross-sectional study. The results were used to plan the intervention15 by using the CBPR approach.24
The intervention focused on promoting systems change at the local tribal clinic as well as at the broader community. The clinic-based component included the following strategies: (a) A patient–doctor guided commu- nication on mammography screening; (b) the delivery and subsequent discussion with the patient of a mammography brochure developed specifically for this project31; (c) a follow-up recommendation letter on mammography screening sent to the patient by the phy- sician; (d) the use of a mammography screening flowchart that assisted the clinic’s physicians when advising patients on mammography screening; and (e) the development of a bulletin board poster at the clinic that encourages women to get a screening mammogram.
The mammography brochure was designed based on the 6-page letter-fold template and contained 573 words. It contained information on basic scientific facts about breast cancer and mammography, including survivorship, and a testimonial from an AI breast can- cer survivor who was also a member of the CSC. The purpose of the brochure was to enhance awareness of the importance of breast cancer screening among AI/ AN women and to assist the medical providers during their communications with the patient on breast cancer
Implementation via Community Based Participatory Research
Intention to get future mammogram
Doctor-patient guided communication, recommendation letter, breast cancer brochure, bulletin board poster, mammogram screening flowchart
Intergenerational discussion groups
Bracelet as a reward for getting a mammogram
Intergenerational discussion groups
Intergenerational discussion groups
Perceived behavioral control
Figure 1. Theory of change based on an expanded model of the theory of planned behavior (TPB). Squares refer to the TPB, circles to the secondary constructs, 3D diagrams to the constructs addressed in the intervention, diamonds to the clinic-based activities, and hexagons to the community-based activities.
54 E. L. TOLMA ET AL.
screening. The reading level of the brochure is at the ninth-grade level based on the SMOG readability for- mula. More information about the development of the brochure can be found in a previous publication.31
The community-based component included the follow- ing: (a) participation in a related discussion group called the Intergenerational Discussion Group (IGDG) accompa- nied by a younger female relative, preferably a daughter or friend, and (b) provision of a congratulatory gift (ie, a bracelet) upon receiving a screening mammogram.
At the entry point of the intervention, the partici- pant filled out a form called the decision stage form, which she then gave to the medical provider. The medical provider then discussed the topic of mammo- graphy screening with the patient according to the decision stage she was in, based on the precaution adoption process model.32 For instance, if the patient had never heard of screening mammogram (stage 1), then the content of the medical provider’s conversation would focus on providing basic information on the pros and cons of screening mammography. On the other hand, if the patient had heard of screening mam- mogram and was considering getting one (stage 3), then the conversation would focus on discussing the logistics of scheduling a mammogram and identifying possible barriers and solutions to attend the mammo- gram. Within a week of the visit, a clinic administrator would send a letter to the participant recommending mammography screening. Within a month of the visit, the participant was invited to an IGDG.
During the IGDG, the research participants watched a 10-minute excerpt from a video called The Pink Shawl Project, which is a project that depicted AI breast can- cer survivors who made pink shawls and gave them as gifts to their daughters. The shawls in the AI culture signify protection and care of the children by their mothers. The video conveyed the importance of regular mammography screening and passing these important preventative breast health messages as an act of care and love from the older generation of AI women to the younger generation, the same way a mother would give her daughter a pink shawl. Based on the Freire philo- sophy of raising consciousness,33,34 the excerpt acted as a “code” that triggered a structured discussion around the issue of breast cancer. The discussion was facilitated by the project coordinator (CT) and all participants were sited in a circle. The discussion started by first reflecting on the broader AI community’s status and relationship with breast cancer, then moved to a personal level where women discussed how breast cancer affected their personal lives, and ended with a menu of action steps suggested by the women them- selves. The IGDG lasted for approximately 90 minutes
and was scheduled monthly. In addition to the above discussion, it included storytelling through singing by a breast cancer survivor and other educational, inter- active, fun activities such as the palpation of a breast model with irregular masses. More information about the IGDG can be found elsewhere.15
The intervention program was launched in June 2014 and lasted until December 2014. However, actual participation in the intervention by the partici- pant (meeting with the medical provider and participa- tion at the IGDG) only took a few hours. It is also important to note that during the implementation of the intervention there was a change in leadership of the clinic’s administration.
A detailed recruitment protocol was developed and only active patients from the clinic were enrolled in the study. Active patients were defined as those patients who have used the clinic’s services at least once during the last 2 years. An invitation to the program signed by the clinic director was sent by mail to each potential participant. Within a week of mailing out the invitations, the project coordinator (CT) called each participant to find out whether they had received the letter and whether they were interested in the study. The eligibility criteria for the study participation were as following: (a) was American Indian; (b) was aged between 52 and 74 years old; (c) had no history of breast cancer; (d) had not had a mammogram during the last 2 years or more; and (e) at the time of recruitment, the patient must have had a prescheduled visit to the clinic within the following month. All women who participated in the intervention were also eligible to participate in the evaluation portion of the intervention.
The recruitment process was monitored on a monthly basis with the help of a flowchart that depicted the number of women who were contacted and, among those who were potentially eligible to be in the study, the number of women who were truly eligible for the study and the number of womenwhowere actually interviewed. Despite our efforts to contact the potential participants and recruit them to the study, the overall response rate was 59%. The moderate response rate was due primarily to the fact we were not able to contact many of the participants either due to outdated contact information or just being unable to reach them via phone.
Process evaluation and process evaluation plan
All staff members hired for the project were local AI women. As part of the CBPR process, the first author
THE NATIVE WOMEN’S HEALTH PROJECT 55
(ET) conducted various training sessions with the staff, project coordinator, and community research assistants (CRAs) on data collection. The staff and members of the CSC also received training in CBPR by 2 experts in the field and collaborators on this project. An approval by the tribal authority in the form of a tribal resolution was obtained and maintained during the project. All staff members received an ethics-related training via the University of Oklahoma Health Sciences Center Institutional Review Board.
A comprehensive process evaluation plan was devel- oped to assist with the implementation of the evaluation. This evaluation plan was guided by the framework developed by Steckler and Linnann.35 It consisted of a table that included the intervention component that would be evaluated (eg, mammography brochure), the questions to be answered according to the main dimen- sions of process evaluation (eg, reach, dosage delivered, dosage received, fidelity), and the measures that would provide the information needed to answer the questions (eg, physician log, discussion with physicians at the end of the study; Table 1).
A secondary evaluation planning framework used was the “Plan, Do, Study, Act” (PDSA) change cycle.36,37 The PDSA cycle is an iterative process where a task is first introduced at a small scale and then is evaluated by those who actually carried out the task; then, based on the assess- ment, the task is modified and a new cycle begins. As one moves through the cycle, the task is tested at a larger scale.
The evaluation survey was a modification of the Women’s Health Survey used earlier in the study to assess AI/AN women’s beliefs on screening mammogram.29 This survey included the constructs/ scales that the intervention intended to address, such as social modeling, social norms, fatalistic attitudes, and knowledge on mammography screening. It also mea- sured the two most important outcomes of the inter- vention; that is, the intention of getting a future mammogram within the next 3 to 5 months, and the actual mammography behavior.
Table 1. Process evaluation component matrix. Component and definition What will be evaluated/related research questions How will it be measured?
Context (aspects of the larger social, political, and economic environments that may have influenced intervention implementation)
Barriers influencing the delivery of the components of the program: 1. Patient–doctor guided communication 1. Physician log 2. Educational brochure 2. Interview with physicians at the end of the
program 3. Mailing out recommendation letter by physician 3. Tracking logs by project staff 4. Constraints to attending intergenerational discussion group
4. Follow-up phone calls with participants who missed the intergenerational discussion group
Reach (proportion of intended target audience that participated in an intervention)
1. Percentage of research participants who had a patient–doctor guided communication
1. Copies of the decision stage forms returned
2. Percentage of women participating in the discussion group
2. Sign-up sheet, comparison of those women who were supposed to come vs those who actually showed up
Dose delivered (number or amount of intended units of each intervention or each component delivered or provided)
1. Extent to which physicians have guided communication with patients, length of discussion
1. Physician log
2. Did the medical providers find the guided communication meaningful/beneficial?
2. Interview with physicians
3. Number of brochures distributed by physicians 3. Keep track of brochures delivered, physician log
4. Number of brochures given out to clinic or sent to patients by mail or in person via discussion groups
4. Post-evaluation survey, focus groups with research participants
5. Number of physician recommendation letters mailed out
5. Tracking by project coordinator
6. Extent to which all components of the intergenerational discussion groups took place
6. Facilitator checklist for discussion groups
Dose received (extent to which participants actively engaged with, interacted with, were receptive to, and/or used materials or recommended resources)
1. To what extent did the participants have a guided communication with the physician?
1. Post-survey evaluation, focus group discussions with research participants, physician log
2. Number of participants receiving the brochure 2. Post-evaluation survey 3. Number of participants receiving the recommendation letter
3. Post-evaluation survey
4. To what extent did the participants benefit from the overall program?
4. Focus groups with participants
Fidelity (extent to which the intervention was delivered as planned)
Did all of the intervention components take place as planned?
Physician log, discussion with physician, focus groups with participants, post-evaluation survey, facilitator checklist, review of the logic model
56 E. L. TOLMA ET AL.
The behavioral variable was defined as a composite outcome including the women who had a mammogram at the time of the post-survey and women who reported having a mammogram during the follow-up call within 6 months after the end of the intervention. The inten- tion variable was measured with the question, “How likely is it that you will obtain your next mammogram within the next 3 to 5 months?” (coded on a 5-point scale from very unlikely to very likely).
There were two versions of the survey: the pre- and post-evaluation surveys. They were exactly the same with only a few additional questions on the post- evaluation survey. The post-evaluation survey included process evaluation-related questions such as whether the participant received the mammography brochure or not, which was 15 pages long and included 105 items. The average time for completion of the survey was 20 minutes and the survey was self-administered. However, a member of the research team was nearby to assist with any questions the participant might have.
Quantitative data analysis
Descriptive statistics were calculated, followed by the completion of univariate and bivariate analyses com- paring the mean scores of the quantitative measures (attitude, self-efficacy, social norms, perceived control, breast cancer fatalism) between those who received a mammogram and those who did not during the follow-up period. Data analysis was focused on the 21 participants who completed both the pre- and post- surveys among the 29 who participated in the project.
Because of the small sample size, nonparametric methods were used, including a Wilcoxon rank sum test to compare median values between independent groups. Fisher’s exact test was used to compare propor- tions between independent groups and an exact form of the Cochrane-Armitage test for trend was used to com- pare the distribution of ordered, categorical responses between independent groups. Multivariate modeling, with adjustment for confounding factors, was not fea- sible. A 2-sided alpha level of .05 was used to define statistical significance for the analysis. The data were entered by a graduate research assistant and every entry of the record was double-checked either by a different graduate research assistant or by the first author (ET). The data were cleaned before data analysis.
The quality of the program was also assessed through focus group research and an interview with the medical director of the entire health system of the tribe. We
invited all women who had participated in the clinic and community-based components of the intervention regardless of their intention or behavioral background (ie, whether they received a mammogram or not). Among the 17 women who were eligible to participate, only 4 actually accepted our invitation. Two focus groups were conducted 5 months after completion of the project. They were conducted by the first author (ET) with the help of the CRA who was the assistant moderator/note-taker. Each interview lasted about 60 minutes. The note-taker (Ms. Vickie Canfield) was trained by the principal investigator on how to take notes during a focus group discussion. Member check- ing was done at the end of each focus group by having the note-taker summarize the information collected during the focus group and confirm with the partici- pants that the information collected accurately reflected what was said in the group. Content analysis was done with the help of two graduate students trained in qua- litative research and according to standard guidelines.38
Each researcher first identified the main ideas/themes individually and then the group came together to dis- cuss their findings and derive conclusions. A theme was defined as an idea/belief noted in both focus groups that were completed. Due to the small number of participants and focus groups, we felt that we did not reach data saturation. However, the information gained from the focus group was still valuable in the evaluation of the NWHP.
Process evaluation centered on ensuring the quality of the two intervention components: the clinic-based component (patient–doctor guided communication, recommendation for mammography letter, screening flowchart, and delivery of mammography brochure) and the community-based component (IGDGs and the receipt of a bracelet as a congratulatory gift upon receiving a screening mammogram).
All women participated in the guided communica- tion with the medical provider; however, only 17 out of 21 (80%) attended the IGDG. In regards to the clinic- based component, only 12 women indicated that their medical provider discussed mammography screening with them, and 13 out of the 21 said that they had the opportunity to ask her or him questions about mammography screening. Eighteen women said that they received the mammography brochure; 6 women noted that they received the mammography brochure via mail, 3 during their visit with the medical provider,
THE NATIVE WOMEN’S HEALTH PROJECT 57
and 9 during the IGDG they had participated in. When asked to write down what they remembered from read- ing the brochure, 7 women said the importance of mammography as a breast cancer early detection tool, one woman wrote “truths and myths about breast can- cer,” 2 women said “nothing,” and one woman said, “I don’t remember.” Twelve out of 20 women (one person did not answer this question) said that they had not received the physician’s mammography recommenda- tion letter in the mail. In addition, logs were prepared and given to the medical providers to complete at the end of each visit with the patient. Among the 29 parti- cipants who entered the study, we received only 9 logs. We also received 14 copies of the decision stage forms. Finally, we only received 5 copies of the recommenda- tion letter sent by the medical provider to the patient.
In regards to the community-based component, we conducted 6 IGDGs. Seventeen of the 21 research par- ticipants participated in them, but only half of them brought a younger female guest. Based on our facilita- tor checklist, all topic/activities were fully implemented with the exception of the survivor’s story, which was to be delivered by one of our volunteers. Overall, based on the evaluations the participants completed at the end of each session, the participants enjoyed the IGDGs. All participants rated both the discussion group and the facilitator as “excellent.” In terms of knowledge gained, the participants commented on the importance of being familiar with one’s body, of regular mammography, and of support groups. They also acknowledged the pervasiveness of breast cancer across ages and races and said that they would share information they received through the NWHP with their family mem- bers and friends.
Additional information that we received from the post-evaluation survey revealed that 13 women out of the 18 who answered the related question stated that they had read the bulletin board poster on mammogra- phy screening at the clinic. When asked at which point the women decided to get a screening mammogram, 6 out of the 15 women who answered this question said that it was after they completed the pre-evaluation survey, 3 after talking to the medical provider, 3 after participating in the discussion group, one after receiv- ing the physician recommendation letter, and 2 after finding out that the local tribal clinic was about to start a new breast cancer center. Finally, when asked what was the most educational activity for them, 8 of the 16 who answered this question indicated the IGDG, 5 indicated the discussion they had with the medical provider, and 3 noted the mammography brochure.
Another tool that we used during the formative research was the PDSA cycle.36,37 The PDSA cycle was
primarily used to test two activities; the physician recommendation letter and the IGDG. Preparation and mailing of the physician recommendation letters took place as planned and no further modifications were needed. Certain modifications took place with regard to testing the implementation of the IGDG, such as reducing the time to answer each self- reflection question to 3 minutes, and a short break was added to the agenda after the self-reflection portion was completed.
A summary of the results of the process evaluation can be found in Table 2.
The qualitative portion of the evaluation consisted of two focus groups (N = 4 participants in total) with research participants who completed both the pre- and post-surveys and one key informant interview with the medical director of the clinic. The results of the qualitative research cover both process and impact evaluation.
The demographic characteristics are presented below. Two of the participants were 50 to 59 years old, one was 60 to 65, and one was older than 65. In regards to education, 2 participants indicated complet- ing some college work, one completed an associate degree, and one had some high school education. Half of them lived in town and half in a rural community. Two participants were divorced, one was married, and one was widowed. In regards to employment status, 2 were fully employed, one was employed part-time, and one was unemployed. Moreover, half of the participants had an annual gross general income of up to $19 000, one less than $10 000, and one over $50 000. Three of them stated that they had already received a screening mammogram and the fourth had scheduled one (which was later confirmed as “received”).
All participants were familiar with the program, its components, and its purpose. When asked what was particularly useful about the program, an emergent theme was the IGDGs. A few participants noted that the program needed more exposure in the community and that some women needed to participate in more than one IGDG so that they could start “opening up” and sharing information with others. Overall, all parti- cipants were satisfied with the implementation of each intervention activity. Some indicated that comments were “detailed, very well put together” in reference to the mammography brochure and “I am so proud to have it (the bracelet) and wear it. It’s an easy way to begin a conversation around cancer and prevention.” Interesting enough, women voiced mixed opinions
58 E. L. TOLMA ET AL.
regarding their experiences with the medical providers. Some women expressed great satisfaction with their medical provider (talked about mammogram, went over the brochure), though one indicated that her pro- vider was not knowledgeable about the project and had to be reminded about it by her. In addition, some women said that some medical providers did not give out the brochure.
In terms of impact, an emerging theme was that through the NWHP the participants became more vocal about breast cancer and that more friends and
family members came to them with questions about mammography. Another idea that arose was that because of the program, some women became more conscious about their own health. As one participant said, “Education is power.” Finally, in terms of recom- mendations for future implementations, women sug- gested a broader exposure of the NWHP in the community and scheduling more frequent IGDGs.
In addition to the focus groups, we conducted an interview with the medical director. The medical director said that all parts of the intervention took place (patient–
Table 2. Process evaluation results. Component and definition What will be evaluated/related research questions Results
Context (aspects of the larger social, political, and economic environments that may have influenced intervention implementation)
Barriers influencing the delivery of the components of the program: 1. Patient–doctor guided communication 2. Educational brochure 3. Mailing out recommendation letter by physician 4. Constraints to attending intergenerational discussion group
Program needed more community exposure and visibility Women should be invited to more than one intergenerational discussion group Physicians needed more time to process the educational tools created Project material was not always in the examining room Some medical providers thought that counseling was too complex Clinic director too overwhelmed with existing clinic tasks to dedicate ample time and resources to this program No “buy-in” from clinic leadership
Reach (proportion of intended target audience that participated in an intervention)
1. Percentage of research participants who had a guided communication with a physician 2. Percentage of women participating in the discussion group.
29 eligible research participants; only 21 (72%) returned the post-survey and participated in the program All (100%) met with the medical provider Approximately 80% participated in the discussion groups
Dose delivered (number or amount of intended units of each intervention or each component delivered or provided)
1. Extent to which physicians had a guided communication with patients, length of discussion 2. Did the medical providers find the guided communication meaningful/beneficial? 3. Number of brochures distributed by physicians 4. Number of brochures given out to clinic and sent to patients by mail or in person via discussion groups 5. Number of physician recommendation letters mailed out 6. Extent to which all components of the intergenerational discussion groups took place
9 completed physician logs on guided communication with patient returned to evaluation team 14 copies of decision stage forms returned to the evaluation team All intergenerational discussion groups took place as planned All components of the intergenerational group discussions took place with the exception of the breast cancer survivor presentation (2/6 sessions)
Dose received (extent to which participants actively engaged with, interacted with, were receptive to, and/or used materials or recommended resources)
1. To what extent did the participants actually have a guided communication with the physician? 2. Number of participants receiving the brochure 3. Number of participants receiving the recommendation letter 4. To what extent did the participants benefit from the overall program?
12 (63%) participants indicated that a physician talked to them about mammography 13 (61%) participants indicated that they had the opportunity to ask the medical provider questions during their visit 12 (60%) participants said that they did not receive the physician’s recommendation letter 18 (85%) participants received breast cancer brochure; among them, 7 still remembered the importance of mammography as highlighted in the brochure Participants enjoyed discussion groups; learned about importance of regular mammograms, knowing one’s body, that breast cancer runs across ages and race/ethnicity, and the importance of sharing information with family and friends Most educational component was intergenerational group discussions Satisfied with most components of the program, mixed feelings about the patient–doctor guided communication All women who got a screening mammogram within the time of the intervention received the bracelet
Fidelity (extent to which the intervention was delivered as planned)
Did all of the intervention components take place as planned?
Most of the activities in the clinic-based component took place as planned with the exception of the physician’s recommendation letter Most of the community components (ie, intergenerational group discussion) took place as planned
THE NATIVE WOMEN’S HEALTH PROJECT 59
doctor guided communication, recommendation letter, and delivery of the brochure), except for discussing the brochure with the patient. Regarding the question, “What was the most useful tool of the clinic-based component?” he indicated the decision stage form because it gave him an idea of the educational gaps that women had. He said, “Sometimes if you don’t ask, you don’t know what the women think about.” When asked about the mixed reac- tions the research participants had in regards to discuss- ing mammography screening with their medical providers, his response was that despite all of the informa- tion disseminated to his colleagues, they thought that the counseling part was too complex.
The medical provider rated the project on a scale of 1 to 5 (5 = excellent) as a “4” because it provided the basics of screening and was culturally appropriate. On the other hand, he said that the NWHP did not last long enough to give him and his colleagues enough time to “process the tools we created.” Finally, in response to our question regarding, how we could promote the clinic’s staff involvement in the study—for instance, through the provision of incen- tives—his response was that providing incentives was unnecessary if the administration truly supported the project. In our case, he reported that the new leader- ship of the clinic viewed the project as external to the clinic’s operation and as inherited from the previous administration. There was no zeal for the project and the new administration never “bought into it.”
Results based on the evaluation survey
Twenty-nine women participated in the study, of whom 21 were retained (72% retention rate) until the end of the study and completed the post-survey. The primary purpose of the quantitative analysis was to determine whether the intervention showed promising positive effects toward the goal of increasing mammo- graphy screening and to identify factors that are pre- dictive of getting a mammogram.
By the end of the project in December 2014, 7 (33%) out of the 21 participants indicated in the post-survey that they received a mammogram (within 3–5 months following the study entry) and 4 received a mammogram within 6 to 12 months following entry into the study. Therefore, 11 (52%) women underwent mammography screening within 1 year of entering the study.
The descriptive baseline data analysis results showed that none of the women had had a mammogram in the last 2 years or more prior to entry into the study. In addition, only 1 out of 5 women had an annual profes- sional breast examination, and only 55% intended to get a mammogram within the next 3 to 5 months. Only 22%
identified screening mammography and professional breast examination as the recommended modalities for early detection of breast cancer, and the majority of the women (40%) indicated that 30 years old was the appro- priate age to start screening mammograms. More infor- mation about the demographic information of the sample can be found in Table 3.
In total, 11 out of 21 (52%, 95% confidence interval [CI], 30% to 74%) women underwent mammography within the 3- to 12-month follow-up up period post entry into the study or 6 months after the end of the intervention. In addition, 6 out of 20 (30%) women improved their intention to undergo mammography during the course of the intervention (95% CI, 15% to 52%). Among the 10 participants who did not undergo mammograms and did not have a mammogram sched- uled, 7 (70%) indicated that they were likely or very likely to obtain their next mammogram within the next 3 to 5 months (95% CI, 35% to 93%). In fact, 3 out of the 7 got a mammogram within 22 months’ of entry into the study.
Due to the small sample size and skewed distribu- tions, the parameter for comparison between the groups across the theoretical constructs as defined by the theory of planned behavior26,27 is the median and not the mean. None of the score differences were sig- nificantly different between the women who got a mammogram and women who did not (2-sided Wilcoxon rank sum test; data not shown). Regarding knowledge, there was no significant difference in knowledge among those who got a mammogram and those who did not. Additional exploratory research (data not shown) revealed that there was not much change in the specific categories of response between the pre and post levels regardless of mammography status. For example, pre-study, 12 women indicated that the first mammogram should be received at the age of 30. Among those, 92% retained their answer post-study.
The study aimed at promoting regular mammography screening and decreasing health disparities by addres- sing multiple system levels; that is, the individual, family/social support, organizational/practice, and community/environmental levels. Research has shown that addressing multiple levels of intervention is the most effective way to decrease health disparities, including breast cancer disparities.39
The primary purpose of this evaluation was to assess the feasibility of this culturally appropriate and theory- based intervention study and its initial impact on
60 E. L. TOLMA ET AL.
a group of AI women who had not have a recent mammogram. We tried to answer two main questions: “Can the intervention work” and “Does it work”? Given some evidence that a treatment might work (feasibility), the “does it work” question refers to whether the inter- vention can be implemented under ideal (efficacy eva- luation) or actual conditions (effectiveness evaluation) compared to other practices.23
Ideally, feasibility studies take place prior to efficacy studies and produce a set of findings to help determine whether an intervention should be recommended for efficacy testing or not.23 However, in this study, we addressed all three levels at the same time. We were able to demonstrate that the NWHP can work and does have some impact. Part of the impact was tested in ideal con- ditions (eg, the IGDGs) and some in actual conditions (eg, the patient–doctor guided communication).
The results of the process evaluation reveal a high reach in both components (clinic and community), ranging from 80% (community-based component) to 100% (clinic-based component). In addition, the results
of the process evaluation indicate that some of the intervention targets (ie, IGDG) were implemented as planned; however, some were not fully implemented, such as the patient–doctor guided communication and mailing of the recommendation letter. This conclusion derives from multiple sources of evidence based on the results of the qualitative and quantitative research. The fact that parts of the intervention were not fully imple- mented may have led to a type III error, which states that an intervention might not be proven effective not due to its theoretical design but due to the fact it was not fully implemented.40 On the whole, it was more difficult to implement the clinic-based component (ie, patient–doctor guided communication, conversation concerning the breast cancer brochure, and mailing of the recommendation letter) than the community-based component because the research team did not have much control of the implementation of these activities.
These activities were implemented solely by the clinic staff in a real-world setting, which meant that the implementation was prone to external factors that
Table 3. Descriptive survey results at baseline (N = 29). Variable Levels N %
Age Mean 59 years (min = 52, max = 71, SD = 5.56) 29 Marital status Divorced 14 48 Employment status Disabled and do not work 8 29
Unemployed or homemaker 6 22 Employed full-time 8 29
Education Some college/junior college but did not graduate from 4-year college 13 48 Graduated from high school or completed GED 8 29
Income Under $45 000 25 86 Rural area Yes 6 21 Private health insurance Yes 16 55 Treated for breast cancer Yes 0 0 Family history–mother Yes 8 28 Ever heard of screening mammograms Yes 27 93 Ever mammogram Yes 27 93 When was the last mammogram? 2 years ago 15 54
3–5 years 9 32 More than 5 years 4 14
Where was the last mammogram? Hospital 17 60 Mobile clinic 4 14 Public health center 2 7
Time travel required to mammogram location Between 31 minutes and one hour 14 50 Over an hour 4 14 Between 16 and 30 minutes 4 14 Less than 15 minutes 2 7
Professional breast exam Every year 5 18 Every 2 years 5 18 Every 3–5 years 9 32 Only once 4 14 Never 5 17
Primary health care provider Yes 26 90 Intent to have a mammogram within next 3–5 months Likely/very likely 15 55 Knowledge of age at 30 years old 18 67 first mammogram 40 years old 9 33 How often has a screening mammogram Every year 11 41
Every 2 years 9 33 Every 3 years 5 18 Never 2 7
Early detection of breast cancer Screening mammography and professional breast examination only 6 22 Screening mammography, professional breast examination and breast self-exam 11 41 Professional breast exam and breast self-exam only 4 15 Breast self-exam only 3 11 Screening mammography only 3 11
THE NATIVE WOMEN’S HEALTH PROJECT 61
might have occurred during the implementation such as staff turnover. Ideally, we would have liked to test the patient–doctor guided communication through the PDSA cycle; however, this was no feasible with respect to protecting patient privacy.
Another useful observation derived from process evaluation was the fact there was not a strong linkage between the clinic- and community-based components. In other words, after they had completed their visit with the physician, some women were reluctant to participate in the IGDG. Based on successful interven- tions on tobacco control, approaches for linking clinical and community services include computer-linked sys- tems where referrals are automatically made from a clinician to a community-based program and vice versa or a fax referral system that links providers with community-based quitlines and vice versa.41 In future replications of our study, one possible way to link the community and clinical components is to create a referral from the medical practitioner’s office to attend the IGDG. Integration of effective clinic- and community-based strategies across the multiple levels of the socioecologic framework expands the availability of services at the levels of influence that may be most accessible to different individuals, thus making utiliza- tion of available services more likely.41
In terms of behavioral outcomes, at least half of the research participants got future mammograms, a finding that indicates a moderate success of the study. Similarly, intention to get a future mammogram was increased, which further provides evidence of the success of the study. This finding is particularly impor- tant considering the fact that none of the research participants had had a mammogram within 2 years prior to entry into the study. The results of this study are similar to the results of English,11 the only similar study we found. During this study, all 39 women who participated in the study got a future mammogram; however, participation in mammography screening was promoted by the research team in an organized way through designated “mammography days” and through the provision of transportation. In our study, the uptake of future mammograms was left entirely to each individual woman, which made the task more challenging but more realistic.
Qualitative research revealed that, through the IGDGs, the research participants became more con- scious about breast cancer and ready to undertake pre- ventative action. In fact, some women became “resources” themselves and were eager to talk about breast cancer to other women and encourage them through their own example of having a mammogram to also have a mammogram. Using the Freire33,34
approach of raising consciousness was an innovative way to spark the women’s interest in and enthusiasm for breast cancer prevention and early detection.
Interesting enough, there was no enhancement of knowledge on mammography screening among the research participants. However, this is not discouraging or surprising because the main public health message we were trying to convey was for women to consult with their primarymedical provider and get regular screeningmam- mograms according to his or her recommendation. In addition, it was evident from the evaluation surveys we completed at the end of each IGDG and during the focus group discussions that women became more knowledge- able regarding other aspects of breast health. For example, they became more knowledgeable of the importance of being aware of breast-related changes and that breast cancer affects all women and not only Caucasians.
From a system perspective, the study promoted sus- tainable changes at the clinic with a potentially long- term impact. Specifically, the bulletin board poster and the mammography screening flowcharts are still being used at the clinic after completion of the project. Moreover, we developed educational tools such as the decision stage forms and the breast cancer brochure, which were left with the clinic personnel for their use as needed. The study also introduced new practices at the clinic in relation to patient counseling on mammogra- phy screening.
Our study also highlights the importance of context and context evaluation.42 The NWHP was launched at a time when there was clinic leadership turnover. This resulted in delays in staff recruitment, project imple- mentation, and, ultimately, a partially engaged partner- ship between the university partner and the clinic administration. Nevertheless, the partnership survived primarily due to the continuous support from the tribal leadership and the existence of the tribal resolution. Overall, the administration helped us in multiple ways, such as providing us with resources (eg, office space), access to the clinic’s medical records, and, most important, through its participation in the CSC.
Another example of the importance of context eva- luation refers to the launching of a mammography center at the tribal clinic toward the end of the study. We hypothesized that the presence of the center might influence the uptake of screening mammography. A related question was included in the post-evaluation survey to assess that possibility. Only 2 women indi- cated that they decided to get a mammogram after they found out that a mammography center would be launched at the clinic, which might indicate that the center’s effect was not as great as we thought it would be.
62 E. L. TOLMA ET AL.
From a methodological perspective, the use of CBPR was instrumental in conducting this evaluation. Having community members conduct research (survey admin- istration, participant recruitment, and qualitative research) requires training, supervision, and patience on behalf of the investigators. On the other hand, CBPR is the only way to conduct this kind of research, especially within AI/AN populations. For example, com- munity members can lend their wisdom as to how to best recruit participants through existing linkages in the community. Community members can also lend their expertise in doing research. For instance, one CRA was a stenographer and thus became the focus group note- taker. This method of matching raw community skills with desired research skills is in line with the concept of building communities from the inside out or the asset approach.43 Through this study we were able to demys- tify the word research in the community and thus build local research capacity to some extent.
From a theoretical perspective, the theory of planned behavior26,27 is a useful tool for identifying the motiva- tors and barriers regarding getting regular mammo- grams. However, it is premature to state that building interventions based on its theoretical concepts will lead to increased intentions and future behaviors. Further replication of this study is needed with a larger sample size and, most important, a longer implementation time so that the intervention become more robust or mature as its components are fully integrated within the clinic and community settings.
The study has a few limitations. The first limitation is the lack of a comparison group. Such a design would have allowed us to assess whether the observed out- comes were due to secular changes or the intervention itself. Despite our intense efforts, we were unable to secure a similar tribal community that would serve as the comparison group.
A second limitation is the small sample size and the moderate response rate, which further led to a lack of generalizability. In other words, the results of this study only pertain to this specific AI community in Oklahoma. Replication of the study with different AI/AN commu- nities, in different geographical regions, and with larger sample sizes can enhance generalizability of the results.
Difficulties in recruiting AI/AN research participants are not uncommon.44 In fact, a similar study also utilized a small number of research participants.11
Recruitment was limited by the fact that only active patients at the clinic were allowed to enter the study. In addition, the potential participants needed to have scheduled an appointment with a medical provider at the clinic within a month. These restrictions minimized our pool of potential research participants. We were
faced with the dilemma of either having a small num- ber of research participants or not having participants at all (in the event that we did not abide by the clinic’s guidelines). We opted for the first option, which allowed us to conduct the study, albeit with a small number of research participants.
In addition, we experienced difficulties in contacting the potential research participants, which was done primarily via phone. The use of lay health advisors could have enhanced the recruitment process, but this option was not feasible at the time of study implemen- tation. In future replications of similar studies, we strongly recommend that more emphasis and resources be allocated to participant recruitment.
Translation to Health Education Practice
The study has several implications for Health Educators. From a programmatic perspective, multicomponent interventions, provided that they are well implemented, can promote sustainability. Health Educators need to be thinking about sustainability as early as in the planning phase and always strive, with their community members, for programs that do not only focus on short-term impact (eg, increase in knowledge) but mostly on long- term impact (eg, policy changes). On the other hand, multicomponent interventions are complex and require many resources. Ample time is needed for the interven- tion’s full implementation so that it becomes mature and fully integrated within the settings or entities in which it is taking place.
A second implication is the importance of active participation of the community members who will ulti- mately benefit from the program. Active participation of the community throughout the stages of program development and evaluation is vital to ensure that the programs we create are culturally appropriate, well supported by community members, and ultimately fea- sible. Health Educators need to create supporting struc- tures such as community-based advisory boards or local community research teams who will assist them in the implementation and evaluation of a program.
Another implication of the study relates to the evalua- tion of health promotion programs, especially in commu- nity settings. Evaluation can be a “messy” undertaking because researchers are constantly trying to balance scien- tific rigor with practicality. This is not surprising because communities are diverse, dynamic, and complex. Therefore, it is important that evaluation is rigorous while at the same time easy to implement and helpful for the communities we work with. Health Educators can work closely with community members to design appro- priate evaluation tools and methodologies to collect the
THE NATIVE WOMEN’S HEALTH PROJECT 63
required data. Emphasis should also be placed on process evaluation by incorporating a planning model such as the one by Steckler and Linnan.35
A final implication relates to the importance of quali- tative research, especially in the evaluation of commu- nity-based programs. Quantitative research does not always produce statistically significant results that will allow us to decipher whether the program is effective or not. Qualitative research can be complementary to quan- titative research to enhance its results or shed light on areas that the quantitative research cannot capture. Evaluation of health promotion programs is like a “puzzle picture,” and to complete the puzzle all pieces need to be present. The use of both qualitative and quantitative research can provide most of the pieces of the evaluation puzzle picture and help us understand why the intervention worked or did not work.
To conclude, theory-based interventions promoting breast health among AI communities by using multiple- level intervention targets at the clinic and community set- tings via CBPR are feasible and efficacious.More research is needed to test the implementation of similar interventions with an ecological focus among different AI/AN popula- tions and by utilizing more complex evaluation designs.
Our sincere gratitude goes out to the leadership of the parti- cular American Indian tribe and its clinic administration for helping accommodate this project and to all of the women who participated in the study. We are also grateful to Drs. J. Neil Henderson and L.D. Carson and Vickie Canfield for their contribution to this study.
No potential conflict of interest was reported by the authors.
This work was supported by a grant from Susan G. Komen®. Partial funding was provided by the National Institutes of Health, National Institute of General Medical Sciences (Grant 1 U54GM104938), and the National Institute on Minority Health and Health Disparities (Grant R25MD011564). The content is solely the responsibility of the authors and does not necessarily represent the official views of the above funding sources.
1. American Cancer Society. Breast Cancer Facts and Figures 2017–2018. American Cancer Society, Inc. https://www.cancer.org/content/dam/cancer-org /research/cancer-facts-and-statistics/breast-cancer-facts-
and-figures/breast-cancer-facts-and-figures-2017-2018. pdf. Published January 1, 2018. Accessed May 21, 2018.
2. National Cancer Institute. Surveillance, epidemiology, and end results program SEER), cancer statistics, inter- active tools, fast facts. https://seer.cancer.gov/faststats/ selections.php?run=runit&output=1&data=1&statis tic=12&cancer=553&year=201804&sex=3&age=1&ser ies=race&race=2;5. Published January 1, 2016. Accessed February 19, 2016.
3. Oi SL, Martinez ME, Li CI. Disparities in breast cancer characterisitcs and outcomes by race/ethnicity. Breast Cancer Res Treat. 2010;127:727–738.
4. Campbell EJ, Martinez SA, Janitz AE, et al. Cancer inci- dence and staging among American Indians in Oklahoma. J Okla State Med Assoc. 2014;107(3):99–107.
5. Campbell J, Gandhi K, Pate A, et al. Five-year cancer survival rates in Oklahoma from 1997 to 2008. J Okla State Med Assoc. 2016;109(7,8):318–332.
6. Indian Health Services. Area summary report government performance and results act (GPRA) 2017. https://www. ihs.gov/quality/includes/themes/responsive2017/display_ objects/documents/GPRASummaryReport-2017.pdf. Published January 1, 2018. Accessed August 18, 2018.
7. Office of Disease Prevention and Health Promotion. Disparities overview by race and ethnicity-Mammography within past 2 years. Healthy People 2020 website. https:// www.healthypeople.gov/2020/data/disparities/summary/ Chart/4055/3. Published January 1, 2017. Updated August 16, 2018. Accessed August 16, 2018.
8. Martinez SA, Janitz AE, Erb-Alvarez J, Mowls DS, Campbell JE, Anderson T. Cancer among American Indians-identifying priority areas in Oklahoma. J Okla State Med Assoc. 2016;109:374–384.
9. Burhansstipanov L, Bad Wound D, Capelouto N, et al. Culturally relevant “navigator”, patient support: the native sisters. Cancer Pract. 1998;6(3):191–194.
10. Burhansstipanov L, Krebs LU, Grass R, Wanliss EJ, Saslow D. A review of effective strategies for native women’s breast health outreach and education. J Cancer Educ. 2005;20(Sup1):71–79. doi:10.1207/ s15430154jce2001s_15.
11. English KC, Fairbanks J, Finster CE, Rafelito A, Luna J, Kennedy M. A socioecological approach to improving mammography rates in a tribal community. Health Educ Behav. 2008;35(3):396–409. doi:10.1177/ 1090198106290396.
12. Haozous EA, Eschiti V, Lauderdale J, Hill C, Amos C. Use of talking circle for comanche women’s breast health education. J Transcult Nurs. 2010;21 (4):377–385. doi:10.1177/1043659609360847.
13. American Cancer Society. Circle of life: a breast cancer awareness project for native American women, a handbook for program planners. http://circleoflife. cancer.org/. Published February 1, 2016. Accessed May 10, 2018.
14. Banner RO, DeCambra H, Enos R, et al. A breast and cervical cancer project in a native Hawaiian commu- nity: wai’anae cancer research project. Prev Med. 1995;24(5):447–453.
15. Tolma EL, Engelman K, Stoner J, et al. The design of a multi-component intervention to promote screening mammography in an American Indian community: the
native women’s health project. AIMS Public Health. 2016;3(4):933–955. doi:10.3934/publichealth.2016.4.933.
16. Strickland JC, Hillaire E. Conducting a feasibility study in women’s health screening among women in a pacific northwest American Indian tribe. J Transcult Nurs. 2016;27(1):42–48. doi:10.1177/1043659614526251.
17. Friederichs-Fitzwater MM, Navarro L, Taylor SL. A value-based approach to increase breast cancer screen- ing and health-directed behaviors among American Indian women. J Cancer Educ. 2010;25(4):582–587. doi:10.1007/s13187-010-0111-0.
18. LaFrance J. Culturally competent evaluation in Indian country. New Direct Eval. 2004;2004(102):39–50. doi:10.1002/ev.114.
19. Usera JJ. The efficacy of an American Indian culturally-based risk prevention program for upper elementary school youth residing on the northen plains reservations. J Prim Prev. 2017;38(1–2):175–194. doi:10.1007/s10935-016-0462-3.
20. Winer RL, Gonzales AA, Noonan CJ, Buchwald DS. A cluster-randomized trial to evaluate a mother-daughter dyadic educational intervention for increasing HPV vacci- nation coverage in American Indian girls. J Community Health. 2016;41(2):274–281. doi:10.1007/s10900-015- 0093-2.
21. Karanja N, Lutz T, Ritenbaugh C, et al. The TOTS community intervention to prevent overweight in American Indian toddlers: a feasibility and efficacy study. J Community Health. 2010;35(6):667–675. doi:10.1007/s10900-010-9270-5.
22. Green L, Glasgow R. Evaluating the relevance, general- ization, and applicability of research: issues in external validation and translation methodology. Eval Health Prof. 2006;29:126–153. doi:10.1177/0163278705284445.
23. Bowen D, Kreuter M, Spring B, et al. How we design feasibilty studies. Am J Prev Med. 2009;36(5):452–457. doi:10.1016/j.amepre.2009.02.002.
24. Minkler M, Wallerstein N, eds. Community-Based Participatory Research for Health. San Francisco, CA: Jossey-Bass; 2003.
25. Daponte BO. Evaluation Essentials: Methods for Conducting Sound Research. San Francisco, CA: Jossey- Bass; 2008.
26. Ajzen I. The theory of planned behavior. Organ Behav Hum Decis Process. 1991;50:179–211. doi:10.1016/ 0749-5978(91)90020-T.
27. Fishbein M, Ajzen I. Belief, Attitude, Intention and Behavior: An Introduction to Theory and Research. Reading, MA: Addison-Wesley; 1975.
28. Tolma EL, Batterton C, Hamm RM, Thompson D, Engelman KK. American Indian women and screening mammography: findings from a qualitative study in Oklahoma. Am J Health Educ. 2012;43(1):18–30. doi:10.1080/19325037.2012.10599214.
29. Tolma E, Stoner JA, Li J, Kim Y, Engelman KK. Predictors of regular mammography use among American Indian women in Oklahoma: a cross-sectional study. BMC
Womens Health. 2014;14(101). doi:10.1186/1472-6874- 14-101.
30. Langdon SE, Golden SL, Arnold EM, et al. Lessons learned from a community-based participatory research mental health promotion program for American Indian youth. Health Promot Pract. 2016;17 (3):457–463. doi:10.1177/1524839916636568.
31. Tolma E, Thomas C, Neely N, et al. Development and evaluation of a culturally tailored breast cancer bro- chure for American Indian women. http://ces4health. in fo/ f ind-products/v iew-product .aspx?code= CJ5FMZ7P Published 2016. Accessed May 12, 2018
32. Weinstein ND, Sandman PM, Blalock SJ. The precau- tion adoption process model. In: Glanz K, Rimer BK, Viswanath K, eds. Health Behavior and Health Education: Theory and Practice. 4th ed. San Francisco, CA: John Wiley; 2008:123–145.
33. Freire P. Pedadogy of the Oppressed. New York, NY: Seabury Press; 1970.
34. Freire P. Education for Critical Consciousness. New York, NY: Seabury Press; 1973.
35. Steckler A, Linnan L, eds. Process Evaluation for Public Health Intervention and Research. San Francisco,CA: Jossey-Boss; 2002.
36. Walley P, Gowland B. Completing the circle: from PD to PDSA. Int J Health Care Qual Assur. 2004;17 (6):349–358. doi:10.1108/09526860410557606.
37. Cleghorn G, Headrick L. The PDSA cycle at the core of learning in health professions education. J Comm J Qual Improv. 1996;22:206–212.
38. Kreuger R, Casey M. Focus Groups: A Practical Guide for Applied Research. 3rd ed. Thousand Oaks, CA: Sage; 2009.
39. Legler J, Meissner HI, Coyne C, Breen N, Cholette V, Rimer BK. The effectiveness of interventions to pro- mote mammography among women with historically lower rates of screening. Cancer Epidemiol Biomarkers Prev. 2002;11:59–71.
40. Basch C, Sliepcevich E, Gold R, Duncan D, Kolbe L. Avoiding type III errors in health education program evaluations: a case study. Health Educ Q. 1985;12:315–331.
41. Ockene JK, Edgerton EA, Teutsch SM, et al. Integrating evidence-based clinical and community strategies to improve health. Am J Prev Med. 2007;32 (3):244–252. doi:10.1016/j.amepre.2006.11.007.
42. Clarke AM, O’Sullivan M, Barry MM. Context matters in programme implementation. Health Educ. 2010;110 (4):273–293. doi:10.1108/09654281011052637.
43. McNight JL, Kretzmann JP. Mapping community capa- city. In: Minkler M, ed. Community Organizing and Community Building for Health and Welfare. 3rd ed. New Brunswick, NJ: Rutgers University Press; 2012:171–186.
44. Burhansstipanov L, Dignan MB, Bad Wound D, Tenney M, Vigil G. Native American recruitment into breast cancer screening: the NAWWA project. J Cancer Educ. 2000;15(1):28–32. doi:10.1080/08858190009528649.
- Description of the intervention
- Process evaluation and process evaluation plan
- Evaluation survey
- Quantitative data analysis
- Qualitative research
- Process evaluation
- Qualitative research
- Results based on the evaluation survey
- Translation to Health Education Practice
- Disclosure statement