Abstract
Background
Despite evidence that eHealth approaches can be effective in reducing HIV risk, their implementation requirements for public health scale up are not well established, and effective strategies to bring these programs into practice are still unknown. Keep It Up! (KIU!) is an online program proven to reduce HIV risk among young men who have sex with men (YMSM) and ideal candidate to develop and evaluate novel strategies for implementing eHealth HIV prevention programs. KIU! 3.0 is a Type III Hybrid Effectiveness-Implementation cluster randomized trial designed to 1) compare two strategies for implementing KIU!: community-based organizations (CBO) versus centralized direct-to-consumer (DTC) recruitment; 2) examine the effect of strategies and determinants on variability in implementation success; and 3) develop materials for sustainment of KIU! after the trial concludes. In this article, we describe the approaches used to achieve these aims.Methods
Using county-level population estimates of YMSM, 66 counties were selected and randomized 2:1 to the CBO and DTC approaches. The RE-AIM model was used to drive outcome measurements, which were collected from CBO staff, YMSM, and technology providers. Mixed-methods research mapped onto the domains of the Consolidated Framework for Implementation Research will examine determinants and their relationship with implementation outcomes.Discussion
In comparing our implementation recruitment models, we are examining two strategies which have shown effectiveness in delivering health technology interventions in the past, yet little is known about their comparative advantages and disadvantages in implementation. The results of the trial will further the understanding of eHealth prevention intervention implementation.Free full text
Keep it up! 3.0: Study protocol for a type III hybrid implementation-effectiveness cluster-randomized trial
Abstract
Background:
Despite evidence that eHealth approaches can be effective in reducing HIV risk, their implementation requirements for public health scale up are not well established, and effective strategies to bring these programs into practice are still unknown. Keep It Up! (KIU!) is an online program proven to reduce HIV risk among young men who have sex with men (YMSM) and ideal candidate to develop and evaluate novel strategies for implementing eHealth HIV prevention programs. KIU! 3.0 is a Type III Hybrid Effectiveness-Implementation cluster randomized trial designed to 1) compare two strategies for implementing KIU!: community-based organizations (CBO) versus centralized direct-to-consumer (DTC) recruitment; 2) examine the effect of strategies and determinants on variability in implementation success; and 3) develop materials for sustainment of KIU! after the trial concludes. In this article, we describe the approaches used to achieve these aims.
Methods:
Using county-level population estimates of YMSM, 66 counties were selected and randomized 2:1 to the CBO and DTC approaches. The RE-AIM model was used to drive outcome measurements, which were collected from CBO staff, YMSM, and technology providers. Mixed-methods research mapped onto the domains of the Consolidated Framework for Implementation Research will examine determinants and their relationship with implementation outcomes.
Discussion:
In comparing our implementation recruitment models, we are examining two strategies which have shown effectiveness in delivering health technology interventions in the past, yet little is known about their comparative advantages and disadvantages in implementation. The results of the trial will further the understanding of eHealth prevention intervention implementation.
1. Introduction
Young men who have sex with men (YMSM) account for nearly 70% of all new HIV diagnoses among adolescents and young adults in the United States, and are the only group showing increasing incidence [1]. Despite this increased HIV risk, few interventions in the Centers for Disease Control and Prevention (CDC) Compendium of Evidence-Based Interventions and Best Practices for HIV Prevention Programs [2] are focused on YMSM [3]. The current arsenal of behavioral evidence-based interventions primarily includes face-to-face individual and small-group programs [2], and their reach has been limited by economic and structural barriers to implementation [4]. eHealth approaches may represent a new cost-effective critical modality for engaging YMSM as funding shifts away from health education and behavioral risk reduction to medical approaches [5], and deliver critical intervention content while overcoming barriers to access (e.g., geography [6, 7]), structural issues like stigma in health settings [8], and delivery challenges (e.g., fidelity [9]).
Despite emerging evidence that eHealth approaches can be highly effective in reducing HIV risk or improving care engagement [10, 11], implementation requirements for scale up – namely the cost-effective means to reach a large real-world audience – are not well established. To date, little to no implementation science studies have examined strategies to effectively scale up eHealth HIV prevention programs, an important component of the CDC Ending the HIV Epidemic initiative [12]. To that end, Keep It Up! (KIU!), a CDC designated best-evidence HIV prevention intervention made with and for YMSM, is an ideal candidate to develop and evaluate novel strategies for the implementation of eHealth HIV prevention programs.
KIU! was previously studied in several contexts. A pilot trial (KIU! 1.0) among 102 YMSM, ages 18–24, found a significant decrease in condomless anal sex (CAS) compared to a knowledge-based control [13]. A service-based implementation (KIU! 1.5) at a Chicago community-based organization (CBO) showed significant pre—post reductions in condom errors, total number of male sex partners, and CAS among 579 YMSM, ages 15–24 [14]. A multi-city randomized clinical trial (KIU! 2.0) with 901 YMSM, ages 18–29, demonstrated a significant 40% decrease in sexually transmitted infection (STI) incidence as well as a significant decrease in self-reported CAS [15, 16]. Intervention acceptability was high across all three studies, and KIU! was designated a CDC best-evidence HIV prevention intervention [17].
In this manuscript, we describe the design of the KIU! 3.0 trial, a Type III Hybrid Effectiveness-Implementation cluster randomized trial [18], which aims to 1) compare two implementation strategies, CBO versus direct-to-consumer (DTC) delivered approaches, using a cluster-randomized trial and 2) examine the effect of strategies and determinants on variability in implementation outcomes using a mixed-methods research approach [19, 20] on the domains from the Consolidated Framework for Implementation Research, version 2.0 [21]. The KIU! 3.0 trial will also explore sustainment of KIU! at the completion of the study.
2. Methods
The KIU! 3.0 trial launched in October 2019 but experienced slow participant recruitment and poor retention due, largely, to the COVID-19 pandemic. For example, many HIV testing clinic sites temporarily closed for in person testing or slowed down HIV testing to accommodate COVID-related services. To address these issues and improve recruitment and retention, several changes were made to the KIU! study design “post-launch” in March 2021. The following sections describe the initial design as well as the modifications implemented. At the time of writing, retention for the trial was ongoing.
All procedures involving human participants were in accordance with the ethical standards of Northwestern University Institutional Review Board and with the 1964 Helsinki declaration and its later amendments. All participants provided informed consent.
2.1. Sample frame – County selection and randomization
CBO and DTC delivery are widely used health programming implementation strategies [22, 23] however their relative advantages and disadvantages are unknown. Our sample size target was 66 counties to be randomized 2:1 to the CBO and DTC strategies, respectively, to account for the assumption that there would be counties where no CBOs applied to the request for applications (i.e., there were 44 counties from which 22 CBOs were recruited). We selected US counties with an estimated number of YMSM ≥1,500 [24], which yielded an initial sample frame of 113 counties. An iterative process was developed to narrow the sample frame to 66 counties, which included eliminating counties in which KIU! was previously studied and eliminating nearly all bordering counties to reduce clustering that could produce unintentional exposure across county lines. For more information, see supplementary material entitled County Selection Iterative Process.
2.2. Community-based organization eligibility and recruitment
To select CBO recruitment sites, we published a request for proposals modeled on those created by the CDC and other HIV prevention funders. An objective scoring rubric was developed, and reviewers were trained to score applications on nine criteria. Each application was scored by three review panelists. Once panelists submitted scores, we averaged each application’s scores and selected the highest scoring applicants for funding. For more details on the request for proposals process, see supplementary material entitled Request for Proposals Process.
2.3. Direct to consumer recruitment catchment area
It became evident that certain constraints of our research study design, which was intended to facilitate statistical comparisons of KIU! effectiveness and implementation across arms at the county level, were at odds with running a pragmatic implementation trial. Specifically, nearly a quarter of people interested in joining the DTC arm screened ineligible due to county of residence. As such, post-launch, we expanded the DTC catchment area from 22 counties to nationwide to improve study enrollment and make DTC delivery more pragmatic – in other words, aligned with how other self-guided digital health interventions or applications are often delivered outside research settings (e.g., available to all through the App Store or accessible online).
2.4. Participant inclusion criteria and recruitment
The study has two components: a service component (intervention content) and a research component (surveys and STI testing). All participants were allowed to take part in the service component of the study but had to meet eligibility criteria to complete the research component. YMSM were eligible for participation in the research component if they met the following criteria: 1) tested HIV-negative at the time of registration, 2) reported they were assigned male at birth, 3) identified their current gender identity as male or non-binary, 4) were between the ages of 18–29 at registration, 5) reported CAS with a male partner in the previous six months, and 6) were not on pre-exposure prophylaxis (PrEP), had been on PrEP for less than six months, or reported missing a PrEP dose in the previous six months. Post-launch, participant inclusion criteria were expanded and it was no longer necessary for participants to 1) have received a negative HIV test at registration in order to create additional enrollment channels (e.g., at community events where testing was not occurring on site), or 2) report CAS with a male partner in the previous six months because COVID disrupted patterns of sexual behavior [25]. In addition, the upper age limit of eligibility was increased from 29 to 34 years given older men’s interest in KIU.
Participants were recruited using different strategies based on the arm of the study (CBO versus DTC strategy). For the CBO arm, recruitment was primarily integrated into routine HIV testing and prevention services. Using this approach, after a negative test result, CBO staff explained KIU! And registered interested YMSM, who automatically received a confirmation email that allowed them to set a password and access the KIU! Welcome page. Participants were prompted to complete a baseline survey embedded within the welcome page, where their eligibility for the research component of KIU! Was verified. All participants, regardless of eligibility, could access intervention content upon completing the baseline survey as KIU! Was being delivered as a service while also being studied.
In the DTC arm, participants were recruited primarily through social media, dating apps, and supplemental approaches (e.g., referrals and print ads). Participants registered for KIU! Using a brief online form. Participants who indicated they lived in a county assigned to the CBO arm were marked as ineligible and not provided access to the intervention. All other participants received a confirmation email to set a password and access the KIU! Welcome page where they completed a baseline survey to assess eligibility for the research component. If eligibility requirements were met, participants were sent an at-home HIV test kit and materials to self-administer testing for urethral and rectal gonorrhea and chlamydia. Individuals ineligible for the research study were still offered access to the online intervention content. For additional details on the DTC-mediated recruitment, screening, and testing strategies, see the supplementary material entitled Direct-to-Consumer Recruitment.
2.5. Intervention delivery
For this trial, KIU! was redesigned using multi-tenant architecture into a customizeable mobile, tablet, and desktop responsive web-based participant-facing application and online administrative dashboard. KIU! consisted of four user roles with separate login pages and varying levels of access to the KIU! application: Superusers, Organization Implementers (i.e., Project Directors and Project Coordinators), Organization Recruiters, and YMSM participants. Superusers were comprised exclusively of Northwestern University project staff who oversaw the application’s functionality, content, and data. CBO Project Directors had the ability to register participants, add coordinator and recruiter user roles, and had access to participant data relevant for intervention retention for their CBO (such as participant contact information and contact logs, intervention progress and application usage, etc.); coordinators had similar access as project directors but were limited to only adding and removing recruiter admin roles to KIU!; recruiters were only able to register participants and did not have access to any other application data. DTC project staff had the same level of access as CBO Project Directors. Post-launch, a to-do list feature was created within the adminstrative dashboard for Project Directors and Coordinators to streamline and improve participant retention. The to-do list indicated which participants needed follow-up at different stages of intervention progress.
After registration into the application (either automatically from the DTC online screener or manually by CBO staff), automatic confirmation emails were sent to YMSM participants directing them to set a password and log into the KIU! application to complete a baseline survey hosted in REDCap [26]. After baseline survey completion, YMSM participants were directed to the KIU! homepage consisting of the KIU! content, a technical support page, resources pages, a page of custom-made tools, and page to review their favorited content. YMSM participants were required to complete KIU! content in sequence, although they were able to return to previously viewed material using back buttons, a learning map, the tools page and the favorites pages.
As previously published, we iteratively developed the KIU! intervention with input from YMSM and CBOs [27]. Initially, KIU! was comprised of seven modules completed across three sessions with mandatory 8-h breaks between sessions. Additional content and follow-up surveys were provided in two booster sessions 3 and 6 months after the main intervention. Participants also completed a final follow-up survey and STI testing 12-months post-intervention. Table 1 provides an overview of subject matter covered within each module of the intervention as well as the structured breaks between modules. Post-launch, the 8-h timed breaks between the three main intervention sessions were made optional in response to CBO and YMSM feedback that moving through the sessions without breaks is more consistent with the current media landscape. The time between the main intervention content and the booster sessions was reduced from 3 months (Booster 1) and 6 months (Booster 2) to 6 and 12 weeks, respectively, to allow more time for enrollment due to COVID-related slow-downs while staying within the project budget. Follow-up surveys were removed from Booster 1 and only collected during Booster 2 (12-weeks post-intervention). STI testing data was also only collected at 12-weeks post-intervention. For more details on intervention delivery, including data collected by the application and the experience of YMSM, see the supplementary material entitled KIU! Intervention Delivery.
Table 1.
Session | Module | Main Content |
---|---|---|
1 | In Your Community | Candid interviews with young gay and bisexual men about their communities, family, sex, and relationships that situate these relationships as important aspects of health. Similar interviews on various topics appear throughout the program. |
Hooking Up Online | Three comic book vignettes about meeting men online for sex that focus on identifying factors that may lead to increased HIV risk, such as mood and substance use, as well as preparatory strategies to reduce those risks. | |
With Friends | The first chapter of a scripted soap opera that highlights the risks of making assumptions around HIV status and monogamy and promotes positive norms around getting regular HIV testing and utilizing prevention strategies. Remaining chapters appear throughout subsequent modules. | |
| ||
8-h break (later removed) | ||
| ||
2 | In Bars and Clubs | An interactive game that addresses the consequences of excessive alcohol consumption and drug use, as well as decisional balance around condom use. |
On Dates | An animated story that explores how power dynamics in a dating relationship can affect sexual risk taking. Interactive risk calculators that demonstrate differing levels of HIV and STI risk based on various sexual behaviors. | |
| ||
8-h break (later removed) | ||
| ||
3 | In Relationships | An interactive animated story and supporting videos that model using good communication skills in relationships to help meet one’s sexual, emotional, and health needs. |
In the Future | A goal-setting activity that helps participants identify ways to meet their sexual, emotional, and health needs and troubleshoot obstacles to achieving those goals. | |
| ||
3-month break (later changed to 6-week break) | ||
| ||
4 | Knowing Your Status | New videos and activities that focus on the importance of regular HIV testing in combination with prevention strategies like condoms and PrEP while adding additional layers of nuance, such as sexual pleasure and preventing condom use errors. There is also a check-in on participants’ goals. |
| ||
3-month break (later changed to 6-week break) | ||
| ||
5 | In Love | In addition to checking in on participants’ goals, new videos demonstrate when and how to discuss stopping condom use with a partner in a long-term relationship, as well as how to restart condoms after stopping. |
2.6. Implementation strategies - Community-based organization – Training
Northwestern project staff developed and disseminated online training to CBO implementers. Training modules were developed using the KIU! platform, to offer seamless integration of KIU! content, and covered topics ranging from how to use the KIU! administrative dashboard to recruitment strategies. Ongoing technical assistance and training was provided to CBOs to meet emergent needs of the trial. See the supplementary material entitled Training for more details.
2.7. Implementation strategies – Incentives
Offering financial incentives to encourage participation in HIV prevention and care service programs is a common and pragmatic practice [28] replicated in this intervention. Initially, participants were entered into raffles for $200 gift cards when they completed research surveys at baseline, 3-, 6-, and 12-months post-intervention. In the CBO arm, each organization offered incentives at their own discretion for completion of the baseline survey and service components of the program. Across all CBOs, incentives for baseline survey completion ranged from $5 to $30, with many CBOs choosing not to incentivize baseline completion. For the first three sessions, CBOs offered participants between $10 and $50 total. For completing the two booster content sessions post-intervention, CBOs offered participants between $15 and $75 total. CBOs mostly offered gift card incentives, however, several included non-monetary incentives such as free STI testing and bottles of lube.
After a study protocol modification in July 2020, DTC participants received a $10 or $25 virtual Visa gift card for completing the first three intervention sessions. Incentive amount was randomized by county to determine if a threshold amount must be met for incentives to be effective. DTC participants also received a discount code for an online store that sells adult toys and products. For completing the two boosters, participants were entered into an e-raffle for prizes donated by various vendors. These strategies were based on those successfully used by CBO partners in previous KIU! service implementations to keep YMSMs engaged [14].
The incentive structure was updated twice more due to retention difficulties and youth advisory board feedback that guaranteed payment comparable to other HIV prevention services they participated in would increase retention. Incentives were increased in Spring 2021 with DTC participants receiving a $50 Amazon gift code for completing the first three sessions of the intervention, $10 Amazon gift code for completing 6-week booster content, and $15 Amazon gift code for completing 12-week booster content. Participants in both arms of the study also received a $10 Amazon gift code or virtual Visa gift card for completing 12-week STI testing. The second update in August 2022 increased 12-week incentives for participants in both arms to receive $20 for completing the survey and $20 for completing STI testing.
2.8. Measures
The RE-AIM framework [29, 30] broadly guided our outcome measurement. The main comparison between the CBO and DTC implementation approaches used quantitative data on Reach and Effectiveness. Reach to YMSM was defined as the absolute number of participants enrolled (i.e., completed baseline and screened eligible), as this metric was both comparable across arms and provided the necessary demographic data to examine differences between arms. Measures for calculating effectiveness at reducing HIV risk among engaged YMSM [29, 31, 32] are depicted in Table 2. Mixed-methods assessments for examining the effect of strategies and determinants on implementation variability [33] as well as for measuring sustainability are depicted in Table 3, and constructs related to delivery cost [34, 35] are depicted in Table 4.
Table 2.
Outcome | Construct | Measure | Measurement Schedule | |||||
---|---|---|---|---|---|---|---|---|
Original design | Amended design | |||||||
| ||||||||
Baseline | 3-month | 6-month | 12-month | Baseline | 12-week | |||
Primary | HIV Risk Behavior | Number of condomless male assigned at birth (MAAB) sex partners | X | X | X | X | X | X |
Number of casual condomless MAAB sex partners | X | X | X | X | X | X | ||
Behavior associated with most recent sexual encounter: condoms, PrEP, substance use | X | X | X | X | X | X | ||
STI Incidence | Urethral and rectal Chlamydia and Gonorrhea - Aptima Combo 2 Assay on the Panther system | X | X | X | X | |||
Secondary: Prevention Continuum | HIV/STI Testing | Assessing past 3-month HIV and STI testing history | X | X | X | X | X | X |
PrEP Use & Adherence | Current & past 3-month PrEP use; adherence | X | X | X | X | X | X | |
Secondary: Substance Use | Alcohol Problems | AUDIT Alcohol Consumption Questions (AUDIT-C); Alcohol Use Disorders Identification Test (AUDIT) branched to those scoring >3 | X | X | X | X | X | X |
Other Drug Use | Past 3-month use of marijuana and illicit drugs | X | X | X | X | X | X |
Table 3.
Level | Construct | Measure | Measurement Schedule | ||||
---|---|---|---|---|---|---|---|
Baseline (YMSM)/Pre-Launch (CBO) | 3-month (YMSM)/4-month (CBO) | 6-month | 12-month | 24-month | |||
Participant a | Self-Efficacy | Confidence in HIV testing, condom use, and ability to get on PrEP | X | X | X | X | |
Resource Access | Access to HIV testing, HIV prevention services, and other sexual health services | X | X | X | |||
Stigma | Sexual healthcare avoidance due to LGBTQ stigma | X | |||||
Satisfaction | Satisfaction with DTC or CBO staff and services | X | X | X | |||
Technology Center b | Engagement | Mean number of KIU! modules completed by participants | X | X | X | X | |
System analytics; Metrics of activity within modules | X | X | X | X | |||
CBO | Demographics | X | |||||
Training Effectiveness and Satisfaction | Effectiveness and satisfaction with training | X | X | ||||
Normalization Process Theory | NoMAD: Normalization Process | X | X | X | X | ||
Organizational Readiness for Change | OCRBS: Organizational Change Recipient Beliefs Scale | X | |||||
Leadership Engagement | ILS: Implementation Leadership Scale | X | X | X | X | ||
Implementation Climate | ICS: Implementation Climate Scale | X | X | X | |||
Readiness for eHealth | Readiness for eHealth | X | |||||
Sustainability | Program Sustainability Assessment Tool | X | X | ||||
Adaptation | Items based on the FRAME measure | X | X | ||||
CFIR domains | Administrative Data | X | X | X | X | X | |
Qualitative Interviews | X | X |
Table 4.
Cost Type | Construct | Measures | Source | Measurement Schedule |
---|---|---|---|---|
Start-Up | Recruitment | Staff time, consultant, and other costs | Staff activity logs; study records and CBO interviews/surveys | Monthly for start-up period; one-time |
Initial training | Number and duration of trainings | Study records | One-time | |
Technology development | Staff time, content development costs | Staff activity logs | Monthly for start-up period; one-time | |
Variable | Shipping and processing HIV and STI test kits | Number of units shipped, STI lab cost per test, staff time per shipment | Study records; staff interviews | Ongoing; one-time |
Participant contacts | Number of participant contacts, staff time per contact | Participant contact logs; staff interviews | Ongoing; one-time | |
Time-Dependent | Social media advertisement | Total spending | Study records | Ongoing |
Booster training and technical assistance | Staff time by arm | CBO and DTC participant tracking logs; staff activity logs | Ongoing; monthly | |
Other staff activities | Staff time by arm | Staff activity logs; CBO interviews/surveys | Monthly; one-time | |
Space and occupancy costs | Square feet | Staff interviews | One-time |
2.9. Data analysis – Public health impact
Our primary outcomes were public health impact, defined as reach X effectiveness [29, 31, 32], and cost per infection averted [36, 37]. For this trial, public health impact will be assessed by 1) number of YMSM community reached weighted by their HIV risk, and 2) effectiveness at reducing HIV risk [29, 31, 32]. By measuring individual-level change on these modifiable factors, our index of public health impact allowed for heterogeneity in response to KIU! across individuals, race and age groups, counties, and implementation conditions. Effectiveness was measured by the estimated change in that person’s risk for HIV from baseline to follow-up surveys and as determined by observed changes in target risk factors (such as CAS, STIs, and adherent PrEP use, all of which have major impacts on HIV transmission in YMSM) [38–40]. We based our modeling of HIV risk on published single-exposure probabilities and account for multiple exposures using binomial modeling [41]. Because these are individual-based measures, we will use two-level mixed-effect modeling [42].
We followed established guidelines for collecting cost data and conducting economic evaluations. Cost analyses were conducted for each arm from the perspective of the healthcare sector [34, 35]. We followed a micro-costing approach, a technique in which all inputs consumed in an intervention are identified and quantified in detail and then converted into fiscal terms to produce a cost estimate. Cost analysis was used to compute cost per infection averted, which is analogous to the measure used by CDC to decide which effective HIV prevention intervention would be supported as part of a high impact prevention plan [36, 37].
2.10. Data analysis – Compare CBO and DTC implementation strategies
Analyses involve Multivariable Generalized Linear Mixed Models, as they account for nested individual observations within counties and are commonly used in cluster-randomized trials [35, 43]. The multivariable aspect of the model allows for the control of factors that are unbalanced between arms, either because they were not accounted for in the randomization process or because they may have become unbalanced due to loss to follow-up. To account for potential selection bias in constructing an efficient (one degree of freedom) adjustment for measured differences in county-level covariates, we will adjust for the linear combination forming the first canonical covariate that maximally distinguished the sites in the two arms [44, 45]. Similar to propensity score analysis, we will also formally include the model that predicts selection effects [46]. We will estimate standard errors using nonparametric bootstrapping techniques within the multivariable framework treating arm as a dichotomous indicator [35]. The differences between the intervention arms will be evaluated by examining the statistical significance of the level-2 (i.e., cluster level) dichotomous indicator for the intervention arms. We will estimate and compare the difference in predicted mean cost per participant between the arms and will also use those estimates of mean cost to characterize the cost per HIV infection averted in each arm of the study.
For cost-effectiveness analysis, parameters obtained from bootstrapping will be used to estimate cost-effectiveness acceptability curves which will indicate the probability that either intervention implementation strategy is a good value for different willingness-to-pay thresholds (i.e., incremental cost per infection avoided) [47]. Finally, all outcomes analysis will conform to best practices in analysis of randomized trials, including intention-to-treat analysis and sensitivity analysis of missing data. Multilevel multiple imputation models will be used to examine the potential effect of missing responses on the results [48–50].
3. Discussion
Cluster-randomized trials are increasingly used in public health and medicine [51], but with a few important exceptions [52–55], they have not seen significant application in HIV implementation science. Our trial compares the implementation of two competing implementation strategies – CBO versus DTC. The CBO model, with CBOs directly funded by the CDC or their local health department [56], is the current backbone of implementing HIV prevention programs. The DTC model introduces a novel method of mitigating barriers to regular HIV/STI testing by facilitating self-testing [23], which has been demonstrated to be accurate, acceptable, and feasible, especially among those who may not otherwise seek an HIV test [57]. While both the CBO and DTC arms are two viable implementation strategies, their relative advantages and disadvantages are unknown. By evaluating both strategies head-to-head in this cluster-randomized trial, we will be able to discern their overall effectiveness and identify key moderating factors [33, 58] that could help boost and shape future dissemination and implementation of eHealth HIV prevention interventions [59] and contribute to greater scalability, reach, and public health impact.
This paper described the proposed trial protocol and the post-launch amendments made to the protocol to combat extenuating real-life circumstances that contributed to recruitment and retention challenges in the CBO and DTC arms. The COVID-19 pandemic may have negatively impacted KIU! recruitment and retention through numerous mechanisms: 1) disrupting CBOs’ ability to provide onsite services and recruit individuals; 2) causing a decrease in sexual activity and overall HIV/STI risk and prioritization of sexual health; 3) creating other physical and mental health issues that became higher priorities than HIV prevention (e.g., sudden joblessness, fear of COVID, changing personal and professional responsibilities); and 4) YMSM experiencing “screen fatigue,” as many interactions moved into the virtual space. To address these issues and improve recruitment and retention, several changes were made to the KIU! study design (see Table 5).
Table 5.
Component | Original Design | Amended Design |
---|---|---|
Eligibility | ||
Age | 18 – 29 | 18 – 34 |
HIV status | HIV negative test result | HIV negative test result OR self-report HIV negative or unknown status |
Condomless anal sex | Self-report in previous 6 months | No requirement |
Location (DTC arm) | 22 US counties | Nationwide |
Delivery | ||
Session Breaks (Sessions 1 – 3) | 8-h breaks | Optional breaks |
Follow-up Content Timepoint | 3- and 6-months | 6- and 12-weeks |
Follow-up Survey Timepoint | 3-, 6-, and 12-months | 12-weeks |
Follow-up Testing Timepoint | 12-months | 12-weeks |
Incentives | ||
CBO | $200 survey raffle; At CBO discretion (sessions 1 – 3 and at 6-weeks) | Original design incentives plus $20 (12-week survey) and $20 (12-week at-home STI testing) |
DTC | $200 survey raffle | Original design incentives plus $50 (sessions 1 – 3); $10 (6-week); $20 (12-week survey); $20 (12-week STI testing) |
First, study effectiveness was de-emphasized post-launch. Effectiveness was already demonstrated in KIU! 2.0 and based on emerging data, changes in sexual risk may be attributable to COVID effects. Disentangling COVID effects from the effects of the KIU! program would be challenging, if not impossible. This change allowed the DTC arm to recruit nationally, which more closely approximates a pragmatic DTC implementation strategy. Second, the 8-h timed breaks between the three main intervention sessions were removed, and the time between the main intervention content and the booster sessions were reduced from 3 months (Booster 1) and 6 months (Booster 2) to 6 and 12 weeks, respectively. This change was made for the following reasons: 1) previous CBO-based implementations of the intervention were delivered successfully without timed breaks, 2) long-term maintenance of intervention effects were previously shown in KIU! 2.0, 3) requests by CBOs currently enrolled in the KIU! 3.0 trial and 4) this change will not affect implementation aims.
Third, participant inclusion criteria were expanded. KIU! was originally designed to be delivered following a negative HIV test result to utilize testing as a means of recruiting diverse YMSM and leverage YMSM’s attitudes during that teachable moment. However, that specific theoretical mechanism has never been empirically tested to demonstrate that it is a core component of the intervention effect. In fact, in KIU! 1.5, the program was decoupled from testing, and pre—post intervention results remained similar to the KIU! 1.0 pilot effects. Furthermore, not requiring an HIV test allowed CBOs to adopt additional marketing and engagement strategies appropriate to the COVID pandemic. Thus, the requirement of a negative HIV test result before intervention access was removed. Due to the marginal cost of delivery and interest in KIU! from YMSM who had not recently engaged in CAS, this criterion was also removed, as it will not affect study implementation aims, and the COVID-19 pandemic may have reduced CAS. Lastly, the age of eligibility was increased from 29 to 34 since the CDC estimates that the highest-risk YMSM are up to age 34. Again, due to the marginal cost of delivery and interest in KIU! from older YMSM, this criterion was relaxed as it will not affect study implementation aims.
Effective implementation has been described as the greatest challenge to HIV prevention [60, 61] yet there has been insufficient research testing implementation strategies that will ensure effective interventions get to the right individuals at the right time in the right dose [4]. While prior implementation science studies have compared capacity-building approaches for individual and small-group prevention programs [62–65], there have been little-to-none on eHealth HIV prevention [22, 66–68]. The comparison of two viable strategies in this trial will be a contribution to the eHealth literature as well as the implementation science literature that rarely tests markedly different delivery strategies against one another in the same trial. This design led to limitations in what could be reasonably compared between arms and could pose challenges in determining which strategy would be preferred as results could indicate specific strengths and weaknesses for each strategy. Thus, not a clear “winner” as is often sought in head-to-head trials such as this. The use of incentives is another limitation because it may introduce bias into the trial. However, we will do empirical tests to assess if incentives matter, the amount necessary to affect intervention completion, and for which participants incentives encourage intervention completion. The proposed study will lead the way to understanding the implementation of eHealth HIV prevention interventions to ensure that the promise of cost-effective scalability is realized [69].
Supplementary Material
Request for Proposals Process
Direct to Consumer Recruitment
Training Documents
KIU Intervention Delivery
County Selection Iterative Process
Acknowledgments
Research reported in this publication was supported by the National Institute of Mental Health (NIMH), National Institute on Drug Abuse (NIDA), and Office of the Director (OD) of the National Institutes of Health under Award Number R01 MH118213. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. We would like to thank the staff at the 22 CBOs at which KIU! is being implemented as well as the many community partners who are assisting in recruitment for the DTC arm. Additionally, we thank the following centers and institutions for their immeasurable contributions to our trial: Center for Prevention Implementation Methodology for Drug Abuse and HIV; Prevention Science and Methodology Group; Third Coast Center for AIDS Research; Research Application Design and Development, Institute for Sexual and Gender Minority Health and Wellbeing; the Institute for Sexual and Gender Minority Health and Wellbeing, Feinberg School of Medicine, and the Department of Medical Social Sciences at Northwestern University; Center for Health Economics of Treatment Interventions for Substance Use Disorders, HCV, and HIV (CHERISH); Department of Population Health Sciences at Weill Cornell Medicine; Boston Medical Center; and Erik Munson of Clinical Laboratory Science at Marquette University.
Funding
This work is supported by the National Institute of Mental Health (NIMH), National Institute on Drug Abuse (NIDA), and Office of the Director (OD) of the National Institutes of Health [R01MH118213].
Abbreviations:
CBO | community based organization |
CAS | condomless anal sex |
CDC | Centers for Disease Control and Prevention |
DTC | direct-to-consumer |
KIU! | Keep It Up! |
PrEP | pre-exposure prophylaxis |
STI | sexually transmitted infection |
YMSM | young men who have sex with men |
Footnotes
CRediT authorship contribution statement
Brian Mustanski: Methodology, Writing – Review & Editing; Rana Saber: Software, Writing – Original Draft, Writing – Review & Editing; Justin Patrick Jones: Writing – Original Draft, Writing – Review & Editing; Kathryn Macapagal: Methodology, Writing – Original Draft; Nanette Benbow: Methodology, Writing – Review & Editing; Dennis H. Li: Methodology, Writing – Original Draft; C. Hendricks Brown: Methodology; Patrick Janulis: Methodology; Justin D. Smith: Methodology; Elizabeth Marsh: Methodology; Bruce R. Schackman: Methodology; Benjamin P. Linas: Methodology; Krystal Madkins: Writing – Original Draft, Writing – Review & Editing; Gregory Swann: Methodology, Writing – Original Draft; Abigael Dean: Writing – Original Draft; Emily Bettin: Methodology, Writing – Original Draft; Alexandra Savinkina: Writing – Original Draft
Trial Registration: NCT03896776, clinicaltrials.gov, 1 April 2019
Ethics approval and consent to participate
This protocol was approved by the IRB at Northwestern University, IRB reference number STU00207476. YMSM and CBO staff participants were informed of the aims of the study as well as data protection; all participants were consented to participate in our trial. Reports derived from our trial will be delivered in aggregate such that participants cannot be identified.
Consent for publication
Not applicable.
Declaration of Competing Interest
The authors declare that they have no competing interests.
Availability of Data and Materials
The datasets generated and/or analyzed during this trial will be available from the investigators upon reasonable request.
References
Citations & impact
Impact metrics
Alternative metrics
Discover the attention surrounding your research
https://www.altmetric.com/details/142941840
Article citations
A mixed-methods approach to assessing implementers' readiness to adopt digital health interventions (RADHI).
Implement Sci Commun, 5(1):91, 27 Aug 2024
Cited by: 0 articles | PMID: 39192345 | PMCID: PMC11348516
Data
Data behind the article
This data has been text mined from the article, or deposited into data resources.
Clinical Trials
- (1 citation) ClinicalTrials.gov - NCT03896776
Similar Articles
To arrive at the top five similar articles we use a word-weighted algorithm to compare words from the Title and Abstract of each citation.
Internet-Based HIV Prevention With At-Home Sexually Transmitted Infection Testing for Young Men Having Sex With Men: Study Protocol of a Randomized Controlled Trial of Keep It Up! 2.0.
JMIR Res Protoc, 6(1):e1, 07 Jan 2017
Cited by: 25 articles | PMID: 28062389 | PMCID: PMC5251167
Folic acid supplementation and malaria susceptibility and severity among people taking antifolate antimalarial drugs in endemic areas.
Cochrane Database Syst Rev, 2(2022), 01 Feb 2022
Cited by: 11 articles | PMID: 36321557 | PMCID: PMC8805585
Review Free full text in Europe PMC
Barriers and Facilitators to Implementing Keep It Up!, A Digital Health Intervention, in Community-Based Organizations.
AIDS Behav, 10 Oct 2024
Cited by: 0 articles | PMID: 39387999
Behavioral interventions to reduce risk for sexual transmission of HIV among men who have sex with men.
Cochrane Database Syst Rev, (3):CD001230, 16 Jul 2008
Cited by: 99 articles | PMID: 18646068
Review
Funding
Funders who supported this work.
NIDA NIH HHS (1)
Grant ID: P30 DA040500
NIMH NIH HHS (1)
Grant ID: R01 MH118213