2.1. Sample frame – County selection and randomization

CBO and DTC delivery are widely used health programming implementation strategies [22, 23] however their relative advantages and disadvantages are unknown. Our sample size target was 66 counties to be randomized 2:1 to the CBO and DTC strategies, respectively, to account for the assumption that there would be counties where no CBOs applied to the request for applications (i.e., there were 44 counties from which 22 CBOs were recruited). We selected US counties with an estimated number of YMSM ≥1,500 [24], which yielded an initial sample frame of 113 counties. An iterative process was developed to narrow the sample frame to 66 counties, which included eliminating counties in which KIU! was previously studied and eliminating nearly all bordering counties to reduce clustering that could produce unintentional exposure across county lines. For more information, see supplementary material entitled County Selection Iterative Process.

2.2. Community-based organization eligibility and recruitment

To select CBO recruitment sites, we published a request for proposals modeled on those created by the CDC and other HIV prevention funders. An objective scoring rubric was developed, and reviewers were trained to score applications on nine criteria. Each application was scored by three review panelists. Once panelists submitted scores, we averaged each application’s scores and selected the highest scoring applicants for funding. For more details on the request for proposals process, see supplementary material entitled Request for Proposals Process.

2.3. Direct to consumer recruitment catchment area

It became evident that certain constraints of our research study design, which was intended to facilitate statistical comparisons of KIU! effectiveness and implementation across arms at the county level, were at odds with running a pragmatic implementation trial. Specifically, nearly a quarter of people interested in joining the DTC arm screened ineligible due to county of residence. As such, post-launch, we expanded the DTC catchment area from 22 counties to nationwide to improve study enrollment and make DTC delivery more pragmatic – in other words, aligned with how other self-guided digital health interventions or applications are often delivered outside research settings (e.g., available to all through the App Store or accessible online).

2.4. Participant inclusion criteria and recruitment

The study has two components: a service component (intervention content) and a research component (surveys and STI testing). All participants were allowed to take part in the service component of the study but had to meet eligibility criteria to complete the research component. YMSM were eligible for participation in the research component if they met the following criteria: 1) tested HIV-negative at the time of registration, 2) reported they were assigned male at birth, 3) identified their current gender identity as male or non-binary, 4) were between the ages of 18–29 at registration, 5) reported CAS with a male partner in the previous six months, and 6) were not on pre-exposure prophylaxis (PrEP), had been on PrEP for less than six months, or reported missing a PrEP dose in the previous six months. Post-launch, participant inclusion criteria were expanded and it was no longer necessary for participants to 1) have received a negative HIV test at registration in order to create additional enrollment channels (e.g., at community events where testing was not occurring on site), or 2) report CAS with a male partner in the previous six months because COVID disrupted patterns of sexual behavior [25]. In addition, the upper age limit of eligibility was increased from 29 to 34 years given older men’s interest in KIU.

Participants were recruited using different strategies based on the arm of the study (CBO versus DTC strategy). For the CBO arm, recruitment was primarily integrated into routine HIV testing and prevention services. Using this approach, after a negative test result, CBO staff explained KIU! And registered interested YMSM, who automatically received a confirmation email that allowed them to set a password and access the KIU! Welcome page. Participants were prompted to complete a baseline survey embedded within the welcome page, where their eligibility for the research component of KIU! Was verified. All participants, regardless of eligibility, could access intervention content upon completing the baseline survey as KIU! Was being delivered as a service while also being studied.

In the DTC arm, participants were recruited primarily through social media, dating apps, and supplemental approaches (e.g., referrals and print ads). Participants registered for KIU! Using a brief online form. Participants who indicated they lived in a county assigned to the CBO arm were marked as ineligible and not provided access to the intervention. All other participants received a confirmation email to set a password and access the KIU! Welcome page where they completed a baseline survey to assess eligibility for the research component. If eligibility requirements were met, participants were sent an at-home HIV test kit and materials to self-administer testing for urethral and rectal gonorrhea and chlamydia. Individuals ineligible for the research study were still offered access to the online intervention content. For additional details on the DTC-mediated recruitment, screening, and testing strategies, see the supplementary material entitled Direct-to-Consumer Recruitment.

2.5. Intervention delivery

For this trial, KIU! was redesigned using multi-tenant architecture into a customizeable mobile, tablet, and desktop responsive web-based participant-facing application and online administrative dashboard. KIU! consisted of four user roles with separate login pages and varying levels of access to the KIU! application: Superusers, Organization Implementers (i.e., Project Directors and Project Coordinators), Organization Recruiters, and YMSM participants. Superusers were comprised exclusively of Northwestern University project staff who oversaw the application’s functionality, content, and data. CBO Project Directors had the ability to register participants, add coordinator and recruiter user roles, and had access to participant data relevant for intervention retention for their CBO (such as participant contact information and contact logs, intervention progress and application usage, etc.); coordinators had similar access as project directors but were limited to only adding and removing recruiter admin roles to KIU!; recruiters were only able to register participants and did not have access to any other application data. DTC project staff had the same level of access as CBO Project Directors. Post-launch, a to-do list feature was created within the adminstrative dashboard for Project Directors and Coordinators to streamline and improve participant retention. The to-do list indicated which participants needed follow-up at different stages of intervention progress.

After registration into the application (either automatically from the DTC online screener or manually by CBO staff), automatic confirmation emails were sent to YMSM participants directing them to set a password and log into the KIU! application to complete a baseline survey hosted in REDCap [26]. After baseline survey completion, YMSM participants were directed to the KIU! homepage consisting of the KIU! content, a technical support page, resources pages, a page of custom-made tools, and page to review their favorited content. YMSM participants were required to complete KIU! content in sequence, although they were able to return to previously viewed material using back buttons, a learning map, the tools page and the favorites pages.

As previously published, we iteratively developed the KIU! intervention with input from YMSM and CBOs [27]. Initially, KIU! was comprised of seven modules completed across three sessions with mandatory 8-h breaks between sessions. Additional content and follow-up surveys were provided in two booster sessions 3 and 6 months after the main intervention. Participants also completed a final follow-up survey and STI testing 12-months post-intervention. Table 1 provides an overview of subject matter covered within each module of the intervention as well as the structured breaks between modules. Post-launch, the 8-h timed breaks between the three main intervention sessions were made optional in response to CBO and YMSM feedback that moving through the sessions without breaks is more consistent with the current media landscape. The time between the main intervention content and the booster sessions was reduced from 3 months (Booster 1) and 6 months (Booster 2) to 6 and 12 weeks, respectively, to allow more time for enrollment due to COVID-related slow-downs while staying within the project budget. Follow-up surveys were removed from Booster 1 and only collected during Booster 2 (12-weeks post-intervention). STI testing data was also only collected at 12-weeks post-intervention. For more details on intervention delivery, including data collected by the application and the experience of YMSM, see the supplementary material entitled KIU! Intervention Delivery.

2.6. Implementation strategies - Community-based organization – Training

Northwestern project staff developed and disseminated online training to CBO implementers. Training modules were developed using the KIU! platform, to offer seamless integration of KIU! content, and covered topics ranging from how to use the KIU! administrative dashboard to recruitment strategies. Ongoing technical assistance and training was provided to CBOs to meet emergent needs of the trial. See the supplementary material entitled Training for more details.

2.7. Implementation strategies – Incentives

Offering financial incentives to encourage participation in HIV prevention and care service programs is a common and pragmatic practice [28] replicated in this intervention. Initially, participants were entered into raffles for $200 gift cards when they completed research surveys at baseline, 3-, 6-, and 12-months post-intervention. In the CBO arm, each organization offered incentives at their own discretion for completion of the baseline survey and service components of the program. Across all CBOs, incentives for baseline survey completion ranged from $5 to $30, with many CBOs choosing not to incentivize baseline completion. For the first three sessions, CBOs offered participants between $10 and $50 total. For completing the two booster content sessions post-intervention, CBOs offered participants between $15 and $75 total. CBOs mostly offered gift card incentives, however, several included non-monetary incentives such as free STI testing and bottles of lube.

After a study protocol modification in July 2020, DTC participants received a $10 or $25 virtual Visa gift card for completing the first three intervention sessions. Incentive amount was randomized by county to determine if a threshold amount must be met for incentives to be effective. DTC participants also received a discount code for an online store that sells adult toys and products. For completing the two boosters, participants were entered into an e-raffle for prizes donated by various vendors. These strategies were based on those successfully used by CBO partners in previous KIU! service implementations to keep YMSMs engaged [14].

The incentive structure was updated twice more due to retention difficulties and youth advisory board feedback that guaranteed payment comparable to other HIV prevention services they participated in would increase retention. Incentives were increased in Spring 2021 with DTC participants receiving a $50 Amazon gift code for completing the first three sessions of the intervention, $10 Amazon gift code for completing 6-week booster content, and $15 Amazon gift code for completing 12-week booster content. Participants in both arms of the study also received a $10 Amazon gift code or virtual Visa gift card for completing 12-week STI testing. The second update in August 2022 increased 12-week incentives for participants in both arms to receive $20 for completing the survey and $20 for completing STI testing.

2.8. Measures

The RE-AIM framework [29, 30] broadly guided our outcome measurement. The main comparison between the CBO and DTC implementation approaches used quantitative data on Reach and Effectiveness. Reach to YMSM was defined as the absolute number of participants enrolled (i.e., completed baseline and screened eligible), as this metric was both comparable across arms and provided the necessary demographic data to examine differences between arms. Measures for calculating effectiveness at reducing HIV risk among engaged YMSM [29, 31, 32] are depicted in Table 2. Mixed-methods assessments for examining the effect of strategies and determinants on implementation variability [33] as well as for measuring sustainability are depicted in Table 3, and constructs related to delivery cost [34, 35] are depicted in Table 4.

2.9. Data analysis – Public health impact

Our primary outcomes were public health impact, defined as reach X effectiveness [29, 31, 32], and cost per infection averted [36, 37]. For this trial, public health impact will be assessed by 1) number of YMSM community reached weighted by their HIV risk, and 2) effectiveness at reducing HIV risk [29, 31, 32]. By measuring individual-level change on these modifiable factors, our index of public health impact allowed for heterogeneity in response to KIU! across individuals, race and age groups, counties, and implementation conditions. Effectiveness was measured by the estimated change in that person’s risk for HIV from baseline to follow-up surveys and as determined by observed changes in target risk factors (such as CAS, STIs, and adherent PrEP use, all of which have major impacts on HIV transmission in YMSM) [38–40]. We based our modeling of HIV risk on published single-exposure probabilities and account for multiple exposures using binomial modeling [41]. Because these are individual-based measures, we will use two-level mixed-effect modeling [42].

We followed established guidelines for collecting cost data and conducting economic evaluations. Cost analyses were conducted for each arm from the perspective of the healthcare sector [34, 35]. We followed a micro-costing approach, a technique in which all inputs consumed in an intervention are identified and quantified in detail and then converted into fiscal terms to produce a cost estimate. Cost analysis was used to compute cost per infection averted, which is analogous to the measure used by CDC to decide which effective HIV prevention intervention would be supported as part of a high impact prevention plan [36, 37].

Research reported in this publication was supported by the National Institute of Mental Health (NIMH), National Institute on Drug Abuse (NIDA), and Office of the Director (OD) of the National Institutes of Health under Award Number R01 MH118213. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. We would like to thank the staff at the 22 CBOs at which KIU! is being implemented as well as the many community partners who are assisting in recruitment for the DTC arm. Additionally, we thank the following centers and institutions for their immeasurable contributions to our trial: Center for Prevention Implementation Methodology for Drug Abuse and HIV; Prevention Science and Methodology Group; Third Coast Center for AIDS Research; Research Application Design and Development, Institute for Sexual and Gender Minority Health and Wellbeing; the Institute for Sexual and Gender Minority Health and Wellbeing, Feinberg School of Medicine, and the Department of Medical Social Sciences at Northwestern University; Center for Health Economics of Treatment Interventions for Substance Use Disorders, HCV, and HIV (CHERISH); Department of Population Health Sciences at Weill Cornell Medicine; Boston Medical Center; and Erik Munson of Clinical Laboratory Science at Marquette University.