2.2 Introduction: Background and Objectives (Scientific background and explanation of rationale, 2a; Specific objectives or hypotheses, 2b)

Researchers need to state clearly the objectives of their research. This refers not only to the hypotheses invoked, but also to the ultimate objectives. Is the eHealth intervention intended to be a stand-alone intervention or is it to be incorporated into a broader health care program? Is the intervention intended for a particular patient population? Is the intervention ultimately intended to be cost-effective relative to other interventions – if so, relative to what interventions in particular? Answers to these questions will enable the reader to determine how similar the current evaluation context is to the ultimate, intended use of the intervention, and the extent to which outcome measures assess relevant domains.

Researchers should not only provide the information discussed above, but they should also provide a clear rationale for the choices that they made in designing the eHealth intervention for the specific health problems, populations, or outcomes that they intended to address: i.e., why did they select the type of goals, delivery platform, and presentation styles that they did? That is, the researcher should ideally provide the reader with a rationale that explains why the intervention form and content were chosen to achieve the goals identified by the researcher.

The rationale should often include why a particular population is targeted for the research and intervention, and the relevance of this to the design of the intervention and how relevant participant/person factors might drive treatment use and outcomes in eHealth research vs. traditional delivery systems. For instance, while outcomes in traditional interventions might be related to the tendency or ability to form a meaningful or warm interpersonal relationship with the therapist, success in an eHealth intervention might reflect the ability to use computers well, literacy, availability of naturally occurring social support, access to efficient applications (e.g., with high-speed Internet access), outside time obligations that might compete with heavy use of the Internet, lack of prior exposure to Internet interventions (there is a greater novelty effect), and so on. Thus, the rationale should indicate the individuals for whom the intervention is intended, why it is especially appropriate for those individuals, and how this conceptualization or model influenced the choice of measured participant characteristics. As noted above, these characteristics might not be those typically measured in research, and their assessment should allow the researcher to mount ancillary research on subpopulations and suggest the population to which the research findings might be extrapolated (see point 18, Table 1). The important point is that the rationale should focus on population issues as well as other design considerations.

If the researchers have proposed relevant theory, they should describe the use parameters that they believe should lead to optimal outcomes. For instance, what is the optimal timing of use? If the intervention is designed to encourage smoking cessation, is it important for the user to begin using the program before the quit day? And, if the user is successful in quitting, how long should the user continue to use the program after s/he has quit? What instructions or recommendations were given to the user regarding timing, frequency, or heaviness of use? In the case of an eHealth intervention for breast cancer, when should the woman gain access to the intervention: as soon as possible after diagnosis or after initial treatments are finished? In addition, what does the researcher's theory say about use-benefit relations: e.g., is there a minimum dose (overall duration of use) that is needed to obtain benefit? Is there some pattern of use or combination of services that is targeted? Consideration of these characteristics of eHealth interventions can help the researcher sharpen his or her thinking about a tested intervention as well as improve communication of trial results. Thus, while most trials list major hypotheses or objectives (e.g., 18), there may be benefit to elaborating such discussion.

2.3. Methods: Trial Design (Description of trial design (such as parallel, factorial) including allocation ratio, 3a)

While this CONSORT reporting element refers in particular to the experimental design used (e.g., fractional factorial design), it seems especially important and difficult in eHealth research to select a control condition that reflects the objectives of a study (point 2b above) and then to describe it adequately (see point 5 below). Choice of a control or comparison condition demands that the researcher identify the features or facets of an intervention that s/he wishes to test (see also (4). For instance, unguided access to the Internet has been used as a control condition (19). Presumably, this controls for access to the Internet per se, and the availability of computer equipment. This also may control for such nonspecific effects such as positive expectations. However, it is important to note in such cases that the control condition is messy; the control subject may be exposed to many putative “active” elements of treatment. For instance, the eHealth intervention to be evaluated may contain person-to-person support services and up-to-date information. These services may also be readily available on the Internet for the specific disease or condition that is targeted. In this case, the control condition is actually “controlling for” many active elements of treatment. This would be acceptable if the researcher clearly believed that the evaluated treatment were sufficiently different in some way so as to make it highly distinct from other Internet resources. For instance, the investigator may surmise that while the basic services comprised by an intervention are not unique, the intervention's interface and access to services will make the intervention superior (see 20). Otherwise, the researcher is courting Type II error and possibly promoting false inferences based upon his/her research; i.e., that is, the researcher would mistakenly infer that components of his/her eHealth intervention (person-to-person support via discussion groups) do not produce significant health benefits. Unfortunately, using statistical control methods to partial variance related to fortuitous resource/intervention use by control subjects will result in confounded inferences (21).

It is also important to note that the impact of a control condition (to say nothing of an active intervention) may be influenced by population characteristics such as baseline access to resources. For instance, if computer provision and Internet access are used as a control condition, this might have very different effects on low-SES as compared to middle-class populations. The latter might have pre-existing Internet access so that the control treatment affords no material benefit. In the former case, however, the control condition might constitute an important new resource. Researchers may, therefore, attempt to provide information on the relevant resources that participants have prior to study participation.

Other sorts of control or comparison conditions are possible. One might compare eHealth interventions with traditional information resources such as books or pamphlets, or with treatment-as-usual, or with treatment that meets some standard-of-care. These choices should be driven by the inferences that the investigator wishes to make. This latter point is crucial. For example, if the investigator wishes to address the question of whether Internet delivery of information is superior to a book as a delivery system, s/he would try to arrange an experimental test so that control group members receive the same information as experimental participants, but a book(s) is the sole delivery system. However, eHealth investigators often confound a design intentionally. The investigator might decide that it makes no sense to use a computer platform to deliver the same material that would be suitable for a book. Features of computer delivery might afford much more diversity in content, tailoring of content, and so on (e.g., iPhone delivery with visual presentation and audio backup). In this case, the investigator is interested in the pragmatically important question of how well the “best book” does vis-à-vis the “best” computer intervention. This may be an important question, but the book vs. computer comparison per se does not allow the investigator to make strong inferences about whether the special content of the computer system was effective, or whether any differences in outcomes could be attributed to the different delivery systems alone. This design by itself affords little insight into critical intervention features. The point is, as is the case with all research, that the investigator must exercise care in making inferences from results, ensuring that the inferences drawn are permitted by the design.

A researcher may wish to address a very pragmatic question: e.g., does encouragement to use a particular eHealth intervention produce results superior to those produced by patients' fortuitous and normal access to information and resources (access that occurs outside of experimental influence, perhaps via the person's own initiative or via the influence of treatment personnel or family members). Note that in this case, strong inferences cannot be made about effect sizes of types of eHealth intervention components, since exposure to such components may be spread across both the experimental and control conditions.

In considering comparison conditions, it should also be noted that there may be differences in the uniqueness of an eHealth-based intervention as compared to a traditional clinical intervention. For instance, if an RCT involves pharmacotherapy, it is likely that this is a very distinct experience for the participant and one that does not occur outside the experimental context. However, with eHealth interventions, the participant may be exposed to many similar interventions – as Internet resources become more common, and more commonly used, a person may readily access a “therapy” experience that is very similar to the one tested in the RCT. Thus, it would be desirable if the researcher could provide some data on the extent to which participants used similar sorts of eHealth interventions beyond those “prescribed” in the RCT.

Additionally, the researcher might wish to use research findings to make a policy case: e.g., an eHealth intervention should be used in lieu of a traditional or usual care intervention (see discussion of point 2b above regarding objectives). In this case, the traditional therapy would need to be used as a comparison condition. Obviously these different control conditions may exert profound effects on the apparent efficacy of the experimental intervention. Thus, while an eHealth intervention may appear efficacious relative to unstructured Internet use, it may perform less well than a normal face-to-face intervention (22). Global statements about efficacy will be possible only once an intervention has been compared with multiple control conditions.

Researchers may also wish to make inferences about particular aspects of an eHealth intervention. For instance, they may wish to determine if a particular platform characteristic (e.g., high-speed Internet access) or a presentation feature (e.g., design of web pages, use of games) is efficacious. In such cases, the researcher might provide users in control/comparison conditions with the eHealth intervention that is devoid only of the to-be-tested feature or characteristic. Similarly, some eHealth interventions are supplemented by intervention components that are delivered via non-web media (e.g., written handouts, audiotapes: (23). If the researcher wishes to make inferences about the eHealth intervention per se, such non-web components must be provided to control or comparison participants.

The key is that the researcher must clearly identify the particular intervention feature(s) that s/he is interested in testing, and then explain why the control condition is appropriate for supporting strong inferences regarding the targeted feature.

A final word about control conditions: regardless of whether a “treatment-as-usual” control condition is used, the researcher should disclose fully the sorts of interventions that are provided participants in addition to the targeted eHealth intervention. eHealth interventions may not be intended to serve as stand-alone treatments; therefore, it is common for users to receive considerable additional intervention. Thus, if an eHealth intervention for asthma is being tested, the researcher should characterize the sorts of asthma treatments that the participants are receiving in addition to the experimental eHealth interventions. It is conceivable that eHealth interventions will exert very different effects as a function of the intensity or type of adjuvant treatments (e.g., interact with such treatments).

2.4a. Participants (Eligibility criteria for participants, 4a)

The researcher should consider and disclose all exclusion and inclusion criteria used and justify any criteria that are not obvious. It is important to note that de facto exclusion may occur in eHealth research: if recruitment occurs over the Internet, it perforce means that all participants will have access to the Internet. Such de facto selection criteria should be clarified for the reader.

In any area of applied research, a conflict often exists between internal validity and experimental control (lack of bias and accrual of high-quality data) on the one hand and external validity (generalizability to the target population) on the other hand. Many eHealth interventions are designed for use by large segments of the population, which would naturally foster recruitment of “all comers” via Internet recruitment. However, characteristics of a population recruited in this manner might reduce the sensitivity of the Internet evaluation due to heterogeneity of the sample population. Moreover, it might be difficult to get some recruitment information that would allow the researcher to complete the recruitment diagram in Figure 1; e.g., an eHealth study may involve self-determination of eligibility or involve screening and consent only for assessments, and in either case, it might be difficult to determine the proper denominator for enrollment figures (also see (4).

Open in a separate window

Figure 1

Recruitment Consort Flowchart

To the extent that trial participation is taken to reflect interest in a treatment, or treatment acceptability, it is vital that the researcher disclose all of the steps taken to effect recruitment. There is also some evidence that type of recruitment channel can affect the use or success of the intervention (24, 25). The researcher should disclose steps such as Web-based advertisement, personal or telephone contacts by clinic personnel, and so on. The researcher should present a figure that depicts flow of participants through all stages of a randomized trial (see example in Figure 1; (17). The researcher needs to justify the particular population recruited and the recruitment strategy used, and provide a clear rationale for the approach adopted.

2.4b. Participants (Settings and locations where the data were collected, 4b)

In addition to obvious information (e.g., collaborating clinics), it may be helpful to report how institutional affiliations are displayed to potential participants on eHealth media. Affiliations with prestigious hospitals or universities might affect volunteer rates, use, and reactions with regards to an intervention. Any feature or activity of affiliated institutions that might have affected the eHealth intervention use or effects should be reported.

2.5. Interventions (The interventions for each group with sufficient details to allow replication, including how and when they were actually administered, 5)

2.6. Outcomes (Completely defined pre-specified primary and secondary outcome measures, including how and when they were assessed, 6a)

While medical interventions may have single pre-eminent outcome (e.g., reduced hypertension, lowered lipid levels) that are of pre-eminent importance, many eHealth interventions do not. For instance, an eHealth intervention for breast cancer patients (35) might have as its goals: improvement in quality of life, reduction of negative affect or depression, greater knowledge (and confidence in knowledge) regarding the disease and treatment options, improved decision-making on the part of the patient, reduced health-care expenditures and utilization, fewer somatic symptoms, greater satisfaction with health care, more effective utilization of health care resources (e.g., more efficient visits), and so on. This multiplicity of outcomes is probably the norm rather than the exception with most eHealth interventions. Therefore, it is incumbent that the researcher take steps to reduce the threats posed by multiple outcomes and that these steps be accurately reported. The steps may include: 1). Report clearly the outcomes that were identified, a priori, as primary and secondary. This could serve as a basis for deciding which tests should be conducted with alpha protection and would also alert the reader to the tests that were most strongly supported by theory or that have greatest clinical importance (e.g., (36). 2). Analyze latent variables that capture reliable covariance in the outcomes and reduce the number of tests conducted. 3). Disclose tests conducted, but whose results are not necessarily reported, so that the reader has a sense of the population of outcomes that was analyzed. Of course the investigator should do his or her best to prioritize outcomes to identify the smallest number of highly meaningful ones.

In addition to investigating theory-based outcomes, the researcher may wish to examine more peripheral dimensions of the intervention. Investigators may want to ask questions about difficulties in gaining Internet access, ease of use, speed of information access, and so forth. It is possible that the different effects of contrasted eHealth interventions have more to do with such peripheral features than with hypothesized content differences. Finally, investigators may wish to gather subjective ratings of factors such as general satisfaction with the eHealth application, interface, frustration with program bugs, server downtime, or impatience with the system. Such might be especially useful when standardized content is used but presented via different delivery systems.

In closing, while some methodologists recommend having only a single primary outcome (18), there may be circumstances where the eHealth researcher cannot legitimately prioritize outcomes that concisely. In these cases, it is especially incumbent upon him/her to defend the outcomes selected and discuss the relation of these outcomes with the initial experimental hypotheses.

3.2. Recruitment (Dates defining the periods of recruitment and follow-up;14a)

Investigators should provide dates for critical events in an RCT trial (17). These would include dates for the trial launch, the pattern of recruitment over time (e.g., a breakdown of number of participants recruited in each of the study duration quartiles), the completion of trial recruitment, the length of the intervention, and how long the time period is for following up with study participants. As noted earlier, eHealth RCTs are often conducted in a world of flux (e.g., rapid technological change). Therefore, in addition to dates, the investigators could indicate where critical secular events fell during the study. These critical events might occur in at least three domains: 1). Changes in the treatment or assessment of the health condition that is the target of the intervention (e.g., FDA approval of a new medication, new clinic procedures), 2). Significant changes in Internet resources related to the target condition (introduction of a new widely available website), and 3). Changes in computer hardware or Internet delivery resources such as speed (e.g., slower speed dial-up vs. high-speed DSL or cable) or type of access (e.g., wired vs. wireless). The investigator could evaluate whether such events were related to outcomes. Such information will help the reader evaluate the context in which the eHealth intervention was tested.

3.3. Numbers analyzed (For each group, number of participants [denominator] included in each analysis and whether the analysis was by originally assigned groups, 16)

The problem of what to use as a denominator in outcome analyses is a crucial one in eHealth trials. For instance, while thousands of individuals may be exposed to an eHealth intervention, this may provide a very imprecise estimate of those for whom the intervention is relevant (e.g., targeted disease sufferers vs. the curious), those who actually use the intervention, and those who can generate meaningful outcome data. Because users or participants have such control over exposure to the intervention, investigators may be tempted to analyze data only from those individuals who were actually exposed to (i.e., used) the intervention. It is important that investigators realize that if they engage in such a practice, they can no longer describe their trial as adhering to the intent-to-treat principle (18). Perhaps the best strategy is to follow the general CONSORT principle of reporting N's across a range of study participation thresholds. That is, to the extent possible, investigators should report the number of individuals exposed to the intervention invitation, the number who formally consented, and the number who used the intervention beyond a limited number of strategically selected use levels (determined a priori), but the intent-to-treat principle should be used in primary analyses.

Non-response bias is an especially critical issue in eHealth research since it may occur for many reasons unrelated to outcome status. Non-response during follow-up in many clinical trials may be attributed to a poor outcome; e.g., in smoking cessation research (38, 39). However, in eHealth research, non-response may be due to such factors as spam filters, email address changes, or because there is little personal contact with intervention staff (40); participants may not develop loyalty to the intervention or research program. Finally, participants of eHealth intervention studies may misinterpret a follow-up query as intervention-related assessment, and not respond because they feel that they are no longer in need of treatment. Thus, meaningful levels of nonresponse should encourage investigators to report outcomes in multiple ways (e.g., survival methods with proper censoring, data imputation procedures, (41) as appropriate (18).

4.2. Conclusion

The recommendations made in this paper encourage evaluators to consider carefully their evaluation design and to be explicit in describing what they did and why. One key issue faced is how to package this very detailed description in a way that still allows the whole publication to fit within the increasingly more stringent 2,000-3,000 word limitations of many journals. Many aspects of our recommendations are nonobligatory and will not be relevant to some eHealth studies. This will reduce the reporting burden. In addition, authors may make some of the information available via addenda that can accompany the manuscript when it is submitted for review. This information could also be made available to readers via a website sponsored either by the authors or by the journal. Even if it is not possible to include all of this information in the article or addenda, it is worthwhile for scientists to consider these issues as they apply to eHealth research.

Timothy B. Baker has served as an investigator on research projects sponsored by pharmaceutical companies including Pfizer, Glaxo Wellcome, Sanofi, and Nabi. No other author had potential conflicts to disclose. The study was funded by P50 CA095817-05 from the National Cancer Institute.