Your browser lacks required capabilities. Please upgrade it or switch to another to continue.
Loading…
<h2>Design Phase</h2>
<h3>Research question and hypothesis</h3>
1) A hypothesis is developed to allow experiments that elucidate specific aspects of the research question
2) A hypothesis is a predictive statement stating how an experimental condition(s) is expected to affect a measured outcome(s)
3) A good hypothesis should be testable, concise, and name key constructs
4) When constructing experiment hypotheses for HRI studies, it is useful to consider the various design factors can be altered to affect HRI
5) To improve rigour, the experimenter may consider registering the hypotheses prior to conducing the study
/*For detailed information and guidelines see [[Section 3.1|https://tianleimin.github.io/HRI-Methodology-Guidelines/main.html#x1-70003.1]] of the paper.*/
<hr>
What are your research questions?
<<textarea "$rq" $rq>>
What are your hypotheses?
<<textarea "$hypo" $hypo>>
Tick this box if the venue you are aiming for requires pre-registration of hypotheses. <<checkbox "$preregisteration" "not required" "required" `$preregisteration ? 'checked' : ''`>>
<hr>
|This step is not applicable to my study|<<checkbox "$hypoflag" "" "not applicable" `$hypoflag ? 'checked' : ''`>>|
[[Next (study context)->Study context]]
[[Previous (Design phase)->Design phase]]<h2>Design Phase</h2>
<h3>Study context</h3>
1) Identify appropriate location, scenario and other context of HRI
2) Match between participants and users, study and application
/*For detailed information and guidelines see [[Section 3.2|https://tianleimin.github.io/HRI-Methodology-Guidelines/main.html#x1-80003.2]] of the paper.*/
<hr>
What type of HRI study are you planning to conduct?
|Level and behavior of autonomy?|<<listbox "$HRIauto" autoselect>><<option "Full manual control">><<option "Action support">><<option "Batch processing">><<option "Shared control">><<option "Decision support">><<option "Blended decision making">><<option "Rigid system">><<option "Automated decision making">><<option "Shared controlSupervisory control">><<option "Full automation">><</listbox>>|
|What is the nature of information exchange?|<<listbox "$HRIinfo" autoselect>><<option "lexical">><<option "audio">><<option "visual">><<option "physical">><<option "physiological">><<option "multimodal">><</listbox>>|
|What is the role of the human(s)?|<<listbox "$HRIhuman" autoselect>><<option "supervisor">><<option "operator">><<option "mechanic">><<option "bystander">><<option "teammate">><<option "tutor">><<option "student">><<option "other">><</listbox>>|
|Other human roles not specified above:|<<textbox "$HRIhumanother" $HRIrobotother>>|
|What is the role of the robot(s)?|<<listbox "$HRIrobot" autoselect>><<option "assistant">><<option "companion">><<option "supervisor">><<option "tutor">><<option "student">><<option "bystander">><<option "teammate">><<option "other">><</listbox>>|
|Other robot roles not specified above:|<<textbox "$HRIrobotother" $HRIrobotother>>|
|Tick this box if there will be adaptation, learning, or training of people and robot|<<checkbox "$HRIadapt" "no adaption" "has adaption" `$HRIadapt ? 'checked' : ''`>>|
|What are the task's features?|<<listbox "$HRIrisk" autoselect>><<option "high risk">><<option "low risk">><</listbox>><<listbox "$HRIcogload" autoselect>><<option "high cognitive requirements">><<option "low cognitive requirements">><</listbox>><<listbox "$HRIphyload" autoselect>><<option "high physical requirements">><<option "low physical requirements">><</listbox>>|
|Other context not specified above?|<<textbox "$context" $context>>|
<hr>
|This step is not applicable to my study|<<checkbox "$contextflag" "" "not applicable" `$contextflag ? 'checked' : ''`>>|
[[Next (study model)->Study model]]
[[Previous (hypothesis)->Hypothesis]]<h2>Design Phase</h2>
<h3>Study model</h3>
1) ''Within-group designs'': best used when researchers want to evaluate a measure across multiple time points, or test the effects of different conditions. It is best used with limited sample size. Researchers must note the possibility of sequence effects, fatigue and practice effects.
2) ''Between-group designs'': best used when researchers want to test the difference between two conditions, behaviours or tasks, and they do not want participants to be influenced by similar conditions. The design can strongly identify effects across-groups compared to within-group. Between-subjects design requires a larger sample size to produce statistically significant results.
3) ''Mixed-Model designs'': best used when researchers are seeking to assess both between and within-group effects. Mixed-model designs require a larger sample size and more research skill to implement correctly, but produce rigorous results.
4) ''Control conditions'': help to provide a comparison for the research condition of interest. In HRI studies, there are many options to benchmark the robot performance. A control may not be necessary in an initial study, but later studies that are looking to substantiate the claims in more detail should consider using one.
/*For detailed information and guidelines see [[Section 4.1|https://tianleimin.github.io/HRI-Methodology-Guidelines/main.html#x1-120004.1]] of the paper.*/
<hr>
Which study model are you planning to use?
<<checkbox "$modelwithin" " " " Within-group design " `$modelwithin ? 'checked' : ''`>> Within-group design
<<checkbox "$modelbetween" " " " Between-group design " `$modelbetween ? 'checked' : ''`>> Between-group design
<<checkbox "$modelmixed" " " " Mixed group design " `$modelmixed ? 'checked' : ''`>> Mixed group design
What are the control conditions?
<<textarea "$modelcon" $modelcon>>
What are the dependent variables?
<<textarea "$modeldep" $modeldep>>
What are the independent variables?
<<textarea "$modelind" $modelind>>
<hr>
|This step is not applicable to my study|<<checkbox "$modelflag" "" "not applicable" `$modelflag ? 'checked' : ''`>>|
[[Next (design of evaluation methods)->Design of evaluation methods]]
[[Previous (Study context)->Study context]]<h2>Design Phase</h2>
<h3>Design of evaluation methods</h3>
1) ''Task performance metrics'' should not be used alone to substantiate claims about the entire HRI experience.
2) ''Behavioural measurements'' quantify human-related constructs. When using these measurements, psychology effects should be taken into account and sufficient training should be given to the raters.
3) ''Psychophysiological measurements'' give detailed information about the participant’s internal state. They require significant expertise. Researchers should complete relevant training before using it in trial.
4) ''Interviews'' are useful when researchers are interested in developing a deeper understanding of participants' responses during the experiment. Best practice is to use a research team member not invested in the final outcome of the study to conduct the interview to avoid unconscious bias towards reporting of experiences that are more favourable towards the intended outcome.
5) ''Self-assessments data'' are often collected through the use of surveys and questionnaires. Similar to interviews, self-assessments can also be influenced by various human factors.
6) ''The design of a questionnaire'' involves selecting the what to measure and how to measure them. Psychometric validation is lengthy, complex and time-consuming. Researchers should carefully consider the need to create a new scale, and investigate other literature for suitability.
7) ''Metric reliability'' captures the quality of the data with respect to measurement errors. Test-retest reliability, internal consistency reliability, inter-rater reliability, and intra-rater reliability are the common measures that are applicable to HRI studies.
8) ''Validity'' is the extent to which the data-gathering tool measures the intended measurement. Face validity, content validity, and criterion validity are three important metrics to consider when exploring metric validity in the HRI setting.
/*For detailed information and guidelines see [[Section 4.2|https://tianleimin.github.io/HRI-Methodology-Guidelines/main.html#x1-130004.2]] of the paper.
For a summary of constructs and metrics see [[Table 2|https://tianleimin.github.io/HRI-Methodology-Guidelines/main.html#x1-16003r2]] of the paper.*/
<hr>
What evaluation methods are you planning to adopt?
|Task performance metrics:|<<textbox "$evatask" $evatask>>|<<click "(?)">><<toggleclass "#hint31" "hidden">><</click>><div id="hint31" class="hidden">e.g., task completion time, error rate, efficiency, percentage of successful actions</div>|
|Behavioural measurements:|<<textbox "$evabeh" $evabeh>>|<<click "(?)">><<toggleclass "#hint32" "hidden">><</click>><div id="hint32" class="hidden">e.g., audio/video recordings of the experiment which will then be transcribed for analysis</div>|
|Psychophysiological measurements:|<<textbox "$evapsy" $evapsy>>|<<click "(?)">><<toggleclass "#hint33" "hidden">><</click>><div id="hint33" class="hidden">e.g., heart rate, blood pressure, brain activity, skin conductance, muscle activity</div>|
|Interviews:|<<textbox "$evaint" $evaint>>|<<click "(?)">><<toggleclass "#hint34" "hidden">><</click>><div id="hint34" class="hidden">e.g., structured, open-ended</div>|
|Self-assessments:|<<textbox "$evasel" $evasel>>|
|Custom questionnaires included?|<<checkbox "$evaque" "no questionnaires or only standard questionnaires" "custom questionnaires will be used (check their validity)" `$evaque ? 'checked' : ''`>>|
|Other methods not specified above:|<<textbox "$evaoth" $evaoth>>|
<hr>
|This step is not applicable to my study|<<checkbox "$consflag" "" "not applicable" `$consflag ? 'checked' : ''`>>|
[[Next (selection of tools, equipment, and locations)->Selection of tools and Equipment]]
[[Previous (Study model)->Study model]]
<h2>Design Phase</h2>
<h3>Selection of tools, equipment, and locations</h3>
Selection of tools and Equipment
1) HRI studies can take place in the laboratory, in public, or online. Researchers should select the location that is appropriate to the hypothesis under investigation. They can also work incrementally, by first conducting studies in the laboratory then moving towards a more realistic setting.
2) When possible and feasible, it is best to use robot techniques and behaviours in trial that a robot can deliver on its own to avoid deception or assessment of effects that are not yet achievable. Instead, wizard of Oz is best used if the researcher must evaluate a technique that cannot be delivered on its own for the purpose of theoretical exploration, or the researcher is using the robot as an intentional avatar for experiments.
/*For detailed information and guidelines see [[Section 4.3|https://tianleimin.github.io/HRI-Methodology-Guidelines/main.html#x1-140004.3]] of the paper.*/
<hr>
Where do you plan to conduct the experiments?
|Lab testing:|<<checkbox "$loclab" "no" "yes" `$loclab ? 'checked' : ''`>>|
|Field testing:|<<checkbox "$locfie" "no" "yes" `$locfie ? 'checked' : ''`>>|
|Online testing:|<<checkbox "$loconl" "no" "yes" `$loconl ? 'checked' : ''`>>|
What is the frequency and duration of each session?
<<textbox "$locfre" $locfre>>
Which robot(s) are you using?
<<textbox "$locrob" $locrob>>
What is the HRI task?
<<textbox "$loctas" $loctas>>
What will the robot(s) be doing?
<<textbox "$loctasr" $loctasr>>
What will the human(s) be doing?
<<textbox "$loctash" $loctash>>
<hr>
|This step is not applicable to my study|<<checkbox "$locflag" "" "not applicable" `$locflag ? 'checked' : ''`>>|
[[Next (failure and contingencies)->Failure and contingencies]]
[[Previous (Design of evaluation methods)->Design of evaluation methods]]<h2>Design Phase</h2>
<h3>Failures and contingencies</h3>
Planning for failure is important due to the dynamic nature of experiments involving human participants. The best approach is to perform multiple test runs prior to the start of the experiment to minimise the risk of potential challenges and failures, by being able to identify them before participant recruitment. Researchers may choose to use a small sub sample of participants that are discarded from the final analysis (e.g., the first 3 test runs with live participants) or request other staff to complete the session without collecting their data formally in the experiment.
Common sources of errors and bias:
1) Inappropriate, inaccurate, or incorrectly configured measurement instruments
2) Inappropriate or unclear experimental procedures
3) Characteristics of the participants
4) Non-intended robot errors due to lack of technical robustness and functionality
/*For detailed information and guidelines see [[Section 6.2|https://tianleimin.github.io/HRI-Methodology-Guidelines/main.html#x1-420006.2]] of the paper.*/
<hr>
What errors or failures may occur in your study?
<<textarea "$errfai" $errfai>>
How do you plan to recover from or prevent the above failures?
<<textarea "$errrec" $errrec>>
<hr>
|This step is not applicable to my study|<<checkbox "$errflag" "" "not applicable" `$errflag ? 'checked' : ''`>>|
[[Next (participants)->Participants]]
[[Previous (Selection of tools and Equipment)->Selection of tools and Equipment]]<h2>Design Phase</h2>
<h3>Methods of participant / observer recruitment and training</h3>
1) Balancing potential confounding variables can help with noises introduced by the participant selection process, especially when non-random sampling techniques are used
2) Simple randomization such as a coin toss or random number generator can be used for a large number of participants, but should not be used for a small participant group
3) Blinding should be reported. Allocation concealment is an important step for demonstrating that the experiment is not influenced by the researcher's bias.
4) Each recruitment platform comes with different advantages and trade-offs. Researchers need to consider the platform in relation to the participants relevant to the HRI study. Suitable incentivization can be provided given that it does not affect the participant’s consent
5) Participant retention is important especially in longitudinal studies and participant drop-out should be reported
6) Participant preparation, such as training for teleoperators, should be noted in the study. Researchers should devise strategies to ensure that the participants are sufficiently trained
/*For detailed information and guidelines see [[Section 5|https://tianleimin.github.io/HRI-Methodology-Guidelines/main.html#x1-250005]] of the paper.*/
<hr>
What type(s) of participants will your study include? (see [[Section 5.1|https://tianleimin.github.io/HRI-Methodology-Guidelines/main.html#x1-260005.1]] for more information on participant type)? <<click "(?)">><<toggleclass "#hint21" "hidden">><</click>><div id="hint21" class="hidden">interactors, observers, WoZ operators, teleoperators, confederates</div>
<<textarea "$parwho" $parwho>>
How many participants do you plan to recruit? (see [[Section 6.3.3|https://tianleimin.github.io/HRI-Methodology-Guidelines/main.html#x1-460006.3.3]] if you plan to perform small sample size studies)
<<textarea "$parnum" $parnum>>
How will the participants be allocated to groups? (see [[Section 5.2|https://tianleimin.github.io/HRI-Methodology-Guidelines/main.html#x1-270005.2]] for more information on participant selection) <<click "(?)">><<toggleclass "#hint22" "hidden">><</click>><div id="hint22" class="hidden">single participant group or multiple groups? For multiple groups how is sampling, randomization, blinding and allocation concealment performed</div>
<<textarea "$parsel" $parsel>>
How will you recruit the participants? (see [[Section 5.3|https://tianleimin.github.io/HRI-Methodology-Guidelines/main.html#x1-310005.3]] and [[Table 3|https://tianleimin.github.io/HRI-Methodology-Guidelines/main.html#x1-31001r3]] for more information on participant recruitment)
|Contacts:|<<checkbox "$parcon" "no" "yes" `$parcon ? 'checked' : ''`>>|
|Community groups, professional organizations:|<<checkbox "$parcom" "no" "yes" `$parcom ? 'checked' : ''`>>|
|Mailing list:|<<checkbox "$parmai" "no" "yes" `$parmai ? 'checked' : ''`>>|
|Volunteer banks:|<<checkbox "$parvol" "no" "yes" `$parvol ? 'checked' : ''`>>|
|Crowd-sourcing platforms:|<<checkbox "$parcro" "no" "yes" `$parcro ? 'checked' : ''`>>|
What preparation or training will you provide the participants with? (see [[Section 5.5|https://tianleimin.github.io/HRI-Methodology-Guidelines/main.html#x1-330005.5]] for more information on participant recruitment) <<click "(?)">><<toggleclass "#hint23" "hidden">><</click>><div id="hint23" class="hidden">information, training, rater agreement</div>
<<textarea "$partra" $partra>>
<hr>
|This step is not applicable to my study|<<checkbox "$parflag" "" "not applicable" `$parflag ? 'checked' : ''`>>|
[[Next (study protocol and ethics)->Study protocol and ethics approval]]
[[Previous (Failure and contingencies)->Failure and contingencies]]<h2>Design Phase</h2>
<h3>Study protocol and ethics approval</h3>
1) Participant privacy should be maintained through data anonymization and data management
2) Informed consent may not be possible for WoZ studies. Appropriate procedures need to be put in place to debrief the participants after the study
3) Given HRI is an emerging field, there are many long term consequences of the experience that are unknown. Considerations need to be given to all possible participants, from interactors such as children vulnerable groups, to the researchers participating in the study
/*For detailed information and guidelines see [[Section 5.6|https://tianleimin.github.io/HRI-Methodology-Guidelines/main.html#x1-x1-340005.6]] and [[Section 5.7|https://tianleimin.github.io/HRI-Methodology-Guidelines/main.html#x1-380005.7]] of the paper.*/
<hr>
Ethics approval received (Check your national guidelines for ethics approval requirements): <<checkbox "$ethapp" "no" "yes" `$ethapp ? 'checked' : ''`>>
<hr>
|This step is not applicable to my study|<<checkbox "$ethflag" "" "not applicable" `$ethflag ? 'checked' : ''`>>|
[[Next (piloting)->Conduct pilots to refine study design]]
[[Previous (Participants)->Participants]]<h2>Design Phase / Experiment Phase</h2>
<h3>Conduct pilots to refine study design</h3>
In addition to the careful design and planning of a research study, researchers should aim to conduct a good pilot study prior to the main study. This will help them identify critical problems and deficiencies in the study protocol as well as plan and take all the corrective actions
/*For detailed information and guidelines see [[Section 6.1|https://tianleimin.github.io/HRI-Methodology-Guidelines/main.html#x1-410006.1]] of the paper.*/
<hr>
What is your plan for conducting pilot studies? <<click "(?)">><<toggleclass "#hint61" "hidden">><</click>><div id="hint61" class="hidden">Do you need ethics approval before running the pilot?
who will you recruit for pilot studies? what is the study protocol (be consistent with protocols of the main study)?</div>
<<textarea "$pil" $pil>>
<hr>
|This step is not applicable to my study|<<checkbox "$pilflag" "" "not applicable" `$pilflag ? 'checked' : ''`>>|
[[Previous (Study protocol and ethics approval)->Study protocol and ethics approval]]
Do you want to revise the study design based on outcomes of the pilot studies?
[[No (continue to the experiment phase)->Experiment phase]]
[[Yes (return to hypothesis)->Hypothesis]]
[[See a summary of your study design specified so far->Study Plan]]
[[See a check-list for conducting your study->Check list]]
<h2>Experiment Phase</h2>
<h3>Conduct study</h3>
Follow your designed methods to conduct the study
[[Review study models->Study model]]
<hr>
Experiments conducted: <<checkbox "$cond" "no" "yes" `$cond ? 'checked' : ''`>>
Notes on the experimental process (to be filled in during experiments):
<<textarea "$condnote" $condnote>>
<hr>
|This step is not applicable to my study|<<checkbox "$condflag" "" "not applicable" `$condflag ? 'checked' : ''`>>|
[[Next (collect measurements)->Collect measurements]]
[[Previous (Recruit Participants)->Recruit Participants]]<h2>Experiment Phase</h2>
<h3>Collect measurements</h3>
Follow your designed methods to collect data
[[Review evaluation methods->Design of evaluation methods]]
<hr>
Data collected: <<checkbox "$col" "no" "yes" `$col ? 'checked' : ''`>>
Notes on the data collection process (to be filled in during data collection):
<<textarea "$colnote" $colnote>>
<hr>
|This step is not applicable to my study|<<checkbox "$colflag" "" "not applicable" `$colflag ? 'checked' : ''`>>|
[[Next (analysis phase)->Analysis phase]]
[[Previous (Conduct study)->Conduct study]]<h2>Analysis Phase</h2>
<h3>Data annotation and processing</h3>
Data cleaning and post-processing is an important step prior to data analysis, since the validity and generalizability of the observations and conclusions made in a study strongly depend on the quality of the data being analyzed. Researchers should formulate a set of predefined rules for dealing with data errors, and missing or extreme values beforehand. It is also important to provide detailed documentation of all the data-cleaning methods employed, the types and sources of error identified in
the data as well as the excluded data and the reasons for exclusion
# Data cleaning: screening, diagnosis, editing
# Data visualization
# Small sample size studies: use the right statistical tests and visualization
# Measurement scales
/*For detailed information and guidelines see [[Section 6.3|https://tianleimin.github.io/HRI-Methodology-Guidelines/main.html#x1-430006.3]] of the paper.*/
<hr>
Small sample? <<checkbox "$annsml" "no" "yes" `$annsml ? 'checked' : ''`>>
Data cleaning performed: <<checkbox "$anncl" "no" "yes" `$anncl ? 'checked' : ''`>>
Data visualization performed: <<checkbox "$annviz" "no" "yes" `$annviz ? 'checked' : ''`>>
Notes on data annotation and processing (to be filled in during data annotation and processing):
<<textarea "$annnote" $annnote>>
<hr>
|This step is not applicable to my study|<<checkbox "$annflag" "" "not applicable" `$annflag ? 'checked' : ''`>>|
[[Next (evaluation)->Evaluate and analyse hypotheses]]
[[Previous (Analysis phase)->Analysis phase]]<h2>Analysis Phase</h2>
<h3>Evaluate and analyze hypotheses</h3>
Researchers should not blindly apply statistical hypotheses tests, such as ANOVA, seeking for statistical significance. For a well-designed experiment, researchers should pay more attention to statistical power analysis that includes the relationship between the sample size and effect size. It is important to not only report p-values, but also, where possible, report confidence intervals and effect sizes
# Hypothesis testing: standard parametric methods, post-hoc tests and corrections, non-parametric methods, robust statistical methods
# Confidence interval
# Bayesian inference methods
# Regression analysis
Decision flow chart for choosing the statistical method that best fits the characteristics of the data to be analyzed:
<img src="https://raw.githubusercontent.com/tianleimin/HRI-Methodology-Guidelines/master/StatisticsFlowChart.png" width="600">
/*For detailed information and guidelines see [[Section 6.4|https://tianleimin.github.io/HRI-Methodology-Guidelines/main.html#x1-470006.4]] of the paper.*/
<hr>
Which statistical tests are you performing?
<<textarea "$anasta" $anasta>>
Notes on data analysis (to be filled in during analysis):
<<textarea "$ananote" $ananote>>
<hr>
|This step is not applicable to my study|<<checkbox "$anaflag" "" "not applicable" `$anaflag ? 'checked' : ''`>>|
[[Previous (Data annotation and processing)->Data annotation and processing]]
Do the analyses align with your hypotheses?
[[Yes. Continue to post-study phase->Post study phase]]
[[No. Don't worry, let's review your hypotheses->Hypothesis]]
<h2>Post-study Phase</h2>
<h3>Reproducibility</h3>
It is important to include all the data necessary to reproduce your results. Consider releasing your data and code for the research community to bench-mark on.
<hr>
Data released: <<checkbox "$repdat" "no" "yes" `$repdat ? 'checked' : ''`>>
Code released: <<checkbox "$repcod" "no" "yes" `$repcod ? 'checked' : ''`>>
<hr>
|This step is not applicable to my study|<<checkbox "$repflag" "" "not applicable" `$repflag ? 'checked' : ''`>>|
[[Next (iterative design and follow-up studies)->Iterative design and follow-up studies]]
[[Previous (Post-study phase)->Post study phase]]<h2>Post-study Phase</h2>
<h3>Iterative design and follow-up studies</h3>
<hr>
<<button [[See a check-list for conducting your study|Check list]]>><</button>>
|This step is not applicable to my study|<<checkbox "$iteflag" "" "not applicable" `$iteflag ? 'checked' : ''`>>|
[[Previous (Reproducibility)->Reproducibility]]
[[The study life cycle restarts->Hypothesis]]
[[End adventure and generate summary->Thanks]]<h2>Welcome!</h2>
Welcome to the interactive version of the HRI study methodology guidelines!
This webpage will walk you through the life cycle of a typical HRI study, and list recommendations for best practice at different steps.
<h2>Tips</h2>
1) You can jump to specific steps through the map in the right side panel. You can also follow the instructions within each step.
2) The left and right arrows on top of the left side panel will take you to the previous steps you visited or jump forward to the step you are currently at.
3) Your progress will be automatically saved. You can also save manually. The save and load functions may glitch due to browser cookies. You can manually save or print the current progress too through the right side panel's "Summary of study design" or "Check-list for conducting the study" links.
5) The left and right side panels can be collapsed or expanded with the "<" and ">" pointers on there.
6) You may toggle between light and dark mode through the switch on the left side panel.
<hr>
Please tell us what is the title of your research proposal:
<<textbox "$StudyTitle" $StudyTitle>>
[[Let's begin the HRI study life cycle!->Design phase]]
<h2>Design Phase</h2>
You have some brilliant ideas and would like to conduct an HRI study to investigate it.
[[Begin the HRI study life cycle->Hypothesis]]
[[Return to the Welcome page->Introduction]]<h2>Experiment Phase</h2>
You have a well-established research design now.
Time to move on to the experiment phase and conduct your HRI study!
[[Next (recruit participants)->Recruit Participants]]
[[Previous (Conduct pilots to refine study design)->Conduct pilots to refine study design]]<h2>Experiment Phase</h2>
<h3>Recruit participants</h3>
Follow your designed methods for recruiting participants
[[Review recruitment methods->Participants]]
/*For detailed information and guidelines see [[Section 5.3|https://tianleimin.github.io/HRI-Methodology-Guidelines/main.html#x1-310005.3]] and [[Section 5.4|https://tianleimin.github.io/HRI-Methodology-Guidelines/main.html#x1-320005.4]] of the paper.*/
<hr>
Participants recruited: <<checkbox "$rec" "no" "yes" `$rec ? 'checked' : ''`>>
Notes on the recruitment process (to be filled in after the recruitment):
<<textarea "$recnote" $recnote>>
<hr>
|This step is not applicable to my study|<<checkbox "$recflag" "" "not applicable" `$recflag ? 'checked' : ''`>>|
[[Next (conduct study)->Conduct study]]
[[Previous (Experiment phase)->Experiment phase]]<h2>Analysis Phase</h2>
You have now collected all the experimental results.
Time to move on to the analysis phase!
[[Next (data annotation and processing)->Data annotation and processing]]
[[Previous (Collect measurements)->Collect measurements]]<h2>Post-study Phase</h2>
You have now finished the experimental analysis. Time to report on your amazing HRI study!
Not so fast, there are some extra post-study steps you can perform.
[[Next (reproducibility)->Reproducibility]]
[[Previous (Evaluate and analyse hypotheses)->Evaluate and analyse hypotheses]]<h2>Thank you for playing!</h2>
Below is a review of the HRI study you have specified so far, which you can copy or print to keep a record of your HRI study design at this point (Note: for a better view please hide the left and right side panels before printing)
Best of luck with your research!
<hr>
<hr>
Version date: <<set $CurDate = new Date(Date.now())>>
<<= $CurDate.toLocaleString("en-UK", { weekday: "short", month: "short", day: "numeric", year: "numeric", hour: "numeric", minute: "2-digit" } )>>
I. Design
1. Proposal title:
* $StudyTitle
2. Hypotheses: $hypoflag
* Research questions: $rq
* Hypotheses: $hypo
* Pre-registration of hypotheses: $preregisteration
3. Type of HRI study and study context: $contextflag
* Level of autonomy: $HRIauto
* Information channel: $HRIinfo
* Role of human(s): $HRIhuman ($HRIhumanother)
* Role of robot(s): $HRIrobot ($HRIrobotother)
* Apatation and learning: $HRIadapt
* Features of the task: $HRIrisk, $HRIcogload, $HRIphyload
* Other contexts: $context
4. Study model: $modelflag
* Study design structure: $modelwithin $modelbetween $modelmixed
* Control conditions: $modelcon
* Dependent variables: $modeldep
* Independent variables: $modelind
5. Design of evaluation methods: $consflag
* Task performance metrics: $evatask
* Behavioural measurements: $evabeh
* Psychophysiological measurements: $evapsy
* Interviews: $evaint
* Self-assessments data: $evasel
* Custom questionnaires: $evaque
* Other methods: $evaoth
6. Study tools, equipment, locations: $locflag
* Where do you plan to conduct the experiments?
** Lab testing: $loclab
** Field testing: $locfie
** Online testing: $loconl
* Frequency and duration of each session: $locfre
* Robot(s) to be used: $locrob
* HRI task: $loctas
* Robot(s) role: $loctasr
* Human(s) role: $loctash
7. Failures and contingencies: $errflag
Possible failures: $errfai
Plan of recovery and prevention: $errrec
8. Participants: $parflag
* Participants: $parwho
* Sample size: $parnum
* Selection process: $parsel
* Recruitment:
** Contacts: $parcon
** Community groups, professional organizations: $parcom
** Mailing list: $parmai
** Volunteer banks: $parvol
** Crowd-sourcing platforms: $parcro
* Participant training: $partra
9. Study protocol and ethics approval: $ethflag
* Ethics approval received: $ethapp
10. Piloting: $pilflag
* Pilot study plan: $pil
II. Experiment
11. Participant recruitment: $recflag
* Participants recruited: $rec
* Notes: $recnote
12. Conduct study: $condflag
* Experiments conducted: $cond
* Notes: $condnote
13. Collect data: $colflag
* Data collected: $col
* Notes: $colnote
III. Analysis
14. Data annotation and processing: $annflag
* Small sample size? $annsml
* Data cleaning performed: $anncl
* Data visualization performed: $annviz
* Notes: $annnote
15. Evaluate and analyze hypothesis: $anaflag
Statistical tests: $anasta
Notes: $ananote
IV. Post-Study
16. Reproducibility: $repflag
Data released: $repdat
Code released: repcod
17. Iterative design and follow-up studies: $iteflag
<hr>
<hr>
Generated by HRI study methodology guidelines (version 0.1):
https://tianleimin.github.io/HRI-Methodology-Guidelines/<img src="https://seekvectorlogo.com/wp-content/uploads/2018/01/monash-university-vector-logo.png" width="128">
/*[[Full HRI study guidelines paper|https://tianleimin.github.io/HRI-Methodology-Guidelines/main]]*/
<<ToggleTheme>>
Title of your research proposal:
<<textbox "$StudyTitle" $StudyTitle>>
<<button "Generate a summary of my adventure so far" "Thanks">><</button>>
[[Summary of study design->Study Plan]]
[[Check-list for conducting the study->Check list]]
<hr>
Click a step in the HRI study life cycle to jump there
(visited steps are shown in gray but you can always revisit them)
[[Return to the Welcome page->Introduction]]
<html>
<?xml version="1.0" encoding="UTF-8"?>
<svg class="map" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="500px" height="auto" viewBox="0 0 525 1400" version="1.1">
<defs>
<marker id="arrowhead" markerWidth="5" markerHeight="5" refX="0" refY="2.5" orient="auto">
<polygon points="0 0, 5 2.5, 0 5" fill="blue" />
</marker>
</defs>
<rect x="30" y="30" rx="20" ry="20" width="440" height="675" style="fill:rgb(255,180,180);stroke:none;opacity:0.75" />
<rect x="30" y="640" rx="20" ry="20" width="440" height="335" style="fill:rgb(180,255,180);stroke:none;opacity:0.75" />
<rect x="30" y="975" rx="20" ry="20" width="440" height="175" style="fill:rgb(180,180,255);stroke:none;opacity:0.75" />
<rect x="30" y="1150" rx="20" ry="20" width="440" height="200" style="fill:rgb(255,255,180);stroke:none;opacity:0.75" />
<line x1="250" y1="100" x2="250" y2="115" stroke="blue" stroke-width="2" marker-end="url(#arrowhead)" />
<line x1="250" y1="175" x2="250" y2="190" stroke="blue" stroke-width="2" marker-end="url(#arrowhead)" />
<line x1="250" y1="250" x2="250" y2="265" stroke="blue" stroke-width="2" marker-end="url(#arrowhead)" />
<line x1="250" y1="325" x2="250" y2="340" stroke="blue" stroke-width="2" marker-end="url(#arrowhead)" />
<line x1="250" y1="400" x2="250" y2="415" stroke="blue" stroke-width="2" marker-end="url(#arrowhead)" />
<line x1="250" y1="475" x2="250" y2="490" stroke="blue" stroke-width="2" marker-end="url(#arrowhead)" />
<line x1="250" y1="550" x2="250" y2="565" stroke="blue" stroke-width="2" marker-end="url(#arrowhead)" />
<line x1="250" y1="625" x2="250" y2="640" stroke="blue" stroke-width="2" marker-end="url(#arrowhead)" />
<line x1="250" y1="700" x2="250" y2="715" stroke="blue" stroke-width="2" marker-end="url(#arrowhead)" />
<line x1="250" y1="775" x2="250" y2="790" stroke="blue" stroke-width="2" marker-end="url(#arrowhead)" />
<line x1="250" y1="850" x2="250" y2="865" stroke="blue" stroke-width="2" marker-end="url(#arrowhead)" />
<line x1="250" y1="925" x2="250" y2="990" stroke="blue" stroke-width="2" marker-end="url(#arrowhead)" />
<line x1="250" y1="1050" x2="250" y2="1065" stroke="blue" stroke-width="2" marker-end="url(#arrowhead)" />
<line x1="250" y1="1125" x2="250" y2="1190" stroke="blue" stroke-width="2" marker-end="url(#arrowhead)" />
<line x1="250" y1="1250" x2="250" y2="1265" stroke="blue" stroke-width="2" marker-end="url(#arrowhead)" />
<line x1="250" y1="1325" x2="250" y2="1375" stroke="blue" stroke-width="2" />
<line x1="250" y1="1375" x2="500" y2="1375" stroke="blue" stroke-width="2" />
<line x1="500" y1="1375" x2="500" y2="20" stroke="blue" stroke-width="2" />
<line x1="500" y1="20" x2="250" y2="20" stroke="blue" stroke-width="2" />
<line x1="250" y1="20" x2="250" y2="41" stroke="blue" stroke-width="2" marker-end="url(#arrowhead)" />
<line x1="50" y1="672.5" x2="20" y2="672.5" stroke="blue" stroke-width="2" />
<line x1="20" y1="672.5" x2="20" y2="72.5" stroke="blue" stroke-width="2" />
<line x1="20" y1="72.5" x2="42" y2="72.5" stroke="blue" stroke-width="2" marker-end="url(#arrowhead)" />
<g transform="translate(50,50)" onclick="$.wiki('<<goto [[Hypothesis]]>>')">
<rect width="400" height="40" style="fill:white;stroke:rgb(0,0,0);stroke-width:1.5;fill-opacity:0;" />
<svg width="400" height="50">
<a xlink:href="#0">
<text x="50%" y="50%" font-size="13pt" dominant-baseline="middle" text-anchor="middle">Research questions and hypothesis</text>
</a>
</svg>
</g>
<g transform="translate(50,125)" onclick="$.wiki('<<goto [[Study context]]>>')">
<rect width="400" height="50" style="fill:white;stroke:rgb(0,0,0);stroke-width:1.5;fill-opacity:0;" />
<svg width="400" height="50">
<a xlink:href="#1">
<text x="50%" y="50%" font-size="13pt" dominant-baseline="middle" text-anchor="middle">Study context</text>
</a>
</svg>
</g>
<g transform="translate(50,200)" onclick="$.wiki('<<goto [[Study model]]>>')">
<rect width="400" height="50" style="fill:white;stroke:rgb(0,0,0);stroke-width:1.5;fill-opacity:0;" />
<svg width="400" height="50">
<a xlink:href="#2">
<text x="50%" y="50%" font-size="13pt" dominant-baseline="middle" text-anchor="middle">Study model</text>
</a>
</svg>
</g>
<g transform="translate(50,275)" onclick="$.wiki('<<goto [[Design of evaluation methods]]>>')">
<rect width="400" height="50" style="fill:white;stroke:rgb(0,0,0);stroke-width:1.5;fill-opacity:0;" />
<svg width="400" height="50">
<a xlink:href="#3">
<text x="50%" y="50%" font-size="13pt" dominant-baseline="middle" text-anchor="middle">Design of evaluation methods</text>
</a>
</svg>
</g>
<g transform="translate(50,350)" onclick="$.wiki('<<goto [[Selection of tools and Equipment]]>>')">
<rect width="400" height="50" style="fill:white;stroke:rgb(0,0,0);stroke-width:1.5;fill-opacity:0;" />
<svg width="400" height="50">
<a xlink:href="#4">
<text x="50%" y="50%" font-size="13pt" dominant-baseline="middle" text-anchor="middle">Selection of tools, equipment, and locations</text>
</a>
</svg>
</g>
<g transform="translate(50,425)" onclick="$.wiki('<<goto [[Failure and contingencies]]>>')">
<rect width="400" height="50" style="fill:white;stroke:rgb(0,0,0);stroke-width:1.5;fill-opacity:0;" />
<svg width="400" height="50">
<a xlink:href="#5">
<text x="50%" y="50%" font-size="13pt" dominant-baseline="middle" text-anchor="middle">Failure and contingencies</text>
</a>
</svg>
</g>
<g transform="translate(50,500)" onclick="$.wiki('<<goto [[Participants]]>>')">
<rect width="400" height="50" style="fill:white;stroke:rgb(0,0,0);stroke-width:1.5;fill-opacity:0;" />
<svg width="400" height="50">
<a xlink:href="#6">
<text font-size="13pt" dominant-baseline="middle" text-anchor="middle">
<tspan x="50%" dy="0.9em">Methods of participant or</tspan>
<tspan x="50%" dy="1.1em">observer recruitment and training</tspan>
</text>
</a>
</svg>
</g>
<g transform="translate(50,575)" onclick="$.wiki('<<goto [[Study protocol and ethics approval]]>>')">
<rect width="400" height="50" style="fill:white;stroke:rgb(0,0,0);stroke-width:1.5;fill-opacity:0;" />
<svg width="400" height="50">
<a xlink:href="#7">
<text x="50%" y="50%" font-size="13pt" dominant-baseline="middle" text-anchor="middle">Study protocol and ethics approval</text>
</a>
</svg>
</g>
<g transform="translate(50,650)" onclick="$.wiki('<<goto [[Conduct pilots to refine study design]]>>')">
<rect width="400" height="50" style="fill:white;stroke:rgb(0,0,0);stroke-width:1.5;fill-opacity:0;" />
<svg width="400" height="50">
<a xlink:href="#8">
<text x="50%" y="50%" font-size="13pt" dominant-baseline="middle" text-anchor="middle">Conduct pilots to refine study design</text>
</a>
</svg>
</g>
<g transform="translate(50,725)" onclick="$.wiki('<<goto [[Recruit Participants]]>>')">
<rect width="400" height="50" style="fill:white;stroke:rgb(0,0,0);stroke-width:1.5;fill-opacity:0;" />
<svg width="400" height="50">
<a xlink:href="#9">
<text x="50%" y="50%" font-size="13pt" dominant-baseline="middle" text-anchor="middle">Recruit participants</text>
</a>
</svg>
</g>
<g transform="translate(50,800)" onclick="$.wiki('<<goto [[Conduct study]]>>')">
<rect width="400" height="50" style="fill:white;stroke:rgb(0,0,0);stroke-width:1.5;fill-opacity:0;" />
<svg width="400" height="50">
<a xlink:href="#10">
<text x="50%" y="50%" font-size="13pt" dominant-baseline="middle" text-anchor="middle">Conduct study</text>
</a>
</svg>
</g>
<g transform="translate(50,875)" onclick="$.wiki('<<goto [[Collect measurements]]>>')">
<rect width="400" height="50" style="fill:white;stroke:rgb(0,0,0);stroke-width:1.5;fill-opacity:0;" />
<svg width="400" height="50">
<a xlink:href="#11">
<text x="50%" y="50%" font-size="13pt" dominant-baseline="middle" text-anchor="middle">Collect measurements</text>
</a>
</svg>
</g>
<g transform="translate(50,1000)" onclick="$.wiki('<<goto [[Data annotation and processing]]>>')">
<rect width="400" height="50" style="fill:white;stroke:rgb(0,0,0);stroke-width:1.5;fill-opacity:0;" />
<svg width="400" height="50">
<a xlink:href="#12">
<text x="50%" y="50%" font-size="13pt" dominant-baseline="middle" text-anchor="middle">Data annotation and processing</text>
</a>
</svg>
</g>
<g transform="translate(50,1075)" onclick="$.wiki('<<goto [[Evaluate and analyse hypotheses]]>>')">
<rect width="400" height="50" style="fill:white;stroke:rgb(0,0,0);stroke-width:1.5;fill-opacity:0;" />
<svg width="400" height="50">
<a xlink:href="#13">
<text x="50%" y="50%" font-size="13pt" dominant-baseline="middle" text-anchor="middle">Evaluate and analyse hypotheses</text>
</a>
</svg>
</g>
<g transform="translate(50,1200)" onclick="$.wiki('<<goto [[Reproducibility]]>>')">
<rect width="400" height="50" style="fill:white;stroke:rgb(0,0,0);stroke-width:1.5;fill-opacity:0;" />
<svg width="400" height="50">
<a xlink:href="#14">
<text x="50%" y="50%" font-size="13pt" dominant-baseline="middle" text-anchor="middle">Reproducibility</text>
</a>
</svg>
</g>
<g transform="translate(50,1275)" onclick="$.wiki('<<goto [[Iterative design and follow-up studies]]>>')">
<rect width="400" height="50" style="fill:white;stroke:rgb(0,0,0);stroke-width:1.5;fill-opacity:0;" />
<svg width="400" height="50">
<a xlink:href="#15">
<text x="50%" y="50%" font-size="13pt" dominant-baseline="middle" text-anchor="middle">Iterative design and follow-up studies</text>
</a>
</svg>
</g>
<g transform="translate(460,250)" onclick="$.wiki('<<goto [[Design phase]]>>')">
<rect width="30" height="200" style="fill:white;stroke:none;fill-opacity:0;" />
<svg width="30" height="200">
<text x="50%" y="50%" font-size="15pt" font-weight="bold" dominant-baseline="middle" text-anchor="middle" transform="rotate(90 15,100)" fill="lightcoral">Design Phase</text>
</svg>
</g>
<g transform="translate(460,700)" onclick="$.wiki('<<goto [[Experiment phase]]>>')">
<rect width="30" height="200" style="fill:white;stroke:none;fill-opacity:0;" />
<svg width="30" height="200">
<text x="50%" y="50%" font-size="15pt" font-weight="bold" dominant-baseline="middle" text-anchor="middle" transform="rotate(90 15,100)" fill="green">Experiment Phase</text>
</svg>
</g>
<g transform="translate(460,980)" onclick="$.wiki('<<goto [[Analysis phase]]>>')">
<rect width="30" height="150" style="fill:white;stroke:none;fill-opacity:0;" />
<svg width="30" height="150">
<text x="50%" y="50%" font-size="15pt" font-weight="bold" dominant-baseline="middle" text-anchor="middle" transform="rotate(90 15,75)" fill="cornflowerblue">Analysis Phase</text>
</svg>
</g>
<g transform="translate(460,1150)" onclick="$.wiki('<<goto [[Post study phase]]>>')">
<rect width="30" height="200" style="fill:white;stroke:none;fill-opacity:0;" />
<svg width="30" height="200">
<text x="50%" y="50%" font-size="15pt" font-weight="bold" dominant-baseline="middle" text-anchor="middle" transform="rotate(90 15,100)" fill="orange">Post-Study Phase</text>
</svg>
</g>
</svg>
</html>
<<widget "ToggleTheme">>
Switch between light/dark mode: <div class="toggle-wrapper">
<div class="rect_2"></div>
<div class="rect_1">
<div class="rect_1_inset"></div>
</div>
<div class="rect_3"></div>
<div class="toggle_handler">
<div class="toggle_ellipse"></div>
</div>
</div>
<</widget>><h2>Study Design</h2>
Below is a summary of the HRI study design you have specified so far, which you can copy or print to keep a record of your HRI study design at this point (Note: for a better view please hide the left and right side panels before printing)
[[Return to the design phase->Design phase]]
[[Continue to the experiment phase->Experiment phase]]
[[See a check-list for conducting your study->Check list]]
<hr>
<hr>
Version date: <<set $CurDate = new Date(Date.now())>>
<<= $CurDate.toLocaleString("en-UK", { weekday: "short", month: "short", day: "numeric", year: "numeric", hour: "numeric", minute: "2-digit" } )>>
I. Design
1. Proposal title:
* $StudyTitle
2. Hypotheses: $hypoflag
* Research questions: $rq
* Hypotheses: $hypo
* Pre-registration of hypotheses: $preregisteration
3. Type of HRI study and study context: $contextflag
* Level of autonomy: $HRIauto
* Information channel: $HRIinfo
* Role of human(s): $HRIhuman
* Role of robot(s): $HRIrobot
* Apatation and learning: $HRIadapt
* Features of the task: $HRItask
* Other contexts: $context
4. Study model: $modelflag
* Study design structure: $modelwithin $modelbetween $modelmixed
* Control conditions: $modelcon
* Dependent variables: $modeldep
* Independent variables: $modelind
5. Design of evaluation methods: $consflag
* Task performance metrics: $evatask
* Behavioural measurements: $evabeh
* Psychophysiological measurements: $evapsy
* Interviews: $evaint
* Self-assessments data: $evasel
* Custom questionnaires: $evaque
* Other methods: $evaoth
6. Study tools, equipment, locations: $locflag
* Where do you plan to conduct the experiments?
** Lab testing: $loclab
** Field testing: $locfie
** Online testing: $loconl
* Frequency and duration of each session: $locfre
* Robot(s) to be used: $locrob
* HRI task: $loctas
* Robot(s) role: $loctasr
* Human(s) role: $loctash
7. Failures and contingencies: $errflag
Possible failures: $errfai
Plan of recovery and prevention: $errrec
8. Participants: $parflag
* Participants: $parwho
* Sample size: $parnum
* Selection process: $parsel
* Recruitment:
** Contacts: $parcon
** Community groups, professional organizations: $parcom
** Mailing list: $parmai
** Volunteer banks: $parvol
** Crowd-sourcing platforms: $parcro
* Participant training: $partra
9. Study protocol and ethics approval: $ethflag
* Ethics approval received: $ethapp
10. Piloting: $pilflag
* Pilot study plan: $pil
<hr>
<hr>
Generated by HRI study methodology guidelines (version 0.1):
https://tianleimin.github.io/HRI-Methodology-Guidelines/<h2>Check-list for conducting your HRI study</h2>
Below is a check-list for you to follow when conducting your HRI study, which you can copy or print (Note: for a better view please hide the left and right side panels before printing)
Best of luck with your research!
[[See a summary of your study design specified so far->Study Plan]]
[[Continue to the experiment phase->Experiment phase]]
<hr>
<hr>
Version date: <<set $CurDate = new Date(Date.now())>>
<<= $CurDate.toLocaleString("en-UK", { weekday: "short", month: "short", day: "numeric", year: "numeric", hour: "numeric", minute: "2-digit" } )>>
1. Proposal title:
* $StudyTitle
2. Hypotheses: $hypoflag
* Research questions: $rq
* Hypotheses: $hypo
I. Experiment
3. [[Participant recruitment->Recruit Participants]]: $recflag
* Participants recruited: $rec
* Notes: $recnote
4. [[Conduct study->Conduct study]]: $condflag
* Experiments conducted: $cond
* Notes: $condnote
5. [[Collect data->Collect measurements]]: $colflag
* Data collected: $col
* Notes: $colnote
II. Analysis
6. [[Data annotation and processing->Data annotation and processing]]: $annflag
* Small sample size? $annsml
* Data cleaning performed: $anncl
* Data visualization performed: $annviz
* Notes: $annnote
7. [[Evaluate and analyze hypothesis->Evaluate and analyse hypotheses]]: $anaflag
Statistical tests: $anasta
Notes: $ananote
III. Post-Study
8. [[Reproducibility->Reproducibility]]: $repflag
Data released: $repdat
Code released: repcod
9. [[Iterative design and follow-up studies->Iterative design and follow-up studies]]: $iteflag
<hr>
<hr>
Generated by HRI study methodology guidelines (version 0.1):
https://tianleimin.github.io/HRI-Methodology-Guidelines/