Evaluation Criteria for Robotic Care Devices
We compiled the effectiveness, performance, and safety criteria—the outcomes of this study—into review criteria and conducted interim reviews and stage-gate reviews.
1. Development of application form templates
Document Review and Prototype Review in the Stage-Gate Evaluation Process
We have established the following Stage-Gate review document templates: 1-1 Development Concept Sheet (Template)
2-1 Prototype Development Goals and Issues Confirmation Sheet (Template)
3-1 Risk Assessment Sheet (Template)
3-2 "Minimum Safety Verification" Report (Template (by Field))
4-1 Demonstration Test Implementation Plan (Form)
4-2 Demonstration Test Results Report Sheet (Form)
4-3 Demonstration Test Data, Analysis Results, etc. (Free Format)
4-4 Report on Accidents, Incidents, Mechanical Failures, etc.
(Form)
5. Ethics Review Documents (Application, Results Report, etc.)
6. Instruction Manual (Instruction manual intended for commercialization, or operating instructions for equipment used during demonstration tests, etc.)
Please note that if demonstration tests were not conducted during the relevant fiscal year, submission of items 4-1 through 4-4 and 5 is not required.
The latest version of the forms (password-protected compressed file) can be downloaded from the Care Robot Portal.
2. Development of Evaluation Criteria and Methods
We have established evaluation criteria for both the document review and the hands-on evaluation in the stage-gate review process for each priority area. We have decided to place greater emphasis on the document review as the primary evaluation criterion and, as shown in the figure, to use the hands-on evaluation to verify aspects that cannot be determined or assessed through the document review alone.
Therefore, the evaluation will be conducted according to the following procedure:
1) Reviewers score the evaluation documents submitted by the grant recipients.
2) After the hands-on evaluation is conducted, the same reviewers revise their scores based on the results of the document review.
We have established evaluation criteria—including evaluation items and their respective weightings—for the document review (and the hands-on evaluation that supplements it) for each priority area.
The following major categories were established for the document review:
– "Practical Applications in Real Life" within the development concept
– Equipment requirements derived from "Practical Applications in Real Life"
– Objectives and methods of safety verification testing
It was decided that
reviewers with specialized knowledge would conduct the review for each of these major categories. Specifically, a cumulative scoring method was adopted, in which the scores for the scoring items (sub-items) set for each major category were totaled.
Furthermore, to incorporate evaluations from the user’s perspective, we decided to conduct a separate evaluation focused solely on the actual device, distinct from the above review. We requested that evaluators other than those mentioned above conduct the review based on the following criteria:
1. Evaluate whether the target robotic care device contributes to the realization of “enhancing care” (i.e., not merely “supplementary care” that simply assists with physical limitations, but rather improving daily living functions, particularly participation and activity).
2. Conduct the evaluation primarily from the user’s perspective.
The evaluation criteria consisted of the following four items. After inspecting the actual device during the hands-on evaluation, the assessors were asked to rate each item on a scale of 1 to 5.
1. Potential for clinical application
2. Mechanical functionality
3. Clinical safety
4. Potential for commercialization
We required judges to provide a written justification for their scores on each item. Additionally,
5. We requested comments on “Other” items,
including advice for the grantees regarding future development.
We compiled the scoring results above to determine the stage-gate review results for each grant recipient. We decided to calculate the final score by allocating 80 points out of 100 to the document review (and the supplementary physical device review) and 20 points to the physical device review alone.
3. Development of a Method for Evaluating Actual Devices
We have established the procedures for the physical unit evaluation as part of the stage-gate review. During the physical unit evaluation, evaluators will verify and assess the performance and safety of the prototype (or final product) installed at the evaluation site in accordance with the aforementioned evaluation criteria and methods. It has been decided that the physical unit evaluation will be conducted at a simulated care facility located within the driving test area of the Life Support Robot Safety Verification Center (Tsukuba City). The floor plan and interior view of the simulated care facility are shown in the figure.
