Well? What do you think Doc? “Hmm, looks like an instrument for engineers designed by engineers.” Gee, glancing over at my fellow engineer with the look of - “Do you think that is a compliment, after all, we are engineers?” When it comes to clinical usability, it was not a compliment. Unfortunately, we thought we were about done. How could we get this far and end up here?!
You see, we engineers had been tasked with taking two legacy predecessor instruments and combining them into a new next generation. At this time the product line was the organization’s “red-headed stepchild”, and we had no marketing input (i.e., we would bring an actual bobblehead doll to our cross-functional team meetings). We didn’t know which of the predecessor product’s controls and buttons to exclude so we included them all! We thought we had organized them well with various tabs and by physiological function – sigh. The product was revamped and released and ultimately very successful.
A good usability process is intended to prevent this scenario and to reduce use-related risks. We can have the best darn widget but if the operator doesn’t use it correctly it becomes a liability. There are a couple of key standards (IEC 60601-1-6, IEC 62366-1) and guidance (FDA Guidance - Applying Human Factors & Usability Engineering to Medical Devices) to comply with when developing a product. They are wise, they align with design controls, and they are required. For this overview, we will use IEC 62366-1 as it satisfies the EU, FDA, and the ROW.
Shown in Figure 1 are the main steps to the useability engineering process. Many times, there will be iterations between the steps as the product’s usability gets refined during development.
Figure 1: Main Useability Engineering Steps
The first step is to create a User Specification and both IEC 62366-1 and the FDA Guidance on Human Factors & Usability Engineering (HF&UE) prescribe a minimum content for a User Specification (see Table 1) below. A good place to put this is in a User Needs document. This may sound obvious, but Qserve often encounters clients who don’t have such a document, or it is written as a convolution of product requirements and marketing needs. User needs should be in user terms and testable (i.e., able to be validated). User needs are few and typically spawn a large quantity of product requirements – see the example of Figure 2.
Figure 2: User Need versus a Product Requirement
Table 1: Use Specification Content
Use Specification | Example |
Intended Medical Indication | Intended Health Care Use: Clinical conditions and contraindications. |
Intended Patient Population | Gender, Age, Weight, Skin Color, Cognitive Abilities, Physical Abilities etc. |
Intended User Profile | Level of training, Physical Ability, Cognitive Ability, wearable constraints, |
Intended Use Environment | Health care facility (OR, Cath. Lab., Emergency Room, outpatient clinic), Home Use, Emergency Vehicles, layperson wearable etc. |
Operating Principle | Data acquiring capabilities, therapeutic delivery of energy, monitoring capabilities. |
Use Specification
The Use Specification is to include the Intended medical indication, intended patient population, intended user profile, the use environment, and the operating principle. It is a key specification as it informs much of how the product will be assessed for usability. Perhaps one example that may not be so obvious involves cybersecurity. A couple of years back I developed an instrument that was to have multiple user profiles. In Figure 3 you can see the implications of these different profiles with respect to managing privileges. Service personnel could not access patient data. Researchers could access patient data, but it had to be anonymized when exported (just age and gender allowed). These different privileges were defined (specified) and the UI was designed for the various permutations of privileges.
When conducting formative and summative evaluations of the user interface, it is important to evaluate with adequate samples of the user profiles and patient populations. Reviewers look for this and it can be a significant setback to have to redo summative evaluation because the study didn’t adequately cover the users of the user specification.
Figure 3: Example of how User Profile influences authority privileges
User Interface (UI) Specification
It is important to remember that the user interface is all aspects of the product that the user comes into contact with. In 60601-1 parlance these could be considered accessible parts. Software’s graphical user interface (GUI) is rather obvious but service access panels, transport handles, connector placement, and connector type may not come to mind. Because the UI is broad, the specification is often a section of a product specification with details partitioned down into lower-level functional specifications (Hardware and Software). It is the UI’s role to facilitate the intended use by the various user profiles in the intended use environment. This is where the breakdown was in the initial scenario shared in the blog and spring forward, this is where a couple of years back a well-defined user specification informed an elegant GUI means of presenting just what was needed for the logged-in user group to easily perform their tasks. For some niche products, it can be critical to have “obvious” and simple controls with clear prompting for the next steps. Remember to think about whether the control’s indicator is indicating action or status. Niche products that are not used multiple times a day have the UI challenge of requiring the need to repeatedly become acquainted with them – make it intuitive.
Examples of UI specifications include viewing angles, accommodation of a sterile drape or operation with gloves, the center of gravity for handheld and/or mobile units, non-interchangeable cable connections for cables supporting different functions, etc. Packaging and the layout/readability of the instructions for use are also considered part of the UI. All these attributes need to be specified in a way that can be verified and subsequently validated.
Task Analysis
A task analysis is a step-by-step breakdown of the user tasks for a given use case. Remember there are different types of users so a) initial installation, b) updates and repairs, c) device configuration, d) device operation, e) data review and f) archiving may all be using cases to analyze. At Qserve we often provide templates to help get clients going. The use scenario is assigned a process ID and name. For a given use case, the user/actor is defined (from the use specification), the trigger for the use scenario is defined as well as the end of objective. It is good to state the preconditions such as device is already installed and configured (because this was covered in a preceding case). Or in downstream use cases, device is powered up and initial setup and connections are all made – etc. Then the step-by-step analysis is performed stating the user action, the system action (response) and possible errors.
Table 2: Use Flow – Task Analysis
* = Critical Task
Step | User Action(s) | System Action(s) | Possible Use Error (s) |
1 | | | |
2* | | | |
It can seem a bit arduous to write down every step, but it can be very important since developers are typically very familiar with their devices and often can assume things are obvious i.e., remember: “Looks like an engineering instrument designed by engineers….”. Have the team review the task analysis. Often mockups of the device or screen captures of early software GUI’s can be used internally to refine. Capture this activity in review minutes! Remember normal use includes use errors. Evaluation of error reporting interpretation and effectiveness is also important. Abnormal use (i.e., cyber threats) should also be considered.
Looking back at Figure 1, it is then important to perform a use risk analysis against the task analysis. We already identified error conditions that may create use errors, but we need to assign severities and likelihoods. For the USA, it is important to identify critical tasks. This term is also used in IEC 62366-1. A critical task is a user task that, if performed incorrectly or not performed at all, would or could cause serious harm to the patient or user. If the device has no critical tasks document, it and the rationale. All critical tasks are to be evaluated during human factors validation testing. Ideally in design, we do what we can to reduce the number of critical tasks but in many cases, the skill of the user is a big part of the diagnostic and/or therapeutic process involving the device.
Formative Evaluation
This is where manufacturers identify potential users (per the user specification) and solicit their feedback on the user interface. This can be an interesting exercise and it is very important (i.e., fail fast). The formative evaluation plan should state the evaluation methods used, which part of the user interface is being evaluated, and the useability engineering process to perform each of the UI evaluations.
Mockup devices, storyboards, and prototypes are all ways to give users a hands-on experience of how the user interface works before a final product exists. Test the use flow, especially critical tasks. Document the results and use them to inform user interface revisions/refinements. Update the UI Specification and UI, the use flow, and the use risk analysis. This effort can be broken down into individual use cases and/or user types. Plan to iterate and document the reviews and feedback. It is important to include the user manual as it needs to be evaluated for effectiveness. Often the user manual is used as part of the training and a “decay period” is applied prior to the UI's actual evaluation. The “decay period” is meant to assess the IFU’s effectiveness in training/educating the user. This shows up in the summative evaluation, but it can be helpful to consider starting it in the formative evaluation phase.
Figure 4.
Figure 5: Usability Evaluation
Both the formative evaluation and the summative evaluation can be conducted in a similar manner albeit the summative evaluation is with the final device and IFU.
Figure 6: The 7 Steps to Uability Testing (Formative and/or Summative)
Summative Evaluation
Summative evaluation occurs on the final user interface embodiment. For each hazard-related use scenario, the UI evaluation plan for summative evaluation shall specify the evaluation method being used, which part of the UI is being evaluated, and where applicable the criteria for determining whether the information for safety is perceivable, understandable, and supports correct use. This summative evaluation may be part of the system validation but is really focused on validating the usability of the product’s intended use by those called out in the user specification. It must evaluate the critical tasks and contribute to the overall risk acceptability.
Usability Engineering File
The Usability Engineering file contains the sum of all the usability activities conducted on the device. This file can be a combination of documents or pointers to documents that make up the design history file.
Figure 7: Usability Engineering File
Human Factors (HF) & Usability Engineering (EU) Report
Both IEC 62366-1 and the FDA Guidance on HFE & UE provide prescription and guidance on what constitutes a complete Usability Engineering effort. The figure below maps the elements called out in Appendix A of the FDA HF/UE Guidance against IEC 62366-1.
Figure 8: HF & UE Report Content
HF and UE Engineering Process
Aligning and integrating usability within your design controls and risks processes is shown below and is important for cohesive product development (see webinar and blog for the Cybersecurity and Software Development alignment topics).
Figure 9: Alignment of Usability with Design Controls & Risk Management
At Qserve we help clients with achieving such alignment and supporting evidence. We provide training, coaching, templates, or just resources to help get it done. Usability is crucial to MD and IVD safety and effectiveness and Qserve is here to help!