Stakeholder Concept Review

This Concept Review was the first wearable iteration of the design, incorporating device hardware components into a cross-body bag. The concept review posed as the first opportunity for the team at Project Ver to observe members of our end user group interact with the device and collect feedback on the overall user experience. There were two main objectives of the concept review: 

1. AB testing of button functionality, and,  

2. Software and Hardware benchmarking. 

AB Testing: Button Functionality Mapping 

The purpose of the testing was to conduct AB testing on two different button interaction designs. While the physical placement of the buttons remained the same in both designs, the functions triggered by each button press were different between Configuration A and Configuration B.  In addition, the testing process aimed to offer greater insight into general stakeholder engagement with the product by collecting a range of qualitative and quantitative data that will be actioned to finalise the product design before the conclusion of the project.   

Methodology:  

The test was split up into 3 individual case studies that attempted to mimic increasing levels of difficulty within a grocery store context. These cases were as follows: 

1. Case 1 Item: Weet-Bix Box  Objective:  To determine LLM accuracy with respect to labels on clear and flat packaging  
   

2. Case 2  Item: Bagel Bag  Objective: Further testing LLM accuracy when reading information off distorted, soft packaging.  
   

3. Case 3  Item: Coconut Yogurt v Coconut Flavoured Greek Yogurt  Objective: Testing devices ability to accurately differentiate between similarly marketed products.  
   

In addition, each case was composed of the same four individual tasks. These tasks were constructed with the aim of providing quantitative and qualitative to be leveraged in software benchmarking at the conclusion of the testing period. The format of these tasks were as follows:  

Test	Description
Object Recognition Test	Validates how accurately the device identifies the common grocery items based on their visual characteristics
OCR Accuracy test	Validate how accurately the device converts captured text into spoken word.
Query Response Test	Validate how accurately the device understands the information necessary to user.
Contextual Understanding Test	Validates the devices' ability to understand follow up questions and/or provide clarifications based on the initial query.

Table 1: describes the tests implemented to benchmark software requirements  

Test ID	RID	Description	Evaluation Criteria	Test	Implementation
T1.1	R2.2	Detect objects/features in the immediate environment.	LLM recognises grocery store signs/ understands context of image being taken	Object Recognition Test:	“Describe”
T1.2	R2.3.1	Locate and accurately process text.	Location and processing accuracy of text is satisfactory for the stakeholder’s needs.	OCR Accuracy test	“Read”
T1.3	R2.3.2	Aid with detailed text analysis for activities such as shopping (label reading, sign reading).	Depth and utility of text analysis is assessed by the stakeholder.	Query Response Test	"Chat”
T1.4	Contextual Understanding Test	Additional follow-up “chat” question to add clarification based on initial query.

Table 2: describes the tests implemented to benchmark software requirements

AB Case Configurations: 

During the review the team compared two different button function implementations.  

Testing Case A 

• Features: Two distinct buttons for different functionalities. 

• Read Button: Toggles reading of immediate features in the field of view (FOV). 

• Describe & Speak Button: Single tap to describe features in FOV and hold to activate conversation with the LLM.  

Testing Case B 

• Features: Voice-activated controls with audio prompts. 

• Function Activation: Responds to spoken keywords "Read" for signs and labels, "Describe" for surroundings, and "Chat" for engaging with the LLM. Prompts user to repeat if the keyword is not detected. 

Software and Hardware Benchmarking: 

In alignment with the Human Centred Engineering (HCE) Model, the team at project Ver has integrate system benchmarking within stakeholder reviews.  This decision was made to involve project stakeholders in the process of validating the prototype against the project requirements.  

The Concept Review aimed to re-evaluate three software benchmarks and six hardware benchmarks. Requirements that did not receive an “acceptable” grade would be iterated and tested again in the final concept review before project completion. The benchmarks under assessment were as follows: 

System	R.ID	Description
Software	R2.2	Detect objects/features in the immediate environment.
R2.3.1	Locate and accurately process text.
R2.3.2	Aid with detailed text analysis for activities such as shopping (label reading, sign reading).
Hardware	R1.1	The device must not interfere with social interactions nor user activities.
R1.6	The device must not obstruct other senses.
R1.6.1	The device must be ergonomic for the user during operation.
R4.2.1	The device must only share data with approved responsible parties / offline data processing
R5.2.3	The device must complete commands with minimal delay
R6.4	The device must be deemed portable by users in user testing and minimum weigh 200g.

Conclusion and Findings: 

Based off the results collected in AB testing, the Ver team has decided to combine the best features from both configuration A and B. The team will use the results and stakeholder feedback collected in testing in the final design iteration before the final design review at the conclusion of Week 9. The list below details the major findings and feedback that the team will look at implementing: 

• Stakeholders re-iterated the importance of design aesthetics, smaller casings with rounded edges would help seamlessly “disguise” the design.  

• Buttons felt “clunky”, the team will investigate the possibility of incorporating tactile buttons with seamless button covers. 

• Device needs to consider the implications of being designed for right hand dominant users.  

• The device should include audio feedback to indicate that the request is being processed and prevent the user from overloading the system. 

• Implementing and autofocus function to improve LLM response and accuracy. 

• Improving input prompt associated with user functions to make LLM responses better at identify and conveying important and relevant information to the user. 

• Implementing an autofocus feature to improve camera focus and allow identification and description of foreground and background information. 

• Reducing the word limit to prevent excessive verbiage.  

• Consider moving buttons to a separate module to reduce camera wobbling and make the button use accessible to both left and right hand dominant users.  

• Modifying the dual-stream server to prevent LLM hallucinations. 

Image Description: The image above depicts one of the team's ongoing stakeholders testing the prototype Concept 1. He holds a pouch of yogurt in front of himself as the he wears the device across his body. He is wearing his Sony Bluetooth headphones to communicate with the device, asking questions and receiving responses.