Proof of Concept Review

Image 1 above depicts our ongoing, involved stakeholder Joe standing in front of a Coles grocery store. while he stands front on to the camera, the audience can see a prototype of the Ver Device strapped to his chest using a cross body sling bag.

Proof of Concept Update:

To nicely wrap up all the work completed throughout the holidays, the team held the proof of concept (PoC) last Friday the 19th July with one of our ongoing stakeholders, Joe. The PoC offers a unique opportunity for the design team to see firsthand how the project is progressing. From the software side, the PoC provides valuable feedback on the project development and indicates whether the current system architecture effectively meets the stakeholder's expectations and requirements. For the hardware team, the PoC allows us to understand how to better design future protoypes for the end user.  

This proof-of-concept design has been designed to replicate an end use case within a grocery shopping scenario. The ability to shop for groceries is an important part of achieving a sense of independence and is closely tied into their social role valorisation. We also found through our end-user and advocate interviews that shopping in general was often a difficult and overwhelming task, but also one that is unavoidable. Therefore, this use-case was chosen for proof-of-concept validation.    

What did the PoC look like? 

For the purpose of the PoC, the testing process was broken into software and hardware as the design is yet to be fully integrated. A fully integrated design will be explored in Design Review 1 and 2. The following descriptions below provide a brief insight into the different components and strategies used for the testing process. 

As the hardware team continues work on programming the Raspberry Pi, the video input was streamed from a smartphone to a handheld laptop using an application called ‘vdo ninja’. This allowed the image processing to take place on the local laptop before being sent to our server. Until the raspberry Pi is up and running, a laptop will be used to replicate the role of the client, transmitting input information to the server and communicating the LLM with the user. The LLM response was communicated to the test user through Bluetooth connection to their headphones.  

As the team is yet to establish a reliable output from the audio to text module, the PoC was conducted using several pre-programmed prompts each allocated to a different key on the laptop keyboard. The prompts used in PoC can be seen below, we would love to hear what prompts you would find the most useful in your life! 

• Describe what can be seen in 10 words. Use only Natural language. 

• Describe what can be seen in 20 words. Use only Natural language. 

• Describe what can be seen in 30 words. Use only Natural language. 

• Describe what can be seen in 40 words. Use only Natural language. 

• Describe the signs that can be seen in as few words as possible. Use only Natural language. 

• Describe the signs that can be seen in as few words as possible. Use only Natural language. If no signs can be seen reply as such. 

• Describe the signs that can be seen in as few words as possible. Use only Natural language. If no signs can be seen reply with ‘There are no signs that can be seen A T M’. 

• Describe the colours that can be seen. Use only Natural language. 40 words maximum. 

• What locations are listed on the sign. Use only Natural language. 

• What am I pointing at, reply in 10 words? 

• What am I holding, reply in 10 words? 

• Where are the bananas, reply using clock-face coordinates with 10 o’clock being leftmost and 2 o’clock being rightmost. 

Results from the Proof-of-Concept testing: 

The PoC testing at Coles Manuka verified the accuracy and effectiveness of 12 different prompts that were pre-set into the user interface. The conclusions were as follows: 

The positives...  

• The device was highly proficient in understanding context and location. When asked about the exterior of the shopping centre, it had a good understanding of their being a local bakery present and easily identified the Coles shopping sign.  

• The device could accurately read signs and further deduce information such as “specials” and provide information on the sale price.  

• The device effectively read aisles and locations signs and could further deduce what the user might find in the respective locations e.g. the Asian foods section would likely find noodles, rice, curries etc. 

• The device could quickly locate itself by understanding context, e.g. being in a supermarket based off the apparent presence of aisles.   

• The team deduced from user testing that a description between 20-30 words was the most effective response that did not increase verbiage unnecessarily.  

What needs to be improved... 

• The software had some troubles identifying/differentiating between some food items such as pancakes that were not clearly labelled and vegetables such as leeks, spring onions and cucumbers.  

• Without a more specific prompt, the device tended to become too focused on the environment and describing context. For example, while holding a jar of vitamins up to the camera and asking the device to describe the image, the response mainly focused on features such as, being in a supermarket, the background image of an aisle etc.  

• Testing concluded that the device does not currently understand a point command such as “what am I pointing at?”.  

• When using pre-set prompts, the team needs to determine a way of teaching the device to understand the user's presence. This being that if the user asks the device to describe an object they are holding, the device does not need to describe the hands holding it. 

• Camera quality was incredibly poor as there was minimal Wi-Fi connection within the supermarket. This is something that will hopefully be improved with the introduction of the Raspberry Pi board and camera module.  

• As there is little Wi-Fi connection within supermarkets, the team will need to finalise an approach that is not reliant on the presence of a stable Wi-Fi connection.  

What’s next? 

Taking the results from the PoC, it is time for the team to start moving forward with Concept 1. This will be the first time the hardware team and software team will integrate their efforts to hopefully produce a working prototype. Concept 1 will aim to achieve the defined minimal viable product (MVP), a working prototype that meets all the project requirements by the start of semester 2.  

While the software team continue working on features such as the audio to text and voice commands, the hardware team will begin ordering hardware components and printing 3D prototypes.  

Exciting updates to come! 

Images from Proof of Concept Testing:

Image 2: The Ver team stand with Joe who is pointing at a pile of pancakes at the supermarket. While Joe points at the food, everyone is looking at a computer screen.

Image 3: The Ver team are sitting with around a cafe table with Joe. They are all drinking coffee and have laptops and tablets in front of them. Joe is pointing at a tablet while the team reflects on the testing that has just occurred.