Spark

Web Server · OpenCV · IoT · Python · C

Overview

The Spark project was a coursework project I participated in during my study of UX design, in a group of 5. It was about a social interaction enhancement device, designed to break the social barrier between strangers during a social event. It's intended to facilitate the natural discovery process between individuals and groups by establishing common topics via a shared questionnaire, LED lights and a vibrating necklace. The group also produced a prototype that was demonstrated during the faculty exhibit to fellow students and guests.

The Problem

Many people find social interaction with strangers challenging, although they know its importance. This results in significant difficulties in building connections with others in social events, where the primary goal is to build connections with others. Consequently, they find it difficult to actively approach people, especially when they don't know much about their counterpart. However, research found that social interaction with strangers is often much better than what people expect.

The Design

Conversation is the most common form of interaction between individuals during social events, hence the device was aimed to assist its users with initiating new conversations. Conversation is an exploratory process for each participating party, allowing them to obtain more information, express themselves and build up the image of others. While what's considered a 'successful conversation' would differ between individuals, it is difficult to determine if the conversation will be successful when there is no information about the counterpart. Therefore, we designed a device that gives its users additional context about others around them, allowing them to have more confidence with the conversation they are about to initiate, hence feel less challenged when they do so.

Spark asks the users to answer a questionnaire upon entering a social networking venue. The responses entered by the user are then displayed on a LED hat, which is visible to other users. The user is also given a necklace that can vibrate, which has a different vibration pattern depending on the questionnaire response of the person they are looking at. The necklace vibrates more frequently when they have similar responses, allowing the users to identify people who are similar to them.

In addition, the hat on the user's head has LED lights, and the colour of each light represents a different kind of interest. However, the user does not know what kind of interest each colour represents. To discover the meaning of the LEDs, the user must have a conversation with others in the event, who would also be interested in discovering them. By allowing everyone to participate in a guessing game, Spark gives everyone a common conversation topic as well as motivating them to start conversations with others.

Spark asks the users to enter their basic information and topics of interest before entering the social networking venue. The user is then given a LED hat and necklace, which are connected to the central control system. The necklace notifies users about the similarity between them and others using vibration. When a user looks at another person, the necklace will vibrate differently depending on the number of interests they share with each other. This vibration was inspired by a fastening heartbeat from excitement and is intended to give the user additional information about the person they are looking at.

The LED hat has a number of different LEDs that represent questionnaire responses of the wearer. By having this information visible, the questionnaire becomes a great conversation prop for users to easily start conversations with strangers, knowing some common topics they can talk about. The sequence can be randomised to further enhance this feature, allowing users to play a guessing game by talking to each other about their answers.

The Prototype

To demonstrate the core concepts of the Spark device, a functional prototype was produced. The prototype consists of the necklace, the hat and the main application to control both devices. The devices were constructed using off-the-shelf materials and were connected to each other via WiFi. It was built to imitate the use case during a social event with a generic questionnaire. The user location and directions were calculated using fiducial markers and a camera.

Main system

The main system is what does most of the processing and communicates the processed data to the LED hat and necklace. It was written using Python and hosted on a laptop, utilising OpenCV to calculate who each user is facing. The main system could be broken down into UI, server, and position calculator.

UI

A questionnaire UI was built as part of the application, so the questionnaire responses from users could be easily used. The UI was made using Python's Tkinter, which consisted of generic questions such as 'pineapple on pizza?'. The responses were saved with an ID that can be configured by this UI, which is then used to determine the colours on the hat and calculate the vibration strength of the necklace when the user is facing others, or when others are facing this user.

Locator

The locator detects who each user is looking at using ArUco fiducial markers, a webcam and OpenCV for Python. Each marker's 3D location and orientation are calculated from the marker, which are then mapped in 2D space to determine who each user is facing. The sampling frequency and mapping were tuned to stabilise the detection.

Server

The communication between the device and main system was handled by a Python web server and web sockets in a local network. The server responded to incoming HTTP requests by starting a handling thread with every request, then went back to a busy waiting state. The requests are handled by either sending the vibration frequency or LED sequence of a specific user, depending on the information included as part of the request.

LED Hat

The main functionality of the LED hat was to query the main system about a colour sequence, then display those colours using a Neopixel LED strip. Each hat had a static ID used to retrieve the corresponding user's LED sequence from the main system via WiFi. An 'ESP-32 Thing Plus' development board was used as the central processor due to its wireless connectivity. Arduino core was used to simplify the development process. Fiducial markers were also positioned on top of the hats, which were used to detect who each user is facing.

Necklace

The necklace of the prototype consists of a vibration motor and microcontroller, and its only job is to vibrate the motor based on the signal response from the main system. For the prototype, vibration frequency was altered based on shared interests while the vibration strength was kept as strong as possible. The necklace queries the main system about the vibration frequency update 4 times per second, then updates the vibration frequency if there is a change.

The Exhibit

The final version of the prototype was demonstrated during the Physical Computing Exhibit 2022. During the exhibit, the team positioned a webcam on top of the venue and allowed guests to try the Spark system with a demo questionnaire.

The exhibit was successful, and we successfully presented our idea to the guests and confirmed the effectiveness of the device in a social event context. As we were open to feedback, some guests suggested different use cases of the project that the team didn't consider, allowing us to discuss how the idea could be iterated.

My Thoughts

It was an interesting project because it didn't stop at the design phase — a physical prototype had to be developed and demonstrated. Team organisation and workload division were the biggest challenges, because the team had both internal and external students. It was very challenging for external students to contribute to physical development of the prototype because they were in a different country.

Challenges

Could've done better

What I enjoyed

Literature