## Inside Toboggan Labs' Montreal-Inspired Viam Hackathon _(This is part 2 of a two-part series. You can [read part 1](/en/blog/2023-06-14-bagels-robots-and-metro-adventures-part-1) here)._ > I appreciate my Roomba so much more now" > > \- Flo Herra-Vega, CEO and CTO, Toboggan Labs ## Monday, May 23, 2023. Hackathon day. Participants created a Viam account, requested access to our org, and tried connecting to robots via `ssh`. However, most tasks were accomplished via the [Viam SDK](https://docs.viam.com/tutorials/get-started/try-viam-sdk/) with the credentials accessible through the [Viam Application](https://app.viam.com/), where the two rovers and the large arm had been made available. Most participants used the [Python SDK](https://pypi.org/project/viam-sdk/) , but there was one rebel who chose Golang. ![Software engineer at work, debugging code.](/media/dscf1269-683x1024.jpg) ## The Green Line We marked out a "metro simulation" on the floor using green tape. Unfortunately, the tutorial [code](https://github.com/viam-labs/line-follower/) used an outdated version of Viam’s library. However, we successfully managed to [update](https://github.com/TobogganLabs/viam-rover) it to get our robot up and running. Our first challenge was to fine-tune the green detector's sensitivity. We selected the colour from an image taken by the robocam and adjusted the hue accordingly. Before we learned how to digitally crop images to ignore irrelevant background noise, the robot ended up wearing a paper plate hat, much to everyone's amusement. Eventually, the robot successfully followed the line from one end to the other.

![](/media/PXL_20230529_155911228-771x1024.jpg)

Julien checking the rover's camera view of the route.

To indicate stop stations visually, we initially used pink post-its. We defined the pink hue in the colour detector’s vision service. However, we encountered a problem—the camera wouldn't start. After some tinkering, we discovered that our chosen hue was too close in temperature to [colours that are not detected (white, gray, and black)](https://docs.viam.com/services/vision/detection/#configure-a-color_detector). By selecting a hue closer to red, we finally detected the sticky notes.

![Two software engineers focused on their laptops with a manager supervising.](/media/dscf1264-1024x790.jpg)

Hackathons are challenging but really fun when things finally work.

We faced an "early stop" problem where the robot would interpret pink hues on the brown carpet as stations. We overcame this by increasing the size threshold for detection. But this introduced a new issue: when the robot was almost at the station, the marker would move partially out of frame, and the in-frame portion wasn't large enough to trigger detection. To resolve this, we switched to using blue station markers, increasing contrast with the carpet. A remaining issue was that sometimes the stop station was detected at a slight distance from the actual station. To fix this, we implemented a state machine.

![](/media/robot-still1-683x1024.jpg)

Our robot ended up wearing a paper plate hat...

## A State Machine for Metro Station Stops We defined four states: **`drive`, `approach`, `station`**, and **`bagel`**. In the `**drive**` state, the robot follows the green line while scanning for bagels and stations. Upon bagel detection, it enters the `**bagel**` state. If a station is detected, it moves to the `**approach**` state. In the `**bagel**` state, we take a pause to ponder the glory of bagels before returning to the `**drive**` state. We had to manually remove the bagel from the frame, or the contemplation session would restart immediately. The `**approach**` state operates similarly to the `**drive**` state. The robot still searches for bagels, but the station detection is reversed. We wait until a station is no longer detected, assume this indicates we are over the station, and enter the **`station`** state. The `**station**` state is a brief pause accompanied by the harmonious tunes of the STM, followed by a return to `**drive**` state. If there is no line or station detected for three consecutive cycles, the robot becomes lost and ceases operation. We later introduced a `**lost**` state where the robot tries to locate the track for a set number of cycles before giving up.

![](/media/PXL_20230526_152344127_TS_2_AdobeExpress-1.gif)

Our company mascot, Nordy, taking a joy ride on the rover.

## Sound We configured a bluetooth speaker to work with each of the rovers. We had two custom sounds: The metro chime to play at stops, and the Montreal snow plow siren to play as the second rover zoomed around the room. We managed to play sound via the command line and using python libraries executed from the raspberry pi. To play sounds on the rover via the SDK while running the rover scripts from our laptops, we attempted to write a sound service to be called via viam-server. We had some issues here as viam-server did not have access to the libraries we had installed on the pi, but there was no clear documentation on these limitations or default configurations for speakers or sound. We also encountered issues with the way custom modules were implemented, requiring us to manually quit a crashing service to prevent the entire system from failing. ## Bagel Detection Viam allows the addition of “detector” services that use an ML model to return the bounding boxes of identified objects. Here’s how we got that working: Being resourceful ML engineers, we searched for pre-trained computer vision models in the right format (Tensorflow Lite! aka ‘model.tflite’). TF Hub had a tiny [EfficientNet](https://tfhub.dev/tensorflow/lite-model/efficientdet/lite0/detection/metadata/1), great for RPIs. Trained on the COCO dataset, this model had labels for donuts, cakes, and pizzas—bagel's close-enough culinary relatives! [Details](https://developers.google.com/mediapipe/solutions/vision/object_detector/customize) are available for fine-tuning. - We created a Model with Viam’s UI and uploaded the weights (tip: match the Model Name and filename!) - Under the Services tab, [we created a Model](https://docs.viam.com/services/vision/detection/) that connected to those model weights. - We crafted a Vision Service that leveraged that ML Model (basically: Model → ML Model → Vision Service with ML Model). - We added the camera transform to plot the bounding boxes and voila, instant bagel detector!

![bagel with overlay of detection dimensions](/media/bagel-detection-1024x769.png)

Bagel detection

For the extra-curious, here are [mor](https://gopro.com/v/m4E3BJGZpV4G8)[e](https://gopro.com/v/m4E3BJGZpV4G8) [time-lapse videos of the hackathon](https://gopro.com/v/m4E3BJGZpV4G8). ## Next Goals We have plenty of ideas for future projects, such as getting the Yahboom arm working, setting up our custom arm/servos, creating a custom command line as a service, and creating some custom components such as a temperature sensor. From a project standpoint, we'd love to have a dedicated office Pi, that we can keep upgrading into a little assistant. To learn more about how your organization could incorporate robotics and machine learning, [reach out](mailto:info@tobogganlabs.com?subject=reach-out) to see how we can help! Thanks to _Michelle King and Hailey Ellbogen for handling event logistics._ _Thanks also to Arnaud Fosso Pouangue, Becks Simpson, Brock Jenken , Eren Bilgin, Julien Blin, Kim Phan, Ksenia Nadkina, Louis-Philippe Perron, Mona Ghassemi, Raphael Souza, \_and\_\_ \_Rob Linton for their participation in the event and for contributing their notes to this blog post._ Our code: [https://github.com/TobogganLabs/viam-rover](https://github.com/TobogganLabs/viam-rover)

Search

Ready to build?