Spatial LED Scan using ARKit and Vision models

The idea

The idea behind this all was to make 3D effects shareable using the website/app. Rather than having users manually input the shapes of their setups, I figured a nice challenge would be using Augmented Reality to scan and plot the positions of points in the real world.
As you can see from the demonstration video below, I was able to take periodic snapshots, and using Swift vision requests, determine likely locations of LEDs by their detected contours (see below), and finally compare them to the known LED positions. We use Baxter permutations, which I'll describe below.

Baxter's Permutation

I wanted a pattern displayed on the LED strips to uniquely identify the position on the LED strip without requiring time, or multiple screenshots. This meant capturing an image, extracting the colors in a sequence, and matching the sequence to as small a pattern as possible.
This is the point of Baxter permutations- they are a permutation designed for pattern avoidance. From Wikipedia below, we can see that a pattern containing 2074 LEDs in a strip can be uniquely identified using only 7 LEDs in a row. The max number of LEDs is 2048 anyway, so this is perfect. The pattern of 7 is represented using different colored LEDs.

Tying These all Together

The video below showcases a prototype of the scan in action. It is still work in progress, but the prototype has shown promising results for identifying where in 3D space the LEDs are and I have been able to make promising shapes from it.

The Goal

The goal for this is to make both 3D renders for sharing and for use in the database to make spatial-aware 3D effects (where verticality could matter, for example in a fire or raindrop effect). Below is one of the shareable 3D gifs using nearly all of the projects I have touched on in This portfolio.