CookieVision is an AR platform developed and implemented for Tiff’s Treat’s. If you have their iOS or Android app and one of their cookie boxes nearby, you can use CookieVision to enjoy an AR animation on the box itself. Additionally, you can use the app to record video messages and include them when sending gift deliveries and the recipient can then use CookieVision to watch a special AR animation including the video message on a projector screen.

Throughout the development of CookieVision, there have been three particularly challenging obstacles.

The first was to be able to embed our builds into an already existing iOS app as well as an Android app (both written natively for their respective platforms). Achieving this required a significant amount of communication between myself and the lead developer of the apps. We had to make sure that “our app” and “their app” were always providing and expecting the right API calls and information from each other. Even with that solved however, embedding Unity builds inside of native apps proved to require extensive knowledge of both platforms, especially Android.

The next technical challenge was much more fun. The Halloween animation sequence required Chip, the cookie mascot, to “push” up the surface of the box to try and escape from inside and then use a chainsaw to cut a hole out and then jump up. The “bump up” effect required me to take color samples from the camera feed of where on the real world box the bump would be happening. I then applied these colors to the virtual bumps and created a new directional-light that only affected the bumps to prevent the regular lighting from ruining the effect. Then for the virtual hole being cut out of the real world box, I would capture the specific area of the camera feed’s frames where the hole would appear and then would project those captured images back onto the material of the virtual hole’s cutout skewed exactly to match the orientation. Then once the cutout started to fall, I would stop updating this image. Unfortunately, customers would only see this effect if their box’s ribbon were placed over where the hole would be cut. Otherwise, they would just see a uniformly white section get cut out from the box and fall, no more impressive than the bump effect earlier.

The final challenge was allowing for video recording and playback for the gift-message functionality. Third party assets were used to achieve this but due to the very specific nature of the project, problems and bugs arose. In the case of the video playback API we were using, I identified a bug and created a whole separate Unity project for recreating it. The developers of the API were able to use this and fix the problem. Another API for recording videos created problems on Android that forced me to rework how CookieVision was being launched and closed. On top of that, a wide variety of different Android phones were behaving differently when playing back different gift messages recorded by different means (within the app, within the website, uploaded through website etc.). So to ensure more uniformity in the nature of the mp4 files, I embedded FFMPEG into CookieVision so that it actually duplicates and re-encodes the video files before attempting to play them.