Jeremiah Baker - Relief Mapping

Concept:
For this assignment, I wanted to stick with the visual concept from my previous compositions, but incorporate sound somehow. I really liked the shifting panels from the Lion tape mapping, and decided to use that idea for this project. I thought it would be cool to use the abstract piece as a volume meter with a music video running in the background.

Process:

building the relief

First, I teamed up with a couple other students from the class to create an abstract piece. After they had finished the model, David and I fabricated it and placed it in context. I then set up a template file to work from. To do this, I ran a spatial scan on the piece to get its position on the wall, and then lined up the 3D model in MM. This took a lot of tweaking to get right. After I got the model 90% there, I used 'edit mesh' to get it to 100%.

original music video

template file setup

Next was the fun part, the content. The first thing I had to do was pick a music video to use. I decided to go with 'Rescuer' by Rend Collective. I then rebuilt an After Effects document using the lines from the 3D model and the reference image from the spatial scan. Once I had all of the layers set up, I wrote a little bit of code to make everything work properly. I created a null object that moved based on audio levels. I then added opacity switches to each of the layers, and created a rule that they would be on if the null was above them.

That by itself was pretty cool, but I thought it could be a little more interesting, so I added a light that casted a shadow on the wall behind the piece. This turned out way cooler than I was expecting. The shadows ended up only casting behind the layers that were turned on, made for some pretty cool effects. I also added the music video onto the background, so that I was using as many pixels as I could.

In MM, I added the virtual light on the 3D object and controlled it with the audio from the video. This made the lighting on the sculpture change with the music, composited over the AE video. I wanted to make the project repeatable, so I built the file in such a way that any music video could be dropped in to be used.