Not a new idea here, but it was still fun to make. While in NYC last week, I spoke at the Boards Summit. They asked if Andrew Bell and I would give a talk on Processing. I made the following demo because I wanted an easy way to showcase what Processing can handle in real-time and also wanted a way to segue into showing off a couple audio-visual pieces I made a while back.
I decided to just start with the standard FFT waveform. I figured it was something the audience (of mostly advertising folks) might be familiar with. The point was to show that the raw FFT data can be a bit unruly with tons of seesaw and spikes. First I pointed out the FFT data is heavily biased towards the lower frequencies which presents itself as a sharp falloff as you go from low to high. As a quick fix, I multiplied the FFT data by a section of a sine wave and leveled off the data. This put the majority of the action in the middle of the waveform instead of the left side.
I then applied an averaging pass which smoothed out a lot of the spikes. Next, a decay attack and falloff was added to keep the spikes from popping in and out too frantically. Finally, I threw the whole data set into a 2D ripple algorithm to give the whole thing a bit more of an organic feel (one of these days, Id like to retranslate this back into audio to see what it might sound like).
Suddenly, the once frantic FFT data was turned into smooth rolling peaks that still reflected the quality of the incoming audio. The look was somewhat landscapey so I decided to push that aesthetic by showing the history of the audio (in the video example below, I am keeping track of the last four seconds of audio). I then added in some appropriate textures and got a nice scrolling landscape that responding fine to real-time audio input. It is particularly entertaining to swing the camera around to the back of the passing landscape and turn it into a rudimentary flight simulator over dynamically generated, audio driven landscape.
Audio-generated landscape from flight404 on Vimeo. Definitely recommend seeing the HD version on vimeo.com. Audio by Akufen (”Even White Horizons” off the album My Way).
And yeah, it does look a bit Bryce3D. Next time, maybe I will even throw in some crystal spheres rising out of a mountain lake! Retro-Awesome!!
(My apologies if the previous description is both too simple and too complex. I don’t actually know much about FFT and how it is derived. I know what the data looks like and that is plenty for me to play around with it.)
How did you make the FFT integrated with Processing? Thanks!
I use the audio analysis engine Sonia by Amit Pitaru which offers access to the FFT data.
Thank you so much! This is an absolutely awesome project & great website. One last thing – was this done real time?
Man, you just converted what was beating on my mind for several months.
I will need to speak with you.
It’s impressive how effectively a simple height-based coloring scheme can create tropical islands or a volcanic landscape–those color palettes are fantastic. I’ve been playing around with using FFTs in some of my recent Processing experiments, and found the “jumpiness” of the data to be unsatisfactory, even after using Ess’s built in damping. You’ve given me a few ideas to fool around with.
Once again, you blow past the boundaries of what I thought was possible in real-time with Processing. I need to work on my OpenGL-fu.
Very impressive and inspirering. It’s not that it is revolutionary – but it’s just so damn elegant. From the choice of music to colors to view angles, there is just so much beauty. Thanks!
Hehe!
and because it remind me a bit an experiment of my friend Iñigo! (you should see it).
That’s nice Robert!
You should try to compute the fft something like 2 or 3 seconds in advance so the peaks hits the camera at the same moment as it is in the music,… a bit more like “Michel Gondry’s Star Guitar” …
Just a thought in the air
Take care!!
[...] n’est pas nouveau, ni révolutionnaire. Mais c’est simplement beau à voir. Fligth404 nous a faire une nouvelle démonstration de Processing à partir d’un son, d’une [...]
Cool man! Not really new, but you made your part making it look pretty. The guys at parkstudio.ru already did that last year with their voxel engine ( http://www.cleoag.ru/labs/flex/musicscape/TerrainSnd.swf ), and I’m sure someone 10 years ago did that already. The tube one was my favourite.
Regarding FFT, I researched a bit about it some months ago. Checked all the Bass and Fmod forums and there was a lot of people saying that usually the first value of the signal is the value you have to multiply the other values with and daba daba. In the end it didn’t make much sense.
So, ignoring that idea I tried a couple of formulas… The red one is just the original data, and green and blue different formulas.
http://mrdoob.com/temp/fft/SoundSpectrum.swf
The formulas aren’t really important here tho. I though the FFT was kind of -compressed- data, well, seems like it isn’t really compressed it just happens to be the actual data and we, humans, are used to spectrum’s like winamp/itunes which makes the signal easier for the eyes. But when searching for Frequency-test MP3 on google I found this page:
http://www.dr-lex.be/software/testsounds.html
Which has some theory about audio, now, the interesting bit here was trying the “16Hz-20kHz-Lin-CA-10sec” mp3, which you can listen here (DON’T LISTEN IT TOO LOUD):
http://mrdoob.com/temp/fft/16Hz-20kHz-Lin-CA-10sec.swf
As you can see, the red box shows the frequency at the same Amplitude all the time. So, seems like our ears have a kind of filter which needs the low-frequencies sounds to be amplified in order to be at the same level as the hi-freq sounds.
So, this quote “These are the same sweeps as above, except they have a constant amplitude. *BEWARE:* this means that the power of the /sound wave/ increases with the square of the frequency, even though the power of the /electrical signal/ remains constant!” I think you’ll be able to get an idea to see how to equalize the values in a theorical way.
I’m not 100% sure of all this, but in case I am, I’m pitty for the animals out there that doesn’t follow our ear system, we must be killing them with that amount of bass!!
[...] all manner of distractions » Blog Archive » Audio-driven landscape. (tags: music video processing visualization) Visit my other sites: Photo Gallery | Insane in the Membrane | Main website « Am I in Some Strange Parallell Universe? [...]
[...] Robert’s at it again! Lindo – a paisagem é formada de acordo com a música! Veja o vÃdeo abaixo. [...]
Wow, that’s cool! I was playing with making just a regular old histogram thingie in Flash; I’m really intrigued about the algorithms you used to manipulate the FFT data. I mean, in my head I think I understand what you mean by “multiply the FFT data by a section of a sine wave and leveled off the data,” but I’d love to know more. Or about the averaging pass, or the decay attack and falloff, or the 2D ripple algorithm. Probably all stuff that will make my head a’splode.
Definitely badass stuff.
Hi: nice discussion.
Nice job also…
I’m really starting with FFT and image mapping and what I found the hardest it;s to make something really expressive (beyond the audio spectrum like a sony cd pl;ayer) happens in a image processing enviroment. Its just all about mapping and the constrains of values you use what makes it work.
I’m working rigth now in a project that uses a particle system controlled by a liric singer but besides I got really got FFT data and good visualizations I cant make it looks fine…
dx
Maybe you can make a kind of “fly your music” flight sim. Hard grinding rock would be really difficult to fly through versus Mozart.
[...] Video and Reference: http://www.flight404.com/blog/?p=143 [...]
[...] Not a totally innovative idea, but the production is really breath taking and hypnotic. THIS is how you do it, it’s a very technical article, I am not much of a programmer, but I was [...]
[...] Audio-generated landscape from flight404 on Vimeo. More info on using Processing to create landscapes from FFT waveform data is here. [...]
absolutely awsome!!!
i really love your work
Now reverse the process! Take the topography of an actual landscape and convert it to the associated soundscape?! The REAL ‘Music of the Spheres’.
Another side thought – Mythbusters did a program where they showed that (at a low level of resolution) the sounds (song) that a potter made while creating a clay pot on the wheel was ‘recorded’ by the micro-topography of the surface of the clay. This could later be recovered (again, at very poor quality) from the surface of the dried pot in much the same way as vinyl records work.
Similar idea in the exploration of the topography of sounds. (ala YES’s 1974 album, Tales from Topographical Oceans)
[...] made with Processing. Audio by Akufen (”Even White Horizons” from the album My Way). More here at Flight404 Tags: Akufen + audio + Björk + Flight404 + landscape + processing + [...]
[...] Audio-generated landscape by flight404, more info: http://www.flight404.com/blog [...]