Moving Pictures!
Final Project

This project was coded entirely in MatLab. This right here is the first thing I would change. Matlab had poor built in functionality for video or audio data. This entire process only took about 15 seconds to run (except for the saving of the video (tack on another 45 to 60 seconds).

As I mentioned in the intro, I consulted a few, real life, artists for creative input. My grandfather, Herb Hogle, and cousin Christopher Tomlinson, semi-professional and professional videographers (they would most likely tell me otherwise, but they work with motion pictures, and they are good at it). And Gywn Rohrer, a local Pittsburgh digital print artist. They were all very helpful in providing ideas, and solidifying features. As a word to them: Yes, it is true, I was not able to implement everything we talked about; however, your input was certainly appreciated, and did in fact help the process of this project, so thank you!

The overall themes for this project are MatLab structures and horrible MIDIs.

Loading Images

To make a slideshow you need images. This process was handled in the program, in a fairly scalable way. A sturcture was made that held the name of the image and its path, the image itself, the date taken and the mean color for the image. The location of the image wan inputted by the user, this is one of the only parameters that needs to be setup before running. The rest is loaded automatically. In this process a normalization of image sizes is also processed. Each image is bound by a max width and height (I used 500 x 400 most of the time). The image is resized and filled in with a black back ground. The information on when the picture was taken is also pulled from the EXIF tags in the files. This can cause some problems, as some viewing programs write over this when changing the image in anyway. For example the images my sister gave me, all of the images that were rotated, had a new date. This information had the strong potential to cause problems later on. Also the mean value for each of the RGB channels is computed and stored for later use.

Color Pairs

There were two ways the images where grouped. First was by color. The average values of color for each image is compared against the rest of the images, to give a 'color distance'. So given an image 'A,' a list is populated of the rest of the images compared to image 'A.' This list is ordered so that the first value is the most 'a like' image, then on to the next 'nearest neighbor,' with the last value being the image that is the most 'un-like' image A. For each and every image a list like this is formed.

Time Grouping

The next way to sort images, is by the date the image was taken. This proved to be a little finicky. A minimum time of 75 seconds was used to determine if images should be grouped together. Every image was computed to every other image, to compute a list of time distances. This list was sorted and ordered groups were formed to be used later.

Establishing Order

Once two metrics were used to establish order, special care had to be taken to make sure that images were not counted twice, and that time grouping was preserved. The user had to select one single image as the starting point and the rest of the images were established as a result of that. Basically the next image would be the next image on the starting image's color pair list. Then the third image would be from the second imaged color pair list... and so on. Inserting entire groups of images if they were in a time grouping. This process if followed until all of the images are collected in to an ordering.

Dealing with Music

To avoid complications with sound, MIDI files were used. This allowed for a more precise definition of the sound. On the other hand this lead to very bad sounding music. The loading of the MIDI file was made possible using functions I pulled off of Ken Schutte's site , I give full credit to him. There is a lot of information contained in the MIDI files, this had to be parsed out and sorted. From this hard coded info, the tempo and time measure of the file could be established. This helped set the basis for the transitions. Some MIDI files are broken up in to tracks with each track representing a different instrument, while others are all contained in one single file. This differentiation caused some problems, but the general idea is that the transitions need to be spaced out over a minimum time distance, and need to occur on a certain length note. These two parameters were also selected by the user, and had a fairly major effect on the outcome. The notes at any given time were also used to compute the fade time, longer transition note = longer fade.

Assembly: Putting it all together

With the information from the ordering and the sound side of things, a movie could finally be constructed. From talking to some experts, simple fades were the best to use, explaining that fancy fades take away from the photos. So a set of transition functions were created to deal with generating the frames necessary for the movie. This is where the ordering and the sound timing were put together. If an image was part of a time group the transition could be half as short as the other notes, giving the quasi-moving image look.

Once all of the frames were assembled, then you would think that the soundtrack would be added to the movie and then exported. But alas, MatLab does not support anything but soundless AVI files, so the actual combining of movie with sound had to be taken care of outside of the program.

Conclusion

This project was really interesting to work on. Unfortunately I had to spend quite a bit of time working on the sound aspect, thus taking up time to work on the photography side. I was happy with the outcome, it was what I was planning. I do think there is a lot of room for improvement.

Future work

Certainly face recognition. This can go a long way. From my talks with real artists, there can be a visual flow established with faces, so one image would have a face in the same area as the next image. This can also lead to cool transitions like the Picasa face movie. Another major down side for the whole project, to me, is the sound. Given a semester in sound recognition and this project could be awesome and actually useable. As it is, right now, the low quality of the MIDI files just distracts from the images, instead of adding to them.

Pros:
Going from Pictures to Movies!
Getting to implement some topics from class

Cons:
MIDI files
MIDI files

Again, Special thanks to: