Preliminary Project Research - Wekinator

For my project in this class I've solidified on creating a program that will respond to a series of input parameters and game controller inputs to create a sound scape that corresponds to an individual's gameplay. 

The initial proposal for this project included creating a program in Chuck, hardcoding the inputs from the controller and creating the response matrix. While straightforward, there is certainly a limitation in the variety of inputs that can be processed, specifically in games with fixed sequences, like many fighting based games.

To provide myself more flexibility and variety of inputs without having to hardcode combination detections into Chuck, I explored a program called Wekinator, which is a machine learning engine designed to rely not on code but on more unique inputs, including both musical or sonic inputs and game controller inputs. 

The basic process as I've come to understand it thus far is that the user will first select the type of input Wekinator will be processing. This could be visual, such as for a gesture based project, or discrete inputs like button presses, which will be what I look to use. This input is then connected to some typeof electronic output, such as a Chuck program. From there, a series of "classes" are specified for Wekinator to begin identifying; in this case, these classes would include a variety of single and multi-button inputs. The user then needs to provide the Wekinator with a variety of training data examples, allowing the algorithm to begin classifying the examples in their appropriate classes. Once this training has completed, the user then evaluates the accuracy of the Wekinator's identification, by examining the class distribution and testing it on new inputs. New training data will then be provided, or training sensitivity or class distinctions adjusted, and the machine retrained. This process would then continue until the algorithm has the desired level of accuracy and consistancy. 

From there, these class outputs would be taken by the Chuck procedure I will write and thus, generate a soundscape. The benefits of using Wekinator are not only apparent in its design being tailored towards unique inputs like a game controller, but also in the program's interface not relying on code, making it more streamlined to learn for someone like myself who isn't as well versed in some of the common programming languages.

In the future I'll be looking into open source games to determine the feasibility of pulling game event data from the gameplay, or if this program will need to really only on the controller inputs. Specifically, this could provide some unique sound cues, such as a special move triggered by a game state, or a character death. Additionally, for games specifically involving a character selection, I would like to investigate the possibility of the initial parameters (which will help shape things like the general tone, pitch range, and note duration for the project) depending on the character chosen, thus providing a different texture to the sounds available for the soundscape depending on the character selected, in addition to the differences created by both that character's and the user's play style.