bjarthur/songexplorer repository overview

You have an audio recording, and you want to know where certain classes of sounds are. SongExplorer is trained to recognize such words by manually giving it a few examples. It will then automatically calculate the probability, over time, of when those words occur in all of your recordings.

Applications suitable for SongExplorer include quantifying the rate or pattern of words emitted by a particular species, distinguishing a recording of one species from another, and discerning whether individuals of the same species produce different song.

Underneath the hood is a deep convolutional neural network. The input is the raw audio stream, and the output is a set of mutually-exclusive probability waveforms corresponding to each word of interest.

Training begins by first thresholding one of your recordings in the time- and frequency-domains to find sounds that exceed the ambient noise. These sounds are then clustered into similar categories for you to manually annotate with however many word labels naturally occur. A classifier is then trained on this corpus of ground truth, and a new recording is analyzed by it. The words it automatically finds are then clustered as before, but this time are displayed with predicted labels. You manually correct the mistakes, both re-labeling words that it got wrong, as well as labeling words it missed. These new annotations are added to the ground truth, and the process of retraining the classifier and analyzing and correcting new recordings is repeated until the desired accuracy is reached.

https://github.com/JaneliaSciComp/SongExplorer⁠