Upload a video, get a score and timed sound effects: ElevenCreative aligns SFX to on-screen moments and can use the original audio as reference
Synopsis
ElevenCreative launched Instant Soundtracks, which generates an original score for an uploaded video, then analyzes the footage for sound-effect cues and places each generated effect at the matching timestamp on the timeline. Previously music and effects were produced separately and aligned by hand; this update also tightens timing for short-form content and mood switching across scenes, and lets the video's existing audio shape the new score through its genre, crescendos, beats per minute or instrumentation. Effects appear as separate layers in the Music Video Editor, and the API adds add_sfx, separate_tracks and use_audio_as_reference, supporting up to 10 videos per request, 200 MB and 10 minutes combined.
Interpretation
Uploading a video yields both an original score and sound effects synchronized to the picture, with each effect placed automatically at its timestamp on the timeline. Previously music and sound effects were separate production steps, and effects had to be judged and aligned to the picture by hand. The stated example is a whoosh timed to the logo at the end of an ad, indicating effect placement follows moments the model identifies in the footage.
The model first scores the whole video, then analyzes the footage for cues that call for an effect and places each one at the timestamp it identifies. Scoring and effect generation now run in one pass, so effect placement no longer depends on manually marked time points. In the ElevenCreative Music Video Editor, sound effects appear as separate layers that can be reviewed and adjusted independently of the soundtrack.
A new score can take the video's existing audio as reference, with the model analyzing its genre, crescendos, beats per minute or instrumentation to shape the new music. Previously generated scores started essentially from scratch and did not use the audio already present in the video. The stated result is music matching the character of the original rather than starting entirely from scratch.
Perspective
The capability targets users producing video scores and effects in ElevenCreative, ElevenMusic and the ElevenCreative Studio API, with up to 10 videos per request, 200 MB and 10 minutes of video combined. The material states no follow-up experiments or expansion plans.
The material gives one whoosh alignment example and reports no accuracy for effect detection or mood switching; the size of the improvement in short-form timing and cross-scene mood switching is also not quantified.