Semantically consistent Video-to-Audio Generation using Multimodal Language Large Model


Gehui Chen1 †, Guan'an Wang† ‡ , Xiaowen Huang1, 2, 3 *;, Jitao Sang1, 2, 3,

1School of Computer Science and Technology, Beijing Jiaotong University

2Beijing Key Lab of Traffic Data Analysis and Mining, Beijing Jiaotong University

3Key Laboratory of Big Data Artificial Intelligence in Transportation(Beijing Jiaotong University), Ministry of Education


†Equal Contribution

‡Project Lead

*Corresponding Author


[paper] [github]

Abstract

Existing works have made strides in video generation, but the lack of sound effects (SFX) and background music (BGM) hinders a complete and immersive viewer experience. We introduce a novel semantically consistent video-to-audio generation framework, namely SVA, which automates the process of generating audio semantically consistent with the given video content. The framework harnesses the power of multimodal large language model (MLLM) to understand video semantics from a key frame and generate creative audio schemes, which are then utilized as prompts for text-to-audio models, resulting in video-to-audio generation with natural language as an interface. We show the satisfactory performance of SVA through case study and discuss the limitations along with the future research direction.

Introduction

The overview of our proposed SVA framework is as follows. Initially, a key frame (highlighted in red) is randomly selected from the video frames. We then prompt Gemini Pro to generate a SFX and BGM scheme, which comprising three different SFX descriptions and one BGM description. Subsequently, these descriptions are inputted into AudioGen and MusicGen respectively, resulting in corresponding SFX and BGM waveform files. Then we run post-processing for noise removal and reduction. Finally, all multimodal data, including video frames, SFX waveform files, and one BGM waveform file are merged to create the video with audio.

Samples

Here are some samples using videos generated by Sora. The "User Input" means user's query guiding the SFX & BGM style. The "Generated prompts" are generated by Gemini Pro following the user input automatically.

example 1

User Input Make me some proper and impressive schemes Create blue and sorrowful atmosphere in this video I wanna happy and energetic experience
Output
Generated Prompts (auto) Seagulls calling and waves crashing gently(SFX) Seagulls crying plaintively amidst a storm(SFX) Seagulls calling as they fly overhead(SFX)
Wind blowing softly through grass(SFX) Waves crashing against the lighthouse, creating a thunderous roar(SFX) Waves gently lapping against the rocky shore(SFX)
A serene and calming piano melody with the gentle sound of ocean waves in the background(BGM) A somber cello melody, accompanied by a gentle piano arpeggio. The music swells gradually, with a hint of longing and regret in its notes(BGM) A cheerful and optimistic folk song with acoustic guitar, harmonica, and uplifting vocals(BGM)

example 2

User Input Make me some proper and impressive schemes Create blue and sorrowful atmosphere in this video I wanna happy and energetic experience
Output
Generated Prompts (auto) Faint, high-pitched humming (SFX) Wind howling like a lost soul across icy tundra (SFX) A cheerful melody played on a ukulele (SFX)
Flickering holographic lights (SFX) Snow crunching beneath heavy, weary boots (SFX) distant clinking of wind chimes (SFX)
Otherworldly and enigmatic music with a mixture of electronic drones, ethereal vocals, and eerie sound effects, creating a sense of mystery and awe (BGM) A sparse and haunting piano melody with ambient drones, evoking the frigid and desolate expanse of the Arctic wasteland (BGM) A serene and uplifting piano melody, accompanied by the gentle strumming of a guitar and subtle, shimmering synths (BGM)

example 3

User Input Make me some proper and impressive schemes Create blue and sorrowful atmosphere in this video I wanna happy and energetic experience
Output
Generated Prompts (auto) Mammoths stomping through the snow (SFX) Wind howling desolately across an icy plain (SFX) hums and chirps of birds overhead (SFX)
Wind whistling through the trees (SFX) Muffled crunch of snow beneath heavy paws (SFX) distant clinking of wind chimes (SFX)
A slow and majestic orchestral piece with a steady drumbeat, soaring strings, and haunting vocal harmonies, evoking the grandeur and resilience of the mammoths' journey (BGM) A sparse and haunting soundscape, with atmospheric pads, ethereal chimes and occasional, mournful string or piano notes (BGM) A gentle and heart-warming acoustic guitar melody, with light percussion and a touch of strings to create a cozy and inviting atmosphere (BGM)

example 4

User Input A scheme for a McDonalds ad short video Star War Blaster Battle, space ship firing I wanna happy and energetic experience
Output
Generated Prompts (auto) The sound of a crackling fire (SFX) Laser swords clashing and humming (SFX) Waves gently lapping at the shore (SFX)
The sound of rain falling on leaves (SFX) Blaster fires (SFX) Seagulls squawking and circling overhead (SFX)
A cozy and heartwarming folk track with an acoustic guitar melody and a soft violin accompaniment (BGM) A grand orchestral arrangement with thunderous percussion, epic brass fanfares, and soaring strings, creating a cinematic atmosphere fit for a heroic battle (BGM) A cheerful and upbeat reggae song with a steady beat and a melody that evokes the warmth and positive vibes of a tropical beach (BGM)

example 5

User Input Give me a soundtrack for a documentary about the underwater world, showcasing the awesomeness of nature The mysterious and profound depths of the ocean that inspire awe and a sense of dread A Joyful Underwater Adventure
Output
Generated Prompts (auto) Crackling of coral reefs underwater (SFX) distant whale calls echoing in the depths (SFX) rustling leaves in a soft breeze (SFX)
Gentle splashing of waves (SFX) cracking and groaning of unseen creatures (SFX) soothing trickle of a nearby stream (SFX)
A serene and majestic composition featuring shimmering synths, elegant piano melodies, and sweeping strings (BGM) A haunting and ominous drone with dissonant strings and pulsing bass, evoking a sense of dread and the vastness of the ocean (BGM) A bright and uplifting melody with acoustic guitar, playful flute, and gentle percussion (BGM)

example 6

User Input Make me some proper and impressive schemes a Cyberpunk citypop a sweet dream leading to old memory
Output
Generated Prompts (auto) Wind rustling through trees (SFX) Whooshing of cybernetic enhancements (SFX) Flickering sound of an old film projector (SFX)
Birds chirping merrily (SFX) Mechanical whirring of robotics (SFX) Gentle crackling of a fireplace (SFX)
A soothing and calming acoustic guitar melody with a gentle rhythm and light strings accompaniment (BGM) An atmospheric electronic track with synthwave arpeggios, distorted guitars, and a cyberpunk vibe (BGM) Soothing and nostalgic piano piece with a warm and comforting sound (BGM)