Virtual Filmmaking Inside AI-Generated Gaussian Splats
The world of virtual filmmaking is evolving at lightning speed, and AI-generated 3D environments are at the forefront of this transformation. One exciting tool leading the charge is Spell by Spline, which allows creators to generate 3D spaces known as Gaussian Splats, or "splats," with minimal effort. These low-fidelity environments provide a groundbreaking way to experiment with 3D storytelling and real-time filmmaking techniques—and not a polygon or voxel in sight! Here, we explore how to use splats effectively for virtual production, integrate them with Unreal Engine, and consider the future potential of this technology.
What Are Gaussian Splats?
Gaussian Splats are AI-generated 3D environments that can be created from simple inputs like images. Using Spell, filmmakers can quickly generate immersive worlds and then navigate through them with ease. While current splats are relatively low-resolution compared to traditional 3D assets, they offer tremendous creative possibilities and are improving rapidly as AI tools evolve.
How to Use Gaussian Splats for Virtual Filmmaking
1. Generate a Splat
Using Spell, create a Gaussian Splat based on your concept. This could be a cityscape, a forest, or an abstract environment tailored to your story.
2. Navigate and Record
In Spell, you can "fly" through the splat using a mouse and keyboard to frame and capture your shots. While the current navigation system can feel limiting for smooth camera motion, it provides a great starting point for experimenting with virtual filmmaking.
3. Exporting Splats to Unreal Engine
To move beyond Spell’s limitations and achieve professional-quality results, you can export the splat and import it into Unreal Engine:
Export the Splat: Spell allows you to download your splat as a
.splinegsfile. Use the "Export" option in Spline to save your project in a format compatible with Unreal Engine, such as GLTF, GLB, or FBX.Import to Unreal Engine: Open your Unreal project, navigate to the Content Browser, and import the exported file. Unreal’s tools will let you refine and enhance the environment further.
4. Professional Shooting with Camera Tracking
Once the splat is in Unreal Engine, you can integrate real-time camera tracking equipment (e.g., CINCRAFT or MARS) to shoot live-action footage inside the virtual set. This combination of real-world and virtual elements allows for seamless real-time compositing, where actors can perform against a green screen and be "placed" into the virtual world in real time.
Current Challenges and Future Potential
Low-Fidelity Limitations
Today, Gaussian Splats are not as detailed or polished as traditional 3D assets. Navigation within Spell can also be clunky when relying on a mouse and keyboard, making smooth camera motion difficult.
The Future of Splats
Despite these limitations, the potential for growth is immense. As AI tools like Spell evolve, we can expect:
Higher Resolution and Detail: AI will learn to generate photorealistic environments with intricate textures and lighting.
Real-Time Changes: Dynamic updates to the environment during production, enabling on-the-fly adjustments.
Animation: Integration of animated elements such as moving water, characters, and weather effects.
Advanced Lighting: Realistic lighting systems that respond to changes in the virtual set, enhancing the cinematic quality of the scene.
Why This Matters
This approach to virtual filmmaking is not just innovative, it’s democratizing. Tools like Spell and Unreal Engine empower indie filmmakers and small studios to achieve results that rival major productions, all while reducing costs and expanding creative possibilities.
