Posted on by — Leave a comment

Updated Insights into 3D Gaussian Splatting Techniques for Real-Time Rendering

3D Gaussian Splatting for Real-time Rendering

Introduction

The 3D graphics industry is experiencing a transformation thanks to groundbreaking techniques like 3D Gaussian Splatting (3DGS). Traditionally, creating highly realistic 3D scenes demanded extensive computing resources and required powerful hardware for real-time rendering, limiting access for many artists and developers. However, 3DGS represents an innovative shift, utilizing translucent ellipsoids (“Gaussian splats”) to efficiently represent objects and scenes, allowing for real-time rendering with photorealistic quality.

This article will guide you through the basics of Gaussian Splatting, explore the technical processes involved, and examine its integration in today’s leading software.

What is 3D Gaussian Splatting?

Unlike traditional 3D models based on polygons, 3D Gaussian Splatting represents scenes as clouds of ellipsoids, each known as a “Gaussian splat.” These splats are tiny, semi-transparent 3D objects with properties such as position, color, size, and opacity. By layering these splats together, 3DGS recreates lifelike visuals from any angle, and its unique approach enables high-quality rendering without the computational demands of detailed polygon meshes or complex neural networks.

In practice, this means that 3DGS can render images and animations with a photorealistic quality at speeds suited for interactive applications, from virtual reality (VR) environments to augmented reality (AR) simulations.

3D Gaussian Splatting
3D gaussian splatting - bicycle

How Does It Work?

Creating a scene with 3D Gaussian Splatting involves a series of sophisticated steps, integrating various machine learning techniques and optimization algorithms to ensure that rendered scenes are accurate and visually coherent.

  1. Structure from Motion (SfM): This technique is similar to photogrammetry where multiple images of a scene are captured from different angles to generate a 3D point cloud using software like COLMAP.
  2. Gaussian Transformation: Each point in this cloud is converted into a Gaussian splat with defined parameters.
  3. Differentiable Rasterization: The splats are projected onto a 2D plane for rendering, mimicking how a camera perceives them.
  4. Optimization: Techniques like Stochastic Gradient Descent (SGD) are used to refine the accuracy of the splats.

This structured approach allows 3D Gaussian Splatting to represent complex scenes with lifelike depth, colors, and transparency, while keeping computational demands relatively low.

Here’s a video about 3D Gaussian Splatting from SIGGRAPH 2023:

Benefits and Limitations of 3D Gaussian Splatting

Key Benefits:

  • Efficiency: Requires less storage and processing power compared to dense polygon meshes.
  • Realism: Replicates complex lighting effects, including reflections and depth, with a high degree of accuracy.
  • Speed: Suited for real-time rendering, making it ideal for VR and AR applications.
  • Scalability: Efficiently manages large scenes without sacrificing performance.

Challenges and Limitations:

  • Memory Requirements: While optimized, large-scale scenes can still demand substantial memory resources.
  • Fine Detail Representation: Ultra-fine details may not be captured as precisely as traditional models.
  • Editing Limitations: Adjusting and manipulating Gaussian splat models is less flexible than editing standard 3D models with polygons.

3D Temporal Gaussian Splatting: Extending to Dynamic Scenes

An exciting extension of 3D Gaussian Splatting is 3D Temporal Gaussian Splatting (3DTGS), which incorporates a time component to handle dynamic scenes. This allows for the real-time rendering of high-resolution, dynamic environments. 3DTGS represents motion by modeling shape and position deformations across different timestamps, using a predictive framework to track the movement of each splat.

The technique is sometimes referred to as 4D Gaussian splatting, although most implementations still use 3D Gaussian primitives, simply adding time as a parameter for optimization. Achievements of this approach include the ability to maintain high rendering quality even as dynamic scenes evolve in real time, showcasing potential applications in film, autonomous driving simulations, and other media.

Applications of 3D Gaussian Splatting

3D Gaussian Splatting has been successfully adapted across various domains in computer vision and graphics. The technique’s flexibility allows it to be used for a wide range of applications, from dynamic scene rendering to autonomous driving simulations and even 4D content creation.

  • Text-to-3D using Gaussian Splatting: This application uses 3DGS to convert text descriptions directly into 3D models, making it a powerful tool for rapid 3D creation from textual input.
  • Autonomous Driving Simulations: 3DGS is used to generate realistic, novel views of a scene for autonomous driving, improving the simulation of sensor data for vehicle systems.
  • SuGaR: This method allows for the rapid extraction of precise meshes from 3D Gaussian splats, aiding in the conversion of splat-based representations into mesh-based ones for further manipulation.
  • SplaTAM: A technique applying 3D Gaussian-based radiance fields to Simultaneous Localization and Mapping (SLAM), which enhances real-time environment mapping with high optimization capabilities.
  • 4D Content Creation: 3DGS is also used to generate 4D content, enabling the creation of time-varying 3D models, ideal for animation and simulation purposes.

These applications demonstrate the broad potential of 3D Gaussian Splatting in transforming how we create and interact with dynamic 3D environments.

3D Gaussian Splatting in Modern Tools

With its growing popularity, 3D Gaussian Splatting has been incorporated into a range of software solutions, each leveraging the method’s benefits for various types of projects. Below are several key applications:

1. NVIDIA NeRF (Neural Radiance Fields)

NVIDIA’s Instant NeRF is a well-known application of 3DGS principles. NeRFs use neural networks to render complex environments from multiple images, capturing intricate lighting and textures. NVIDIA’s solution enables real-time rendering, which is especially useful for VR and AR applications, where high realism and fast processing are paramount.

Advantages: Optimized for immersive applications; effective for creating volumetric captures that interact well with lighting and motion.

2. Luma AI and Polycam

Luma AI and Polycam have integrated 3D Gaussian Splatting into their platforms, providing accessible ways to capture and render 3D models. Luma AI focuses on high-quality model creation through a web-based interface, while Polycam allows users to capture environments and create models using mobile devices. These tools open up 3DGS to a broader audience, from beginner 3D artists to professionals seeking quick scene representations.

Advantages: User-friendly interfaces for model creation; supports web-based, high-quality 3D visualizations.

3. Nerfstudio

Nerfstudio is an open-source toolkit for advanced users interested in creating, visualizing, and training 3DGS models. Offering command-line controls and extensive customization options, it allows technical users to experiment with and refine their Gaussian splatting models. Nerfstudio is highly flexible and suited for users comfortable with coding and experimenting.

Advantages: Open-source with high customizability; ideal for researchers and developers seeking a customizable, advanced platform.

4. PlayCanvas and Gauzilla

For those aiming to integrate 3DGS in web applications, PlayCanvas (through SuperSplat) and Gauzilla offer convenient tools. PlayCanvas, a web-based rendering engine, supports Gaussian splats with real-time performance in browsers, while Gauzilla, written in Rust, uses WebAssembly for smooth browser-based rendering.

Advantages: Optimized for web-based rendering; allows for interactive applications directly in web browsers.

5. V-Ray 7

As the first commercial ray tracer to support 3D Gaussian Splatting, V-Ray 7 integrates 3DGS directly with its powerful ray-tracing capabilities. Artists and designers using V-Ray 7 can place Gaussian splats in real-world environments, blending them with 3D models for highly realistic, dynamic compositions. V-Ray is particularly valuable in film, animation, and design fields, where achieving photorealism is critical.

Advantages: Photorealistic integration with ray tracing; ideal for high-end production environments needing realistic visuals.

6. Unreal Engine

Unreal Engine has started exploring the use of 3D Gaussian Splatting, though native integration remains limited and experimental. Currently, the engine allows for Gaussian Splatting through custom techniques and adaptations, often relying on external scripts or community-developed shaders. While Epic Games has not yet released official support, there are plugins and tools in development that let interested users experiment with this technology, particularly for projects that require advanced optimization of complex scenes and real-time visualization.

Advantages: Faster rendering speeds and efficient handling of highly detailed scenes.

The Future of 3D Gaussian Splatting

3D Gaussian Splatting is still developing, and improvements are underway to enhance its rendering quality, versatility, and accessibility. Expected advancements include GPU compatibility, shadow rendering support, and additional rendering options for high-quality photorealism. As more software begins to integrate 3DGS, it will likely become a foundational technology in 3D graphics, supporting use cases across film, virtual and augmented reality, simulations, and beyond.

Conclusion

3D Gaussian Splatting offers an innovative approach to 3D rendering, merging quality with efficiency. This makes it an exciting development for anyone interested in creating immersive, interactive digital worlds. With its increasing presence in leading platforms and applications, 3DGS is transforming workflows in industries that demand high-quality visuals and quick rendering times.

For 3D artists, designers, and developers, the time to explore 3D Gaussian Splatting is now. This method opens up new possibilities for creating, experiencing, and interacting with digital scenes in ways previously considered out of reach.

Sources:

Share
Posted on by — Leave a comment

“Ballerina”. A personal challenge to try 3dsMax + UE5 workflow for quick photorealistic rendering

Ballerina animation - 3ds Max - Unreal Engine

Hello, everyone! Today, I want to share an exciting project where I tested a new workflow using 3ds Max and Unreal Engine. After years of handling every part of production in 3ds Max—from modeling, materials, and lighting to final rendering—I was eager to explore Unreal Engine as a tool for shading, lighting, and rendering. My goal? To speed up my workflow and create photorealistic animations faster than ever. Here’s how the experiment went, step-by-step, and what I discovered along the way.

Project Idea: A Stone Sculpture Ballet Dancer in Nature

For this test, I wanted a small but impactful project that could showcase the capabilities of Unreal Engine 5 for photorealistic rendering in a 3ds Max-based workflow. I envisioned a scene featuring a ballerina sculpted entirely from stones, surrounded by a sunny, natural environment. The camera would move around the sculpture in a gentle spiral, gradually revealing the dancer’s form to the viewer. I wanted to capture ambient sound, subtle musical elements, and finish the entire project within a set timeframe. Here’s how it went down!

Step-by-Step Workflow

Step 1: Learning Unreal Engine Basics

I started by taking an excellent free course on UE5 filmmaking. My goal was to get a solid understanding of Unreal’s photorealistic rendering, material, and lighting capabilities. This foundation was essential to maximize Unreal’s features in my workflow.

Step 2: Writing the Animation Script

Next, I wrote a simple script for the animation (similar to the description above) to keep everything organized. Instead of adding complex animation to the subject, I decided to focus on camera movement around the sculpture to keep the project manageable and stay focused on the new workflow—3ds Max for modeling and animation, and Unreal for shading, lighting, and rendering.

Step 3: Gathering Resources for Modeling the Sculpture

For the dancer model, I used Mixamo to find a 3D mannequin in a ballet pose. I also sourced high-quality photorealistic PBR stone models from Sketchfab to use as the building blocks for the sculpture.

Step 4: Constructing the Stone Sculpture in 3ds Max with TyFlow

To build the sculpture, I used the mannequin from Mixamo as a container, essentially like a mold for placing the stones. I needed a way to “fill” this form with stones, so I turned to TyFlow, a particle simulation plugin for 3ds Max. TyFlow allowed me to quickly set up the stone arrangement to follow the form of the dancer, giving it an organic, lifelike look. I learned the basics from a simple tutorial, which was enough to achieve the effect I wanted.

Step 5: Camera Animation with Spiros in 3ds Max

Creating a complex, spiral camera motion around the sculpture was key to this animation. I wanted full control over the distance and movement of the camera, so I used my own Spiros plugin for 3ds Max. Spiros let me create a logarithmic spiral path for the camera with the exact flexibility and control I needed. I then applied a “path constraint” to the camera and animated both the camera and its target. I also adjusted the animation’s timing and pace to sync well with ballet music, adding to the fluidity of the final result.

Ballerina stone sculpture animation 3ds Max and Spiros plugin

Step 6: Exporting Models and Camera to Unreal Engine

With the model and animation ready, it was time to export to Unreal Engine. I first exported the dancer model as an FBX and imported it into Unreal, where the PBR stone textures needed minimal adjustment—they already looked great. For the camera, I used the tutorial “How to Transfer 3ds Max Animated Camera to Unreal Engine 5” along with the “Unreal Engine 4 – Camera Animation Exporter” script. This combo allowed me to successfully export and integrate the camera animation into Unreal.

Step 7: Setting Up the Scene in Unreal with Quixel Bridge and Polyhaven

In Unreal, I found a stone pedestal model in Quixel Bridge that worked perfectly for the dancer’s base. For the background, I chose a high-resolution park HDRI from Polyhaven. The HDRI provided realistic global lighting and created a natural environment that made the sculpture feel truly embedded in its surroundings. I experimented with a few different HDRIs and settings until I was satisfied with the look.

Step 8: Visual Effects and Final Render in Unreal Engine

I then added some visual effects to the camera in Unreal Engine: autofocus on the tracked actor, bloom, lens flare, and motion blur—all of which contributed to a polished, cinematic feel. After a quick test render, I moved on to the final render. Unreal’s rendering speed was astonishing, completing the 705 frames of animation (1920×823 resolution) in just 50 seconds. The quality and efficiency of Unreal’s renderer completely exceeded my expectations.

Step 9: Post-Production in After Effects

Finally, I added sound effects and music in After Effects for the finishing touches. Ambient sounds of birds, soft ballet music, and some brief closing credits completed the piece.

Ballerina animation - 3dsMax - TyFlow - Unreal Engine

Final Thoughts: Is This Hybrid Workflow Worth It?

This project showed me that a 3ds Max + Unreal Engine workflow is not only feasible but also highly efficient for photorealistic animation. Unreal provided the speed and quality I was hoping for in a renderer, making it an excellent option for projects with tight deadlines. I’ll definitely keep exploring this hybrid approach for future work!

If you’re thinking about using Unreal for rendering in a 3ds Max pipeline, give it a shot! You might just be amazed at the results.

Share