Nvidia’s next-generation DLSS technology uses artificial intelligence to directly generate characters, objects and even NPCs in games

Nvidia CEO Huang Renxun revealed at the Computex 2024 conference that the future DLSS (Deep Learning Super Sampling) technology will use AI to generate textures, characters, and objects in games. This means that the game’s graphics and characters can be automatically generated by AI, thereby improving the performance of the game.

Improve your gaming performance :

• By offloading work to a dedicated AI processing unit, it reduces the pressure on the main processing unit and increases the game frame rate.

Automatically generate game content :

• DLSS can automatically generate high-quality game textures and objects, improving the quality of game graphics.
• AI-generated NPCs (non-player characters) can make in-game conversations and interactions more vivid and interesting.

New technology :

• Nvidia is developing new texture compression technology that can significantly improve texture quality without increasing video memory requirements.

During a Q&A session at Computex 2024 (reported by More Than Moore ), Huang answered a DLSS-related topic, saying that in the future we will see textures and objects generated entirely through AI. Huang also said that AI NPCs will also be generated entirely through DLSS.

Generating in-game assets via DLSS will help improve gaming performance on RTX GPUs. Offloading the work to the tensor cores will reduce the need for shader (CUDA) cores, freeing up resources and increasing frame rates. Huang explained that he sees DLSS being able to generate textures and objects on its own and improve object quality, just as DLSS super-resolutions frames today.

Nvidia’s DLSS (Deep Learning Super Sampling) technology is one of the most promising features of the Turing architecture. DLSS improves performance and picture quality by leveraging deep neural networks to extract multi-dimensional features from the rendered scene and intelligently combine details from multiple frames to construct a high-quality final image.

Comparison between DLSS and TAA :

• Performance improvement : DLSS provides better performance than traditional anti-aliasing (TAA) at QHD and 4K resolutions, and the effect of DLSS is particularly noticeable at high resolutions.
• Image quality analysis : In some scenes, DLSS’s image quality exceeds TAA, especially in details such as background vegetation. However, in some scenes, DLSS’s edge processing still has jagged edges.

How DLSS works :

• Multi-frame detail fusion : DLSS extracts detail information from multiple rendered frames through a deep neural network and intelligently fills in missing pixels. This method not only improves image quality, but also reduces the pressure on the GPU core.
• AI image processing : Using AI technology for image processing allows Turing GPUs to use fewer samples for rendering, supplement details through Tensor cores, and ultimately generate high-quality images.

Actual effect :

• Advantages of 4K resolution : At 4K resolution, DLSS performs significantly better than QHD, providing clearer and more detailed picture output. In some games, such as Final Fantasy XV, DLSS performs nearly perfectly.
• Limitations : In the first frame of some new scenes, DLSS will expose its true resolution, causing a slight decrease in image quality. In addition, aliasing still exists in some intermediate frames.

Technical challenges and future development :

• Detail processing : While DLSS performs well in most situations, there is still room for improvement in edge processing and detail preservation in some complex scenes.
• Introduction of new technologies : Nvidia is working on developing new technologies, such as AI-driven texture compression, to further improve image quality and rendering efficiency.

Nvidia has been developing a new texture compression technology that takes into account trained AI neural networks, significantly improving texture quality while preserving the video memory (VRAM) requirements of today’s games. Traditional texture compression methods are limited to 8x compression ratios, while Nvidia’s new neural network compression technology can increase texture compression ratios up to 16x.

This approach can provide four times higher resolution than traditional methods while maintaining the same storage requirements. The new technology uses neural networks to compress and decompress textures in games, making the picture quality higher.

Key Highlights :

1. Higher image quality :

• The new approach provides 16 times more texture pixels than traditional block encoding methods, thereby achieving 4 times the resolution (supporting resolutions up to 8192×8192) while maintaining similar storage requirements.
• It can achieve ultra-high resolution up to 8192×8192.

2. Neural network decompression :

• Use a specially trained neural network to decompress textures.
• These neural networks run on the Tensor Cores of Nvidia GPUs and require no special hardware.

3. Actual effect :

• NTC is more time-consuming than traditional methods, but can provide higher texture quality. In 4K image rendering, NTC texture takes 1.15 milliseconds, while traditional BC texture takes 0.49 milliseconds.
• In complex game scenes, this technology can partially offset its computational overhead by simultaneously performing other tasks (such as ray tracing).

Huang’s more interesting aspect of future DLSS iterations is in-game asset generation. Nvidia’s DLSS 3 frame generation technology improves performance by generating frames between real frames. Asset generation is a step beyond DLSS 3 frame generation, with DLSS generating in-game assets completely from scratch. (DLSS needs to know where assets need to be placed in the game world and which assets need to be rendered, but they will be generated completely from scratch.)

**DLSS 3 (Deep Learning Super Sampling)** is an innovative technology from Nvidia that uses AI technology to improve the image quality and performance of games.

AI-driven frame generation

• New frame generation technology : DLSS 3 introduces frame generation technology that uses AI to generate new frames between existing frames to increase frame rates. This process reduces the computational burden on the GPU and makes the game run more smoothly.
• Performance improvements : By inserting AI-generated frames between existing frames, DLSS 3 can significantly increase frame rates in games, especially at high resolutions.

High-quality images

• Detail Enhancement : DLSS 3 intelligently combines details from multiple frames to construct a high-quality final image. Compared to traditional anti-aliasing techniques (such as TAA), DLSS 3 performs better in maintaining image clarity and detail.
• Texture Quality : DLSS 3 uses AI technology to improve texture quality, reduce blur and jagged edges in images, and make game images more realistic.

Wide range of game support

• Compatibility : DLSS 3 is already supported in popular games such as Need for Speed: Unchained, Warhammer 40,000: Dark Tides, Portal RTX, Jurassic World Evolution 2, and The Witcher 3: Wild Hunt, with more to come.
• Backward compatibility : Although DLSS 3 is primarily targeted at the latest Ada Lovelace architecture graphics cards, DLSS 3 games are also compatible with DLSS 2 on previous generation GeForce RTX graphics cards.

Latency Optimization

• Nvidia Reflex : DLSS 3 combines Nvidia Reflex technology to ensure gaming responsiveness by optimizing input latency. This combination increases frame rates while minimizing the increase in latency.

Easy to use

• One-Click Optimized Settings : Through GeForce Experience, users can easily apply one-click optimized settings for games that support DLSS 3, simplifying the process of adjusting image quality and performance.

Huang also discussed the future of DLSS in terms of NPCs. Not only does Huang foresee DLSS being able to generate in-game assets, he also envisions DLSS being able to generate NPCs. He gave an example of six people in a video game, two of whom are real characters, while the other four are completely generated by AI.