2022 was the miracle year of generative AI: mid-year saw Stable Diffusion’s open-source release ignite text-to-image generation, and by year-end, ChatGPT emerged. Sandwiched between these two milestones in September, Google published a paper that stirred waves of equal magnitude within the research community, though with less public visibility—DreamFusion, the first truly viable solution for "text-to-3D."

The problem it solved can be explained in one sentence: Text-to-image works because there are billions of images with textual descriptions on the internet serving as training material; yet high-quality 3D models with text annotations number only in the hundreds of thousands worldwide—without teaching materials, how do you teach AI to learn 3D? DreamFusion’s answer was beautifully, almost cunningly simple: stop teaching it directly. Instead, let the already-trained 2D model act as the teacher.

What is DreamFusion?

DreamFusion is a research project published by Google Research and UC Berkeley in September 2022 (paper: DreamFusion: Text-to-3D using 2D Diffusion, authors Ben Poole, Ajay Jain, et al.). The official website, dreamfusion3d.github.io, serves as the paper’s showcase page. It is not a product—there is no upload box, no subscription button—but it is the technological ancestor of all subsequent text-to-3D products.

The paper received the Outstanding Paper Award at ICLR 2023, underscoring its academic significance.

Core Idea: SDS, Letting the 2D Teacher Train the 3D Student

DreamFusion’s soul innovation is called Score Distillation Sampling (SDS). Its principle can be clarified with an analogy:

Imagine an art teacher who cannot see in 3D but only judges 2D images (a pre-trained Imagen text-to-image model), and a lump of random 3D sculpting clay (a 3D scene represented by NeRF, or Neural Radiance Fields). The process is:

Take "photos" of the clay lump from random angles (differentiable rendering produces 2D images)
The teacher looks at the photos, judges "does this look like 'a corgi wearing sunglasses'?" and points out the gaps
Adjust the clay lump in reverse based on the teacher’s feedback
Change the angle, take another photo, judge again, adjust again—iterating thousands of times

Eventually, the lump satisfies the teacher from every angle—it has "grown" into the 3D object described by the text. No 3D training data is needed at all; the 2D model’s knowledge is "distilled" into the 3D representation. This approach, which transforms a "generation problem" into an "optimization problem," is elegant enough for textbooks.

Effects and Three Major Woes of That Era

The results showcased in the paper—rainbow-colored peacocks, raccoons on motorcycles, medieval castles—were stunning in 2022: 3D objects conjured from nothing, viewable from 360 degrees. But the limitations were equally conspicuous:

Blurry quality: Details were fuzzy, surfaces rough, with a vast gap compared to the exquisite quality of contemporary 2D generation;

Slow speed: Optimizing a single object took hours on a TPU, rendering it practically useless;

The Janus problem: The most famous flaw—generated animals often had "a frontal face from every angle," with another face visible from the side, like the Roman two-faced god Janus. The root cause is that the 2D teacher had only seen the "good angles" of objects and lacked supervision for "what the back should look like." This term later became standard terminology across the field.

Historical Significance: The Gate Opener

DreamFusion’s true value lies not in what it generated, but in proving that this path is viable. After the paper’s publication, improvements flooded in like a tide:

Magic3D (NVIDIA): Two-stage optimization, improving both resolution and speed
ProlificDreamer: VSD method significantly improved quality and diversity
Zero-1-to-3 / Multi-view diffusion routes: Addressed the Janus problem at its root
threestudio: An open-source framework integrating various methods, becoming the standard experimental bench for researchers

Today, commercial text-to-3D products like Meshy, Tripo, and Luma can generate usable models with textures in seconds. The entire technological lineage of this industry traces its root node to DreamFusion. It holds a status comparable to the original DALL·E in the text-to-image field: no one uses it directly anymore, but no one can bypass it.

Want to Use Text-to-3D Practically?

Match your needs with the right entry point:

For usable 3D assets (games, printing, design): Use commercial products directly—Meshy, Tripo3D, Luma Genie, etc. Their quality and speed far exceed the original DreamFusion, and most offer free tiers.

For research or hands-on experimentation: The threestudio framework on GitHub integrates open-source implementations of DreamFusion and its successors. With a GPU, you can reproduce this chapter of technological history.

To understand the technical principles: Read the original paper; the derivation of SDS is one of the most worth-reading passages in generative AI. The video gallery on the official website also preserves the "stunning" original state of that era.

Who Should Understand DreamFusion?

Researchers and students in 3D/computer graphics: Foundational literature for the field, an unavoidable starting point.

Technical decision-makers in gaming and film industries: Understanding the evolution of text-to-3D technology allows you to accurately judge "when AI-generated assets can enter production pipelines"—and tracing this lineage must start here.

AI history enthusiasts: SDS is a classic case of "leveraging old capabilities to unlock new dimensions." This paradigm recurs throughout AI development, and DreamFusion is the cleanest teaching example.

Curious general readers: Even if you only remember the analogy of the "2D teacher training 3D clay," your understanding of AI generation technology will deepen by a level.

Pricing

The paper and official website showcase are free and public; open-source reproduction is free (bring your own GPU); actual commercial text-to-3D tools each have free tiers and subscriptions, subject to their respective official websites.

DreamFusion is the type of "milestone research" that is itself rough, slow, and flawed, yet chisels open the first crack for an entire direction. Behind every product today that generates exquisite 3D models in seconds stands that blurry, viewable-from-every-angle corgi from the autumn of 2022—this is how technological history is written.