RenderFormer-V2: Neural Rendering with Heterogeneous Scene Primitives

Overview

One model for effects that usually need one code path each.

We present RenderFormer-V2, a unified transformer-based neural rendering model, complementary to modern physics-based renderers, that handles diverse light transport effects such as refraction, volumetric scattering, environment lighting, textured and displaced surfaces and out-of-distribution materials, all without per-scene training or specialized code.

RenderFormer-V2 models global light transport as a sequence-to-sequence transformation. Following RenderFormer-V1, it uses a two stage process: a view-independent stage that resolves primitive-to-primitive transport inside the scene, and a view-dependent stage that turns the internal neural scene representation into image pixels. Three things are new. The view-independent stage combines windowed attention with a render-aware attention sink, which is what lets the model scale. The scene sequence admits heterogeneous primitives, including environment maps and participating media. And materials are described by a neural embedding of appearance rather than by the parameters of a fixed reflectance model.

What changed since RenderFormer-V1

	RenderFormer-V1	RenderFormer-V2
Scene primitives	Triangles only	Textured triangles, volume elements, triangular light sources, environment maps and camera ray bundles, all as tokens of one sequence
Materials	Hard-coded per-triangle GGX BRDF	A 9-D latent appearance space independent of any reflectance model, plus per-triangle SVBRDF, normal and displacement textures
Lighting	Up to 8 diffuse triangular light sources	Triangular light sources plus HDR environment maps, each in dedicated tokens
View-independent attention	Full self-attention, O(T²)	Hilbert-curve serialization, a local sliding window of 512, and a render-aware attention sink of registers, lights and summaries
View-dependent attention	Full self-attention over all ray bundles	SWIN windowed attention, window 8 and shift 4, so context size is independent of resolution
Positional encoding	Triangle-specific 9-D RoPE, camera in camera space	One centroid-based RoPE in world space for every token type
Demonstrated scale	Under 4K triangles at 512 × 512	Over 128K primitives at up to 2048 × 2048

01 / Architecture

A scene is a token sequence.

Geometry, participating media, area lights, environment maps and camera rays are each represented as tokens of one sequence.
The view-independent stage resolves primitive-to-primitive transport; the view-dependent stage turns ray bundles into pixel values.
No rasterization and no ray tracing anywhere in the model, and no hard-coded camera transformation.

The RenderFormer-V2 pipeline: scene encoding, a view-independent stage with sparse self-attention, and a view-dependent stage — Scene encoding turns every primitive type into tokens of one sequence. The view-independent stage sorts them along a Hilbert curve and applies sparse self-attention. The view-dependent stage cross-attends ray bundles into the result, applies SWIN self-attention and decodes pixels with a DPT decoder.

What each token carries

Triangle

3 vertices and 3 normals; a 9-D material code; a 32 × 32, 13-channel patch (9 material, 3 normal, 1 displacement) through a frozen VAE

Voxel medium

rotation and per-axis scale; a 4³ grid of RGB scattering, RGB absorption and phase anisotropy

Area light

3 vertices and 3 normals; RGB emittance, in its own token rather than fused into geometry

Environment map

LDR and log-HDR 512 × 256, each through a frozen VAE into an 8 × 4 token sequence; a direction map; log-max intensity

Camera bundle

8 × 8 ray directions, in world space

A single positional encoding covers all of them: RoPE on the primitive centroid, with position-less tokens falling back to the scene centroid, so the whole embedding is translation invariant.

02 / Coverage

One model, many light transport effects.

Refraction, environment lighting, volumetric scattering, textures and displacement, all from a single architecture and a single set of weights.
No effect-specific code path: each effect is learned by example.

03 / Material

A data-driven reflectance model.

No hard-coded reflectance model.
A 9-D latent space encodes appearance: a sphere under a fixed light probe, encoded by a CNN.
The code is never explicitly decoded. The model learns to evaluate reflectance from it directly, under the actual view and lighting.

a sphere under a fixed light probeCNN encoder

9-D appearance code

z_m ∈ [−1, 1]⁹, added straight into the primitive token, and never explicitly decoded.

analytical BRDF parametersPrincipled, GGX, transparentMLP

fitted afterwards: skips the render

The key observation is that the latent code does not need to be invertible to the input BRDF parameters. Because the space is defined by rendered appearance, any material you can render is usable, including measured BRDFs never seen in training. Spatially varying materials come from rasterizing per-triangle 32 × 32 patches of latent material properties, local normals and a displacement height offset, then compressing them with a pretrained VAE.

tSNE visualization of the learned 9-D latent material appearance space — **The 9-D space, under tSNE.** Materials that look alike land near each other, which is the only property the model needs from the space.

Reference and RenderFormer-V2 renders of measured BRDFs encoded in the learned material space — **Measured BRDFs, unseen in training.** Reference and ours, interleaved. The encoder was trained only on the Disney principled BRDF, yet the space is expressive enough to hold measured reflectance.

04 / Attention

What is the attention sink?

Nearby primitives dominate transport, so each attends inside a ±256 window of the Hilbert order.
What is not local, such as illumination and coarse occlusion, goes into the sink instead.
The sink is always attended, and it attends back to every token.

View-independent attention mask

16 registersup to 8 lights1 summary per 64 primitivesgeometryO(T²) → O(T (512 + T/32))

Every row is one query and every column one key, painted in the colour of the key it reads. Sink and window widths are exaggerated for legibility: the real sink is about 1.6% of the sequence.

Why Hilbert order

Sorting the primitives along a Hilbert curve makes neighbours in space localized in the sequence, so a one-dimensional sliding window buys a spatial one.

Three kinds of sink token

16 registers

Free scratch space for whatever is globally true of the scene.

Every light

Each primitive sees each light whatever the distance, which is the attention analogue of importance sampling the lights.

Summaries

Mean-pooling every 64 geometry tokens keeps long-range occlusion in reach, coarsely: distant detail does not matter.

What loss of focus looks like at 83K triangles

RenderFormer-V1 at 83.2K triangles, globally darkened — RenderFormer-V1

RenderFormer-V2 at 83.2K triangles — RenderFormer-V2

Dense attention over 83K triangles has to spread itself over too many tokens, which darkens the whole image. The sink keeps the highlight and the contact shadow.

05 / Evaluation

What does each part contribute?

Held-out scenes, with every variant trained through the 16K-primitive stage.
Sparse attention beats full attention, because dense attention tends to lose focus.
All timings are measured on a single NVIDIA A100. Blender Cycles uses 4,096 adaptive samples per pixel, the setting the training images were rendered with.

Quality vs scene size

Lower is better

RenderFormer-V2RenderFormer-V1

Data table

Primitives	RenderFormer-V2	RenderFormer-V1
4,096	0.0198	0.0244
8,192	0.0305	0.0358
16,384	0.0502	0.0604
32,768	0.0729	0.1500
65,536	0.0873	0.3070
131,072	0.0999	0.4772

Runtime vs scene size

Seconds per frame, lower is better

RenderFormer-V2RenderFormer-V1Cycles

Data table

Primitives	RenderFormer-V2	RenderFormer-V1	Cycles
1,597	0.047	0.057	3.63
2,557	0.050	0.061	3.76
6,397	0.069	0.081	4.00
12,277	0.112	0.142	4.24
21,757	0.180	0.259	4.37
63,277	0.538	1.20	4.86
83,197	0.682	1.88	4.95
127,233	1.12	3.92	5.25

Runtime vs resolution

A 64K-triangle scene

RenderFormer-V2RenderFormer-V1Cycles

Data table

Resolution	RenderFormer-V2	RenderFormer-V1	Cycles
512²	0.525	1.31	5.13
768²	0.791	1.62	9.41
1024²	1.23	2.20	15.1
1536²	2.33	4.08	31.1
2048²	3.92	7.78	53.8
3072²	8.50	24.8	127.3
4096²	14.9	65.1	216.8

On a linear y axis you can see RenderFormer-V1 climbing towards Cycles at 128K primitives while RenderFormer-V2 stays low. Cycles is drawn dashed because it is the path-traced reference rather than a third method.

Render quality vs primitive count

RenderFormer-V1 was trained on 4K triangles, and we found training it for larger meshes to be unstable. The two methods are comparable up to 8K triangles. Beyond 16K, RenderFormer-V1 degrades sharply while RenderFormer-V2 keeps rendering.

Triangles	PSNR ↑	SSIM ↑	LPIPS ↓	FLIP ↓
4K	33.84 / 34.51	.9776 / .9823	.0198 / .0244	.1025 / .0838
8K	33.00 / 32.89	.9682 / .9708	.0305 / .0358	.1230 / .1079
16K	29.96 / 29.36	.9457 / .9418	.0502 / .0604	.1932 / .1555
32K	28.07 / 24.17	.9205 / .8451	.0729 / .1500	.2387 / .3080
64K	26.79 / 18.72	.9020 / .6651	.0873 / .3070	.2609 / .5313
128K	25.82 / 15.88	.8847 / .4604	.0999 / .4772	.3220 / .7466

Each cell reads RenderFormer-V2 / RenderFormer-V1.

Resolution

Because the view-dependent stage uses windowed attention, its context size does not grow with resolution, which makes RenderFormer-V2 easy to fine-tune for higher resolutions. Interestingly, at higher resolution the model also resolves fine geometry more faithfully, such as the dragon's claws and Lucy's face and hands, even though both renders contain exactly the same triangles.

Ablation of the sparse attention

Every component matters, and, surprisingly, the sparse variant outperforms full attention, which we attribute to full attention having to distribute itself over too many tokens. Test scenes follow the training distribution but use held-out environment maps, SVBRDFs and shapes.

Attention variant	PSNR ↑	SSIM ↑	LPIPS ↓	HDR-FLIP ↓
RenderFormer-V2	28.25	.8982	.0997	.4200

w/o attention sink	27.36	.8841	.1247	.4359
w/o sliding window	26.92	.8715	.1289	.4517

sink w/o lights	27.40	.8813	.1081	.4250
sink w/o summaries	27.09	.8754	.1210	.4485
sink w/o both	26.13	.8474	.1514	.4959

summaries 1 : 32	27.78	.8923	.1072	.4231
summaries 1 : 128	26.48	.8690	.1346	.4655
summaries 1 : 256	26.93	.8663	.1316	.4580

full attention	27.91	.8953	.1027	.4287

Training

RenderFormer-V2 is trained end to end on about 10M procedurally generated scenes (roughly 70 TB) spanning 256² to 2048² resolutions and 1K to 64K primitives, each rendered offline with Blender Cycles at 4,096 samples per pixel. A five-stage curriculum first teaches coarse-scale light transport with full attention at 1K primitives, then adds SVBRDFs, environment lighting and volumes, raises the resolution to 512², switches to the sparse attention mechanism, and finally scales the primitive budget to 16K and then 64K at 2048². Total training time is 19 days on 32 A100 GPUs, after which no fine-tuning or per-scene training is needed to render a new scene.

Gallery

One model, every scene.

Every clip and image below comes from the same set of weights, with no per-scene training or fine-tuning.

Video results

RenderFormer-V2 is trained on single frames and nothing in the loss asks for temporal coherence, so every frame below is an independent forward pass. It stays stable anyway as the camera, the lighting and the materials change.

Orbiting camera

Environment-lit spheres above a textured floor. The viewpoint moves and nothing else does.

Orbiting camera, interior

A displaced block in a Cornell box, with the colour bleed from each wall tracking the camera.

Orbiting camera, refraction

A transparent torus, with the caustic staying put on the floor as the view swings around it.

Rotating environment map

Camera and geometry hold still while the lighting turns, so only the shading moves.

Roughness sweep

A continuous path through the 9-D appearance space, from near-mirror to near-diffuse.

Animated medium

A bunny-shaped participating medium, rendered frame by frame.

Heterogeneous effects

Transparent Torus
Refraction through a transparent, rough-metallic torus.

Environment Lit Spheres
Environment lighting on textured spheres above a reflective floor.

Smoky Bunny
Volumetric scattering in a bunny-shaped participating medium.

Three Teapots
Spatially varying materials and displaced surfaces, over 100K primitives.

Displaced Cornell Cube
Displacement mapping learned end to end, with no specialized code.

Spaceship in Smoke
A transparent spaceship embedded in a scattering medium.

Dinner Scene
Indirect illumination from two lamps in a large textured interior.

Cube Pile
A dense pile of glossy cubes with strong inter-reflections.

Complex room scenes

RenderFormer-V1 test scenes, re-rendered

The eight scenes used to showcase RenderFormer-V1, rendered with RenderFormer-V2.

Poster

Everything above, on one sheet.

ECCV 2026, 1400 × 1000 mm at 1:1. Open the full PDF (3 MB). This is the RGB version, which is what you want on screen; the FOGRA39 CMYK file is for the printer.

Limitations

What it still cannot do.

RenderFormer-V2 is limited to at most 8 light sources per scene, a constraint inherited from its training data; more complex lighting is better modeled with an environment map. Like RenderFormer-V1, it is trained on single frames and does not explicitly enforce temporal coherence. Texture resolution is fixed per primitive at 32 × 32, so large triangles can show texture-quality degradation, which subdividing them resolves. Finally, while new primitive types are easy to add architecturally, they are most effective when introduced in the first training stage, and therefore typically require significant retraining.

BibTeX

@inproceedings{zeng2026renderformerv2,
    title      = {RenderFormer-V2: Neural Rendering with Heterogeneous Scene Primitives},
    author     = {Chong Zeng and Yue Dong and Pieter Peers and Lvmin Zhang and Maneesh Agrawala},
    booktitle  = {European Conference on Computer Vision (ECCV)},
    year       = {2026}
}

Chong Zeng was supported by the Stanford Graduate Fellowship. This work was partially supported by the Brown Institute for Media Innovation at Stanford University.