In this kind of thoughtful glossary, I will try to clarify a few definitions and explain some operational principles, avoiding overly technical considerations that may become obsolete very quickly given to the extreme young stage and rapid evolution of these solutions.

I would like to thank Pasquale Croce (Virtual Production Designer for STS Communication) and Max Rocchetti (Chief Creative Officer at Luno Studios) for their invaluable time and the information they provided during my research on virtual sets in Italy.

Glossary for Cinematographers

VIRTUAL SET

A system of technologies and tools that enables photographic integration between images rendered on an LED screen and reality. The rendering of images occurs in real-time, taking into account the position of the camera in the space and the lens being used.

LEDWALL

The “Wall of LED” can be flat or concave, sometimes can take the shape of a cylinder, often it is used a ceiling, which can be tilled with mobile LED walls as well. The shape and size of the wall of LED are crucial to ensure that the light emitted falls naturally on the set and that reflections appear realistic (as well as to determine how large the scene can be). If viewed from an incredible steep angle, the light from the LED wall tends to shift in hue and shows a lots of artifacts. It can be used to display photographic images, non-real-time renders, and real-time renders.

DOT (PIXEL) PITCH

The linear distance between the diodes on the LED tiles; the larger the screen, the lower the pixel density can be (currently, a range between 2.5 and 2.9 mm is considered optimal).

COLOR DEPTH AND SPACE

Working ranges are from Rec 709 to Rec 2020 with 8 or 10-bit depth, Rec 2020 cannot be fully reproduced by LED walls, therefore it requires an accurate calibration to avoid inconsistent results. Using in 8-bit may result in banding artifacts in the darker areas of the image.

GENLOCK

In order to minimize artifacts, Genlock is signal that perfectly synchronize all devices. If frame rates outside of broadcast standards are used, It can cause issues.

CONTRAST AND BRIGHTNESS

A good brightness for an LED wall is around 1500 NITS. Panels are built and treated to be as black as possible where the diodes are off; however, any light directed at the LED wall raises the black level, so excellent control over lighting and stray lights is necessary.

TRACKING

TRACKER

A system capable of detecting the camera’s position in the space. The most common technology involves an infrared-sensitive camera and IR illuminators mounted on the camera, with a “constellation” of reflective markers in the studio (usually on the ceiling or floor). A dedicated computer calculates the camera’s position in real-time using photogrammetry and passes it to the render engine. If the markers are not visible enough, the position is not calculated accurately and the render will be inconsistent. Currently, a cable from the camera to the computer is necessary. Other technologies involve multiple IR cameras installed in the studio observing markers on the camera; these systems are more accurate but exponentially more expensive and require extensive time for installation and calibration.

CAMERA

CHROMATIC CALIBRATION

A process where a technical LUT is created to ensure optimal communication between the camera and the LED wall (crucial for good image rendering and cohesion).

LENS CALIBRATION

The geometries/aberrations of the lenses are recorded to apply them to the render (crucial if precise alignment between real and virtual scenery is sought)

MOIRÉ

An artifact caused by the isotropic nature of LED walls and the camera sensor. It occurs depending on resolutions, distances, and focal lengths. As resolutions increase and the sensors OLPFs improve, this phenomenon reduces.

DEPTH OF FIELD

The depth of field of the real camera can be manipulated in traditional ways; the aperture and focus distance data can be encoded and passed to the software for computation in the render. Through the Unreal Engine, it is possible to insert the shape and number of diaphragm blades. Calculating depth of field in real-time (linked to the tracker) is very render-intensive, so values are often manually set.

SOFTWARE

ALIASING

An artifact or distortion from under-sampling. It occurs when the render engine’s output resolution is insufficient to resolve the detail. It appears as jagged edges in the output and as a sudden change in single-pixel values (usually at the edges of a solid). The most recent anti-aliasing systems (DLSS) use local supersampling and AI.

FRUSTUM

The field of view of the virtual camera. Frustum, in Latin, refers to the portion of a solid (in this case, a pyramid) between two parallel planes.

UNREAL ENGINE

The most widely used software for real-time rendering in this type of application. Originally developed as a game engine, it achieves an excellent level of photorealism and simulation. It is crucial to determine the project’s frame rate and the LED wall to be used right from the start to optimize the machines and content. It can communicate directly with LED walls through Ndisplay or pass through other software dedicated to projection (e.g., Disguise).

CONCLUSION

Here is a small list (certainly optimizable and likely to evolve) for the benefit of fellow cinematographers:

What should a DOP pay attention to when shooting on a virtual set?

During preparation:

• FPS and color spaces (these are very relevant as they can strain the machines; processing power and data bandwidth)
• Virtual environment lighting (collaboration with those modeling and preparing CGI is crucial).
• Chromatic calibration
• Lens calibration

During shooting:

• Ensure the tracker functions correctly.
• Maintain correct distances from the LED wall.
• Control stray lights and lighting angles to avoid “dirtying” the LED wall.
• Ensure depth of field consistency between the real and virtual worlds.