subscribe to cinematography world

Assimilate announces open beta of new Live FX for live compositing of virtual productions

Oct 18, 2021

Live Compositing enables real-time, on-set pre-visualisation of final content 

Assimilate has announced an open beta for its new and innovative Live FX software that enables real-time, live compositing for green-screen and LED wall-based virtual productions on set. Live FX is also ideal for creating quick comps when scouting locations. Live FX features the latest technology in keying, camera tracking, DMX light control, Notch Block integration, and a live link to Unreal Engine that simplifies on-set virtual production for not only previsualisation, but even final pixel for in-camera VFX. By recording any incoming dynamic metadata along with the imagery, Live FX can automatically prep all content for VFX/post, saving time and money. Filmmakers and artists – from freelancers to big studios – can now work in a familiar film/video workflow rather than in a programming style environment. 

“The future of filmmaking is in virtual productions for the obvious reasons of time and cost savings in reduced logistics, travel, equipment, shipping, location set-ups, and more,” said Jeff Edson, CEO at Assimilate. “Assimilate is supporting this process by pioneering live compositing on set, which provides a significant leap in productivity and creativity for users. Live FX enables the most realistic pre-visualisation up front in the workflow before the content moves down the pipeline and into post production. Additionally, we wanted to develop a tool that meets both needs of running on laptops for doing quick comps for location scouting or on-set quick checks, as well as on powerful systems to drive entire productions.” 

DMX lighting control: Remote control of ARRI SkyPanels based on image content sent to LED wall

Camera Tracking

Live FX supports a broad range of tracking systems for a variety of on-set situations, such as indoor, outdoor, big stage or small set, tethered or untethered. To track camera movement, Live FX allows use of Intel RealSense tracking cameras, Mo-Sys StarTracker, HTC Vive trackers, and even gyro and ARKit technology inside smartphones mounted to the cinema camera. Even without a dedicated tracking device, Live FX is able to create camera tracks using its own live tracking algorithm. All tracking data can be refined and synced to the incoming live camera signal, so all data stays in sync throughout the entire pipeline. Through its built-in virtual camera calibration, Live FX is able to calculate lens distortion and create final composites live on-set. 

Easy Green Screen Replacement

The green screen can be easily replaced on set with a virtual background of choice – whether it’s a simple 2D texture, 360 equirectangular footage, a 3D Notch Block, or a live feed from another camera or Unreal Engine. This is made possible by a variety of keying algorithms within Live FX. Users can create and combine multiple keyers using the layer stack and add multiple garbage masks to their composite. RGB and Alpha channels can be output separately via SDI or NDI to be captured by other tools on-set, such as Unreal Engine. 

Live FX on-set: Preparing a live green-screen composite

LED Rear Projection Creates Photorealistic Environment

Live FX supports colour-managed playback of any footage, at any resolution, and at any frame rate.
It allows projection of conventional 2D files, 360 equirectangular footage, Notch Blocks, or live feeds, such as 3D environments directly from Unreal Engine, onto LED walls. Once the content is loaded, it can be colour graded, framed and tweaked further.

DMX Lighting Control 

For a realistic and natural look, accurate lighting is a key aspect of virtual productions. The DMX lighting control engine in Live FX allows users to mark a region of interest on the footage that is being projected onto the LED wall. Live FX analyzes and averages the colour inside the marked region for every frame and sends the colour down to the stage lighting, making it easy and straight forward to adjust the lighting of any scene.  Think big – not only bright blue sky, but even dynamic scenes like a house on fire with flickering ARRI Sky panels or a car-chase scene through tunnels are now easy to create and set up.

Live FX control of studio lighting via DMX, based on back-projected video content for a car-chase scene

Live Links

Live FX features Live Links, which allows any piece of dynamic metadata entering the system to be used inside a live composition. The metadata comes in from a dedicated tracker system, via the SDI signal of the camera, or through the Open Sound Control (OSC) protocol from another system. Examples include using the dynamic lens metadata that comes in from the camera through SDI to directly animate the blur of a texture. Or, link the camera’s own gyro metadata to the virtual camera inside Live FX to link camera movement completely without additional tracking devices.

Timecode Sync Signal Routing

One of the most important aspects in live compositing is managing latency. Live FX allows for compositing multiple streams together: Live streams, dynamic renders like those from Unreal Engine, and any pre-recorded footage. If an input is timecode-managed, Live FX can auto sync multiple feeds by delaying the fastest feeds to ensure 100 percent frame sync in the composite. The camera tracking can be manually delayed to more precisely align the tracking data with the video feed. The additional latency from SDI output is kept to a minimum in Live FX and the output can be genlocked. Users can also exchange image data using NDI, or even direct GPU texture sharing with zero latency.

Live FX offers various keyers that can be combined to create the ultimate key. RGB and alpha channels can be output separately

Recording, Metadata, and Post Prep

Anything that goes into Live FX can be recorded and instantly played back for review. The media can be recorded in H264, Avid DNx or even Apple ProRes, including alpha channel. In the case of multiple live feeds, users can decide to record the final baked result or the individual raw feeds separately. And not only the media is recorded; all metadata that is captured and used in the live composite – including camera tracking data – is saved to file as well. This all allows for a unique and highly efficient workflow: Start with the live composite and record the raw feeds and metadata separately. As soon as the recording stops, Live FX automatically creates an offline composite that is ready for instant playback and review. The offline composite is a duplicate of the live composite, using the just recorded files instead of the live signals, and includes all other elements and animation channels for further manipulation. 

Once the high resolution media is offloaded from the camera, Live FX automatically assembles the online composite and replaces the on-set recorded clips with the high quality camera raw media. 

Live FX powerful HSV-Keyer includes camera tracking to create final pixel in-camera

These online composites can be loaded into Assimilate’s Scratch to create metadata-rich dailies or VFX plates in the OpenEXR format, including all the recorded frame-based metadata, which saves time in post-production.

“Live FX is not an island. It’s part of Assimilate’s extensive Scratch ecosystem of advanced digital-cinema tools that are designed not only to simplify workflows and increase productivity for DITs, post-production artists, directors, and all filmmakers, but also to achieve the highest quality productions,” said Mazze Aderhold, chief strategic workflow engineer at Assimilate. “For Live FX, Assimilate is now expanding its connections to third-party software with an Unreal Engine plug-in for metadata communication, support for Open Sound Control (OSC) protocol, and direct GPU texture sharing to enable ultimate flexibility on-set. We’ve also added support for a wide range of different camera tracking devices and workflows that equally serve high-end productions and indie filmmakers with cost-effective solutions for virtual production – all running on off-the-shelf (OTF) hardware. This makes Live FX the easy go-to solution for all virtual productions.”

Camera tracking in Live FX: Supports Mo-Sys StarTracker, Intel RealSense, HTC Vive, and even smartphones via gyro-sensor and ARKit implementations

Participate in Open Beta

To participate in the open beta, simply visit the Assimilate webpage and click the “Try” button for Live FX. Fill in your email address and receive a return email with the download-link and license key. 

For beta users to get started immediately, watch the Live FX short hands-on video tutorial series. 


vortex 4

cinematography world

Related Posts

Euro Cine Expo sells out

Euro Cine Expo sells out

Euro Cine Expo, the premier event for film and TV production in mainland Europe, has officially sold out its floor space for the upcoming exhibition at the...