Lumiera (as seen) from Outer Space

Lumiera is not so much defined in terms of features. Instead it can be better seen as a workshop in which the emphasis is on individual basic building-blocks that can be combined together in interesting ways to form more complex structures. These basic modules, entities or objects each have a distinct type which explicitly identifies the possible connections between modules. Within these constraints, any combination is possible without hidden restrictions.

Lumiera is not a set of Lego bricks, nor is it a business application driven by finite usage stories.

These building blocks within Lumiera create a moderate level of abstraction; a user may use the GUI to manipulate several elements directly — not only the clips and effects, but also individual effect parameter automation, masks, layering, and even the placements ← used to stitch the objects together, which is comparatively low-level. Such moderate abstraction encourages the user to understand the inner workings, while shielding the user from merely technical details such as header format conversion or access to individual channels.

To complement this approach, Lumiera does not rely on hard-wired, global conventions — rather we allow the building up of project specific conventions and rules ← to fit a given requirement and preferred working style.

Lumiera will be supplied with a conventional template and a default configuration to ease the learning curve for new users.

Processing video (and audio) data can be conceived as a graph (more precisely a directed acyclic graph). In this model of video processing, data flows along the edges of the graph and is processed at the nodes.

The following figure depicts manipulating video data as a graph. The nodes of the root of the graph are where data input occurs. From there, the data moves on to the next nodes: the direct siblings of the root. Here, the video data is pre-processed. All other operations on the data can be represented by nodes, and data flows from one operation to the next along the nodes of the graph.

When we look at this model we discover that we only need to build ← such graphs, the nodes themselves can be seen as black boxes and will be implemented by plugins ←. Moreover one can pre-configure subgraphs and handle them as a single entity ←.

In Lumiera everything will be translated into such a graph. Footage will be demultiplexed ← at the first node, thereafter passed to the actual processing nodes for calculating effects, masks and layering, and finally be forwarded to the encoding stage ← and on to the multiplexer ← which assembles the final video.

At a first glance, it looks fairly natural to set up the graphs ← as described above. Data can then be pushed into the system through the input nodes and the final result can be seen at the output node. Several multimedia frameworks use this approach. However this scheme exhibits a number of shortcomings which make it inappropriate for non-linear video editing.

Lumiera takes a different approach. Data is pulled through a pipe: any rendering request starts at the output node and makes its way back up through the graph to the inputs. This scheme offers a number of advantages over the naive scheme.

Rendering A/V data can be computationally intensive. To avoid duplicated or wasted work and ensure it is possible to render on time, Lumiera employs a sophisticated means of using cache ← and profiling ←.

The viewer is an area where footage and the assembled edit can be played back and investigated. Obviously, support for audio will also be provided. As there are many sources that can be displayed, a viewer is attached to a source via the viewer switch board. Timelines, probe points, wire taps and individual clips are examples of sources that can be attached to a viewer. Moreover, the number of viewers open at any one time is only limited by the hardware, and each viewer can be collapsed or hooked up to a beamer or monitor.

The preliminary layout in the current GUI is rather provisional — a final decision has yet to be taken on where the transport controls will be located. In later versions of the standard GUI, the transport controls will change. There is even a possibility that the transport controls will be allocated their own GUI element.

Transport controls are devices that are controlled by widgets or by some input device (MIDI, control surface, etc) so the look-and-feel of the transport control widget may vary, depending on what the transport controls have been assigned to. Irrespective of their look-and-feel, they are connected to a play controller.

A play controller coordinates playback, cueing and rewinding. Transport controls are ganged when they attach to the same play controller.

A timeline is a container that provides a time axis and an output. There may be connections to various output designations, each with a different configuration. A more elaborate production setup will use several timelines, one for each distinct output configuration. A timeline does not temporally arrange material, this is performed by a sequence, which can be connected (snapped) to a timeline.

A typical film will define many sequences, but will only have a few timelines. A sequence contains a number of tracks which are ordered in a hierarchy. Tracks do not have any format associated with them and more or less anything can be put into a track. Consequently, audio and video material can equally be assigned to a track, there is no discrimination between audio and video in the Lumiera concept of a track.

A timeline must be assigned to viewer and transport control if playback viewing is desired. In most cases, these connections are created automatically, on demand: Just playing some footage for preview creates a simple internal timeline.

The GUI provides a separate bus view, showing the global pipes. These are the master busses for each kind of output (image and sound) and collect the output of subgroups together in a manner similar to an audio mixing desk. Any pipe is just a means of collecting, mixing or overlaying output produced by other pipes. A simple (linear) chain of effects can be applied within each pipe.

Most pipes are managed automatically, e.g. the pipes corresponding to individual clips, or the pipes collecting output from transitions, from nested sequences and from groups of tracks. At some point, at the timeline level, all processed data is collected within the aforementioned global pipes to form the small number of output streams produced by rendering and playback. Each timeline uses a separate set of global pipes.

An Asset View can be conceived as the timeline’s book keeper: it manages the various constituents in the timeline. Moreover, in addition to managing timeline constituents, raw material, clips, bins (folders) and effects are managed by the Asset View, i.e., typical management operations including, deleting, adding, naming, tagging, grouping into bins, etc. all occur here.

There are various kinds of assets available in any project:

Actually, the same underlying data structure is used to implement the asset view with folders, clip bins and effect palettes, and the timeline view with tracks, clips and attached effects. Technically, there is no difference between a track or a clip bin — just the presentation appears different. Timeline contents can be viewed, just like assets, for bookkeeping purposes, and the contents of a clip bin can be played like a storyboard.

The current representation of a project resident in memory is internally known as a session, whereas from a GUI user perspective, this is known as a project. In this section, we will use the term “session” as we are discussing the internals of Lumiera here.

Everything is stored in a session. If a user saves a session, then there should be no difference to the user whether they continue working or throw all their work away and load the session that has just been saved. In other words “everything is stored in a session!”

This ambitious goal has a number of advantages. If all steps are to be stored, then all steps must be available as an object to be stored. We can perform operations on these objects. For example, unlimited ability to undo previous steps, selective undo of previous steps or the possibility of merging various steps might even be a possibility. On a practical note, work on a project at the office and work on the same project at home can be merged each morning and evening.

Session storage is envisaged to operate similar to a database or journalling file system. Any operation will be logged prior to execution. This protects the session contents against corruption and allows for automated recovery in case a crash occurs. The actual storage itself is delegated to several backends. One of these backends implements a binary storage format for performance reasons, while another exposes all session contents in textual, human readable form. The storage format is designed in a way to ensure a certain degree of compatibility while the application evolves.

Generic mechanism to stitch together media objects. Any placement may contain a list of conditions on how to locate the placed object, examples being time-absolute/relative, relative to another object, or relative to some specific source media frame.

All of the session model contents are attached by placement, forming a large tree. Placements are to be resolved to find out the actual position, output and further locational properties of an object. Missing placement information is inherited from parent placements in the session tree. This causes a lot of relational and locational properties to be inherited from more global settings, unless defined locally at a given object: time reference point, output destination, layering, fade control, audio pan,…

to be written rendering…

Lumiera will process large quantities of data and it is of critical importance to perform this efficiently. The input and output subsystem are all processed in the backend, in fact, this is one very important function provided by the back end.

The typical Lumiera user will have many clips, in various configurations located at various places. All this data will have to be stored by the backend, moreover all this data will have to be rapidly retrieved from storage and provided to the user. The demands on memory are high: huge chunks of data, that can be quickly stored and equally quickly fetched, even if stored over longer periods of time. Moreover, due to the scalability requirement of Lumiera, this process will have to perform adequately on lower-end hardware, and perform efficiently on higher-end hardware.

Lumiera will break down processes that need to be processed into smaller units called frame job. Typically, there will be many hundreds of jobs waiting for processing at any one time. These jobs are queued for processing and the order in which this is performed is managed by the scheduler. This is all done in the backend.

Apart from memory, the backend will be responsible for accessing and saving files. It is essential to do this efficiently. This will be carried out in the backend using low-level mechanisms.

to be written

The above outline of the design uses a lot of special terms and common terms used with specific meaning within Lumiera. To ease understanding, we’ve collected together a glossary of common terms.

For most users, the next step would be to skim through the user manual and to install the application. But we are not this far yet, unfortunately. As of 1/2013, Lumiera isn’t usable in any way. Later, when we approach the alpha and beta development phase, we’ll devote some attention to a user manual and provide a sections on tutorials.

Developers and Readers curious about the inner workings of the application might want to have a look into the overview document of our technical documentation section. Learn more about The Inner Core…