Lumiera Workflow Proposals

In general, we can observe two extremes when it comes to editing:

Because of its analogue roots, traditionally, NLE’s have been track-based. I’d like to quote Randy Ubillos, original creator of ReelTime and Final Cut Pro:

Source: Interview with Randy Ubillos

Tracks have several advantages:

Tracks have disadvantages as well:

Editors often proudly share screenshots of their timelines on social media, and they do look impressive, but these are in fact pretty fragile structures.

So naturally, a question would be: what will happen if we would let go of the track paradigm? This is what Final Cut Pro has done, starting from the rewrite of Final Cut Pro X. At the time, a disastrous marketing campaign caused many editors to leave the application, although more and more people are starting to realise that many of its ideas were way ahead of its time. It’s still the only big NLE out there that was designed with computers in mind, and not analogue hardware.

It’s not entirely trackless, but it manages to hide the concept of tracks from the user. There are no buttons to enable or disable tracks or other track controls, and instead, the tracks are called “layers”. In a nutshell, it works by having a primary storyline (in a track-based NLE this would be V1+A1) where you build the foundation of your edit. Then the video clips you put on top and the audio clips you put below get connected to one or more clips from the primary storyline. Move a clip on the primary storyline, and all connected clips automatically move with it. With a modifier key you can ignore clip connections, so that you can also easily move a primary clip elsewhere without its connected siblings coming along.

Sounds good. Why not just copy this? One reason is that FCP assumes that all clips that are not on the primary storyline should be connected to this primary storyline. This might work well for fiction films, but not necessarily for other types of video. Earlier I mentioned an example of sound effects that share a connection to b-roll on a higher layer. FCP won’t allow you to connect them. Another limitation of FCP is that it’s primarily designed to be operated by mouse. It’s not possible to use keyboard commands to, for example, swap the position of clips (other than by cut and paste), or to perform slip edits (the trim keys will work, but the clip needs to be selected by mouse using the Trim tool).

How else could we group clips together that ought to be connected?
We could create groups that are directly editable, like so:

However, there are a few problems with this: we expect rendering to happen from top to bottom, in which case the b-roll would cover the subtitles. On top of that, it’s hard to see at which points video clips overlap. Last but not least: it looks rather unorganised.

Can we restructure this? Perhaps like this:

But how exactly would a user interact with a timeline like this? I will leave the answer to that question for an actual design document, as for now I just wanted to show the concept of grouping so that it becomes clear that alternative ways to organise the timeline might be of great value.

If we observe existing NLE’s, we can distinguish several ways to change timeline interactions. Some use a tool-based approach (Adobe Premiere, Final Cut Pro), others a mode-based approach (Avid Media Composer, Lightworks and DaVinci Resolve), and we can also see a view-based approach (FCP’s use of the precision editor, explained in more detail later). Which of these interaction methods would be preferred for Lumiera?

Another example is the Trim View in DaVinci Resolve: whenever you perform a trimming operation, the program monitor displays a trim view (showing the frames left and right of the cut next to each other). As soon as you start doing something else, it exits this trim view. It needs no dedicated user action.

This might seem like a great approach (and can be combined with tools and/or modes), but it does mean a user needs to reorient a bit when they exit the altered view and return to its default state. Perhaps that’s a small price to pay, when not overdone.

When we more closely examine the behaviour of NLE’s, we see that things are not as black and white as they seem. No NLE exclusively uses one method over others. Resolve’s modes do in fact also change the mouse cursor’s look and behaviour and can therefore be considered a hybrid mode/tool system. FCP’s Select and Position tools also change the behaviour of trimming keys and can therefore indeed also be considered modes, which gives us a hybrid design as well. Specialised views can be combined with tools or modes.

To conclude: all methods have pros and cons. Tools, modes and views aren’t mutually exclusive, so I think we should use whichever method or combination of methods is appropriate for different actions in Lumiera. Whenever we can, we should try to avoid all three and design the app to be as consistent as possible. Whenever we use a mode, we should see if we can design it in a way in which a user fluidly enters and exits this mode, without requiring extra actions: contextual modes. Most mode-based NLE’s already implement this for their trim mode in a way where selecting a trim side automatically enters trim mode. However, a user does still need to explicitly exit trim mode.

Regardless of the method, immediate visual differentiation of tools/modes/views is important and I’d like this to extend to any timeline option that can be toggled on/off. Therefore I’d like to propose the use of a contextual bar that appears over the bottom part of the timeline whenever a contextual mode is active.

Currently I’m thinking of using this bar on two or three occasions:

Colours could be used (for example as an outline around the bar) to indicate which contextual mode is active.

A limitation of such a bar is that it might overlap with content that a user wishes to interact with. If this is the case, then clicking on the grid with dots will let a user drag this bar out of the way. As an extra option I would like to suggest to use a modifier key (Ctrl?) which temporarily hides the contextual bar. Another consideration could be to attach this bar to the bottom of the timeline and combine it with the zoom and navigation sliders, although I feel a more central default position will get things done quicker.

All NLE’s offer multiple ways to get your material in the timeline, and those methods fall into the following categories:

All of these options make sense, but in general I rarely remember more than two options (insert and overwrite) and easily forget any surplus options that NLE’s offer. We should also try to limit the amount of keyboard shortcuts that a user needs to memorise, so I would like to propose a single “Add Clip” action. This will insert a clip and will show the different options via the contextual bar. You can then change the desired method after the fact, until you commit by doing something else. In this case, adding a clip will enter and exit another contextual mode. One last thing I would like to address is the destruction that an insert operation can cause. Usually this can be prevented by proper track management, but you can wonder why you would ever want an insert action to cause the following:

In this case, if you’d like the clips on V2 to stay in sync, the overlapping clip will be split in two. If you don’t want this to happen you would have to move the overlapping clip to another track temporarily, or move everything on V2 by hand with the same amount of frames as the duration of the clips that’s inserted. My point is, however, that while the behaviour above makes sense when you strictly follow the logic of how such a timeline works, in real-life you would rarely want a clip to be split only to keep sync. Therefore, I would like to propose insert operations to be nondestructive by nature. Overlapping clips on other tracks or layers will simply move in its entirety.

Another thing to consider, as I put much emphasis on grouping (or linking) clips, is that we might want to differentiate between inserting a clip into the currently active group, or inserting it in a newly created group. This way a clip will always end up in a group without much manual work.

Quick timeline navigation is key in editing. Let’s start by examining how other NLE’s deal with this. We need to make a distinction between:

Moving the viewport, by mouse:

Moving the viewport, by keyboard:

Moving the viewport, by other device:

Moving the playhead, by mouse:

Moving the playhead, by keyboard:

Combining mouse and keyboard:

Using other devices:

Zooming also needs to be mentioned here, because often we might be zoomed in, do some work, zoom out to get an overview of the timeline, and zoom in to another part of the timeline. Zooming in and out, by mouse:

Zooming in and out, by keyboard:

I can imagine that all of the keyboard shortcuts for navigation were invented simply to provide many options for each editor to choose what they need. This way, every editor can pick the shortcuts that best fit their workflow.

Navigation by means of keyboard has very good support in most apps and it’s a matter of providing similar shortcuts in Lumiera. It could be expanded by allowing the creation of navigation markers (a special category of markers) that are bound to keys or numbers. A user can jump to specific markers by pressing a “go to navigation marker” key, followed by the key they bound. This is basically Vim’s way of doing it (m+0-9a-zA-Z to bind, backtick+0-9a-zA-Z to jump). Or we might want to keep it as simple as having the regular “add marker”, “add and edit marker” and “go to next/previous marker” keyboard shortcuts.

Sections (explained later in this chapter) will get navigation markers automatically, so a user doesn’t have to create each marker manually. Or we might want to include keyboard shortcuts for “go to previous/next section”.

Fast forward and fast rewind keys as found in Reaper might be a helpful addition to the aforementioned keys. This would give as a quick way to skim a timeline by keyboard. While fast forward and rewind can be achieved by the regular j/k/l shuttle controls, this requires a lot of tapping to get to the desired speed, over and over again.

Scrollbars and zoom sliders have a different set of reasons for being chosen as the de facto standard widgets for navigation. Familiarity on the one hand, and visibility on the other. By having these controls visible at all times, yet outside of the working area itself, each user will be able to locate them while they remain out of the way. These are very good reasons for their existence and positioning, but they come at the cost of speed: moving your mouse cursor out of your main working area towards the edges and back takes time and interrupts the creative flow. It works, but it’s not ideal. That’s why I would like to propose an additional way to navigate by mouse, which is by popup widgets. These will pop up around the cursor, triggered by a keyboard press (similar to how Blender uses pie menus):

Mouse-only navigation:

Selecting a single clip happens in all NLE’s by left mouse clicking. Selecting multiple clips happens through box-select, by using left mouse and drag. Premiere and Resolve both offer “Selection follows playhead”, where the position of the playhead determines which clip gets selected. Then there are keyboard shortcuts, to select the next or previous clip on a track. Final Cut Pro lets you move the clip selection up and down by Cmd+up/down.

Selecting multiple clips by keyboard has no support that I know of in any big NLE. I would suggest to add this though, because the alternative for keyboard selection is the famous mark in/out range. That’s a pretty arbitrary way to indicate what you’d like to extract, lift or remove, and, in all fairness, this works great as long as your timeline has few tracks. However, once you have multiple overlapping clips on different tracks, combinations of mark/lift/extract/insert/overwrite require careful track selection and there are serious risks of throwing tracks out of sync. It’s not very convenient either, the moment you start creating J- or L-cuts. It’s a butcher’s knife that pays no respect to any kind of clip connections.

While we should support marking in and out points (and I’d advice to copy Lightworks' mark and park, where you only need to set a single mark and the playhead becomes the second mark) simply because most professional editors are so used to them, we should aim to include less destructive ways of arranging clips on the timeline. The natural element in the timeline is the clip, so I would suggest to focus on providing proper ways to select clips and edit the positions and durations of these clips, instead of relying on in and out marks to do the job.

I would propose to use the arrow keys for navigating clips and groups. There can be a cursor (for example, a white dot) which we can move left, right, up, and down by using the arrow keys. Changing the cursor from clip to clip will also move the playhead to the start of the highlighted clip - but this is secondary. A clip-based focus instead of playhead/cut-based focus will be different from what any other NLE does at the moment, but enabling multi-clip selection by keyboard will be a good reason to go in this direction.

For selecting, we can introduce an “add to selection” key which adds the highlighted clip to the current selection (apart from the convention of using Ctrl+mouse click). We could also think of adding box-select by keyboard, by using a system with grid coordinates. One thing to note is that we do still need extra keyboard shortcuts to move the playhead frame by frame, independently from the clip selection. See the navigation subchapter for more details. Selection will be one of the previously mentioned contextual modes. The user can always return to their previous selection until they make a new one, by pressing the corresponding keyboard shortcut (S).

Most NLE’s have functions for selecting many clips at once, called “select forward” and “select backward”. In Adobe Premiere these are tools: click and you select all clips to either the left or right of the mouse cursor, and using Shift will toggle between selecting all clips on a single track or on all tracks (it doesn’t respect track activation status though, which it should). Other NLE’s (Resolve, Avid, Lightworks) only have keyboard shortcuts for these functions, which do only affect activated tracks. This is a bit faster: in Premiere you need to switch to the Select Forward/Backward Tool, make your selection, then switch back to your previous tool. With a keyboard shortcut you only have to press a single key (if your track selection is correct).

Most NLE’s handle the arranging of clips in a destructive way. Drop one clip over another, and the clips underneath disappear. Move one of two clips that are connected by a transition, and the transition disappears. While it’s certainly true that experienced editors will understand the risks involved in moving clips and will therefore avoid such destructive behaviour unless they intend for something to be destroyed, accidents still occur and most probably someone won’t notice until much later, when undo cannot help to recover the lost work anymore.

At the same time, something as fundamental as swapping the position of two neighbouring clips is harder than it should. Premiere has Ctrl+Alt+drag, but the results can be destructive. Resolve has a similar modifier combination, and keyboard shortcuts for swapping left and right, but these also easily mess up parts of your edit if you’re not careful with your track selection. Avid’s smart tool has a Segment Mode that can be set to extract/splice-in, which will do a similar thing. It only has an effect on the tracks that hold the clip selection and results can get messy when there are J- or L-cuts. An alternative is to select a clip by mouse, use “ripple cut”, move the playhead to the next cut, and use “paste insert”, but this can also be destructive when not being careful with track management.

We’ve talked about using in and out marks already in the previous subchapter, so I will skip that for now.

FCP is the only widely used NLE as of now that allows non-destructively changing the order of clips, by having other clips move out of the way: the so-called magnetic timeline (see above). How come the other professional NLE’s make it so difficult to simply change the order of clips? The reason for that, I can only assume, is that the software doesn’t know how clips on different tracks relate to each other. And as a result, it will be very hard to guess how to exactly change the order. FCP solves this problem by having clips always connect to a primary storyline. Lumiera could do a similar thing by utilising its grouping features, in combination with a Ripple-toggle in the contextual bar of the selection mode. As long as clip relations are clear, changing the order of clips should be simple to achieve.

Trimming means extending or shortening the duration of one or more clips (simultaneously), either from its start or end. In most applications, there are two distinct ways to trim: ripple trimming and non-ripple trimming. Ripple trimming means that when you extend or shorten a clip, all other clips on the same track and possibly on other tracks move along. When non-ripple trimming, all clips stay in place, and when a clip is shortened, a gap appears. To choose a trim type, NLE’s use either tools or modes.

Next to ripple/non-ripple trimming there are three other edit operations closely related to trimming that need to be mentioned:

Some apps use their trimming system for slip and slide edits (Avid, Lightworks). Others use a special tool or mode and/or separate keyboard shortcuts for this (FCP, Premiere, Resolve).

Tool-based NLE’s offer a Blade or Cut tool that will cut clips wherever a user clicks. All NLE’s offer a dedicated keyboard shortcut for creating a cut at the playhead as well.

Merging two timeline clips with so-called “through edits” is often possible by selecting the cut for a roll edit and pressing the delete key. A through edit occurs when two successive segments of the same source clip are placed directly next to each other in the timeline (so as a result, it looks as if there wasn’t a cut at all). A Blade tool, a shortcut for Add Cut and the ability to remove through edits with backspace seem all worth implementing.

In general, we find the following options to remove clip from a timeline:

These operations are so common that I would not change them.

I would like to propose the ability to divide a timeline into multiple sections. Each of these sections will have a header in the ruler that can be edited, to give each section a name (similar to how duration markers in Premiere are displayed).

The benefits of sections:

Several other characteristics of sections:

Things to consider:

There’s usually a variety of ways to add transitions:

Editing transitions:

Most of these established ways of working with transitions can be applied to Lumiera. I would like to suggest to also add Avid’s option to add transitions to a large number of cuts while skipping existing transitions (and this should even be the default).

One thing we do need to address is how transitions magically disappear in pretty much any NLE except Avid and Final Cut Pro, the moment you move one of the two adjacent clips elsewhere, or whenever you replace a clip by another. Transitions take time to setup in the right way, and they shouldn’t disappear by themselves. I would suggest to follow Avid and FCP’s behaviour and leave transitions in place by default.

Each NLE has an effect properties panel en most have a generic properties panel for changing all basic properties of clips in the timeline, such as:

Avid is the only NLE out there that still requires adding an effect to change these basic clip properties (“3D DVE”). Quick access to such properties saves a lot of time, so enabling this is essential. Depending on the actual effects workflow we might want basic properties to be part of an effect that is automatically applied to each clip when it’s added to the timeline. A separate Properties panel (as in Premiere) is less ideal: the fewer panels, the better. FCP and Resolve use a tabbed approach for their panels (which is named “Inspector”) to keep different categories of properties separated.