The new emerging NLE for GNU/Linux

the following collection of usage situations helps to shape the details of the time values and time quantisation design.
→ see also more about time quantisation

time position of an object

indeed the term “time position” encompasses two quite different questions

  • a time or timing specification within the object

  • determining the time point in reference to an existing scale

time and length of an object

basically the same situation,
but the length of a time span (duration) could be treated in two ways for quantisation

  • having a precise specification and then quantise the start and endpoint

  • quantise the start position and then establish an (independently quantised length)

moving and resizing an object

this can in itself be done in two different ways, and each of them can be applied in a quantised flavour, which sums up to 8 possible combinations, considering that position and length are 2 degrees of freedom.

  • a variable can be changed by an offset

  • a variable can be defined to a new value

  • another (hidden) degree of freedom lies in how to apply an quantised offset to an unquantised value (and reversed), because this operation might be done both in the quantised or non-quantised domain, and also the result might be (un)quantised

updating the playback position

this can be seen as a practical application of the above; basically we can choose to show the wall clock time or we can advance the playback position in frame increments, thus denoting the frame currently in display. For video, these distinctions may look moot, but they are indeed relevant for precise audio editing, especially when combined with loop playback (recall that audio is processed block wise, but the individual sample frames and thus the possible loop positions are way finer than the processing block size)

dispatching individual frames for calculation

when a render or playback process is created, at some point we need to translate this logical unit (“calculation stream”) into individual frame job entries. This requires to break continuous time into individual frames, and then ennumerating these frames.

displaying time intervals

for display, time intervals get re-quantised into display array coordinates. While evidently the display coordinates are themselves quantised and we obviously don’t want to cancel out the effect of an quantisation of the values or intervals to be displayed (which means, we get two quantisations chained up after each other), there remains the question if the display array coordinates should be aligned to the grid of the elements to be displayed, and especially if the allowed zoom factors should be limited. This decision isn’t an easy one, as it has an immediate and tangible effect on what can be showed, how reversible and reproducible a view is and (especially note this!) on the actual values which can be set and changed through the GUI.

time value arithmetic

Client code as well as the internal implementation of time handling needs to do arithmetic operations with time values. Time values are additive and totally ordered. Distance, as calculated by subtraction, can be made into a metric. Another and quite different question is to what extent a quantised variant of this arithmetics is required.

relative placement

because of the divergence between quantised and unquantised values the question arises, if placement relative to another object refers to the raw position or the already quantised position. Basically all the variations discussed for time and length of an object also do apply here.

notable issues

Direct quantisation of length is not possible. This is due to the non-linear nature of all but the most trivial time grids: Already such a simple addition like a start offset destroys linearity, and this still the more is true within a compound grid where the grid spacing changes at some point. Thus, the length has to be re-established at the target position of an time interval after each change involving quantisation. Regarding the strategy to apply when re-establishing the length, it seems more appropriate to treat the object as an entity which is moved, which means to do quantisation in two steps, first the position, then the endpoint (the second option in the description above). But it seems advisable not to hard wire that strategy — better put it into the quantiser.

We should note that the problems regarding quantised durations also carry over to offsets: it is difficult to define the semantics of a quantised offset. Seemingly the only viable approach is to have a intended offset, and then to apply a re-quantisation to the target after applying the (raw) offset.

When to materialise a quantisation. Because of the basic intention to retain information, we prefer to delay the actual applicatio of the quantisation operation to the values stored internally as much as possible. But not materialising immediately at quantisation has the downside of possibly accumulating off-grid values without that being evident. Most notably, if we apply the raw offsets produced by GUI interactions, the object’s positions and lengthes are bound to accumulate spurious information never intended by the user.

Thus, an especially important instance of that problem is how to deal with updates in a quantised environment. If we handle quantisation stictly as a view employed on output, we run into the problems with accumulating spurious information. On the other hand, allowing for quantised changes inevitably pulls in all the complexity of mixing quantised and non-quantised values. It would be desirable somehow to move these distinctions out of the scope of this design and offload them onto the client (code using these time classes).

Another closely related problem is when to allow mutations, if at all (→ see more here…). We can’t completely do away with mutations, simply because we don’t have a pure functional language at our disposal. The whole concept of reference semantics doesn’t play so well with immutable objects. The Lumiera high-level (session) model certainly relies on objects intended to be manipulated. Thus we need a re-settable length field in MObject and we need a time variable for position calculations. Yet we could make any derived objects into immutable descriptor records, which certainly helps with parallelism.

The problem with playback position is — that indeed it’s an attempt to conceptualise a non-existing entity. There is no such thing like “the” playback position. Yet most applications I’m aware off do employ this concept. Likely they got trapped by the metaphor of the tape head, again. We should do away with that. On playback, we should show a projection of wall-clock time onto the expected playback range — not more, not less. It should be acknowledged that there is no direct link to the ongoing playback processes, besides the fact that they’re assumed to sync to wall-clock time as well. Recall, typically there are multiple playback processes going on in compound, and each might run on a different update rate. If we really want a visual out-of-sync indicator, we should treat that as a separate reporting facility and display it apart of the playback cursor.

An interesting point to note for the frame dispatch step is the fact that in this case quantised values and quantisation are approached in the reverse direction, compared with the other uses. Here, after establishing a start point on the time scale, we proceed with ennumerating distinct frames and later on need to access the corresponding raw time, especially to find out about the timeline segment to address, or for retrieving parameter automation. → see frame dispatching.

Note that the display window might be treated as just an independent instance of quantisation. This is similar to the approach taken above for modifying quantised time span values. When following this line of thought, we should provide a special kind of time grid, the display coordinates. The origin of these is always defined to the left (lower) side of the interval to be displayed, and they are gauged in screen units (pixels or similar, as used by the GUI toolkit set). The rest is handled by the general quantisation mechanisms. The problem of aligning the display should be transformed into a general facility to align grids, and solved for the general case. Doing so solves the remaining problems with quantised value changes and with specifying relative placements as well: If we choose to represent them as quantised values, we might (or might not) also choose to apply this grid-alignment function.

the complete time value usage cycle

The way time value and quantisation handling is designed in Lumiera creates a typical usage path, which actually is a one-way route. We might start out with a textual representation according to a specific timecode format. Assumed we know the implicit underlying time grid (coordinate system, framerate), this timecode string may be parsed. This brings us (back) to the very origin, which is a raw TimeValue (internal time value). Now, this time value might be manipulated, compared to other values, combined into a time span (time point and duration — the most basic notion of an object to be manipulated in the Session). Anyway, at some point these time values need to be related to some time scale again, leading to quantised time values, which — finally — can be cast into a timecode format for external representation again, thus closing the circle.

substantial problems to be solved