Lumiera
The new emerging NLE for GNU/Linux
State Dropped
Date 2008-04-21
Proposed by rick_777

Mistakes to avoid in the Lumiera design

As a multimedia user and experienced programmer, I’ve found various flaws present in open source Non Linear Video editors. Here I will list the problems and their proposed (or mandatory) solutions. Please forgive me if some of the ideas here have already been approved, I wrote this text before reaching this wiki.

Description

As a multimedia user and experienced programmer, I’ve found the following flaws present in open source Non Linear Video editors (your mileage may vary) :

  1. Frequent crashes (which most of the time make you lose your work)

  2. Reinventing the wheel for every new project

  3. Lack of a user-friendly (and extensible) UI

  4. Lack of support for certain video formats or codecs

  5. Lack of documentation

  6. Lack of cross-platform support

  7. Dependency on scripted languages like Python, which make installation a mess

I will expand on the problems and their proposed (or mandatory) solutions.

1. Frequent crashes

Problem Frequent Crashes and unsaved work.
Severity CRITICAL.
Solution Isolating the UI from the rendering and data handling (also
improves the extensibility)
Required Yes
Workarounds Auto-save (however it’s not a real solution for the problem)

Working with multimedia (video / audio) editing is a magnet for segfaults (crashes) due to the handling of pointers and compression algorithms. A bug in a plugin (like in Audacity’s low-pass filter) will crash and you suddenly realize you lost your work - unless you have an auto-save feature, but that doesn’t go to the root of the problem.

My proposal is to move the low-level handling of video to a separate process, which then will do the processing - if it crashes, the UI will only report an error with a dialog (i.e. "the process crashed. Try again?"), but you work will stay safe. I’m not sure of the implementation difficulties that arise from having a shared memory buffer for rendering / processing, but one thing is certain: Whenever you move the cursor or rewind a part of a clip in your resources, the application isn’t supposed to crash. Just moving the cursor isn’t a time-critical task, so perhaps we can use temporary files for this. It’s safer if you’re not doing the final rendering.

Comments

I am not sure yet about separating things into processes, generally it is clear that this would be more robust but there are some performance impacts and programming problems (massisve amounts of data in shared memory). But most importantly, when a subprocess gets a job and crashes on it, it won’t complete the job, we don’t have a choice except gracefully abort it. From a user perspective "It doesn’t work!" there is no much difference to a complete crash. Well and yes we aim to make it crash proof rather, crashes a bugs and have to be fixed, point.

Lumiera will never ever loose work, we don’t plan to make a project file, autosafe way. Lumiera will keep projects in an internal database like format which consists of a Dumpfile and a contingous written logfile. After a crash/powerdown whatever, this log just gets replayed. The advantages are countless, imagine persistent, selective undo and so on. Any other format (cinelerra2 XML, MXF, …) will be realized by importer/exporter plugins.  — ct

2. Reinventing the wheel for every new project

Problem Various projects compete and reinvent the wheel
Severity Serious (Slows down development time. A lot)
Solution Multi-tier design, turn the data handling into a backend and use
whatever UI you prefer
Required Yes. Better now that the project hasn’t started

Imagine the Linux kernel was tied to the window manager. You would have to stick with KDE or GNOME and you couldn’t improve it! Fortunately it’s not like that for Linux, but it is for some projects. If you want to change the user interface from QT to wxWidgets or GTK you’ll need to rewrite every module.

If you separate the UI from the project handling engine, you can simply fork the project and change the UI to one that supports skinning, without having to do the complicated video-processing stuff.

Separating the processes has an equivalent for web programming, it’s called "separation of concerns", or multi-tier design. When you suddenly change the database engine, you don’t need to change the whole program, just the database module. Same goes for changing the UI from HTML to XML or Flash. If they’re separate modules that only communicate through a clearly defined API.

Example case 1: The Code::Blocks IDE. The compiling engine supports various compilers, and the engine itself is only a plugin for the main editor. If the compiler crashes, you only get an error, but the IDE doesn’t crash (unless it’s the UI part that’s doing something wrong).

Example case 2: Chessmaster. The user interface and speech synthesis stuff only call the chess engine, called "theking.exe". Linux chess games also depend on an engine to do the thinking.

So I suggest to split the project into four separate tiers (not necessarily processes):

  1. User interface - communicates with the "project" tier, handles the user events and does the calls.

  2. The project tier - the main part of the video editor. This one invokes the renderer and decides which effects to apply, saving them as mere parameters for later processing. It also tells you where the current pointer for the track view is. Also calls the rendering engine for the current frame, or for previews of a certain special effect. Note that if this process keeps running even if the GUI crashes, later we can restart the GUI and keep working.

  3. The rendering engine - This one must be a separate process for the reasons stated in problem #1. This also gives us the advantage that it can work on the background while we keep working on the project (after all the project is just a set of data stating which effects to apply to which tracks, and which files are used for the tracks) - instead of just having a window saying "Rendering, please wait". Even Adobe Premiere Pro suffered from this problem. This means that if we put enough effort, we can surpass commercial software in certain areas. Note that the rendering engine uses the same API than the project tier, as it works on a copy of the project when doing the final rendering.

  4. The video processing wrapper, which has interfaces for different video processing toolkits (DirectX, GStreamer, etc). This also makes the project cross-platform. Tiers 1 and 2 can go in one process, and the 3 and 4 in another (this would make tier 2 a library which defines a C++ Class, and tier 4 would also be a library which is used by the rendering engine).

By separating the tiers, these can later become their own projects and overall the community would receive great benefits.

Comments

Please look at our design drafts, things will be separated (little different than you describe here). We reuse things which are benefitful (gavl, ffmpeg, ..) but we are also aware that we reinvent the wheel for some things by intention. Lumieras goal is not just to glue some existing libraries together under a new gui, there are already a lot projects trying this way. We rather aim for a 'Professional' high performance Video editing solution which does some things in a different (maybe more complex) way. We do not use existing frameworks like MLT or gstreamer because we believe that these do not fit our goals (gstreamer will be supported through plugins). We do not produce yet another multimedia framework library (this only happen by coincidence) to be used by others.  — ct

3. Lack of a user-friendly and extensible UI.

Problem Often, editors provide a very poor and buggy interface.
Examples: Ja hshaka doesn’t even provide tooltips for the various tools, and
the document ation is poor; In Cinelerra I’ve noticed some bugs when using the
open dialog, I’d rather have the KDE one, thanks.
Severity From Annoying to Serious.
Solution 1 Use a library that allows you to use different widget
libraries, l ike wxWidgets.
Required Recommended, but not obligatory.
Solution 2 Write different user interfaces, but they’d be hard to maintain.
Required, No.

This problem is complicated, we need a good framework for handling the tracks. Perhaps this could become a separate project. Ideas are welcome.

Comments

Joel started working on a GUI recently and making good progress. The UI should finally be quite flexible as it mostly provides a skeletion where plugins render to. We have quite a lot ideas about the UI and user input is welcome. The UI is currently the most separate tier in the design, i’d like to make it a plugin itself which is loaded when lumiera is started in a gui mode, but it is to early to say how exactlly it will be integrated, except that we all agree that GUI is optional and Lumiera can also run headless, script driven.  — ct

4. Lack of support for certain video formats or codecs

Problem Lack of support for certain video formats or codecs.
Severity Critical.
Workarounds 1. Give a help page for the user to do his own conversion, but
this is very inelegant, annoying, and a waste of time. 2. Provide conversion
on the fly, and keep a separate "preprocessed" copy of the imported clip in a
separate dir ectory. This is a nice middle ground, IMHO.
Solution Use a wrapper library as stated in problem # 2, having a
plugin-based design is recommended.
Required Yes.

Some editors like Cinelerra are hardwired into using one format, or have a phobia to certain formats / codecs (i.e. DivX AVI’s). If we separate the project editing engine from the video handling libraries, we can use unknown formats by simply providing an input/output plugin. This would allows us to use files encoded with lossless codecs like Lagarith. This also provides forward compatibility for future formats.

Comments

Lumiera is a video editor we don’t care (cough, not really true) about video formats. Everything which comes In and goes Out is defined in plugins which handle video formats. We currently decided to use gavl because it is a nice small library which does exactly what we want. Later on gstreamer and other such kinds of decoder/encoder/processing-pipe libs will be realized.  — ct

5. Lack of documentation

Problem Some video editors have very poor documentation (and that’s an
understateme nt cough Jahshaka cough )
Severity Critical.
Solution Have a team for the documentation.
Required Yes.

Nuff said.

Comments

Quote from Ohloh.net: (http://www.ohloh.net/projects/lumiera)

Extremely well-commented source code

Lumiera is written mostly in C++. Across all C++ projects on Ohloh, 23% of all
source code lines are comments. For Lumiera, this figure is 46%. This very
impressive number of comments puts Lumiera among the best 10% of all C++
projects on Ohloh.

Nuff saied… Oh well, about user docs we like to get that impressive ratings there too, any helpers?  — ct

6. Lack of cross-platform support

Problem Where’s my Windows version?
Severity Blocker
Solution Use a cross-platform toolkit for the UI.
Required Depends, do you plan to make it Cross-Platform?

A good example for this is the Code::Blocks IDE, which was thought of being cross-platform from the beginning. Curiously, at first the project was Windows-only, and its only F/OSS alternative was Dev-C from Bloodshed (eew). Otherwise you’d have to stick with proprietary applications like Visual C.

In Linux there were various IDE’s, but they were Linux-only. Since Code::Blocks uses a cross-platform toolkit (wxWidgets), it can be compiled either in Windows and Linux. There are RPM’s for various distros now that the first public version (8.02) got out. I’ve heard that QT is also cross-platform, but I haven’t tried it yet.

Of course - if you separate the UI from the project engine, someone could make his own Windows UI for the project. Now what needs to be taken care of, is that the rendering libraries are cross-platform too.

Comments

We refuse to make it cross platform intentionally. Most things are written portable, POSIX compatible, some might need platform specific fixes. But our target is primary Linux (because thats what we use) secondary any other Free OS (hopefully we find some testers/maintainers for that). Lumiera 'might' run on OSX and patches will be accepted, but it is not a free platform so we don’t care by ourself. Windows due its diffrent system interfaces will be hard to port, if someone wants to do that, have fun, we will accept patches to, but we do not support it in any way by ourself.  — ct

7. Dependency on scripted languages like Python, which make installation a mess

Problem Installation can be a mess if we depend on scripted languages.
Severity Annoying, the end user might just conform with another project
that "just w orks".
Solution Make it in C++ or other easily-compilable language.
Required VERY recommended.

I’ve had to install several packages for my distro (whose repository is not as large as others like Ubuntu’s) from source. Some of them depend on very esoteric scripted languages which I also need to install. And then the libraries, etc. My suggestion is to free the end user from this burden, and work on a common language, like C++.

Comments

At some point a scripting language 'will' be required, yet to drive the testsuite, make headless rendering work and so on. We need to provide installation instructions and/or even bundle this language with Lumiera. This will likely become a small embedded language like Lua or some kind of forth (or maybe some scheme?) it should not depend on strange modules which are not part of the core scripting language distribution (or we shall provide them too), needs to be worked out.  — ct

Author’s comments

Some of the measures stated in this document are optional, but separating the processes for the rendering engine, editor and User Interface are the optimal solution and required to avoid common problems.

Discussion

Mostly we agree with the general statements in this Design Entry. But there are some points which don’t stand the test of a detailed technical discussion. For example, you simply can’t state it’s a mistake not to write code which similarly runs on windows and *nix. Well. You could try to write it in Java. See my point? While today it’s quite feasible to write office stuff or banking applications in a cross-platform manner, a video editor still is a different kind of a beast.

A similar argumentation holds true for the question, wether or not to use separate processes and IPC. While it certainly is a good idea to have the X server or a database running in a separate process, the situation is really quite different for editing video. Hopefully it’s clear why.

Could you please rework this Design Entry in a way that we can finalize (accept) it?

  • Please remove the section about windows

  • Please separate out things needing technical discussion and are not just "mistakes", thus retaining only the big picture statements (on which we all agree)

  • How to secure the application against crashes

  • If it is viable/desirable to run the gui in a separate process really needs in-depth technical discussion (create a new Design Entry for this)

  • How to deal with the dependencies problem in combination with plugins/extensions and script languages  — Ichthyostega

Conclusion

The October.2008 dev meeting decided to drop this design proposal as is.

Basically, this text just tells us "to make Lumiera good", and especially it contains a mixture of topics

  • We fully agree to 80% of the statements made there, but we think those statements are so very basic and self-evident as to be considered off-topic here. We are aware of the recurring problems with open source video editing. That’s why we are here.

  • The proposal draws conclusions on two technically substantial points, at which we don’t agree. And it fails to provide sufficient (technically sound) arguments to prove these statements.

While it is certainly desirable to be cross-platform as much as possible and especially ''target Microsoft Windows'', we don’t see much possibilities with today’s mainstream technology to build an application which is as technologically demanding as a video editor is. We would end up developing two or even three sister applications, or we are forced to sacrifice performance for portability. When put up to face such options, we have a clear preference to concentrate on a really free and open platform.

While it is certainly desirable to make the application as robust as possible, we don’t see how ''using multiple separate processes'' could help us with this goal 'without creating major scalability or performance problems' due to the use of shared memory. And, yet more important: we don’t share the basic assumption made in the proposal, namely that video processing is inherently dangerous. We think the basic algorithms involved are sufficiently well-known and understandable to implement them in a sound manner.

Back to Lumiera Design Process overview