node-builder.hpp File Reference

Specialised shorthand notation for building the Render Node network. More...

Go to the source code of this file.

Description

Specialised shorthand notation for building the Render Node network.

During the Builder run, the Render Node network will be constructed by gradually refining the connectivity structure derived from interpreting the »high-level Model« from the current Session. At some point, it is essentially clear what data streams must be produced and what media processing functionality from external libraries will be utilised to achieve the goal. This is when the fluent builder notation defined in this header comes into play, allowing to package the fine grained and in part quite confusing details of parameter wiring and invocation preparation into some goal oriented building blocks, that can be combined and directed with greater clarity by the control structure to govern the build process.

Levels of connectivity building

The actual node connectivity is established by a process of gradual refinement, operating over several levels of abstraction. Each of these levels uses its associated builder and descriptor records to collect information, which is then emitted by a terminal invocation to produce the result; the higher levels thereby rely on the lower levels to fill in and elaborate the details.

• Level-1 is the preparation of an actual frame processing operation; the Level-1-builder is in fact the implementation class sitting behind a Render Node's Port. It is called a Turnout and contains a preconfigured »blue print« for the data structure layout used for the invocation; its purpose is to generate the actual data structure on the stack, holding all the necessary buffers and parameters ready for invoking the external library functions. Since the actual data processing is achieved by a pull processing, originating at the top level exit nodes and propagating down towards the data sources, all the data feeds at all levels gradually link together, forming a TurnoutSystem.
• Level-2 generates the actual network of Render Nodes, which in turn will have the Turnout instances for Level-1 embedded into their internal ports. Conceptually, a Port is where data production can be requested, and the processing will then retrieve its prerequisite data from the ports of the Leads, which are the prerequisite nodes situated one level below or one step closer to the source.
• Level-3 establishes the processing steps and data retrieval links between them; at this level, thus the outline of possible processing pathways is established. After spelling out the desired connectivity at a high level, the so called »Level-3 build walk« is triggered by invoking the [terminal builder operation](ProcBuilder::build() on the processing builder corresponding to the topmost node. This build walk will traverse the connectivity graph depth-first, and then start invoking the Level-2 builder operations bottom-up to generate and wire up the corresponding Render Nodes.

Using custom allocators

Since the low-level-Model is a massive data structure comprising thousands of nodes, each with specialised parametrisation for some media handling library, and a lot of cross-linking pointers, it is important to care for efficient usage of memory with good locality. Furthermore, the higher levels of the build process will generate additional temporary data structures, refined gradually until the actual render node network can be emitted. Each builder level can thus be outfitted with a custom allocator — typically an instance of lib::AllocationCluster. Notably the higher levels can be attached to a separate AllocationCluster instance, which will be discarded once the build process is complete, while Level-2 (and below) uses the allocator for the actual target data structure, which has to be retained while the render graph is used; more specifically until a complete segment of the timeline is superseded and has been re-built.

Remarks

syntactically, the custom allocator specification is given after opening a top-level builder, by means of the builder function .withAllocator<ALO> (args...)

Building Render Nodes

At Level-2, actual render nodes are generated. The NodeBuilder creates a suitably configured Connectivity object, which can be dropped directly into a ProcNode. Managing the storage of those Render Nodes themselves is beyond the scope of the builder; so the user of the builder is responsible for the lifecycle of generated ProcNode objects.

Flavours of the processing function

The binding to the actual data processing operations (usually supplied by an external library) is established by a processing-functor passed to configure the Port builder. The supported signatures of this functor are quite flexible to allow for various flavours of invocation. Data types of parameters and buffers are picked up automatically (at compile time), based on the signature of the actual function supplied. The accepted variations are described in detail here. Basically, a function can take parameters, input- and output-buffers, yet only the output-buffers are mandatory. Several elements of one kind can be passed as tuple.

Handling of Invocation Parameters

Typically, a processing operation can be configured in various ways, by passing additional setup- and invocation parameters. This entails both technical aspects (like picking some specific data format), organisational concerns (like addressing a specific frame-number) and elements of artistic control, like choosing the settings of a media processing effect. Parameters will thus be collected from various sources, which leads to an additional binding step, where all these sources are retrieved and the actual parameter value or value tuple is produced. This specific parameter binding is represented as a parameter-functor. Whenever the processing-function accepts a parameter argument, optionally a such parameter-functor can be installed; this functor is supplied with the TurnoutSystem of the actual invocation, which acts as front-end to access contextual parameters.

synthesised additional parameters

As an extension for (rare and elaborate) special cases, a special evaluation scheme is provided, which relies on a »Param Agent Node« as entry point, to invoke additional functors and compute additional parameter values, which can then be used in a nested delegate Node tree. This special scheme can be configured from the nested Port builder. See the detailed description in param-weaving-pattern.hpp

Todo:

WIP-WIP 1/2025 Node-Invocation is reworked from ground up – some parts can not be spelled out completely yet, since we have to build this tightly interlocked system of code moving bottom up, and then filling in further details later working top-down.

as of 1/2025, the Layer-3 builder seems to be beyond the current integration scope. //////////////TICKET #1389 : develop a processing builder

See also

steam::engine::NodeFactory

nodewiring.hpp

node-basic-test.cpp

Definition in file node-builder.hpp.

#include "lib/error.hpp"
#include "lib/nocopy.hpp"
#include "steam/engine/weaving-pattern-builder.hpp"
#include "steam/engine/media-weaving-pattern.hpp"
#include "steam/engine/param-weaving-pattern.hpp"
#include "steam/engine/proc-node.hpp"
#include "steam/engine/turnout.hpp"
#include "lib/several-builder.hpp"
#include "lib/format-string.hpp"
#include "lib/index-iter.hpp"
#include <utility>
#include <vector>

Namespaces
namespace	steam
	Steam-Layer implementation namespace root.
namespace	steam::engine
	Lumiera's render engine core and operational control.
namespace	steam::engine::anonymous_namespace{node-builder.hpp}

Typedefs
template<class POL , class I , class E = I>
using	DataBuilder = lib::SeveralBuilder< I, E, POL::template Policy >
	A builder to collect working data.

Classes
struct	UseHeapAlloc
class	NodeBuilder< POL, DAT >
	Top-level builder to create a single Render Node. More...
class	PortBuilderRoot< POL, DAT >
	Nested DSL builder scope to define a single Port for the enclosing Node. More...
class	PortBuilder< POL, DAT, WAB >
	Nested DSL-Builder context to define a regular media processing Port. More...
class	ParamAgentBuilder< POL, DAT, SPEC >
	Nested sub-Builder analogous to PortBuilder, but for building a »Param Agent Node«. More...
class	ProcBuilder
class	LinkBuilder

Functions
template<class POL , class D0 , uint siz, class BUILD >
	NodeBuilder (NodeBuilder< POL, D0 > &&, SizMark< siz >, BUILD &&) -> NodeBuilder< POL, PatternData< D0, BUILD, siz > >
	Deduction Guide: help the compiler with deducing follow-up NodeBuilder parameters.
auto	prepareNode (StrView nodeSymbol)
	Entrance point for building actual Render Node Connectivity (Level-2)
auto	retrieve (void *streamType)
	Entrance point for defining data flows and processing steps.

---

Class Documentation

steam::engine::anonymous_namespace{node-builder.hpp}::UseHeapAlloc

struct steam::engine::anonymous_namespace{node-builder.hpp}::UseHeapAlloc

Class Members
typedef HeapOwn< I, E >	Policy

Collaboration diagram for UseHeapAlloc: