Lumiera
The new emerging NLE for GNU/Linux
State Final
Date 2009-11-01
Proposed by Ichthyostega

The situation focussed by this concept is when an API needs to expose a sequence of results, values or objects, instead of just yielding a function result value. As the naive solution of passing an pointer or array creates coupling to internals, it was superseded by the GoF Iterator pattern. Iteration can be implemented by convention, polymorphically or by generic programming; we use the latter approach.

Lumiera Forward Iterator concept

Definition

An Iterator is a self-contained token value, representing the promise to pull a sequence of data

  • rather then deriving from an specific interface, anything behaving appropriately is a Lumiera Forward Iterator.

  • the client finds a typedef at a suitable, nearby location. Objects of this type can be created, copied and compared.

  • any Lumiera forward iterator can be in exhausted (invalid) state, which can be checked by bool conversion.

  • especially, default constructed iterators are fixed to that state. Non-exhausted iterators may only be obtained by API call.

  • the exhausted state is final and can’t be reset, meaning that any iterator is a disposable one-way-off object.

  • when an iterator is not in the exhausted state, it may be dereferenced (*i), yielding the “current” value

  • moreover, iterators may be incremented (++i) until exhaustion.

Discussion

The Lumiera Forward Iterator concept is a blend of the STL iterators and iterator concepts found in Java, C#, Python and Ruby. The chosen syntax should look familiar to C++ programmers and indeed is compatible to STL containers and ranges. To the contrary, while a STL iterator can be thought off as being just a disguised pointer, the semantics of Lumiera Forward Iterators is deliberately reduced to a single, one-way-off forward iteration, they can’t be reset, manipulated by any arithmetic, and the result of assigning to an dereferenced iterator is unspecified, as is the meaning of post-increment and stored copies in general. You should not think of an iterator as denoting a position — just a one-way off promise to yield data.

Another notable difference to the STL iterators is the default ctor and the bool conversion. The latter allows using iterators painlessly within for and while loops; a default constructed iterator is equivalent to the STL container’s end() value — indeed any container-like object exposing Lumiera Forward Iteration is encouraged to provide such an end()-function, additionally enabling iteration by std::for_each (or Lumiera’s even more convenient util::for_each()).

Implementation notes

iter-adapter.hpp provides some helper templates for building Lumiera Forward Iterators.

  • IterAdapter is the most flexible variant, intended for use by custom facilities. An IterAdapter maintains an internal back-link to a facilitiy exposing an iteration control API, which is accessed through free functions as extension point. This iteration control API is similar to C#, allowing to advance to the next result and to check the current iteration state.

  • RangeIter wraps two existing iterators — usually obtained from begin() and end() of an STL container embedded within the implementation. This allows for iterator chaining.

  • PtrDerefIter works similar, but can be used on an STL container holding pointers, to be dereferenced automatically on access

Similar to the STL habits, Lumiera Forward Iterators should expose typedefs for pointer, reference and value_type. Additionally, they may be used for resource management purposes by “hiding” a ref-counting facility, e.g. allowing to keep a snapshot or restult set around until it can’t be accessed anymore.

Tasks

The concept was implemented both for unit test and to be used on the QueryResolver facility; thus it can be expected to show up on the session interface, as the PlacementIndex implements QueryResolver. QueryFocus also relies on that interface for discovering session contents. Besides that, we need more implementation experience.

Some existing iterators or collection-style interfaces should be retro-fitted. See Ticket #349.
Moreover, we need to gain experience about mapping this concept down into a flat C-style API.

Alternatives

  1. expose pointers or arrays

  2. inherit from an Iterator ABC

  3. unfold the iteration control functions into the custom types

  4. define a selection of common container types to be allowed on APIs

  5. use active iteration, i.e. pass a closure or callback

Rationale

APIs should be written such as not tie them to the current implementation. Exposing iterators is known to create a strong incentive in this direction and thus furthers the creation of clean APIs.

Especially in Steam-Layer we utilise already several iterator implementations, but without an uniform concept, these remain just slightly disguised implementation types of a specific container. Moreover, the STL defines various and very elaborate iterator concepts. Ichthyo considers most of these an overkill and an outdated aproach. Many modern programming languages build with success on a very simple iterator concept, which allows once to pull a sequence of values — and nothing more.

Thus the idea is to formulate a concept in compliance with STL’s forward iterator — but augmented by an stop-iteration test. This would give us basic STL integration and look familiar to C++ and Java programmers without compromising the clean APIs.

Comments

Now in use since more then a year, without turning up any serious problems. The only minor concern I can see is that this concept, as such, doesn’t solve the problem with exposing implementation details of the underlying container on the API. Similar to STL Iterators, the actual implementation representation is only disguised behind a typedef. But, generally speaking, this is an inevitable consequence of the “zero overhead” abstraction. For the cases when an indirection (via VTable) is feasible, I’ve created the IterSource template, which sticks to this Lumiera Forward Iterator concept, but provides an opaque frontend, allowing to decouple completely from the actual implementation. Besides that, over time I’ve written several standard adapters for the most common STL containers, plus Map, key and value extractors.

Ichthyostega

Sa 16 Apr 2011 00:20:13 CEST

minor change: removed support for post-increment. It doesn’t fit with the concept and caused serious problems in practice. A correct implementation of post-increment would require a “deep copy” of any underlying data structures.

Ichthyostega

Sa 07 Jan 2012 21:49:09 CET <prg@ichthyostega.de>

Note Lumiera Forward Iterators can be made to work together with the range for loop, as introduced with C++11. The preferred solution is to define the necessary free functions begin and end for ADL. This is best done on a per implementation base. We consider a generic solution not worth the effort

This is to say, these two concepts can be made to work together well. While, at a conceptual level, they are not really compatible, and build on a different understanding: The standard for-loop assumes “a container”, while our Forward Iterator Concept deals with “abstract data sources”. The user needs to understand the fine points of that conceptual difference:

  • if you apply the range-for construct on a container, you can iterate as often as you like. Even if the iterators of that container are implemented in compliance with the Lumiera Forward Iterator concept.

  • but if you apply the range-for construct on a given iterator, you can do so only once. There is no way to reset that iterator, other than obtaining a new one.

See 71167be9c9aaa for the typical bridge implementation

Ichthyostega

Sa 04 Jul 2015 22:57:51 CEST <prg@ichthyostega.de>

Final

Accepted on developer meeting

Do 14 Apr 2011 02:52:30 CEST Christian Thaeter

Back to Lumiera Design Process overview