Embedding GUI Elements inside code

First off, you should read the introduction to the TextEditor. Everything described here takes place inside it. Secondly, the GUI Embedding features are also covered in this downloadable tutorial.

The central problem is one of speed. Field demands code when other environments let you drag a slider, push a button and so on. While writing code is undeniably powerful, it's sometimes not very fast. Typing out your numbers long form is from a bygone age. But the philosophy of Field is that code is everywhere, and dumbed down, pre-canned interfaces aren't. How can these two competing views of interaction be combined?

Click fullscreen (bottom right) and turn scaling "off" (top right) for best viewing.

Field's text editor offers an alternative way of making editing code as a fast as interacting with a graphical user interface — by simply quoting graphical user interface elements directly:

Here we set a variable “equal” to a slider. Slide the slider, and re-execute that line, and the variable has changed (and see below). You get a menu of these with a right click. These elements flow within the body of the text itself and the text insertion point simply skips over them; they can be copied and pasted like text (#20), deleted like text &c. The rule is simple: the graphical object can be as graphical and as interactive as it needs to be, but in return it must to be able to convert itself to text on demand — Python is still a text-based language after all, and, as stated previously, we choose to operate under the constraint that this language remain completely standard. So, at some level this text editor (and anything else in the delegation graph) still believes that it's editing text. To help maintain this charade, the graphical object gets to convert itself to both text and a small persistent “cookie” (a property, of course) which helps with the conversion back to UI element if needs be. Importantly the UI element needs to be dedicated to “being text” convincingly since should this element not be present in the editor, the cost of instantiating UI elements is never paid.

The GUI Element Zoo

Once we’ve broken free of our text-only shackles we have another two axes of extension in Field. Firstly, the lexicon of graphical elements that we can embed in the code, and secondly, the kinds of “objects” that these UI elements masquerade as when they are text.

This first expansionary front is simple to understand — the vocabulary of UI elements is well known, and many if not most of them are useful when embedded inside code:

(A "Point Picker", which selects a Vector2 or a 2-tuple)

(A "Graph widget", which makes some function over 0->1)

Obviously we can keep going through the standard set of graphical user elements: we we can have buttons that do things (because they are given callbacks in the code itself); popup color-wells that select a color; combo-boxes that set and edit strings; and so on. But we can also create elements that select from more structured data. For example, selecting a “joint” of a character from a hierarchically structured 3d scene is tedious and error-prone in code, but easy with a tree view of the data itself:

Note the idiom here, common to all such elements: we pass in the data to be displayed with the call to set(), which returns the object represented by the current selection of the tree view. Obviously, these things can get collapsed down to maintain the momentum and readability of our code. We can also have elements that are more strictly simply structured output — elements that draw histograms from the contents of lists have proved particularly useful — often, in work preparatory to the art, when one is not setting numerical parameters, one is understanding the statistics of signals. Such histograms and other pure visualizations often make appropriate background layers for number and range pickers for this kind of work.

One final class of embedded GUI is the visual element reference. While elements are free to talk about other visual elements using traditional means, care must be taken to permit the clean, complete deletion of elements. Holding on to and ultimately persisting “deleted” elements is generally discouraged. This can be avoided with the appropriate techniques, but some UI, customized for computing (based on some criterion, for example, pattern matching on the name, or matching on the spatial relationships between visual elements) and holding references to other visual elements in the sheet, has proven useful. This makes transparent the long term relationships between visual elements.

Having established a basic vocabulary of UI elements with in the appropriate internal interfaces, they can migrate to other areas of Field, decorating other, traditionally text, components: the output window of the text editor can contain all of these things (again a handful of patterns emerge: it’s sometimes useful to print, or indeed yield(), some buttons that offer up choices of what to do next in what becomes a very simple way of making step-by-step “wizards”). And, of course, these elements can be embedded not in the otherwise textual code inside visual elements but directly on the canvases themselves, effectively exposing a choice set of parameters, a visual front to the code inside. See EmbeddingGuiOnCanvas for a brief tutorial.

But perhaps more powerful, and less obvious, than adding to the standard library of graphical widgets that claim to be text, is thinking about what these graphical widgets can evaluate to. Here the malleability of Python's dynamism comes to the fore.

The simplest thing is to have the slider evaluate to something like “0.75” (that is literally the string “0.75”, excluding the quotes, which is textual way of indicating the number 0.75. It's important to remember that UI objects have to evaluate to a text fragment, not an object — they can evaluate to “Vector2(1,2)” but not some specific Vector2 instance.) This already lets us write things of the form:

But once we execute this assigning a value to a variable it loses its dynamism — any trace of those sliders &c are erased, their values “baked”. When we set and execute ‘x=((slider))*5‘ and the slider is at 0.5, x is forever 2.5 even if we change the slider later. This is the unfortunate result of having graphical elements “evaluate to text”. We can avoid this by making sliders evaluate to a (string that creates a) dynamic proxy that transforms numerical operations into operations that are evaluated lazily. In this way the dynamic UI penetrates deep into the structures set up by the code, rather than just its surface, and by changing the slider things “far away” can automatically update. See LazyFunctionalHelpers for a close up of this magic.

Block Markers

In addition to this growing set of GUI elements directly embedded inside code, there is one more class of graphical object in Field, related to the task of rapidly exploring code — the block marker:

A block marker is just a normal text field, but embedded, in turn, into the text editor. The convention here is that the label ‘(number).name’ starts a block and ‘\(number).name’ ends it. In our original formulation of UI element evaluates to text, these simply evaluate to non-executed comments, but in Field's text editor, pushing command-(number) when the text cursor inside the block all of that block is automatically updated. Now one can slide around the control knob of a slider all the while pressing ‘command-1′ to execute the block it’s in. Very fluid experimentation of code is now possible.

But again, we've added a small amount of additional structure to Field and seek to mine a diverse range of possibilities from it. Now that there is a convention that marks a block of code in the editor, we need to open this possibly back out to code itself. A new set of possibilities suggest themselves: firstly we can use the presence of the block markers to indicate that we’d like to transform the block of text itself.

Here’s a “block transformation” that wraps the text in our generator-executing environment and offers up a place to draw a progress marker:

The fraction returned by yield is drawn as an increasing pie chart, and whether the block is executing at all is indicated by a conventional color coding. Clearly the progress marker code is quite involved (and, ultimately, mostly in Java), but simpler transformations are easy, and implementable in place just by writing functions in python that manipulate text and call exec — Python’s statement that executes strings. Again: these blocks aren’t just visual text editor things — say, like folds in many text editors — rather, they are just like sliders: they “evaluate” to text that becomes part of the source (in this case a function call with a pair of opening and closing python multi-line quotes passing in the block as one of its arguments). So whether one is using the text editor affordance of pressing command-(number) or whether the code is selected and executed, or whether the whole file is executed as a script without any text editor being involved, transformed blocks do actually transform the code inside them — they exist, and transform code, even when there's no TextEditor looking at them.

Three illustrations of these blocks follow, both exploiting this structure and doing so with almost no pre-arranged support from a hidden Field “standard library”. Rather, these transformations are created in-situ, and spontaneously.

The first example is from Forest, a 5 screen, 5 GPU / CPU artwork. In Forest we introduce a block transformation that can cause the text “transformed” to be executed on a different Field process, running on a different computer. This uses networking code that's entirely standard to the Python library. The second example is integration between Field and Maya, which, in the case of its proprietary scripting language “MEL” allows commands to be sent across a telnet port. Using a text transformation we can embed this foreign language directly into Field's text editor:

Note in this example the text transformation automatically uses Python’s string formatting constructs to inject local variables into the foreign language, allowing in the above example for MEL code to be looped over by a Python for-loop. See MayaIntegration for more ideas. We'll simply note here that this sub-language executes not in Field but in the Maya runtime. Similar bridges to Apple's “Open Scripting Architecture” allow the embedding of AppleScript, which in turn has enabled Field's use as an asset manager and workflow organizer — calling upon other software applications to process and update files. Thirdly, and finally, we note a recent and exciting addition to Field — the Scala language — can be currently embedded inside Field using block transforms.

Prompts

Finally, a recent, work in progress, addition that opens up a new class of GUI elements for Field — the Prompt:

Most simply the prompt is a button that masquerades as a python generator / java iProvider that "blocks" until you push it (it doesn't really block in the threading sense — everything else gets to go on executing — it blocks in the coroutine sense). If the above code is executed with -shift-return or inside a generator-transformation block, or inside a plain old generator, the code doesn't advance until you click the button — one advance per click, unless you hold down option-click (just like executing a visual element from The Canvas or a text area from the execution ruler). A few design elements: the prompt turns red when it's actually holding up execution, green when it's been option-clicked. In the future, we should be able to call the whole execution off (by throwing an exception inside PythonGeneratorStack). Finally, we note that, with no UI to interact with the prompt defaults to advancing.

To read more about where this idea comes from and ends up, see GeneratorExamples.