Drawing Transforms: A Unifying Interaction Primitive to Procedurally Manipulate Graphics across Style, Space, and Time

1 Introduction

Figure 1:

2 Related Work

3 Design Space

3.1 Informational Interviews

We interviewed three professional motion graphics artists and designers who work with both direct and symbolic tools. We received IRB approval and participant permission to participants’ identities. Miwa Matreyek ¹ is a performance artist who creates animations with AfterEffects. James Paterson ² is an experimental animator who uses symbolic tools animate hand-drawn animations. Kurt Kaminski ³ is a media artist who uses programming languages to build particle systems and real-time AR animation. Each interview lasted 1.5 to 2 hours. We developed custom interview frameworks for each participant by analyzing samples of their artwork (see fig. 2 for examples) and developing questions targeting specific effects for object animation, layout, and content generation. Across all interviews, we focused on each artist’s use of manual and computational tools and methods to organize animations spatially and visually. We recorded each interview and discussed initial observations and impressions after each interview. We analyzed the recording transcripts to conceptualize themes on distribution strategies, timing manipulation, and manual input. Our approach centered on reflexive analysis with a focus on emergent themes. We also distilled workflow descriptions for specific effects to explore the range of methods in procedural and manual digital visual art.

Figure 2:

3.1.1 Interview Theme #1: Arrangement-Level Visual Design Practices.

All three artists create and manage representations of groups of animations. Matreyek creates groupings of related graphic elements and then animates each group in the direct manipulation tool AfterEffects. She manually adjusts the animation effects of individual graphics to achieve variation. Kaminski uses symbolic programming languages to author custom particle behaviors that integrate fluid simulations with audio synthesis. He described how he envisions a collection of particles as a single entity with properties that can manifest individually. Paterson works across manual illustration and symbolic programming and he also develops his own direct manipulation animation tools. He described the differences between working on manually created artwork and “controlling visual arrangements with code.” He emphasized how he works back and forth between symbolically describing “arrangement patterns” and iterating on manually created forms. Paterson, Matreyek, and Kaminski’s workflows demonstrate how artists work at an arrangement-level to develop coordinated visual effects.

3.1.2 Interview Theme #2: Precise Manipulation of Timing across Manual and Procedural Methods.

Each artist relied on precise timing control regardless of their method. Paterson uses woven loops–variable length loops of animation where the last frame flows seamlessly back to the first– to create phasing effects. He also creates visual density by staggering the starting points of visually related animations, like a cloud of ghosts, across time and space. Matreyek uses manual methods to manipulate loops of rotating images. She manually adjusts their position to correspond with other animated elements like a wave. Kaminski enacts precise timing control over his particle effects to develop interactive visuals for augmented reality. He fine-tunes the timing to generate cascading effects across multiple particles. Collectively all artists increase the visual complexity of their animations by manipulating a group of animations across time and fine-tuning their temporal behaviors collectively, either through manual or procedural methods.

3.1.3 Interview Theme #3: Value of Manual Input.

All three artists valued manual input for enacting control and engaging with their artwork regardless of whether or not they used symbolic languages. Kaminski desired easier mechanisms to leverage physical inputs like drawing curves as input. He described the improvisational opportunities of designing animations that respond to gestural inputs. Matreyek described using her hands as input to create manual variation in the timing of particle-based animation behaviors. She also described the importance of manually manipulating easing functions to produce an effect that is “not just the same mechanical movement.” Paterson developed a custom VR animation tool centered around manual sketching. He emphasized how using drawing as the primary interaction modality made the tool “as physical as possible...leaving that spell of flow and expression unbroken.” Overall using manual inputs created expressive opportunities and facilitated continuous workflows.

4 Drawing Transforms

Figure 3:

4.1 Structure of the DT Interaction Primitive

A drawing transform (DT) is a procedural transformation function that interprets input geometry to transform the spatial, stylistic, and timing properties of a target. Input geometry consists of one or more continuous, closed, or open vector-graphic paths with a start and end point. This can include but is not limited to lines, polylines, arcs, Bezier curves, ellipses, and irregular polygons. A DT procedurally transforms a target: a graphical element. In practice, targets are individual vector graphics or ordered collections of vector graphics. We visually summarize the DT primitive in Figure 3. Any ordered collection can serve as either input or target geometry for a DT. This could include existing vector-graphic collection representations like Illustrator groups–where the order is implicitly assigned, drawing order, or lists in procedural direct manipulation tools like Para [20]–where the order is explicitly set by the artist. Finally, because DTs themselves contain vector graphic input geometry they can also function as targets and be transformed by other DTs.

With DTs, drawing is akin to authoring a function. The DT transformation function evaluates input geometry through five parameters: 1) geometry parameterization, which calculates numerical values from input geometry, 2) property mappings, which control how target artwork is transformed, 3) behaviors, which determine how property mappings function for targets comprising multiple elements, 4) mode which specifies if a transformation is relative or absolute, and 5) playback points, which are used to trigger discrete events when a DT is executed. We detail each parameter below.

Figure 4:

4.1.1 Geometry Parameterization.

The geometry parameterization segments input geometry at even intervals to output a sequence of numerical values. Segmentation depends on the drawing direction or the start and end of the vector path. Each DT segments geometry in one of two ways. The first segmentation approach is x-value where paths are split as even segments along the x-width. X-value is like using a straight edge for measurement (see fig. 4-a). The second segmentation approach is path-length where the path is split as even segments along its length (see fig. 4-c). Path-length is like using a measuring tape that winds along the entire length of the path.

DTs support path-length parameterization for two reasons. First, path-length parameterization is not dependent on traversing an axis from left to right. Instead, the artist can use their direction to describe procedural functionality. Second, path-length parameterization also simplifies describing periodic structure with hand-drawn forms. For example, say the artist seeks to create a seamless looping animation of a bouncing ball. They can do so using an x-value segmentation of a waveform function (see fig 5-a) wherein x-value corresponds to time and y-value is mapped to the ball y-position. However to ensure a seamless loop they must draw a precise waveform. The artist can create an identical transformation using path-length parameterization on a circle wherein path length corresponds to time and y-value corresponds to the ball y-position (see fig 5-b). If the DT is played and looped, the circle serves as a literal representation of a seamless loop.

Parameterized segments are numbered. They start at index 0 (the start point) and continue as the artist draws. For closed shapes, segmentation start and end points depend on the geometry implementation of a given system. For our testbed, we use the Paper.js [23] vector geometry implementation wherein the start and end points of a closed shape are set at the lowest point of the shape on the y-axis. The total segment count is determined by a predefined parameter. In practice we set this parameter dynamically with respect to the mapping in our testbed implementation; however, depending on implementation, the artist could directly specify this value for greater control. The segment index is used to drive the function in an unambiguous fashion. For example, say an artist draws a vertical zig-zag path where there are multiple y-values for every x-value on a standard x-y graph. Because segments depend on drawing direction, the system calculates the y value for each segment with respect to path length and x-intersection. Figures 4-b and 4-d show examples of x-value and path-length parameterization of zig-zag paths respectively.

Each segment outputs four numerical values: segment index, x-value, y-value, and path-length. Artists can use the same input path to specify a function across time or across a collection. Depending on the mapping, the DT will assign values to the target with the segment index corresponding to either time or collection index. If multiple paths are used as input, for every segment index the DT will calculate an array of x-value, y-value, and path-length values to update the target. We provide further detail on interpreting multiple paths in section 4.1.3.

Figure 5:

4.1.2 Property Mappings.

The property mapping controls how the target artwork is updated in response to the values generated by the geometry parameterization. Default property mappings consist of lower and upper bounds and set functions which linearly map a numerical value to a graphical property (e.g. property[min, max] = f(time or index, x, y, path length)). For example, consider a DT that uses the hue property to map output y-values from the geometry to a range of hue property values from 0 to 360. The property mapping sets the hue property of the target to that mapped value.

The min and max ranges can be set to the default ranges for the given property. In our testbed, we include four categories of property mappings specific to visual design applications, listed in Table 1. Property mappings could be expanded or modified to support specific application domains. For instance, in our testbed, we use a property mapping to duplicate new elements where the property duplicates can be used to create or remove copies of an element. Together, parameterizations and property mappings produce different outcomes. For example, when a DT with a position property and path-length parameterization is applied or played, the input geometry is interpreted as a set of position values for multiple graphic elements in the target artwork, or a motion path (see fig. 1-a and c).

We reify how the property and parameterization are used to interpret geometry through a graphical representation, which we call the mapping context. The mapping context maps the x or y-axis of the bounding box of input geometry to the property mapping range and the segmentation basis (x-width or path-length) to time or an indexed list. In our testbed, mapping contexts can be directly manipulated through operations like dragging the y-min and y-max values.

Figure 6:

4.1.3 Behaviors.

Behaviors determine how a DT samples and interpolates mapped property values by index to transform a target collection of vector graphics (e.g. Targets[i] = alternate(i, f[inputs])). In our testbed, we implement three behaviors listed in Table 1: interpolate, alternate, and random.

Behaviors also determine how values are calculated with respect to the geometry of multiple paths. Figure 6 demonstrates sample motion-path transformations for a DT that uses two vertical curved paths as input to control the motion path of a collection of nine circles. In this example, interpolation produces nine different motion paths that correspond with an interpolation of the points across the original two input paths. Alternate produces two motion paths that correspond with the input paths, with odd-indexed circles traversing a path that corresponds with the left-most input path, and even-indexed circles traversing a path that corresponds with the right-most input. Random samples a random interpolated path using the input geometry paths with indices closest to the relative randomized index from the artwork in the target collection. The interpolated path is then used to calculate the value update of the target artwork.

4.1.4 Modes.

Modes enable describing two types of transformations for a property mapping: absolute or relative (see table 1). The mode determines if a property is set with or without respect to its initial value. The transformation begins by either resetting or preserving the original property value for the target based on the mode. In our testbed, artists select a mode with a dropdown.

4.1.5 Playback points.

Playback points determine how DTs elicit discrete events during geometry evaluation to trigger other functions. They consist of trigger-value pairs of pre-defined geometric conditionals like slope change and other functions, including DTs. Each type of playback point, such as slope change, can be calculated based on the outputs from the parameterization (e.g. segment index, x, y, path length) and evaluated as the DT executes. Granularity affects accuracy, so the system may evaluate finer-grained segments to assist with playback points if a threshold is detected. In our testbed, artists can select playback point triggers through a drop-down and then indicate the triggered function by drawing a link to the artwork used as input to a specific DT.

Table 1:

Mappings
Spatial	E.g. position distribution, motion path, scale animation.
Stylistic	E.g. hue distribution.
Geometric	E.g. polygon number of sides distribution, vertex animation.
Instancing	E.g. duplicate each.
Behavior
Interpolate	Generate a Lagrange polynomial using input, sample with target index / total input paths.
Alternate	Cycle through input geometry using the target index % the number of input paths.
Random	Generate a Lagrange polynomial using input, sample using a randomized index.
Modes
Absolute	Set the target state to the start state given by the input.
Relative	Transform with respect to the original target state.

5 Demonstrative Evaluation

5.1 Sample AMES Workflow

Figure 7 shows an artist creating a DT in the AMES UI to animate goldfish. After she draws a goldfish, she uses the collection tool to select the goldfish and create a collection. She then drags the count value (shown in UI on the canvas in a green box) to create additional copies. She draws two paths and adds them to a second collection. Using the DT button, she creates a new DT and corresponding DT editor. She draws links from the input field in the DT editor to the path collection and the target field to the goldfish collection. In AMES, we made a design decision to combine selecting property and parameterization. In our examples, static transformations combine a property with x-value parameterization, and transformations in time combine a property with a path-length parameterization. From a dropdown, the artist selects motion path (path-length parameterization and position property mapping). She adds a playback point using the playback point editor. We use this basic interaction structure in the examples we describe below.

Figure 7:

5.2 Generate Shapes and Animations

DTs enables using manually drawn paths to author different types of procedural behavior, while also supporting the reuse of manually created geometry in multiple ways. In our first example (fig. 8), an artist creates a twinkling starfield by animating procedurally generated artwork. This example recreates and extends the approach used in the Para software [20].

Figure 8:

First, four DTs are used to procedurally generate and arrange artwork. The first DT controls procedural shape generation. This DT uses two short, vertical paths, one drawn up and one down, as the input geometry to modify the vertex position of a regular polygon (fig. 8-a). By applying this animation with alternate and relative behavior, each input path transforms every other vertex inwards or outwards based on its starting position relative to the polygon path. The DT creates a star shape. Moreover, it encodes a procedural function to create a star shape on any regular polygon with any number of sides. The artist can modify the number of sides of the polygon and experiment with different star shapes. The DT uses the drawing direction to specify meaningful information about how the geometry is interpreted. Next, two DTs that use the same input geometry are used to vary the hue and scale across a collection of 15 stars (fig. 8-b). The artist can edit the mapping context of the scale property mapping to modify the range of scale values that the DT uses to map output values that set the scale of each star. Because the same geometry is used to generate output values for both functions, the relative relationships between the hue and scale of the stars will remain the same. The last DT that we use to arrange artwork positions the stars at random locations between two hand-drawn lines by using random behavior to map the outputs of the DTs to generate random values constrained by the input geometry (fig. 8-c). This interaction enables a random position distribution defined by hand-drawn geometry.

We use two additional DTs to animate the stars– one to specify vertex animation and one to control animation rate. The animation transformation DT reuses the same input geometry used in the first DT to specify a vertex animation. Instead of playing this animation once, which would deform the target shapes, we use a circle as input geometry to a second control DT. By playing the control DT, the animation progresses from its start to end state and then returns to its original state, 8. The control DT also modifies the easing of vertex geometry animation: the stars expand slowly at first, then increase and decrease in speed. They contract with the same easing effect, returning to their original state. Repeating the execution of the control DT loops the animation seamlessly. In the final result, shown in fig. 8-e, the stars twinkle perpetually.

Overall this example shows how DTs enable the same parametric artwork generation and layout features as Para, with greater flexibility. We can control both the geometry and the layout with the same underlying representation. Furthermore, unlike Para, this representation can also support the animation of the resulting artwork.

5.3 Recreate Artwork made with Textual Code

DTs can recreate event-driven sequences originally created in symbolic textual programming tools. In our second example (see fig. 9), we use DTs to recreate an animation made with a textual programming language [43] by artist Dave Whyte⁵ [56]. We refer to this artwork as the N-Gon. In the comparison of Whyte’s work and our recreation, we show the animation behavior of the N-Gon (fig.9-h, i): circles traverse as nested regular polygons with different numbers of sides. As each circle changes direction on the corner of a polygon, it is duplicated. The duplicated copy increases in scale and disappears. The circle hue matches the hue of its motion path.

To create the N-Gon with our approach, we first use two DTs to create the nested polygon structure. They both transform a collection of six triangles. The first DT uses a hand-drawn line as input geometry to scale the triangles, creating a set of nested shapes (fig. 9-a). The second DT uses another hand-drawn line as input geometry and uses a property mapping to set the number of sides of each shape (fig. 9-b). Both input geometry paths reify the parameterization of the N-Gon geometry based on the scale and the number of sides respectively. Each one can be individually manipulated to transform a key visual attribute of the N-Gon.

Next, we re-create the procedural event-driven animation where circles traverse the nested geometry as motion paths and scale up at the corners. We use three DTs to do this. The first DT uses a short hand-drawn curve as input geometry to define a scale animation (fig. 9-e). We use a playback point on this DT to remove the circle at the end of the scale animation. The second DT uses the same hand-drawn path to control the duplication of circles (fig. 9-d). We use a playback point on this DT where new instances trigger the scale animation. The third DT uses the nested polygon paths, (the procedurally generated artwork described above), as the input geometry to specify motion paths (fig. 9-c). We add a playback point so that the slope change that occurs at the corners of each polygon triggers the duplication DT.

Together, these three DTs enable recreating the N-Gon animation through graphical representation rather than symbolic code. When we play the motion path DT, each circle is animated around a polygon path. When the slope of its trajectory changes, the circle is duplicated. The newly instantiated circle is scaled up before it disappears. Additionally, because DTs represent procedural relationships as input geometry, an artist can create variations of the DT by editing vector graphics and visualizing geometric relationships (fig. 9-g), as opposed to editing numeric values or using algebraically-defined functions. DTs support reusing geometry in multiple ways which enables defining meaningful procedural relationships directly on the canvas. Here, the nested polygons are used on the canvas as artwork, as motion paths, and to evaluate conditional events.

Figure 9:

5.4 Author Particle System Behaviors

Prior procedural direct manipulation tools like Draco [26] and Kitty [25] rely on predefined particle behaviors to create animated textures. We demonstrate how DTs can be used to author particle system behaviors from the ground up by generating three distinct stylized particle system effects (fig. 10).

5.4.1 Particle Effects: Fireworks.

DTs enable 1) automating the generation of new elements using user-defined events and 2) using new instances to trigger animations represented by expressive, hand-drawn artwork. In Figure 10-a, we combine these two affordances to create an animation of a firework.

This example uses four DTs. The first DT describes a motion path and uses hand-drawn paths in a firework shape to animate particles. The second DT is a control DT that changes the easing of the motion path animation. Instead of the animation playing at a constant rate, the particles accelerate as they move outward. The third DT duplicates the particles. We use a playback point to connect the end of the second DT to the third DT so that after the firework explodes, new particles are generated. The fourth DT specifies a scale animation, and uses a playback point to remove artwork at the end of the animation. New instances trigger the scale animation.

The result is a firework-like particle system: as particles explode outward new particles are created that pulse briefly and disappear. We can loop the execution of the control DT to loop the particle system behavior.

5.4.2 Particle Effects: Rain.

Because DTs enable defining geometric procedural relationships and using geometry as flexible input, an artist can use DTs to interpret geometry to define many kinds of procedural functionality including defining a clock. In Figure 10-b, we create a rain effect that consists of procedurally instantiated droplets at a rate determined by a motion path that acts like a clock. Rain is distributed evenly across motion paths that are positioned randomly between two horizontal lines to create uneven, but structured, animation behavior. The first DT transforms the position of vertical paths across a space given by two horizontal lines. The second DT uses the vertical paths as motion paths to transform raindrops. Raindrops are procedurally instantiated using a DT that maps a hand-drawn curve to a duplication rate. A playback point uses new instances to trigger a motion transformation. Another playback point triggers removing the raindrop at the end of the motion path animation. The last DT creates a motion path-based animation of a small circle that traverses a large circle. This DT provides the timing functionality; at the end of each iteration, it triggers the duplication DT to create more rain. The timing DT determines the frequency and intensity of the rain.

Both the firework and rain example use the same number of DTs. Because DTs support many types of procedural control, the artist can create an entirely different effect using low-level procedural functions as building blocks.

5.4.3 Particle Effects: Smoke.

DTs enable using a few input geometry paths to interpolate transformations of many target elements. We demonstrate how this affordance enables the approximation of the fluid movement of multiple particles. In Figure 10-c, we use DTs to create a particle system that resembles smoke. This example uses four DTs. The first modifies the hue and scale of a collection of translucent circles. The remaining three specify the animation and use interpolate behavior to smoothly interpolate transformation functions across multiple elements. The second and third DTs use two hand-drawn paths that have spirals like smoke as input geometry. The second DT defines a motion path transformation of the particles, which are removed at the end of the animation with a playback point. The third DT specifies the rates of procedural instantiation of the smoke particles. Because one path is shorter than the other, unequal numbers of copies are made. Fewer large purple particles are duplicated in comparison to the small green particles. The fourth DT defines the motion path animation of the smoke particles moving up a chimney. At the conclusion of the motion path animation, the DT triggers the duplication DT to create smoke particles.

The final effect is a smooth animation where smoke particles are emitted continuously. They swirl upwards in circular patterns. This example shows how manually drawn motion paths can serve as a structure to a generative effect in DTs.

Collectively these examples suggest that DTs could offer a means to provide greater expressive control by creators in direct manipulation by enabling them to edit existing behaviors or create their own without resorting to a symbolic representation. When we asked one of the artists from our formative interviews to provide their impressions of a full demo of DTs and its potential application to their work, Kurt Kaminski shared “I wish more content creation applications had interfaces like this. I use Houdini and the interface is not geared toward gestural input. I would love to see DTs integrated into Houdini, or even more so in Photoshop or AfterEffects which lag in both gestural and procedural tools.”.

Figure 10:

6 Limitations

7 Discussion

8 Conclusion

Acknowledgments

A Pseudocode for DTs

For every element in the target…

... If the mode is absolute, call get_transform_value to get the value of this DT at the start state (segment index = 0) and call update_target

... If this DT is static, call get_transform_value based on the at the state mapped to the target index and call update_target

... If the transformation is dynamic (temporal), call playback_helper

A base case evaluates the stopping and looping conditions for the DT: IF (!reverse && state_idx >= stop_state_idx) || (reverse && state_idx <= stop_state_idx) IF loops[target_idx] < max_loops call transform ELSE return

Call get_transform_value based on the curr_state_idx and next_state_idx and pass the output values to update_target

Call playback_helper for the next state

If the DT is static, call calculate_state on curr_state_idx

If the DT is temporal, calculate difference between the outputs from calling calculate_state on curr_state_idx and next_state_idx

// Exact sampling is determined by behavior or if the input is a collection; details omitted

Sample the input: call calculate_state on one or more input paths using an index given by the behavior (e.g.alternate), then interpolate or select across those values according to the behavior

Return calculated tuple (dx, dy, dv) values based on x value, y value, and path length

If the parameterization is path-length return point on the input at the segment s_idx

If the parameterization is x-value...

... If the path is non-looping calculate the intersection point of the x-axis at the segment s_idx and the input artwork path

... If the path is looping use the point of the input path at segment s_idx to calculate the nearest segment on the x-axis to calculate the nearest intersection point

Linearly map the point to the property range given in tf_space and return the new point

Update the execution state and check if any playback point conditions have been met based on the execution state. If so, call the playback point values (functions)

Call the property function of the target passing in dx, dy, or dv

Footnotes

References