Scram Devlog, some fun videos
June 04, 2019
I’ve been learning a lot about matrix math and its applications in 3D rendering. I don’t have time to write a full post about this yet, but here’s some teasers of some of the work I’ve been doing while building Scram — the planning poker app for remote teams.
Mooq first attempt
My original intention was to make an app allowing users to swipe through movies they want to watch, tinder style. These are some early attempts at using the OpenGL perspective transform matrix generators built into Flutter, along with spring physics. I love springs!
Encountering a render bug
I encountered a bug with the render engine of Flutter culling my surfaces that should be visible when nearly perpendicular with the “camera.” I submitted a bug report which got things fixed, but it’s recently cropped up again and I have yet to file another bug.
This is the original error video. You can see that if I use the inspector, the objects are indeed in the render tree, but they are passed up for painting during the paint phase.
I was having trouble figuring out how to handle aspect ratios in a way that would make it easier for implementers of my library (at this point, only me) to specify their object locations and sizes in a device-size invariant manner. To help, I dove deep into learning Quaternion math. They’re essentially vectors that specify a point on a 4d hyper-sphere, used to avoid gimbal lock problems for axis-angle rotations that euclidian rotation would typically suffer from.
Using Quaternions, I was able to implement a rudimentary “debug” viewpoint. No end user would ever see the cards from this angle, but it helped me a lot in determining where things are. Humans primarly rely on our stereo vision to perceive depth, but since screens are flat, being able to defer to perceiving depth through motion really clarified things, and made debugging the matrices much easier.
You can see that things look weird. I had my aspect ratio math completely wrong, so cards are squished horizontally, and stretched vertically. The red boxes show the size of the “parent” view responsible for positioning the cards. The child views are always the same aspect ratio as the parent view, only smaller. This was a limitation I imposed in early iterations to allow me to focus on the basics first. My latest versions allow arbitrary child sizes, and will still position them correctly. In addition, the latest versions also choose a “natural” field of view and z scale based on how far the typical user holds their phone from their face. More details on that later.
Anyway, despite the obvious visual artifacts, the effect is getting there.
Fixing the aspect ratios
Eventually, I figured out what I was doing wrong with the aspect ratios in the perspective matrices. These look a lot more natural. Still experimenting with the debugger view. Also, notice how smooth those springs look. Delicious!
I decided to try wiring this up to TMDB.
You can see that as I drag cards, the cards behind slowly move themselves in. This was done as “hacky” as possible, just to get the concept in. Excuse the jank. It looks awful, honestly, but building this crappy version was instrumental in understanding how to wire up gestures in an environment with 3d transforms + a projection matrix.
The sensor, not the tasty kind. Since I put all the work into learning Quat math, I figured it would be a fun experiment to throw in some parallax using the phone gyroscope.
At some point, I implemented a decay that causes the cards to constantly settle back to their flat position, but I unfortunately did not record a video, and am too lazy to look through my git history for that version.
I decided to move everything over to its own “Cards” repo, separate from Mooq with the intention of open sourcing my work.
I spent a significant amount of time architecting the various layers of the pipeline, and came up with a basic implementation of the
<Transition> element for Flutter inspired by react-spring.
There’s a ton of work that went into making arbitrary operations “just work” in terms of animating, and I will write a detailed post about how everything works after I open source my library.
The gist is that, as a developer, you specify a tree of transformation matrices, much like SceneKit in iOS, with your intended target layout.
<Transition> will do the work of animating your objects there using springs in a way that should look visually pleasing, and is easy on the device CPU and GPU.
Here’s roughly how I got to that point.
Once I realized I could make a generic
<Transition> element, it became clear that my goal of making a developer-friendly 3D-enabled
<Flow> widget would be possible as well without tightly coupling to any particular 3d layout or transforms.
I lifted everything out of Mooq, wrote the
<Transition> element, and hacked together an end-to-end example built with clean layers.
- Gesture wrapper
- Transition element
The “gesture wrapper” sits at the highest point of the widget tree and is responsible for maintaining the state of gestures on cards. It also produces the matrices that define the transforms of each individual cards as callbacks to the Transition element.
The “transition element” is purely responsible for asking its parent (gesture wrapper) for target matrices, and animating some child state when those matrices change.
The “render” layer combines the animated position state from “transition,” and the children widgets themselves from “gesture wrapper,” and renders them in a 3d space.
This is basically just my souped up
This is all very abstract - I’ll write a better explanation when I actually release my library.
Here’s the basic example I hacked together first to get a feel for how everything will fit together. It looks ugly, but that’s obviously not the point. Mostly I just wanted to understand how a developer would integrate touch into a transition-able view.
To start, I made a testbed displaying a single card where I can tweak the individual inputs using a control panel to make sure everything looks right.
I built a second testbed with much more helpful controls.
- parent resizing - to ensure resizing the view will still render children nicely.
- opacity adjustment
- child aspect ratio adjustment
- percent-based padding adjustment
- number of cards to test
It also support rotating the camera using an arcball control that I wrote.
Since the render layer is now completely uncoupled from the animation and business logic, it was trivial to write some fun examples of card layouts. Some of these are incomplete, especially the last few.
Bringing it all together with Transition
Finally, I integrated everything end to end again with the Transition element and gesture detection. This time, significantly less hacked and with each component completely uncoupled.
I want to specifically call out the power of the
<Transition> element to gracefully handle every state transition.
The user can gesture a card off the screen, or use buttons to simulate network requests changing the props passed to the
Transition will automatically diff the elements in both cases and animate appropriately.
It’s difficult to explain this without any code, but the gist is that every animation you see here is driven by simply removing or adding an item from a list.
When an item is removed from a list,
<Transition> builds a “tombstone” widget to animate offscreen.
As a developer consuming this library, it can’t get much easier than this.
Note how when I rapidly right-button cards and put them back, it automatically “cancels” the tombstone animation and puts back the original card in its proper place. Even with the chaos slider, everything “just works.”
This is an unorganized video I recorded early on. I only include it because it shows a bit where I replaced the colored cards with playing card images. They look quite nice in a fanout.
One more unorganized/earlier video. This one just has a little more perspective on the rotation, so the effect is more exaggerated. One side effect of how I implemented the gestures is that the card springs to a flat perspective when the user starts interacting with it. Fancy :)
Release as open source
I want to release what I have as open source when I get around to cleaning it up. Flutter is fun, and 3D UI is fun, so they go well together :)
For my own educational purposes, I’m writing an app called “Scram,” to help remote teams with their scrum process — in particular, planning poker. I’ve always wanted to release an app on the Android or iOS app stores, and since I spent all this time making the tools necessary to build beautiful animated card UIs, a planning poker app seems like the perfect fit.
I’ve been working on mockups in Adobe XD — a design tool I’m teaching myself for the express purpose of making this app — and it’s been pretty fun so far.
Here’s a simple example of starting a new round of planning poke and allowing other developers to join. I love the spring animations!