How a Dreams Creator Makes Mo-Cap and Pool Tables With Just Controllers, Tape, and String

This article first appeared on USgamer, a partner publication of VG247. Some content, such as this article, has been migrated to VG247 for posterity after USgamer's closure - but it has not been edited or further vetted by the VG247 team.

There's no denying it: Tyler "TannicAlloy" Whittaker's latest creation for Media Molecule's Dreams makes him look a little silly. In the game—and let's call Dreams a game, even though it's more like a powerful suite of creative tools—a puppet mimics the facial expressions Whittaker's making in the real world. Newer iPhones with depth-sensing cameras can easily do performance capture on the fly, but there's not a way to use a tool like that with Dreams (or at least there isn't yet). What Whittaker's done is attach two DualShock controllers to a hat, and then attach a few strings from the analog sticks to points on his face with tape.

It's a genius solution that opens up your mind not only to what's possible inside Media Molecule's Dreams, but to how those possibilities are further expanded when you stop thinking of a game controller as a run-of-the-mill input device.

Whittaker's Twitter and YouTube accounts are home to a handful of ambitious projects he's created in Dreams over the past year of early access. He started by recreating various instruments, among them: drums, a guitar with a whammy bar function, a violin with a Move motion controller bow, and a DualShock trumpet with a microphone for a mouthpiece. Some of Whittaker's creations involve complex real world builds, like his Steel Battalion-esque cockpit concept, or the giant cardboard and rubber band bow with a DualShock at its core he made for an archery game.

He's also a bit of a showman: a project where Whittaker pours water "into" his TV and watches the water level of the in-game scene rise is presented with a magician's flair, and even his demo of a simple-but-impressive heart monitor ends with a mischievous whip turn to camera.

Whittaker's taken Dreams and turned it into his own Labo set on steroids, and he's done it all without much prior design or programming experience. With the official release of Dreams on coming up next year, I wanted to pick Whittaker's brain to see if he had any advice for folks who want to follow his lead.

USgamer: Tell me a bit about your experience with design and programming tools prior to picking up Dreams

Tyler "TannicAlloy" Whittaker: I used Maya in my early teens. That only lasted a few years. The Personal Learning Edition was great, but to keep going with that would've been too expensive at the time. As far as software that I've had experience doing this sort of thing with ahead of time, there hasn't been much. I've got something of a knack for logic and reasoning, or at least I think so—my school transcripts have been pretty good, so—but as far as design and programming go, not much. I have a brother who's a pretty decent programmer, but outside of a half-hour of him showing me what he does, I don't have much experience with programming.

When trying out new ideas, do you find yourself mostly relying on experimentation, or do you regularly look to the Dreams community or even resources for other creative mediums?

I usually just keep throwing different prototypes at a project. I'll quite often have the desired effect in mind ahead of time: the easiest examples are my instrument prototypes. I thought, "Okay, if the microphone sends a positive signal, then I can plug it in here, here, or here." Then I learned that it's not sending an on or off signal, but partial signals; I'll remember that. Let's say I have something that requires a full signal as opposed to a partial signal, I'll keep plugging away until I realize, "Oh, it's just not receiving a full signal."

I've looked up a few tutorials from Dreams community members—NeonTheCoder is great for that sort of thing—but mostly it's been me seeing how things work and then, if necessary, have 40 different wires that achieve that effect when there's probably some tool that can do it in 2.

Is there something you've tried creating in Dreams that you haven't quite cracked yet?

My pool table project doesn't work how I originally wanted it to.

In Dreams, you can possess and move pieces of geometry, characters, or puppets with the controllers. I had wanted to have a motion controller tracked with the pool stick, and when I moved the stick there'd be a possessed piece of geometry in the level—an invisible cylinder, so it looks like it's just my pool stick—and have that hit the ball itself and track exactly. So I'd move slowly for a soft shot, quickly for a harder shot, how high I held the stick above the TV would determine if I was doing a spin or a jump. I wanted to essentially translate my movements into the scene itself.

That was just too much trouble to keep calibrated the whole time, so I scrapped that technique. The end product is essentially possessing a piece of geometry that's invisible and doesn't collide with any other geometry in the world. Then, the angle the controller is at determines the angle of a force applier. When I push a button on the controller, it'll pulse that force very strongly for a brief second. So I move the controller around, the angle determines the direction of the force applied, but there's no physical geometry that's impacting the balls themselves in that scene.

Do you have any advice for people who want to start making their own real world contraptions with Dreams when it comes out?

Before Dreams was in beta and early access, I was going wild with ideas and prototypes. Once I got my hands on it I realized, "Okay, some of this doesn't work the way I thought it did." But, I still tried to achieve those effects anyway. So one thing I'd really recommend is to start coming up with ideas now, and once you get into Dreams, mess around with the tools—and if something doesn't work, figure out why it doesn't work. Think of another way to achieve that effect, even if it requires tons of different tools and wires. Try to reach the goal in some different fashion.

My instrument prototypes were the first programming I did in Dreams. Looking back at the logic now—and I'm not done with those prototypes—there's an excessive number of wires. I could've shortened this thing or that, and cut stuff down to a quarter of the complications it has. So, come up with ideas now, but once you get into Dreams, have fun. Go crazy, and try not to have too much of an idea of a perfect project. Find out what doesn't work, why it doesn't work, and then use that discovery for something else.

What about tips for crafting sturdy, effective prototypes with DualShocks and Move controllers?

There have been a lot of things where I've strapped controllers to different parts of my body. One thing that I noticed early on is that I want the controller to be steady if I'm pulling on the thumbsticks with my arm, or with strings running to my fingers. It's kind of a balancing act—you don't want to have it too loose, to where you pull on the string and it just kind of nudges the controller. You want to have it secure enough so it's registering you pulling on that thumbstick, pressing that button, or having the gyroscope sense its at an angle, but not so loose that the controller just rattles around.

Precision is required sometimes, as far as needing to make sure the controller doesn't nudge out of the way, but don't stress things so much that it becomes a lot of work to get set up. With the pool project, for example, I wanted to get things set up and taped at precise angles. I thought I could tape the camera to the ceiling and angle it just right, make sure it's level... and it's actually kind of discouraging to have something be so much trouble. So try to find an easy way to do it, but not in any way that could cut off circulation, damage equipment, or become a hassle.

How I Made a Pool Table Using Dreams on PS4

Watch on YouTube

Are there any features you've found yourself wishing for as you've been working with Dreams?

There are a few I've got on my wishlist. I'd like to have tracking with two motion controllers at once. Even though when you're editing a scene there are two imps [Ed. note: the Dreams UI's cursors] on-screen, when you go into the play mode Dreams only tracks one motion controller. So if I possess something with the motion controller, the other one is rendered useless. This isn't a knock against Media Molecule. It might be an upcoming feature... and I have a couple ideas that would take advantage of tracking two motion controllers.

I want to do a couple projects involving controller rumble that would need the option to pick which controllers vibrate. Let's say you define a zone, and when a possessed object is in the zone, a controller vibrates. Then, you have four people with DualShocks possessing a character in the game—if any person steps on that zone, all the controllers vibrate. Same goes for sounds you play from the controller as well. I was thinking of doing Marco Polo with a couple controllers hidden around my room. When someone yells "Marco" and it detects microphone input, it would yell out "Polo" from all the controller speakers, but I want the different speakers to behave independently from each other.

I also think it'd be really cool to have DualShock lightbar control, being able to decide which color it shines. I'd love to be able to mess with that, maybe do a Simon Says thing where the different colors flash.

While Whittaker and other creators who got in on Dreams early can keep creating to their hearts' content up to and through the official release, newcomers will need to wait until February 14, 2020 to get in on the action. As Whittaker said, though, this is a fine time to start coming up with ideas—they won't go to waste. Thankfully, neither will all the ideas Dreams creators have come up with so far, as Media Molecule plans to bring all those projects forward.

For more advice on working with Dreams, check out USG's beginner's guide and our interviews with some of the biggest creators from its closed beta.

This interview has been edited for length and clarity.