Stepping Into Mixed Reality Is Easy… Creating It Is Not. A Brief Look At Tools, Skills & Insight Into Mixed And Virtual Reality Experiences

New Worlds To Explore

I’ve spent the last year + working on VR and mixed reality. Several years ago I made some concepts with Augmented Reality for several innovative prototypes. The one thing that hasn’t changed is crafting an experience takes time. The other truth is that crafting an engaging experience takes even more time, planning and resources. At this very moment I’m in a process of research, learning, and creating. It’s very reminiscent to the start of my career – the learning curve seems insurmountable. To create for these experiences you have to think beyond possible. I’ve been known to say crazy things like the following:

For experience designers today it’s easy to be stuck in existing mediums. It takes practice and extra effort to separate the activities we wish to complete from the tools we use to complete these activities (keyboards, mice, controllers, etc…). Only through time and experimentation do we learn what works and what doesn’t. At the end of this time period we start to see new commonplace design patterns emerge, up until this point we can only speculate if customers of today will even want to use these new methods.

It’s for this very reason we find it so hard to transition from a keyboard. It’s not that the keyboard isn’t a great invention its just that we have become so reliant upon normal design models to complete our everyday work. In many cases the perception of the new medium can instantly kill it’s progress and momentum beyond it’s availability and price. (I may talk more about this in a future post.)

The Alternate Realities Voyage So Far…

Years ago, it used to take quite awhile to put together an application. Today a robust system of tools, techniques and methods can be used to craft a very basic experience quickly. We are still a few years away from this for Mixed Reality & VR (general population) as it takes time for new mediums to mature and new design patterns to emerge (I’m already seeing a few in the realm of VR, more on this later.).

As you may have guessed this level of technology is something I have been dreaming about for many years.  The space is still very much a new frontier and just like my past pursuits the technology has a pretty high learning curve. Sure, this is a curve that can be countered with a team of skilled creators but, as luck would have it for some odd reason I like to torture myself by understanding all the inner workings and connective tissues of new interface systems. In addition to this, I usually like to apply a practical use cases to what I create. I often feel it’s not enough to just create without a healthy dose of skepticism.

AR / VR / MR The Future Design Tool Sets & Skills

I’ve never been one to shy away from learning all the roles and connective pieces to create digital experiences. This is exactly what I have been actively doing for the last few years  in my spare time.   I’ve been understanding how these technologies work, analyzing design patterns, and then reverse engineering the process to create and build my own. We are still years off from mainstream usage. The price points are pretty high. The technology and skills to create mixed reality experiences require new tools, new training, and patience. Lots and lots of patience.

The tools and methods to create are still very young.  You can check out a few of these here:  Google DayDreamGraybox VR, Unreal Engine.  Like I mentioned most are still extremely experimental. Despite this I put on my VR headset and prepared to do some world creating.

First step was to create my new landscape (yep, I had to learn how to do this all from scratch. It required lots of tool knowledge. I tried the gambit including Blender, Zbrush, Panda, Autocad, Maya, Maya LTD, and a few more. Each had an interface that had been around for generations. This was not a good thing, but many of these tools are still stuck in the professional use landscape (which means high learning curve, and powerful systems – not very accessible).

I pushed on…(more later)…

I must admit it was pretty awesome to step into my 3d modeled landscape and begin shifting mountains. Next, I would take my newly created mountain teleport to the peak and continue to craft new hills in the valley below. (this was something I could only handle for maybe 20 minutes before getting some VR sickness). I endured this until I ran into a slight frame glitch with a mis-placed particle effect. This pretty much + vertigo meant it was time to leave the VR world for today.

A very rough 3d model landscape i forced myself to build.
A very rough 3d model landscape I forced myself to build.

Another few days passed…

I spent some of the day yesterday creating in a tool called Holo Studio. It’s neat, but far from immersive. That being said it still is very magical. I wanted to create that magic, but like the first time I stepped into HTML and then CSS so many years ago. It’s a long road ahead. By now you may be asking yourself what does it take to actually do this as a single person and not a team? Here are the skill sets (thus far), which in time will yield new job roles and bring life to existing ones.

In no particular order:

  • Product Vision Mapping (What the hell does your experience do?)
  • Sketching / Concept Boards
  • Color & Design Theory
  • Game & Scene Level Design
  • Visual Design Tools
  • Game Design Tools
  • 3D Modeling Tools
  •  Texture Mapping
  • 3D Modeling
  • C# / Objective C (or other language programming skills)
  • Logical Flows & Story Boards
  • Illustration
  • Experience Design
  • Playtesting
  • Spatial & Environmental Mapping
  • Architecture Principles
  • Light Theory
  • Cognitive Psychology Design Principles
  • Presence (eventually will be a new rating criteria)
  • Physics

That’s a lot of learning yes? Of course, not every skill or technique will be used for every project. It’s all going to depend on what you want to create. Those with game design backgrounds will take to this field pretty quick. Imagine taking the game world and moving it to the real world. There are a lot of real world factors to deal with and lots of new usability challenges we haven’t even begun to explore (I’ve run into a few already I’ll share latter.)

As always there will be a huge need for UX principles. Here is a sneak peak at some of the items I will be working on for a future class. Every time I try out a new experience whether it be in VIVE, or Holo Lens, Rift, Leap Motion, Meta, etc..  I’m in  state of analyzing the technology, capabilities, limitations, actions, re-actions, micro-interactions, and input / output systems.

I dream of a world of open technology and removed barriers to allow complex systems to speak together, but as you might have guessed there are still a lot of closed platforms going to come down the line before we can get there. Look back to the days of instant messengers until tools like jabber came along.

As I have time to write I’ll be sure to share some of the experiences a long the way.

Here are just a few topics I’ve been exploring. (You may see more of these pop up in future articles)

  • How do you handle object occlusion to not disturb your immersive experience?
  • How do you deal with light and changes in existing environment layout? (a new object enters into a space)?
  • How do we build linked experiences to transition through environmental shifts (important to VR)?
  • How do you understand when it’s appropriate to use text, iconography?
  • How important is way-finding in VR, Mixed Reality, and spatial location?
  • What physical impacts do your decisions have upon the customer (user)?
  • How do you craft experiences that transcend what we know today?
  • How can you rapid prototype in un-familiar mediums with existing mediums?
  • How can you streamline creation flows?
  • How do new people become comfortable with new mediums? (seeing is believing but it doesn’t mean continued usage)

I’ll be sharing more as the weeks pass.  Feel free to reach out on twitter @uidesignguide. For now I’m off to unity land.