Howard Stearns

Howard Stearns works at High Fidelity, Inc., creating the metaverse. Mr. Stearns has a quarter century experience in systems engineering, applications consulting, and management of advanced software technologies. He was the technical lead of University of Wisconsin's Croquet project, an ambitious project convened by computing pioneer Alan Kay to transform collaboration through 3D graphics and real-time, persistent shared spaces. The CAD integration products Mr. Stearns created for expert system pioneer ICAD set the market standard through IPO and acquisition by Oracle. The embedded systems he wrote helped transform the industrial diamond market. In the early 2000s, Mr. Stearns was named Technology Strategist for Curl, the only startup founded by WWW pioneer Tim Berners-Lee. An expert on programming languages and operating systems, Mr. Stearns created the Eclipse commercial Common Lisp programming implementation. Mr. Stearns has two degrees from M.I.T., and has directed family businesses in early childhood education and publishing.

Ben’s Social VR Adventure

being-of-light

There are lots of old and new VR sites with prognostications. This guest blog from a VCpreneur has four “requirements” for social VR, and it sparked some discussion at our office.

It reads to me very much like this investor fellow might be talking with some startup that features his four points, and he just wants to sort out whether the concepts are sticky. I’m imagining a company that uses Facebook for login, uploads your 2D and 3D albums and videos into your 3D space (which is zero-install via WebVR), lets you and your friends like/share from in-world, and folks that click on those shared things in Facebook get brought into your 3D space at the corresponding item. (I pitched such a thing five years ago as “Beyond My Wall”, using the Unity browser plugin instead of WebVR.)

One of the blogger’s “requirements” was that participants use their real-world identity, and this was what interested our gang, both for and against. I think this is a red herring. Although I use my real name online alot, my gut is that an alwful lot of VR is going to be about sex, and folks won’t want to use their real name. But overall, I don’t think it’s killer one way or the other. I’m guessing that he’s trying to turn the limitation of Facebook-login into a feature called real-world identity, and I think it’s a stretch. There’s clearly a lot of things that went into making Facebook win over MySpace, and I’m not persuaded that that real-world identity is the magic ingredient. Indeed, both Twitter and his other example, Youtube, are both counterexamples. I think real-world identity can be a part of many potentially successful killer apps, but I don’t see it being required for all such killer apps. (I think verified identity, whether real-world or not, will be a great service for High Fidelity to offer, but it won’t be required for everything.)

I do think he’s on the right track, though, with his feature set including pre-established friend links and content-sharing. But I’m not sure the guy has really understood why those two things matter or what they are part of. They feed the virality of a lot of social media winners, but the magic is in the viral math, not specifically in the features. For example, “pre-established friends” is helpful, but not necessary for Twitter, Youtube, or EBay. I think that each one of a Facebook liked-story, Twitter hashtag, Ebay auction, and Youtube video/discussion page forms a micro-community of interest with a specific “address” that can iself be shared out-of-band. Each user can and does “belong” to many such micro-communities. I believe that’s the real driver for virtuous-circle virality. High Fidelity is going to be great for this, because anyone can easily create any number of separate, persistent, domains of interest, and each one can have the computation shared by the people who are interested in it. (Our upcoming Beta makes it easy to organize one domain per user’s computer, which I think is a good initial model.) Nothing else I’ve seen (not even Beyond My Wall) can scale like that. This is so very different from social VR startups that offer even a large number of hosted chat rooms.

Of course, none of this is to say what is interesting about any of these domains. That is a separate – and more important – question that remains. The blog (and this response) are about what qualities are necessary, given something interesting, for it to catch on.

Separately, the comment from DiGiCT was interesting, that the huge numbers of Chinese HMD sold are just a series of upgrades to very bad/cheap/unsatisfying units. I wonder if that’s true.

Posted in Inventing the Future, Philosophical business mumbo-jumbo | Comments closed

How To Use High Fidelity

no-cow-tipping

tl;dr

1.Install the “Sandbox” app(We currently do not provide pre-built installers for Linux, but techies can build from open source for all platforms.) The “Sandbox” is the HighFidelity logo that appears in your system tray. (It is in the corner near where the wifi logo is on your screen, and may be behind the arrow if you have a lot of system tray apps. Sandbox starts automatically on Windows.)

2. Click on the Sandbox app and choose “Go Home” to visit your own personal domain. You can also visit another domain by clicking on a “hifi:” link that someone has given you on a Web page or email.

This is your user interface. While you use this, your actions are seen by others through your representation as an avatar.

3. Use the arrow keys to move, or the joysticks/touchpads on a hand controller. (You may need to turn on some controllers using the Avatar -> Input Devices menu.) Change between desktop display and Head Mounted Display (HMD) using the Display menu. (HMDs and controllers need to be plugged in and have their manufacturer’s software installed before starting the user interface.)

4. Most of the system behavior is defined by scripts that you can get from friends or from the Examples, or that are attached to the objects you find in the domains that you visit. Some initial behavior that you get “out of the box” includes:

  • The Go To button, giving you a directory of places.
  • The Examples button, giving you different avatars to choose from, or things to add to domains that allow it (like your own Home domain).
  • The Edit button, which lets you edit any object in in any domain that lets you.
  • Pressing the “return” key gives you an address bar to specify another place or user to visit.
  • The microphone is on! Use a headset.

If you’ve got 10 people sharing wifi, or have a Comcast connection that turns into molasses on a Friday night, things might be a bit slow when your network sucks. This is especially true when visiting a new domain for the first time. Also, some domains are faster than others. If things don’t feel right, try again later, or see “Stats”, below. Read More »

Posted in Inventing the Future, workflow | Comments closed

The Bird Is the Word!

Believe it or not, there’s some great engineering and sportsmanship behind this:

View post on imgur.com

bird-is-the-word

We’re trying to create low-latency, high-fidelity, large-scale multi-user experiences with high-frequency sensors and displays. It’s at the edge of what’s possible. The high-end Head Mounted Displays that are coming out are pretty good, but even they have been dicey so far. The hand controllers have truly sucked, and we’ve been basing everything on the faith that they will get better. But even our optimism had waned on the optical recognition of the LeapMotion hand sensor. We made it work as well as we could, and then left it to bit-rot.

But yesterday LeapMotion came out with a new version of their API library, compatible with the existing hardware. Brad is the engineer shown above, and no one knows more than him how hard it is to make this stuff work. He was so sure that the new software would not “just work”, that he offered a bottle of Macallen 18 year old scotch to anyone who could do so. Like the hardworking bee that doesn’t know it can’t possibly fly, our community leader, Chris, is not an engineer and and just hooked it up.

chris-baring-magic-leap

In just minutes he made this video to show Brad, who works from a different office.

True to his word, Brad immediately went online and ordered the scotch, to be sent to Chris. Brad then dug out his old Leap hardware from the drawer next to his CueCat and made the more articulate version above.

We sent it to few folks, including Stanford’s VR lab, which promptly tweeted it, with the caption, “One small step for Mankind? Today we saw 1st networked avatar with fingers”.

So now we have avatars with full body IK driven by 18-degree of freedom sensors, plus optical tracking of each finger, facial features and eye gaze, all networked in real time to scores of simultaneous users, with physics.  In truth, we still have a lot of work to do before anyone can just plug this stuff in and have it work, but it’s pretty clear now that this is going to happen!

One more dig: Apple has long said that they can only make things work at the leading edge by making the hardware and software together, non-interoperable to anyone else. Oculus has said that networked physics is too hard, and open platforms are too hard. Apple and Oculus make really great stuff that we love. We make only open source software, and we work with all the hardware, on Windows, Mac, and Linux.

Posted in history: external milestones and context, Inventing the Future | Comments closed

Makers’ Mash-Up

As the nascent VR industry gears up for The Year of VR, the press and pundits are wrestling with how things will break out. There are several Head Mounted Display manufacturerers that will release their first products early this year, and they are initially positioning them as extensions of the established games market. The idea is that manufacturers need new content for people to play on their boxes, and game studios need new gizmos in which to establish markets for their content. The Oculus will initially ship with a traditional game controller. The Vive will provide hand sensor wands that allow finer manipulation. They’re both thinking in terms of studio-produced games.

The studio/manufactuer model is well-understand and huge — bigger than the motion picture industry. The pundits are applying that framework as they wonder about the chicken-and-egg problem of content and market both requiring each other to come first. Most discussion takes for granted a belief that the hardware product market enables and requires a studio to invest in lengthy development of story, art, and behavior, followed by release and sale to individuals.

But I wonder how quickly we will move beyond the studio/manufacturer model.

I’m imagining a makers’ mash-up in which people spontaneously create their own games all the time…

  • a place where people could wield their Minecraft hammers in one hand, and their Fruit Ninja swords in the other.
  • a place that would allow people to teleport from sandbox to sandbox, and bring items and behaviors from one to another.
  • a place where people make memories by interacting with the amazing people they meet.

I think there’s good reason to believe this will happen as soon as the technology will enable it.

Second Life is an existence proof that this can work. Launched more than a dozen years ago, its roughly 1M montlhy users have generated several billion dollars of user-created virtual goods. I think SL’s growth is maxed out on its ancient architecture, but how long will it take any one of the VR hardware/game economies to reach that scale?

Ronald Coase’s Nobel-prize-winning work on the economics of the firm says, loosely, that companies form and grow when growing reduces their transaction costs. If people can freely combine costume, set, props, music, and behaviors, and are happy with the result, the economic driver for the studio system disappears.

I think the mash-up market will explode when people can easily and inexpensively create items that they can offer for free or for reputation. We’ve seen this with the early Internet, Web, and mobile content, and offline from Freecycle to Burning Man.

High Fidelity’s technical foundation is pretty close to making this happen at a self-sustaining scale. There are many design choices that tend to promote or restrict this, and I’ve described some in the “Interdimensional conflicts” section at the end of “Where We Fit”. Some of the key architectural aspects for a makers’ mash-up are multi-user, fine-manipulation, automatic full body animation, scriptable objects that can interact with a set of common physics for all objects, teleporting to new places with the same avatar and objects, and scalability that can respond to unplanned loads.

Posted in Inventing the Future, The Age of Imagination | Comments closed

Where do we fit?

computer-dimensions

There are many ways by which to categorize work in Virtual Reality: by feature set, market, etc. Here are some of the dimensions by which I view the fun in VR, and where High Fidelity fits in.  To a first-order approximation, these axes are independent of each other. It gets more interesting to see how they are related, but, first the basics.

Scope of Worlds: How do you count VRs?

As I look at existing and developing products for virtual worlds, I see different kinds of cyberspaces. They can be designed around one conceptual place with consistent behavior, as exemplified by social environments like Second Life or large-scale MMORGs like World of Warcraft. Or they can replicate a single small identical meeting space or game environment, or some finite set of company-defined spaces. Like my previous work with Croquet, High Fidelity is a “metaverse” — a network in which anyone can put up a server and define their own space with their own behavior, much like a Web site. People can go from one space to another (like links on the Web), taking their identity with them. Each domain can handle miles of virtual territory and be servered from one or more computers depending on load. Domains could be “side by side” to form larger contiguous spaces, although no one has done so yet.

Scope of Market: How do you sell VRs?

Many see early consumer VR as a sustaining technology for games, with the usual game categories of FPS, platformers, builders, etc. The general idea is that gaming is a large market that companies know how to reach. Successfull games create a feeling of immersion for their devotees, and VR will deepen that experience.

An emerging area is in providing experiences, such as a concert, trip, or you-are-there story.

Others see immersion and direct-manipulation UI as providing unique opportunities for teaching, learning, and training, or for meetings and events. (The latter might for playing or for working.)

Some make tools and platforms with which developers can create such products.

High Fidelity makes an open source platform, and will host services for goods, discovery, and identity.

Scope of Technology: How do you make and populate VRs?

Software:

By now it looks like most development environments provide game-like 3D graphics, some form of scripting (writing behavior as programs), and built-in physics (objects fall and bounce and block other objects). Some (including High Fidelity and even some self-contained packaged games) let users add and edit objects and behaviors while within the world itself. Often these can then be saved and shared with other users.

A major technical differentiator is whether or not the environments are for one person or many, and how they can interact. For example, some allow small numbers of people on the Internet to see each other in the same “space” and talk by voice, but without physical interaction between their avatars. Others allow one user to, e.g., throw an object to a single other user, but only on a local network in the same physical building. High Fidelity already allows scores of users to interact over the Internet, with voice and physics.

Hardware:

Even Low-end Head Mounted Displays can tell which way your head is turned, and the apps rotate your point of view to match. Often these use your existing phone, and the incremental cost beyond that is below $100. However, they don’t update fast enough to keep the image aligned with your head as it moves, resulting in nausea for most folks. The screen is divided in two for separate viewpoints on each eye, giving a 3-dimensional result, but at only half the resolution of your phone. High-end HMDs have more resolution, a refresh rate of at least 75Hz, and optical tracking of your head position (in addition to rotation), as you move around a small physical space. Currently, high-end HMDs connect to a computer with clunky wires (because wireless isn’t fast enough), and are expected to cost several hundred dollars. High fidelity works with conventional 2D computer displays, 3D computer displays, and head-mounted displays, but we’re really focusing the experience on high end HMDs. We’re open source, and work with the major HMD protocols.

A mouse provides just 2 degrees of free motion: x and y, plus the buttons on the mouse and computer. Game controllers usually have two x-y controllers and several buttons. The state of the art for VR is high frequency sensors that track either hand controllers in 3-position + 3-rotation Degrees Of Freedom (for each hand), or each individual finger. Currently, the finger sensors are not very precise. High Fidelity works with all of these, but see the next section.

Capture:

One way to populate a VR scene is to construct 3D models and animated characters using the modeling tools that have been developed for Computer Aided Design and Computer Generated movies. This provides a lot of control, but requires specialized skills and talent.

There is now pretty good camera/software apps for capturing real scenes in 3D and bringing them into VR as set-dressing. Some use multiple cameras to capture all directions at once in a moving scene, to produce what is called “cinematic VR” experiences. Others sweep a single camera around a still scene and stitch together the result as a 3D model. There are related tools being developed for capturing people.

Scope of Object Scale: What do you do in VRs?

The controllers used by most games work best for coarse motion control of an avatar — you can move through a building, between platforms, or guide a vehicle. It is very difficult to manipulate small objects. Outside of games, this building-scale scope is well-suited to see-but-don’t-touch virtual museums.

In the early virtual world Second Life, users can and do construct elaborate buildings, vehicles, furniture, tools, and other artifacts, but it is very difficult to do so with a mouse and keyboard. At High Fidelity, we find that by using two 6-DoF high-resolution/high-frequency hand controllers, manipulating more human-scaled objects becomes a delight. Oculus‘ “toy box” demonstration and HTC’s upcoming Modbox show the kinds of things that can be done.

Interdimensional conflicts:

These different axes play against each other in some subtle ways. For example:

  • Low-end HMD would seem to be an obvious extension of the game market, but the resolution and stereo make some game graphics techniques worse in VR than on desktop or consoles. The typical game emphasis on avatar movement may accentuate nausea.
  • High-end hand controllers make the physics of small objects a delightful first-person experience, but it’s profoundly better still when you can play Jenga with someone else on the Internet, which depends on software design limitations. (See October ’15 video at 47 seconds.)
  • Game controllers provide only enough input to show avatar motion using artist-produced animations. But 18-DoF (in two-hand-controllers plus high-end HMD) provide enough info to compute realistic skeletal motions of the whole body, even producing enough info to infer the motion of the hips and knees that do not have sensors on them. (See October ’15 video at 35 seconds.) This is called whole-body-Inverse-Kinematics. High Fidelity can smoothly combine artist-animation (for walking with a hand-controller joystick), with whole-body-IK (for waving with your hand controller while you are walking).
  • These new capabilities allow for things that people just couldn’t do at all before, as well as simplifying things that they could not do easily. This opens up unchartered territory and untested markets.

Notice that all the mentioned interactions depend on The Scope of Object Scale (above), which isn’t often discussed in the media. It will be interesting to see how the different dimensions of VR play against each other to produce a consistent experience that really works for people.


 

A word about Augmented Reality: I think augmented reality — in which some form of VR is layered over our real physical experience — will be much bigger than VR itself. However, I feel that there’s so much still to be worked out for VR along the above dimensions and more, that it becomes difficult to make testable statements about AR. Stay tuned…

Posted in history: external milestones and context, Inventing the Future | 2 Comments (Comments closed)

The Big Show

We do a lot of live demos. Our VR software is in alpha and we sometimes run into bugs, but we pride ourselves on showing the true state of things instead of slides and videos. It’s often a build with radical changes created just a few moments before. And yet the most common trouble is the unpredictable network connectivity one finds at demo venues. Our folks have done quite a few tethered to a cell phone.

But this demo from Sony is in a whole other class. Ouch. I’m sure we’ll all have great hand controllers within a year or so, but right now, hand controller hardware really sucks.

By the way, I think their IK looks great, and I especially like the jump. But notice that the avatars are stuck on a pedestal instead of moving around. I haven’t seen anyone combine IK with artist-designed animations the way we have.

Posted in history: external milestones and context, Inventing the Future | Comments closed

Big milestone in our VR today: bow and arrow are cool, and the reasons why are cooler

coming at youWe had our weekly company meeting in HighFidelity today, just like we always do. For a while now, we’ve all been using Oculus VR headsets and trackers for our hands, and we now have avatars that look like like us. We have good multi-user physics and whole-body tracking-animation for our avatars.

Today one of the guys showed off his recurve bow. As long as you hold it, it re-arms with arrows. Pull the string back with your physical hands, aim and let go, and the arrow flies. It’s basic fun with weaponry. People took turns playing with it.

But quickly people started saying, “Oh, shoot me between the eyes!” (including our CEO). It’s pretty cool to see the arrow coming right for you. Can’t do that IRL! (In Real Life.)

But I never said this. Instead, I stood among the circle of avatars in front of the shooter, and kind of stared him down. Without thinking, I conveyed, “Come at me, bro”, but with nothing so obvious as arms wide or a Neo-like “come on” finger-wave. My avatar could do that with just a stretch of my physical arm or squeeze of my hand, but I didn’t gesture or say a word. Our team is pretty competitive, but we’re not the kind of folks who would shoot the person in front of us without being invited. We need to be invited. But multiple shooters did shoot me. Non-verbal communication works!

Posted in history: external milestones and context, Inventing the Future | Comments closed

Avatar Results

Some terrific initial results of our Avatar IK. This is not from post-processing of someone wearing a motion-capture suit with a bunch of ping-pong balls stuck to it.  All the movement is instead driven by a (developer’s edition of a soon-to-be-retailed) gaming VR headset (capturing head position and orientation in 3D), and two (somewhat dodgy) gaming hand-controllers (capturing each hand’s position and orientation, plus a trigger). As described here, we interpret these 3 measurements to trigger an appropriate underlying base animation, which then provides more measurements to drive a realtime inverse-kinematics skeleton.

Posted in history: external milestones and context, Inventing the Future | Comments closed

High Fidelity Avatars: Interpretive Motion IK

meryWe’ve come up with something that I think is quite cool for conveying presence though our avatars. Interpretive Motion IK is a technique in which all kinds of inputs drive a small set of animated motions, and then the animation results and additional inputs both drive an Inverse Kinematic system. This gives us a rich set of built-in avatar behaviors, while also making use of an ever-evolving set of input devices to produce a dynamic and life-like result.

Why Aren’t Avatars More Like Us?

From static “head shots” in a text chat, to illustrations in a book and famous faces in a movie, avatars give us an intuitive sense of who is present and who is doing what. The story of an animated movie (or high end game “cut scene”) is largely shown through the fluid motion of anthropoids. A movie studio can hire an army of artists, or record body actors in motion capture suits. But a virtual world does not follow a set script in which all activity can be identified and animated before use. Avatar animation must instead be generated in real time, in response to a world of possible activities.

This challenge leads some systems to just show users as simplified images, or as just a head, a disembodied “mask and gloves”, or a mostly un-animated “tin can” robot. This may be appropriate for specialized situations, but in the general case of unlimited high fidelity virtual worlds,  the lack of whole-body humanoid animation fails to provide a fulfilling sense of personal immersion.

When it works well, the interpretation of motion is so strong that when another avatar turns to face your avatar, we describe it as “YOU can see ME”. In fact, the pixels on the screen have not turned, and cannot see. Think of the personality conveyed by the Pixar desk lamp hopping across the scene and looking at the camera, or the dancing skeletons of Disney’s early Silly Symphony. Unlike Disney and Pixar, High Fidelity aims to capture this rich whole-body movement as a realtime result of dynamic user input. Alas, today’s input devices give us only incomplete data. Interpretive Motion IK allows us to integrate these clumsy signals into a realistic sense of human personality and action.

Read More »

Posted in Inventing the Future, Software | Comments closed

Hackers and Painters

I went to http://www.paintingvirtualreality.com last weekend.

Billed as the World’s First Virtual Reality Painting Exhibition, it featured:

  • artwork one could view individually, using a Head Mounted Display (HMD) with single-camera tracking;
  • artists at work wearing HMD with dual lighthouse tracking (and the results displayed live on a big monoscopic screen).

The work was done with http://www.tiltbrush.com, which appears to be a couple of guys who got bought by Google. The project – I’m not sure that its actually available for sale – appears to be evolving along several dimensions:

  1. 3d Model Definition: covering stroke capture (including, symmetric stroke duplication), “dry” and “wet” (oil paint-mixing/brushwork effects), texture patterns, volumetric patterns, emmision, particles.
  2. Interactive Model Creation: tool pallette in one hand, and brush in the other.
    1.   Videos at tiltbrush.com suggest an additional moveable flat virtual canvas (a “tilt canvas”?) that one can hold and move and paint against. The art on display was clearly made this way, as they all felt like they were a sort of a rotational 2.5D — the brush strokes were thin layers of (sometimes curved) surfaces.
    2. The artists last night appeared to be working directly in 3D, without the tilt canvas.
    3. The site mentions an android app for creation. I don’t know if it is one of these techniques or a third.
  3. Viewing: HMD, static snapshots, animated .gifs that oscillate between several rotated viewpoints (like the range of an old-fashioned lenticular display).

  I haven’t seen any “drive around within a 3D scene on your desktop” displays (like standard-desktop/non-HMD versions of High Fidelity).

  The displays were all designed so that you observed from a pretty limited spot. Really more “sitting”/”video game” HMD rather than “standing”/”cave” exploration.

My reactions to the art:

  • Emmission and particle effects are fun in the hands of an artist.
  • “Fire… Light… It’s so Promethean!”
  • With the limited movement during display, mostly the art was “around you” like a sky box, rather than something you wandered around in. In this context, the effect of layering (e.g., a star field) – as if a Russian doll-set of sky boxes (though presumably not implemented that way) – was very appealling.

Tip for using caves: Put down a sculpted rug and go barefoot!

Posted in history: external milestones and context, Inventing the Future | Comments closed
  • Connect With Us

    Follow Wetmachine on Twitter!

    Email Updates

    [subscribe2 link='Subscribe with just your email address'] To get updates for all Wetmachine posts. Want more control? Log in using your Wetmachine account (or with your Facebook, Twitter, or other social media account), or register for a Wetmachine account.

Username
Password

If you do not have an account: Register