Umbra Composit – massively complex 3D in mixed reality

Leading Finnish 3D software company Umbra expands into augmented reality with a game-changing product for architects, engineering and construction. We work with them to make significant cost reductions and unforeseen speed increase in complex projects.

Umbra 3D is a Finnish growth company developing an algorithm used to compress and rebuild complex 3D models and environments in real-time. So far their clientele has been at the triple-A gaming industry where they serve over 50% of the market. If you’ve ever played titles such as Destiny or Call of Duty, you’ve seen Umbra’s algorithms in action. What they provide is essentially the best render and optimization engine and a cloud platform in the world for massively complex 3D environments.

At the same time, Mixed reality (MR) is on it’s way to mainstream. Mixed and augmented reality devices such as HoloLens by Microsoft are essentially wearable mobile computers that are limited by both processing power and battery, and thus are incapable of heavy realtime calculations with large and complex data. This is where Umbra steps in — their black box makes it possible to view, edit and share any 3D content anywhere, independent on the device it is being run — including any mixed reality headset.

Intro to Nordkapp’s Umbra Case Study

What have we done so far?

Since early 2017, we’ve been working with Umbra on the product design for their augmented and mixed reality portfolio in architecture, engineering and construction (AEC). Our work covers all touchpoints from the business and pricing models to plugins for architects and interface design on Hololens, desktop (Archicad Revit), mobile (iOS, Android) and browser/WebGL platforms.

The results

— Validated product-market fit. Current users include several industry leaders in AEC, with total turnover more than $25B.
— Go-to-market in progress in the construction industry, where the solution will reshape an old and a very traditional industry with futuristic technology with significant savings in cost, speed and quality.
— Transformation of Umbra’s technology product from a developer tool into an end-to-end platform with a number of end user applications: Umbra Composit, Pryzm and Umbra.io, the web and cloud platform for the products.

(Use+sales numbers undisclosed and confidential for now.)

We have been inspired by being able to design real life, practical frameworks for immersive realities. As the field is just emerging, this has resulted in a series of firsts when it comes to our own toolbox and also to the industry as a whole.

To see what this means, scroll down. For an overview of Umbra’s ambitions in AEC and beyond, have a look at this video:

In this video you can see one real use case and the whole workflow from exporting from architecture software into Umbra.io cloud service for optimization, and finally into seeing the model at the real construction site using the Umbra Pryzm viewing application (iOS version in this case).

The Service

The outcomes of our work so far, are
– Umbra Composit, 3D cloud computing platform automatically optimizes the most complex models in minutes with the click of a button
— Umbra Pryzm, a platform-agnostic viewer that enables 3D model viewing and collaboration from anywhere, on any device. Currently available on iOS, Hololens and WebGL.
— Umbra.io, the content platform acting as a frontend for customers’ data in the cloud.

Based on large study and interviews throughout the industry worldwide, working with 3D models in 2D is difficult because we all have to “render” the image in our heads individually, and thus sometimes very obvious errors can be left unnoticed.

Umbra’s goal is to make AEC industry’s design review sessions and model inspection more effective by:

1) Agile collaboration: Enabling quick and easy review rounds between different disciplines’ models that lead to actual action items (i.e. annotations), thus improvements before the “master” design review sessions

2) Make future design feel real now: Give visual (3D) context to detected issues, thus understand the implications of a proposed change better and help communicate these to other stakeholders too. Clash detection gets you only so far (false positives, confusing visual context on a 2D screen) and 2D plans on paper don’t really serve the purpose of communicating certain type of issues that well.

3) Smart use of technology: Help take most out of the visual capabilities that immersive 3D devices bring to the table, making issue spotting and inspection very effective.

4) Reduce cost and time: Removes the need to use external agencies to create high quality, but static, visualisations of architecture models, as Umbra’s game-level visualisations do the same thing in seconds, and are also interactive.

Once installed, Composit makes it possible for users to “Umbrafy” projects instantly. What this means for the clients is a significant increase in speed and reduction in costs. Instead of first taking several hours to render a single view of a complex architectural model, designers can now plug on Hololens and inspect their work in real-time from all angles. Instead of sending the complex gigabit-3D file to a builder or a 3D printer for a physical model, the whole team can now view the model anywhere, in real time on their headsets, mobile devices or even in the browser. The actual savings vary greatly per case, but we’re easily talking days of work and tens of thousands of euros. No wonder this has got world’s largest firms such as Gensler excited.

Underlying Technology

For the more technologically minded, Umbra Composit is a cloud-based system for optimizing and viewing high detail 3D data in real-time solutions. Not only can Umbra Composit optimize any size 3D data sets, it has the ability to stream in your 3D content based on predefined memory limits, visibility and distance to camera in runtime. What happens in the background is advanced model optimization including geometry decimation, occlusion culling and other 3D magic, literally rebuilding everything in a way it runs everywhere.

The Umbrafied content can be literally anything. From functioning cars with a live engine, architectural drawings of a stadium or even a LIDAR-scanned 3D model of an entire city. Running this kind of data in real-time, on almost any device or mobile devices was simply not possible before Umbra stepped in.

Use of data, and the hard problem we’re solving

So, for over a decade, Umbra has provided the industry’s best automated graphics optimization solutions to the most demanding video game publishers in the world. Now this experience is being taken into enterprise world through AEC, and augmented / mixed reality interfaces and WebGL.

Trying to explain this at scale is difficult, it’s a lot easier to show what this means instead. Two of the more prominent (and public) cases exploring Umbra’s potential on displaying complex realtime 3D data are the 3D model of City of Helsinki here in two forms: Architectural legend Eero Saarinen’s unrealized masterplan of Munkkiniemi district as European style closed-block grid, and the current day LIDAR-scanned 3D model of the city — both now available in augmented reality:

The content is uploaded to Umbra’s powerful cloud, where it is first compressed, and then served in realtime to users worldwide. The content can be shared virtually to any internet-connected device with a web browser and a growing number of VR and AR platforms including Microsoft Hololens, Apple ARKit and Samsung Gear VR. This makes it possible for large, geographically dispersed teams of varying technical expertise to review renderings in real time, greatly accelerating the design review process, and thus reducing costs. In other words, true digitalization in action through Moore’s law.

The Project

About a year ago when we started collaborating with Umbra, the basic technology was starting to come together. The usability and experience of the products required design to be made tangible, at this stage, there was not much to use, or see. To achieve our goals, we first needed a tight integration with the technology team. Our way of working is based on the idea that we first venture out into the unknown and discover the optimal future for our clients, and then make them real. We first bring the entire team onboard and form a singular vision, and then set up a design team building the vision through constant design and prototyping.

Originally we set out to build and recruit a design team inside Umbra, but as it turns out we are hard to replace — the work continues us as a one, singular design and technology team. The collaboration has led to three unique projects, that are actively being designed, developed and built as we speak.

We also work together with Umbra’s business development, in constant dialogue with worldwide leaders of their respective fields such as Gensler and Trimble as validating and testing our ideas. The details of this are confidential.

Once the high level scope was clear, we began by designing the user interface for Microsoft HoloLens. Since design for hologram interfaces and Mixed Reality is still a very new field, we actually had to invent our way around with many details and ways of working. We found ourselves learning new tools such as Unity 3D, discussing units in meters instead of pixels, and keenly observing light and objects and learning from the space around us through lean design and development cycles together with the Umbra technology team. All our knowledge and experience had to distil through a new lens towards the beginning of a new age of computing ahead of us.

Design Principles

All of our design is done in tight collaboration with Umbra’s technology team. This allows us to prototype our ideas directly in the real environment as-is. We came up with a set of design principles to guide us along the project:

1— Follow with Delay: The UI follows the user, and with a delay. This proved a good way to have the UI always near, but also moveable just so that it’s never blocking the content the user wants to see.

2—Consensual Interaction: The UI is shown only when the user wants to interact with it, and other times it fades out quite quickly. As obvious it may sound, after the user has positioned their model the way they want, her main goal is to see and interact with the model itself. Consequently, the UI is immediately brought back up again through an airtap, a Hololens signature interaction. (The headset has an always-on hand tracking)

3—Minimum Binary Interaction: For example, transposing and scaling the model can always simultaneously be done in two different ways: either instantly dragging in real world, or tapping on provided shortcuts or arrows, that move the model automatically. Architects & car-designers, who are the main users of the HoloLens version, need both of these methods to move the model for short and long distances at separate times.

4—Focus on the task, not the UI: The current augmented and mixed reality headsets field of view has its limits. The usable physical area is still rather small and is being shared by the actual content and the UI. For example, designing an automatic hiding/showing of the UI based on user’s hand position makes a huge difference on the experience.

Umbra.io, the platform

Following up, our next challenge to make these products available at Umbra’s website where consumers could buy their licenses, and manage their 3D content and users. We executed the service design, visual design and front-end implementation for umbra.io, which acts as an interface service to user’s 3D data in the cloud. The design process started from a series of workshops, where the initial mutual understanding of the service was created.

It was also an important step in order to start understanding how this very advanced technology works, could be put online to be utilized by the user. Another interesting aspect on this collaboration was the opportunity of developing the business model itself—from aspects ranging from determining the vision for the future to practicalities such as—how much does a license cost and what should it include?

WebGL viewer for modern browsers

All good things also eventually come to the web. Next generation web graphics are almost here in the form of WebGL 2.0 standard. We’re helping Umbra to be future proof by hitting the ground running with it too.

Below are two examples designed by us that demonstrate a huge improvement in 3D quality on WebGL using Umbra’s technology. The demos can be tested in the latest Chrome or Firefox browser.

Point Clouds

A powerful feature of Umbra Composit is to optimize Point Cloud Data. Point Clouds containt 3-dimensional information from real world, that are typically gathered with a LiDAR laser scanner, or by extracting 3D information from hundreds of photographs. The resulting point cloud data may have anything from hundreds of millions to billions of 3-dimensional triangles. Umbra can optimize this data as well, and play it back interactively at smooth 60 fps speed.

Unity 3D plugin

An alternative way to create content on the platform is with Unity 3D. The video also shows the Umbra.io website along with the latest iteration of the iPhone/iPad ARKit user interfaces:

Walkthrough covering Umbra’s Unity plugin to Umbra.io web management and viewing in Umbra’s iOS App at 5:05.

Umbra Pryzm, an iOS ARKit viewer

Umbra Pryzm is the enterprise mobile product bundled with Composit for viewing any 3D content on any iOS or Android device. Since ARKit and ARCore are very similar to the user and from the custom UI design perspective, we initially focused mobile design efforts on different sized iPhones and iPad. The development was done with Unity 3D by Umbra’s developers, and Unity apps can look just the same on all platforms, so simultaneous Android versions were created at the same time.

For reference of the capabilities, here is a Umbra Composit™ ARKit iOS Demo shows multimillion triangle models running in real time:

Umbra Corona DLS

Umbra’s Corona DLS (Design Language System) is the first of its kind cross media design language system that enables seamless digital product interactions that scale all the way up from the boundaries of a smartphone screen to immersive mixed reality devices such as Microsoft HoloLens. Corona DLS serves to complement Umbra’s optimisation engine used to compress and rebuild complex 3D models and environments in real-time. An active work-in-progress, the project encompasses holistic customer experience across Umbra’s products and services on leading platforms such as iOS ARKit, Hololens, desktop (Archicad Revit) and browser/WebGL platforms for clients in architecture, engineering and construction (AEC).

What our users are saying

Umbra Composit and Pryzm were launched in London at Digital Construction Week 2017 – an industry event for digital construction, engineering, design, manufacturing, and operation. Products gained lots of audience interest in Digital Construction Week. Robert Scoble, an AR/VR industry heavyweight also did a video about Umbra, using what was at the time an MVP version of the UI.

There is a great promise in the AEC industry as they can disrupt slow, expensive and manual visualisation work that the industries are currently challenged with their technology and services. The work was been constantly tested and iterated with real customers. Here are a select testimonials and quotes of the success so far:

What’s next

We learnt a lot as we made progress and decided to start documenting our augmented/mixed reality knowledge in a series of blog posts on the topic of Design for Immersion. The first post is out already.

Our work with Umbra continues on bringing Composit to new platforms and industries. In a way, this is just the beginning of a new age of computing, and we couldn’t be more excited about being at the forefront of it. Watch this space.