Build low-code mixed reality apps in Power Apps, AI Builder, and HoloLens

Picture: Microsoft

Microsoft’s low-code Power Platform is a key part of its enterprise software suite. Sitting between Microsoft 365 productivity tools and Dynamics 365 business apps, it’s a way to create custom workflows and add your own user experiences. As a bonus, you get access to Azure Cognitive Services for AI as a Service and connectors linked to third-party apps.

The result is a fast way to deploy code as needed, filling what is often referred to as the “enterprise application gap”. Power Platform tools are ready for use both by users who need an application, but there is nothing to meet that need, and by professional developers who need to deploy a solution quickly . Technologies such as the PowerFX language and support for Visual Studio Code make it easier to integrate the Power Platform into traditional development tool chains, supporting the development of interdisciplinary and cross-functional fusion teams to design, create and deploy apps.

SEE: Windows, Linux, and Mac commands everyone should know (free PDF) (TechRepublic)

If you have created an app in Excel or Access, you can create one in Power Platform, especially using its Power Apps tools. There are three ways to build Power Apps apps: using canvas apps to drag and drop controls onto a design surface, attaching code to them, or using model-driven apps to pass data directly to code. Canvas apps give you the most flexibility because they can be used to create a wide range of different types of apps using many different data types. They also allow you to host mixed reality in your low-code application.

Power Platform in the Metaverse

One technology that’s getting a lot of attention right now is mixed reality or, as it’s increasingly being called, the metaverse. Microsoft has a lot of experience working with 3D content in Virtual and Augmented Reality (VR/AR), with tools built into Windows and Azure, as well as supporting mixed reality content for Android devices and iOS using their Augmented Reality SDKs (Software Development Kits).

Microsoft’s first experiments with enterprise mixed reality were assistive tools, giving frontline workers access to real-time assistance with colleagues looking through their eyes (or at least their HoloLens camera) and annotating the world for them or providing life-size examples of equipment for training and planning.

These concepts are integral to how Microsoft approaches mixed reality in Power Apps, as a tool rather than a gimmick. That’s not to say they can’t be used as part of a marketing application, but there’s a lot more to how Microsoft views technology.

The upside, of course, is that using Power Apps reduces the expertise needed to build a mixed reality app. So instead of having to learn Unity or a similar 3D development environment, all you need is a Canvas app and the right controls. This makes building a prototype a matter of minutes, no matter what devices you are targeting.

Add mixed reality to a Power Apps app

Power Apps Mixed Reality Controls builds on the work of the Babylon.js team. A tool for creating 3D images on the web using JavaScript, it is now integrated with React Native for use on mobile devices outside of the browser. Microsoft was already using the React Native framework to deliver Power Apps Mobile controls, so using it to deliver mixed reality content made a lot of sense.

Work had to be done to integrate Babylon’s native version with React Native as well as iOS’s ARKit and Android’s ARCore. For PCs using Power Apps web tools, there is integration with the WebXR Toolkit.

Although you don’t need to understand the back-end to build Power Apps Mixed Reality, the resulting controls support the key features your apps need, including providing anchors for views as well as hit testing and plan detection. This now means you can test 3D mixed reality content using Babylon on PCs before moving it to Power Apps and using it on devices.

SEE: Recruitment kit: Back-end developer (TechRepublic Premium)

Using the commands is relatively simple. Content is stored in your app’s backend and loaded as needed, where it’s displayed using the View control in MR Power Apps using the dimensions you defined. Your device becomes a window into a mixed reality world, placing objects on a floor or wall for you to move around.

You don’t want to use this tool all the time; it will probably be a way to take an item from a catalog and show it where it could be used. Maybe it’s a piece of furniture in a house, or maybe it’s industrial equipment in a factory. Users click a button in your app to launch the mixed reality view, replacing the view they were using.

Bringing Power Apps, Device Sensors, and AI Together

In addition to using Power Apps to add elements to augmented reality, you can use mixed reality tools to measure the environment around them. This can be useful if you are creating an app to help you calculate construction costs.

The addition of the Measure Camera control allows users to pick a starting point and then pan to measure a space, with an overlay displaying the current measurements. This uses the depth sensors built into most modern phones. you’ll get the best results if a phone has a LIDAR or similar precise sensor. Mixing measuring tools with sight checking can help plan the installation of equipment, showing where something can fit securely.

Things get even more interesting when you mix these tools with Power Apps AI Builder. Here you get low-code access to Azure’s cognitive services, mixing pre-built templates with custom functionality.

You can use Power Apps camera controls and its object recognition capabilities to, for example, detect which version of a piece of equipment needs maintenance, automatically loading the correct mixed reality model for comparison. The app can then launch the appropriate Dynamics 365 guides to help with maintenance, using an AR phone or, more usefully, a hands-free HoloLens. An example from the Azure Developer Defense Team shows how you can use the tools to identify different hardware.

Building the future of app development with Power Apps

Building complex mixed reality apps like this used to be difficult and required a lot of development resources. By using Power Apps, businesses can take advantage of its ability to quickly connect different apps, using device cameras and cloud services along with simple workflows.

The result may not be as pretty or as full-featured as a purpose-built app, but it will get the job done and do it more than well enough. Plus, you can have something up and running in days as a proof of concept or a way to quickly troubleshoot issues.

Rosemary C. Kearney