How Preact used Mixed Reality and AI to win Microsoft Hack@Speed
Last month, we were delighted that the Preact team won our second successive Hack@Speed event hosted by Microsoft!
Aside from our delight at this latest success, it is thrilling to showcase new solutions which can be created quickly using the latest cutting-edge Microsoft technology.
In this post, our Build Manager (Bogdan Ciobanu) and Solutions Architect (Paul Rutter) from our participating team have shared details of their winning solution and what they took away from the experience.
What is the purpose of Microsoft’s hackathons?
These events bring together creative minds, allowing them to work in teams, and really get hands on with technology they may have seen, but not worked with before. Having a theme that explores current challenges faced by many businesses, means that teams are tasked with solving real-life problems in new and innovative ways.
What was the brief?
For the latest event, our brief was to focus on the topic of “return to work”, to understand what challenges organisations have with offices reopening and employees returning to work. Within five hours we were tasked with formulating an idea, designing the solution, and then building and developing this before presenting this to the judges.
The solution should support organisations in handling new processes that are due to Covid-19.
How did you approach it?
Our thought process revolved around the various compliance checks organisations now have to undertake to ensure work environments are safe. These involve many new processes, and often these are done on paper and require a lot of manpower. As a result, they can absorb a lot of time and are liable to errors and inconsistent handling.
We decided to focus on a couple of processes for compliance checks to measure the distance between desks and the cleanliness of desktops.
Our solution would tackle these issues by supporting an operations team, office manager or other personnel in completing these compliance checks and provide reassurance that a high standard of safety is being met.
What did you build?
The starting point of this solution was creating a flow (using Power Automate) that would generate compliance records in the background. Users would then complete these compliance checks and capture supporting photos through our Power App on their mobile device. These checks could be scheduled as required.
The results of the compliance checks are stored in Microsoft Dataverse (previously Common Data Service). This tracks the pass or fail status of each check split by location, room and facility. This data can then easily be managed through Dynamics 365 or any model-driven app to follow up flagged problems and provide reporting so organisations can keep on top of safety for their employees.
Using mixed reality technology, our mobile app measures distances between desks to compare this to compliance standard of two metres. The results are applied to the compliance record with details of the specific building and room location, as well as a picture of the desk for cleanliness checks.
Another exciting piece of a technology used in our solution is the AI capabilities of Azure Cognitive Services. This analyses pictures of desks to check if they are tidy, by identifying items on cluttered desks such as cups, food and stationery. If the AI model recognises a desk as messy, another flow will trigger an email to be sent to the relevant people to alert them that the compliance check failed.
What does it look like in action?
Using the mobile app - simply click “Perform Compliance Check”. Open the camera and point towards the desk so the mixed reality feature identifies the desk. Then click the desk and move the camera towards the next desk. The app will show in real time the distance between these.
The “Distance” and “Unit of Measurement” fields are automatically populated, along with the name assigned to the compliance check. Take another picture of the desk, for a tidiness spot check.
The app will instantly report if the check has passed or failed.
Using a Power Automate flow, the desktop image is sent to Azure Cognitive Services to assess how clean the surface is. Using this object recognition model, if the check fails, it will send an email to operations with the picture of the desk and the list of items identified in the picture.
How could this solution be extended?
The great thing about our app design, is that it’s versatile. We created a framework that allows organisations to build in custom checks to meet their unique needs. The compliance check survey can be as long as needed, and extra fields can be created to capture further detail.
Other elements could include desk booking, for example if buildings have limited capacity for desks, and Power BI reporting to better handle the compliance checks and stay on top of safety standards. We also wanted to potentially look at Dynamics 365 Customer Voice functionality to allow for ad-hoc surveys to be embedded.
In what other ways can mixed reality and AI capabilities be applied?
The new mixed reality features in Power Apps provide not only the capability to measure distance, area and volume, but also allows you to overlay and interact with 3D/2D content over the live camera feed. This opens up lots of potential use cases.
Measurement components could be used by:
- Trades people/ Surveyors to measure up a job; instead of taking pictures and manually adding these and the supporting detail to a Word document, this can all be done via the app and the document automatically generated
- Engineering companies/ Field service scenarios to record dimensions on assets when they are deployed onto a customer’s site
The 3D overlay functionality could be used to:
- Show potential customers your products in their environment
- Determine if a piece of machinery will fit on a factory floor
- Mark up a job to show where the assets should be placed ready for hand over to the installations team
The AI capabilities provided in the wider Azure platform are staggering. Microsoft have made huge investments in building machine learning models to support the different use cases. Models already exist for lots of different scenarios: Decision, Language, Speech and Vision.
For this solution, the service we used was specifically the computer vision service which provides pre-trained models for object detection. This means that images can easily be sent from the mobile device to the Computer vision service. While this service uses a pre-trained model by default, but you can train it to understand your own objects if required.
For example, if you manufacture special components, you could train the model to identify what a conformant component looks like, versus a non-conformant component, helping to improve your quality control.
How does participating in Microsoft’s Hackathon benefit Preact and its customers?
Participating in Hackathons organised directly by Microsoft, allows us to be very close to current problems that organisations have at the time and try to put efficient, cost-effective and future proof solutions in place, specifically on the Power Platform and related technologies.
It’s easy to build things using familiar tools and technologies, but the Hack@Speed event pushes us to explore new technologies, with the support from Microsoft experts. We can really start to push the art of the possible, which ultimately allows us to build innovative solutions for our customers, giving them a competitive edge.
What did you enjoy about this experience and what you took away from it?
The Hackathon was a great experience and one that we would love to repeat and make it three wins in a row! The teams at Preact and Microsoft are so passionate about technology and how it can be used to build innovative solutions.
Sometimes new problems occur unexpectedly, or you might find yourself having to deal with issues unique to your organisation. With Microsoft's Azure and Power Platform capabilities constantly extending, it's becoming easier and more accessible to build individual solutions. Contact us to discuss how we can help you fill in your requirements.
Never miss a post! Subscribe to our blog
Receive our emails about the latest Microsoft Dynamics 365 updates and events