Seeing across the world with Augmented Reality with Patrick Liddy | UtilityAR #33
Augmented Reality sounds so exciting. Put on the magic glasses and escape to another world…..
But let’s come down to earth. It’s certainly a technology that promises amazing things and, with the right application, can bring huge new benefits to business. In this podcast, our host Oliver Dowson is talking with Patrick Liddy, a founder of UtilityAR. They’ve developed a great application of Augmented Reality – or AR as it’s abbreviated – for engineering.
Patrick is an energy engineer by profession, and a true innovator. Back in 2006, he founded Innovation Energy in Dublin, one of the first electricity demand response systems in Europe. He went on to sell that to a US corporation before, more recently, starting UtilityAR.
When you have a moment, take a look at Patrick’s video that we’ve posted on our growinternational.org website. It’ll help you better understand what we’re talking about.
But right now, imagine you’re an engineer going to fix a complex piece of equipment that you’ve never worked on before. Now, think of putting on a pair of special glasses. You can see that piece of equipment through the lenses, just as you would any other glasses, but as well, you can see diagrams, text and video clips projected into a corner of your vision, and hear instructions from a colleague or instructor – who may be thousands of miles away.
When one thinks of the cost and the huge amount of time spent not just travelling, but negotiating backstage access to sites, it’s easy to understand the advantages of being able to kit out one of the site crew with an AR headset and direct them remotely. It could achieve better results, more quickly and at much lower cost.
Even if you don’t relate to engineering, this conversation with Patrick is illuminating. There must be so many practical applications of Augmented Reality waiting to be developed.
Contact details and Links
Patrick’s LinkedIn profile: https://www.linkedin.com/in/patrick-liddy-36559310/
OLIVER: Patrick, welcome to the ‘Grow Through International Expansion’ podcast.
PATRICK: Thanks a lot Oliver. We’re delighted to be here.
OLIVER: So, I know that you’re into a very clever application of augmented reality for engineering. Can you just talk us first through what augmented reality is, how it’s different to virtual reality and what you’re actually doing with it?
PATRICK: Certainly. Well just to explain first of all, virtual reality, which you’re probably more familiar with and is more famous, is where you’re wearing a headset which closes you off from the world and puts you in a virtual space so you can imagine you are anywhere else in the world, and that’s got a lot of uses, but we’re really more interested in augmented reality, which is a technology where you’re wearing transparent glasses, where you can see the world around you and/or virtual information is projected on that glass as well as your still being able to see the real world. So that means that you are safe, you are aware of your surroundings, but you have access to virtual information which might help you get certain things done.
OLIVER: So, you’re sort of seeing two things at the same time?
PATRICK: You’re seeing two things at the same time. So to give you an example that you have experienced in the real world, when you drive the car you sometimes put the newspaper on the dashboard of the car, you can still see the street and you can still see the world around you but you might also be able to see the newsprint which is sort of reflected on the glass of the car quite faintly. Well we do something similar in augmented reality, except it’s being projected on a pair of glasses that you’re wearing. So, we’re interested in helping technicians get their job done. So, what that might mean is a technician who has to follow a procedure or see that I have information about the asset that they’re working on, they’re wearing a pair of our glasses and they can see that information projected on the glass in front of them. So, it could be a step by step procedure of first of all, turn the screw, second of all, open the cabinet, third of all, switch off the power, whatever it might be. They go like that and they can see the steps in text. They can see the steps in images, or they can see short video clips showing them what to do.
OLIVER: So, like technicians like engineers in factories, in maybe electrical plants, maybe engine rooms and ships even?
PATRICK: That’s right. The tasks that these people do or the style of tasks that they do is quite similar, whereby very often what they’re doing is they’re doing a maintenance task which you can predefine the way they wanted it to be done and you want them to follow that procedure step by step. What augmented reality allows you to do is to improve the way you communicate the procedure from just being a text based thing whereby you’re giving someone a clipboard and paper or maybe you’re sophisticated and you give them a clipboard paper and images of what to do. Now you get to go a step further and show them a short video. So, if it is that you want them to turn a handle they can watch a three second clip of someone turning the handle and they see how hard do I have to push this handle. Is this a sticky kind of handle where you have to push kind of hard or is it that it should be easy and I’m finding it hard. So, there’s some problem here that I hadn’t anticipated and I need to go back and see what’s wrong. So. So it’s very very interesting in that in that way. They might also want to be able to see live data of what’s happening. So, let’s say they’re working on a piece of equipment where there’s alarms or there’s flow rate data or there’s electrical consumption data or something like that. They’ll be able to see that information as they go in their vision while they’re doing the task. So, it’s very very useful additional information that they don’t have right now available to them.
OLIVER: So, is this something that’s then self-guided that they choose or they access at the time or to some extent using a human as a remote control robot?
PATRICK: In the first instance it is a self-guided or the glasses guided thing. So it can be that the glasses recognise the asset that you want that the technician should be working on and it can recognise
OLIVER: So this is the style of this is this piece of equipment because it recognises the size and shape.
PATRICK: Correct. So, this is air handling unit one, two, three. We’re validating that it is air handling one, two, three not air handling unit one, two, four and you’re working on the correct one and here is the specific information about this one and and as I said that it can guide you in a more visual way than what would previously have been able to do but then the second instance which is sort of he further guidance is if that technician encounters a problem and they want to get a colleague to a system, right now what they need to do is use the phone to ring a colleague and try and describe verbally what they’re looking at and the problem they’re encountering or they need to call on that colleague to come to site to have a look at the problem they’ve encountered. What you can do with augmented reality is you can do a remote advisor video call whereby the glasses are a computer of themselves and they can do a video call where the colleague at the other end is seeing what the glasses are seeing forward on. So they have a front facing camera which allows you so it’s it’s quite similar to video calls you’ve experienced before but what the adviser is seeing on his computer or her computer at the other end is what the technician is looking at and they can go a step further, they can speak to the technician and say OK we’ll have a look in that cabinet there. They can also go a step further and they can draw annotations on the screen of the technician.
OLIVER: Which is basically the glasses that the technician is looking through?
PATRICK: Which is the glasses and they they draw the annotations which say open that cabinet. I need you to turn that handle clockwise and they can draw an arrow of what they mean by clockwise or they can highlight I’d like to press the green button and they can draw a box around what they mean by green because there can be disagreements about green and red and things like that and simply left and right. So so in that way it’s a whole new paradigm whereby you can communicate much more visually and less text but also you can provide remote advice, whereby you have expertise in one part of the world and you can share that with a colleague who’s in another part of the world.
OLIVER: I was going to say because I mean I in my working life I’ve often experienced engineers or people who have to actually travel to the other side of the world for what would amount to a one hour fix on a piece of equipment or to actually set something right. Simply because it’s been impossible to describe over the phone or it can’t be seen, usually the phones aren’t, well in the old days the phones weren’t anywhere near it and mobile phones can’t be used. Issues like that come up and also there’s a language problem. So, it’s not just you know the color, being colourblind between red and green, it’s also potentially not understanding what the words mean.
PATRICK: You’re completely right and from my past life working in the energy and utility sectors I experienced scenarios, I mean you you alluded to a one-hour fix. I drove across the country to do a 30 second fix in a past life and that was nothing bad about the technician who was there. He just couldn’t see it. He couldn’t see it that this thing was backwards. It’s just a simple thing. So those sorts of things go on. Now you’re a man who likes to travel, I know that and we’re not going to do away with all your travel. There’s times where you still need to send someone to do whatever task it is but in an awful lot of cases this sort of technology avoids unnecessary travel and that’s what we’re thinking about.
OLIVER: I can see that and I can also see the advantage of being able to do something much quicker because if it’s a matter of finding the right person and getting them on a plane and getting them to another place that could be days literally, even if someone is sort of ready to move immediately.
PATRICK: Yeah, so the savings can be significant in all sorts of ways. So, in the first way that I talked about where you want to a technician to follow a procedure, the savings could be measured in percentages. So, you say the technician can do the job 20% because we’re communicating how to do the job more effectively to them. So, you say well there’s a 20% or 50% saving or whatever it is and it depends on the nature and the complexity of the task that you are asking the technician to do. The second aspect is the remote adviser video calling and then the measure is better in days, where you say we have a problem in Germany, we need to send a technician from London to go to Germany to look at this problem. When is the technician going to be able to catch a flight? Well his kid has a birthday party tomorrow or or there just isn’t another flight until tomorrow morning or whatever and the saving can be measured in days which in many businesses is make or break. You know the figures that you talk about are absolutely enormous.
OLIVER: Absolutely I can see it being used in areas other than engineering as well. I think there must be, not sure I can think of them right now you probably know better than me but there must be other areas in which things like this are highly valuable?
PATRICK: Yes. Well I mean my, the company that we are our our DNA and our background is engineering, its technicians, its factories, its utilities that sort of thing, but I know that there are companies looking into this for medical purposes. So, if you have here in London an expert on a particular operation and there is someone in Nigeria who needs assistance on on that surgery or that particular issue, whether he can be getting guidance from the local surgeon here in London on how to do that more effectively. So that’s that’s a really interesting thing. I predict that while many of your listeners, this will be the first time they’ve heard of augmented reality glasses. Ten years from now it will be very much the norm and possibly five years in many applications will be the norm.
OLIVER: Certainly, the speed at which technology advances that doesn’t surprise me at all.
PATRICK: That’s right. I mean if you read some of the trade papers, they will talk about the iPhone 13 being a pair of glasses rather than a phone for a phone and that’s because there’s so many advantages to glasses form factor. Right now the technology isn’t there for consumer products but for enterprise where the value proposition is so clear there’s real opportunities.
OLIVER: Absolutely, it sounds very very clever. I guess it could be that the applications I mean they talk about Big Brother. I mean there are many other things that could be done perhaps in engineering, world of engineering but also that should be used to even without telling somebody what to do, to watch how effective they are actually doing a job.
PATRICK: Yes. So, we get lots of requests around training and I guess in the first instance I described following a procedure to do a particular task and that is sort of a subset of training. Procedure following and training are very linked things but then ultimately at the end of the training you may want to do a test on a technician to validate that they can do their job correctly and definitely like so we’ve spoken to various electricity utilities. They’re interested in the idea that when they send an electrician up a pole to fix a wire, right now they can’t really adequately test, is he okay up there? Does he know..
OLIVER: Does he know what he’s doing when he is up there!
PATRICK: Know what he is doing up there, he knew what he was doing when he was down here but does he still know when he’s up there.
OLIVER: He could be panicking; I don’t like heights.
PATRICK: Correct. Correct. So, it’s a very interesting application and I think you’ll find similar applications in the case of areas which are just difficult to get to and you want to look over the shoulder of someone from a training point of view. Also, in a lot of cases there’s just a liability question. So, if you were involved, we’ve had various conversations with the aviation industry where an aeroplane has many masters, someone financially owns it somebody else operates it, somebody else is the leasing company, somebody else is the maintenance company and when a problem occurs all of these organisations are interested in, well who’s to blame. How are they going to manage that?
OLIVER: and how quickly are they going to get it back in the air again?
PATRICK: Well I think they are interested in that too but there’s parts of their organisations who are just interested the liability. So, the idea of having many parties being able to observe a walk through and a discussion of this is the agreed state of the aeroplane right now and I’m sure there’s many other scenarios you can you can think of. The other thing that we haven’t scratched the surface on and it’s some time before we even begin to think about it will be the big data opportunity that these things will create, whereby imagine you have a technician who you get to service backup generators all day and he’s driving around London or whatever city you’re in servicing backup generators and as part of this you get him to do a short video using his glasses of the state of the backup generator every time he sees it. Now you have a thousand videos of backup generators every year.
OLIVER: Gosh that’s exciting.
PATRICK: Oh well to a geek like me, Yes.
OLIVER: So, this is like the new generators end game?!
PATRICK: Something like that. Well I do think that there will be learnings about seeing where there is wear and tear or seeing common problems that that the manufacturers just can’t imagine right now because they’re just not out there and they’re not getting the feedback. So that could that can be provided. I mean it’s all sorts of silly things within factories where you have your staff and you begin to notice that there is a backup of product at a certain point in the factory at a certain point in the day every time all this sort of new information will be generated and it’s all visual. It’s not what we’re used to which is number based metered information. It’ll be this visual information, but my basic understanding of big data is that it loves or AI I suppose more specifically, it loves images as a thing to analyse and think through, you know and identify trends that are important.
OLIVER: Yes I was talking to somebody in the AI industry or trying to get into the AI industry, developing AI only a few weeks ago and it does, I can well imagine if you were together you’d be discussing how to actually effectively create self-learning tools from this so that you don’t actually need somebody remotely to tell you what to do and which direction to turn the screw in and which handle to push because it would effectively learn it from all of these things and create its own manual.
PATRICK: I think there’s definitely truth in that. I definitely think that if you have an assembly. So if you are sending a piece of kit around the world and having many different people assembling it and if they are being guided on how to assemble it, so I’d use the IKEA furniture as an example but not really that in a more technical sense.
OLIVER: Everyone understands that or at least everyone knows, empathises with it even if they don’t understand the diagram.
PATRICK: So, in the not too distant future on your augmented reality glasses, you will be able to follow a similar IKEA furniture procedure as you have now but then at a certain point they’ll begin to learn oh gosh people encounter a problem at this step. We’ve not explained this step well enough and that’s because of the feedback that the glasses could give them. That’s really really interesting.
OLIVER: It’s actually incredibly powerful and I can think of another application and the world connected to international expansion which is what most of our audience are here listening about and that’s in the world of export because when it comes to exporting technical products, one of the big issues that businesses have in terms of the investments is, is it worth investing in exporting to a certain market given the cost we will have of creating manuals, training and so forth and if you’re selling something that’s you’re probably going to sell in the thousands. That’s probably not such a big issue but if you have something that’s a very valuable asset that sells in very small numerical quantities, then breaking into a new market can be a real challenge. Being able to effectively use this type of technology to help train people on how to actually install this in the first place, maintain it and what to do. I can see huge application for that.
PATRICK: I think you’re right. I think you’re right and that sort of really does come down to our target and I mean we are working with customers from different, looking at this in different ways. So in some cases it is the factory who want, who have a thousand procedures that their technician those each year and they already have procedures for those things but this is a better way of communicating that, those procedures or indeed giving support from their hundred different vendors that their technician can now call upon through the glasses but then we also are talking to companies who have products which they are exporting in the way you described and they and these are technical products so an example would be one that has a renewable energy product that they send to Africa and Asia and local talent installs the product but they provide support through the phone right now or very often through aeroplanes in practice. They wind up having to fly out there and see what’s wrong. Why is it not working? So, the idea that they will give the local talent a pair of glasses, augmented reality glasses and software like ours is very attractive to them.
OLIVER: I can understand that. It makes a lot of sense because it eliminates the loss of the unknown. You say you know going to you know African countries, for example, there happens to be a destination I know that some companies struggle to find people who are prepared to relocate there for months on end and I was working recently with a business that is selling a technical product to Angola and I can estimate that that’s the sort of thing that they would really have loved to have had at the time because having shipped a million dollars’ worth of equipment out to Angola. They then really struggled. It sat around incompletely installed for months before they could actually get out there and discover that in fact, really very simple instructions had been admitted. It’s almost as simple as you know there’s construction manuals 1 to 10 and somebody left number 8 out of the book, out of the packet.
PATRICK: Yes, well what we’re involved in developing new products at the moment to work on these glasses. So, I have some firsthand experience of writing the manual right now and it is interesting to me how when you’re engrossed in a product or you are engrossed in a culture, what we think is obvious isn’t necessarily obvious. So, if you as as a as a British man wrote a manual and handed it to someone in Angola and said follow that, some of the things you would just think is unnecessary to say, is necessary to say.
OLIVER: Absolutely. Absolutely. It’s like does every I cross every T.
PATRICK: That’s right and that that that doesn’t, you don’t have to go as far as Angola for that to be an issue. That is an issue with Belgian customers and some of the things we’re like ‘oh gosh I hadn’t really thought about that being a problem.’
OLIVER: But it doesn’t even get to the visual thing. I mean it’s like, even if we got to staying within the same country and we went to the ordinary retail sector, just dealing with people directly. This is like a step beyond YouTube videos. You know people go to look at YouTube videos, I use them. I mean I went to a YouTube video to learn how to make a repair to my bicycle because I’m not very good with Derailleur gears and I learned how to fix it from the video and did so. I now have a car which has a manual which is 783 pages long. It’s not that wonderful a car but it’s a very and there’s thousands of them on the road.
I couldn’t understand from the manual where to find the button to actually make full beam headlamps work. If I’d been able, looking at the video doesn’t hack it because it’s the American model in the video. So if I’ve actually been able to sort of get in the car and put a pair of AR glasses on look at it and say ‘what’s this button’ and we had effectively with your big data, effectively something that would say that’s the button you have to press to do this and it’s been sitting under my right hand panel for ages but I mean there’s a lot of symbols on them and I didn’t know what they did. So, I can see that there are technical things, even just in a domestic scenario where there are opportunities for using these things, how do I actually fold the back seats down?
PATRICK: You’re completely right, you’re completely right and I mean so much of support is those little things. I know of a British boiler maintenance, well it’s a British utility that offers boiler maintenance and they contract with you basically to maintain your boiler. So that means they get called saying my boiler is broken, please come and fix it and the most common repair is to press the reset button. Now you need to know where the reset button is to press it. You know the reality is is we’d all happily press the reset button because we want our heating back right now. We don’t really demand that they send a technician right now, if the reset button is going to fix it, I’d be very happy to press it. So even within phones there’s opportunity there, but really, the big game here from our point of view is what augmented reality glasses can offer.
OLIVER: Sure and I think there’s huge opportunities for them. We’ve explored a few of them here and I can see that you’re going to have a very successful business working with these and to looking for new opportunities because even as we sit here and talk, I can think of all sorts of new applications and I’m sure others can too. So, Patrick it’s great to talk to you today. I wish you every success with your augmented reality tools. What’s the name of your company again?
Patrick: UtilityAR, you’ll find us on utilityAR.com.
OLIVER: OK and I know that you’ve got a very good little video that actually shows how these things really work and as you’ve said you know you can’t really describe things over the phone. We can’t really describe things visually either and since our listeners don’t have AR glasses with them, we’ll post that video on the growinternational.org site. So, listeners if you look on growinternational.org, click on videos, you’ll see that video there.
OLIVER: So, Patrick, thanks for talking to me and good luck with UtilityAR.
PATRICK: Great stuff. Thank you.
Any facts and opinions presented in this content are those of the author or speaker. The inclusion of this content on the Grow through International Expansion platform does not imply endorsement by the platform owners of such facts and opinions nor by any business represented by interviewees or contributors. Whilst every care is taken to check facts and figures, we accept no responsibility for their accuracy. Please advise us of any discrepancy and we will endeavour to correct the information as quickly as possible.