Lidar technology is similar to radar technology and is already being used in autonomous vehicles.
Modern vacuum robots use it to orient themselves in the apartment.
The abbreviation stands for "Light Detection and Ranging".
Technically, lidar uses light pulses instead of radio waves.
Corresponding lidar scanners are already in the camera module of the iPhone 12 Pro (Max) and the iPad Pro from 2020. The scanner emits laser light and simultaneously records the reflections.
The rays, which are invisible and harmless to the human eye, extend up to five meters.
If a beam hits an object and is reflected, the scanner calculates the distance to the iPhone or iPad based on the transit time of the reflected light.
Since the lidar scanner emits the photons in a grid of points - similar to the infrared rays in Face ID - the sensor records several parts of the object at once and creates a 3D model with depth information in real time.
The camera, among other things, benefits from this: iPads and iPhones can use the LiDAR sensor in poor light conditions and focus more quickly and differentiate between object and background in bokeh shots.
This supports the effect of such pictures with a natural background blur.
Augmented Reality (AR) apps work more precisely with the depth information and recognize more quickly, for example, whether one object is covering another.
Virtual monsters do not float in the air, but stand on the ground.
All apps that access Apple's ARKit framework automatically benefit from the new sensor.
Short and sweet
The iPhone 12 Pro (Max) and the iPad Pro of 2020 use LiDAR sensors to create depth information of people and objects.
In addition to the camera, AR apps also benefit from the sensors.
Apps with LiDAR support can create 3D models, measure rooms or incorporate objects into games.
Other apps have functions that would hardly work or not work at all without lidar.
The tape measure app, for example, works more precisely on lidar iPhones, and the magnifying glass measures the distance to a person.
Some programs can generate 3D models from the captured objects, which can then be used in AR or 3D apps.
If you were to shoot the 3D models exclusively with the camera, they would look pretty flat.
Lidar's capabilities are limited by the sensor's relatively low resolution.
This is particularly noticeable with 3D scans.
The scanner records large surfaces and objects such as a wall, a cupboard or a chair quite reliably.
For smaller or thinner objects, such as a bicycle spoke or a Lego toy figure, the point grid is far too coarse.
And although the lidar sensor measures more precisely than the camera, there can still be deviations in room measurements.
A LiDAR-capable iPhone does not yet reliably replace the folding rule.
Manageable offer
With the iPad Pro, Apple introduced some apps last year that benefit from the lidar sensor, but many of them are still missing from the App Store.
For example, the anatomy app Complete Anatomy is supposed to record patient movements in order to identify motor restrictions.
The manufacturer of the CAD program Shapr3D promises to scan rooms or furniture with the sensor and to create 3D models or 2D floor plans from them.
Both apps with LiDAR support were announced for 2020.
When asked about the editorial deadline for the current issue of »Mac & i« in early 2021, the developers announced that they were still working on the integration of Lidar.
They didn't want to give us a specific point in time for the implementation.
Nevertheless, dozens of programs can already be found in the App Store that integrate Apple's new lidar sensor.
Simply enter »Lidar« in the search to find more copies.
These include many simple apps that are showcasing value: some only show the 3D image of the lidar sensor that can be used for night vision without a lamp, such as Lidar Night Vision.
We present the most interesting apps from our point of view that are worth trying out:
3D scanner app (3D scanner)
Icon: enlarge
The 3D scanner app captures depth information via lidar and creates 3D models of real objects
Photo: Mac & i
Provider: Laan Labs. Price: free
The app has two modes: The low-resolution one scans large areas and creates three-dimensional room models when you turn around your own axis in a room.
The second, high-resolution mode captures free-standing objects that you slowly walk around.
Although the resolution is too coarse for small or detailed objects, the app recorded larger objects such as furniture, statues or cars with a little practice.
In poor lighting conditions or reflective surfaces, slight artifacts formed on the edges.
The scans can be subsequently measured in the app and exported as a wire frame model or as a textured 3D object.
They are sent to other users via AirDrop or iMessage in order to display them in AR programs.
In addition, the user can export the files in various formats in order to further process them in 3D programs such as Blender, Maya or SketchUp or to prepare them for 3D printing.
Teamviewer Pilot (remote maintenance)
Icon: enlarge
Support employees help with Teamviewer Pilot on site and mark objects via AR
Photo: Anthony Boyd Graphics / Mac & i
Teamviewer; free of charge (private use); from 27.90 euros / month (commercial)
Some things are difficult to explain, you have to show them: Unlike the remote maintenance tool Teamviewer, the pilot does not share the screen, but the camera image from the iPad or iPhone for another Teamviewer user who is sitting at the Mac or PC.
So one can help the other with problems in the real world.
Practical when you have to explain to parents in the distance which cable goes into which socket of the new router or how to repair a bike.
Objects can be marked with AR: With a short tap on the display, an arrow appears over an object;
if you swipe across the screen, you draw lines.
This also works on iOS devices without lidar, but it marks itself more precisely with the sensor.
Object recognition also works better: if you move around with the iPhone and turn the camera so that a marking is covered by another object, it appears transparent.
Markings can also be provided with text, for example to name components.
With an instruction mode, even without remote access or network connection, short videos can be recorded in which objects are marked via AR.
The app mostly worked reliably, but on an iPad Pro from 2020 there were occasional alignment problems that could be resolved by turning the tablet briefly.
The Teamviewer Pilot iOS app and the Teamviewer desktop app are free of charge for private use.
Service technicians and field staff can subscribe to the commercial version from EUR 27.90 per month and user.
It allows an unlimited number of connections.
Canvas Pocket 3D Room Scanner
Icon: enlarge
With Canvas you can create 3D models of a room using a lidar scan
Photo: Mac & i
Occipital; free, with in-app purchases
Canvas captures all walls and objects within a room and creates a 3D model from them.
To do this, stand in the middle and turn in a circle while moving the iPhone or iPad up and down.
As soon as the app has detected a piece of furniture, for example, it marks it with guidelines and you can continue to turn.
Tracking occasionally stopped in bright areas and we had to turn back to the last point detected.
After a full scan, Canvas takes a few minutes to compute the model.
In the 3D room, you can then measure furniture yourself at the touch of a finger, add notes or display it as a 360-degree photo.
The dimensions sometimes differed by two centimeters;
there were also artifacts in lamps, mirrors and windows.
Canvas offers a CAD export, but this is chargeable because the developer wants to revise each model, which should take two days to wait.
Depending on the target format, e.g. for SketchUp, Revit or Chief Architect, the exported file costs between 15 and 39 US dollars.
Magicplan (room planner)
Icon: enlarge
Magicplan quickly creates the floor plan of an apartment from a lidar scan.
Photo: Mac & i
Sensopia; free / subscription (9.99 euros per month to 899 euros per year)
If you move or renovate, you should take a look at Magicplan: The app captures rooms with the iPhone camera and creates a floor plan from them.
The floor and the type of room such as kitchen or bedroom are determined for each room.
If the ceiling height is unknown, the lidar sensor determines it by scanning the baselines of the floor and ceiling.
Then you turn around your own axis in the middle of the room until you have scanned all the walls.
Magicplan places corners either automatically or with a tap of the finger.
Finally, you hold the camera again on existing doors and windows to store them in the floor plan.
The automatically created floor plans can later be supplemented with notes or objects such as electrical appliances or cupboards.
The acquisition works best in empty rooms, but also worked in furnished rooms.
Magicplan correctly captured the walls and doors of a 16-square-meter, rectangular bedroom.
One window, however, seemed four centimeters too wide.
The app supports some bluetooth laser meters for more accurate values.
In the free app, the user can only save two floor plans.
With a subscription that is primarily aimed at architects, brokers or craftsmen, there is no limit.
Costs can also be calculated based on the living space, for example for the wall color.
Lidar Sense (rangefinder)
Icon: enlarge
The rangefinder is aimed at people with visual impairments
Photo: Mac & i
Mike Dopsa; free
The app is designed to help people with visual impairments.
To do this, hold the iPhone in front of you so that the lidar sensor is facing forward.
As soon as it detects obstacles, the iPhone begins to vibrate.
If you have AirPods Pro or Max, the app also uses their 3D audio capabilities and plays a sound from the direction of the obstacle.
With other headphones or earphones, there is only a mono signal.
The closer you get to the object, the stronger the vibration and the louder the sound, which worked well in the test.
The sensitivity of the app can be set between one and eight meters.
We did not notice any difference in objects that were five meters or more away - the sensor does not reach any further.
"Hot Lava" (skill)
Icon: enlarge
In the Apple arcade game "Hot Lava", the play figure rotates through the kitchen, living room or children's room with lidar support
Photo: Mac & i
Clay; free (with Apple Arcade subscription)
Stay on the sofa, the hallway is lava!
In the skill game that is part of the Apple Arcade catalog, you actually jump, climb and sprint over the furniture from a first-person perspective.
If you touch the ground during the wild obstacle course, the pawn is burned and you have to start again at the last checkpoint.
The app activates an AR mode on iPads and iPhones with a lidar sensor: Here you can place traps, opponents and helpful objects such as trampolines in the room using AR and finally chase the character through the living room from a bird's eye view.
Thanks to the lidar sensors, the iPhone and iPad also detect differences in height between real objects: for example, the protagonist falls down if he walks over the edge of the table.
"RC Club - AR Racing Simulator" (racing game)
Icon: enlarge
“RC Club” sends a remote-controlled racing car onto the track via AR.
The lidar scanner recognizes when a car hits a real obstacle.
Photo: Mac & i
Abylight; free, with in-app purchases
Anyone who used to race fervently across the playground with remote-controlled cars could enjoy this here: In »RC Club« you can steer a racing car in AR through the kitchen, forest or neighborhood.
The app works on iOS devices from the iPhone 6s;
On an iPad or iPhone with a lidar sensor, the app also integrates real objects as obstacles.
For example, the sensor detects a slope as a slope or lets the bolide crash into a cabinet.
This worked well with furniture, with wet hills of leaves or carpets the spatial integration was barely noticeable.
At the moment there are only two modes available with an open mode (sandbox) and the time trial.
Others like a fight mode or a duel are to follow.
The base game is free, upgrades or other cars can be obtained through long game sessions - or by paying in-app purchases.
"Epic Marble Run" (Skill)
Icon: enlarge
In "Epic Marble Run" the player rolls around AR marbles
Photo: Mac & i
Patched reality; free
With "Epic Marble Run" you let a marble roll from start to finish: Here you set a starting point and a destination using AR, and a tip later the ball rolls off.
This works on the living room table, in the stairwell or in the subway station.
The longer the distance covered, the more points you get when you cross the finish line.
The marble cannot be influenced while driving, only the angle at which it rolls.
The ball rebounds physically correctly against obstacles that the sensor detects.
What sounds boring at first turns out to be more difficult than expected.
And the app is an entertaining pastime, especially on larger stairs, for example at a bus stop.
Icon: The mirror
