Reality Now: Turning Data into Actionable Information
Much industry media coverage has been given to the capabilities of current reality capture (recap) tools such as drones, mobile lidar scanners, hand-held SLAM devices and even smartphone-based capture technology. And surely most readers of this publication are at least casually familiar with recap terminology such as point cloud, photogrammetry, orthomosaic, lidar and 3D mesh.
It seems nearly every AEC and Infrastructure (AEC/I) software and hardware vendor has joined the chorus extolling the virtues of recap and offering a product or service related to digitally capturing the real world. This is a great development for the industry, because a common vernacular is needed for conversation and widespread recap adoption to occur.
During the last few years, you’ve likely come across many articles, courses and webcasts about recap for AEC/I, including those from this publication. This, too, is great for elevating the industry knowledge base and advancing efforts to improve safety, accuracy and efficiency. Most of this content focuses on the capture tools themselves as well as the techniques and workflows associated with achieving these benefits. This article will not regurgitate those previous works.
Rapid advances in recap hardware and processing software—and dramatic decreases in the costs of both—have lowered, or altogether removed, many of the adoption hurdles and democratized its use. No longer the domain of a few specialists or highly trained professionals, the ability to quickly and easily collect vast amounts of highly accurate, highly detailed data has refocused the recap conversation onto the data themselves and how they are and can be used. To that end, this article will look at these data in action now and what’s just over the horizon in the very near future.
A Brief Sidenote
Before jumping into the conversation about the use of data, it’s worth pausing to briefly discuss a common misconception about the computer hardware needed to access and work with recap data. In the AEC/I industry, two of the most-prevalent recap methods are photogrammetry and lidar, and the resultant measurements frequently are delivered in the forms of a point cloud, high-resolution imagery and 3D meshes.
A common misconception is that high-end workstations are needed to take advantage of all these wonderful data. However, with the widespread availability of online processing and storage coupled with affordable pricing, there’s no need to invest in new hardware. A modest CAD workstation (e.g., Intel Core i7 with 20GB of RAM and an Nvidia GForce GTX 960 with 1GB of VRAM) is more than adequate to effectively and efficiently work with the orthomosaics, point clouds, and 3D meshes generated by online processing services.
Furthermore, the cost of the recap hardware itself has plummeted. For example, the highly capable workhorse DJI Phantom 4 photogrammetry drone costs a bit more than $1,000; the Leica BLK combines a high-definition 360-degree camera, a laser scanner and a thermal imager for less than $20,000; and if you own an iPhone 12 or newer, you have a lidar scanner for free with your phone.
Turning Data into Actionable Information
Armed with all this powerful tech, how are the masses of designers, engineers, contractors and other professionals using the data in their everyday work lives? “Each stage [in the design, build, operate cycle] sees very important benefits of reality capture,” says Benoit Fredericque, senior director of product management iTwin Context – Context RealityData and GIS with Bentley Systems. “Design sees a lot of benefits with growing use in construction and operations. Drones are increasingly used at each stage.”
In fact, the base starting point for most AEC/I projects is capturing and creating a model of what is, whether on a site or inside a building.
“All work we do is based on a plan,” says Martin Fischer, director for the Center for Integrated Facility Engineering (CIFE) and Kumagai professor in the School of Engineering at Stanford University. “That’s where the biggest impact of our work happens. Recap gives us the tools to create feedback loops to more-efficiently deploy resources and optimize the design of assets and workflows.” The ability to rapidly develop accurate models and plans drives the adoption of recap in the AEC/I industry.
One well-documented use is in site topographic and structural surveys. Point clouds often are captured for topographic surveys and volume calculations. This use is not new and has been around for more than a decade. The previous challenge was creating a useful digital terrain model (DTM) from the point cloud, as this exercise would tax the capabilities of most workstations.
What is new(ish) is the ability to create usable analysis directly from the point clouds themselves. For example, most point cloud processing tools avoid DTM creation and directly compute differential volumes between two point clouds or between a point cloud and a design DTM. In (most) cases where a DTM of the existing ground is needed for use as the basis for further design, these same processing applications can output TIN, meshes and 3D contours that are easily handled by most design and drafting software.
Nico Bonnafoux, product manager for Esri Site Scan for ArcGIS, shares a specific example: “Arcadis, a global design, engineering and management consulting company based in Amsterdam, reduced field data collection time by a factor of 10 for recap of a 30-mile stretch of a 12-lane highway in Qatar. They did this by moving from traditional survey equipment to using drones. This is a tool that enhances the capabilities of trained and qualified staff.”
Not only was field time dramatically reduced, the quantity and quality of the collected data far surpasses what’s feasible with legacy tools. Better, more-thorough data lead to better decision-making, planning and design.
“Having detailed info [about current conditions] leads to better design and reduces errors during design,” concurs Bryan Williams, portfolio manager for 3D capture and robotic layout at Trimble Inc. “Engineers generally consume the data and see real returns on investment [from recap] by getting things right the first time without making a lot of changes; and by preparing accurate estimates and better designs, especially in an existing situation. Good 3D data help lead to better designs.” The operative words here are “good data,” where good applies to both the quality (high definition and accuracy) and accessibility (timeliness and usability) of the data.
DIY or DaaS?
Although do-it-yourself (DIY) recap for existing site and corridor modeling and planning now is within reach for most industry players, some want the convenience of skipping this step and instead simply licensing the data for immediate use. A benefit of using a “Data as a Service” (DaaS) provider is the data have been prepared for use in advance, saving time and costs of doing this task inhouse, and eliminating the need for purchasing or renting recap equipment.
One such DaaS provider is Nearmap, which captures high-resolution aerial imagery multiple times per year and licenses photogrammetry products to the AEC/I industry, including orthomosaic imagery, 3D meshes and point clouds.
“Previous challenges with recap data include obtaining appropriate data and then using it in a meaningful way,” says Don Weigel, global vice president of product at Nearmap. “While current recap technology has made this easier, sometimes it is preferable to simply stream current baseline model information directly into your design and analysis tools. When engineers and designers have ‘analysis-ready’ data, they don’t have to worry about the accuracy of the inputs, its format, rotation or translation. Instead, they can make data-driven decisions and focus on engineering tasks and outcomes.”
Another DaaS provider is Here Technologies, which offers several location-based products available for subscription and licensing. In addition to traditional base maps, users can access points of interest with enhanced location data; road and infrastructure data on road signage and signals, curvature and steepness, and roughness and elevations; and environmental zones data that include information on area restrictions such as vehicle class or emissions restrictions.
One of the most-impactful changes enabled by the democratization of recap tech is the decentralization of data capture and processing. Recent bottlenecks caused by the need for specially trained recap professionals in the field and data processors in the office are reduced or eliminated by giving more team members simpler (and more-powerful) tools to capture and work with the data at the times and locations needed. (Of course, “garbage in, garbage out” still applies.)
Perhaps the greatest benefit of this is that recap is no longer a special event that occurs at predetermined, prescheduled times. While comprehensive recap of existing conditions still are planned and scheduled ahead of time during the design phase, it’s during the shift from design to construction planning and execution when the new benefits become most apparent. Says Trimble’s Williams, during construction “power moves from the office to people in the field. The field team can communicate back to the engineers in the office, decreasing the communication time between identifying issues and resolving them.” Movement and distribution of data models no longer is a one-way street; instead, they move freely among field and office teams and other stakeholders.
Several real-world examples at every stage of construction—from pre-construction planning to closeout and handover—demonstrate the versatility of recap and its expanding uses.
Site logistics and pre-construction planning benefit from recap in several ways, including evaluating access to the site via public roads, rivers, etc.; and understanding and accommodating buildings, trees and other existing items either enroute to the site, adjacent to the site or on the site itself. Common uses of recap point cloud and photogrammetric data include extracting information about bridge height and width clearances along possible roads leading to the site; evaluating existing overhead wire sag elevations that may impede access to the site; and using a 3D model of the site and adjacent areas to plan for crane location and onsite material storage.
“Recap delivers a richness of data, which is used before construction even starts,” explains Bentley’s Fredericque. “Because the scans are so comprehensive, often contractors will collect data for a specific purpose and then return to the data for additional uses.”
Once construction begins, there are almost too many uses of recap to count and certainly too many examples to list them all here. One such example is weekly construction planning and more-frequent progress validation. Capturing as-is or as-built conditions to compare against as-designed models previously required a special setup to perform the capture and then additional time in the office to process the data and overlay the information on the design model. Any deviations from the design or conflicts in the construction then would be flagged for discussion and remediation. Now these comparisons are done directly in the field.
Using tools such as the Trimble X7 scanner, which includes an inertial measurement unit (IMU) and advanced internal algorithms, field personnel are capturing the as-is conditions at greater frequency and directly comparing the recap scan data with the design model. On mobile devices, field personnel are making measurements, identifying missing components and verifying that items have been built in the correct locations.
“Validation during construction keeps the decision process moving by putting the data and information directly into the hands of those making decisions,” explains Williams. “This reduces rework and change orders, and keeps the project moving.”
With the increased practice of offsite prefabrication of components, recap provides additional value. For example, BIM design models typically are used as the basis for prefabrication models. This workflow is satisfactory until actual as-built conditions begin to deviate from the design, as often happens as construction progresses. Because of the limited space into which many MEP assets are built, even slight deviations from the design can result in prefabricated components not fitting properly. With the widespread use of recap tools during construction, contractors are scanning as-built conditions and relaying this information back to the shop to inform changes to fabrication, eliminating rework and schedule delays.
“The general contractor and the MEP contractors are collecting this recap information,” says Williams. “This is no longer a specialty service. Because the tools have been simplified, more team members have the ability to collect the data and work with the results themselves. QA/QC tasks become decentralized.”
Yet another example of the time and cost savings associated with today’s recap addresses the problems of labor shortage and personnel safety. As Micah Callough, technology director, AEC, at Esri explains, “Inspections, especially specialty inspections like those done for asbestos abatement, are addressing labor shortage using recap technology. For example, demolition and construction schedules can be negatively affected by the limited number and availability of experienced inspectors.”
However, high-resolution, high-defintion recap data captured in the field by less-experienced inspectors or laborers can be reviewed in the office by an experienced inspector. “This saves time, money and keeps the project on schedule,” adds Callough.
The list of additional current uses is long and includes checking aggregate elevations prior to a concrete pour and measuring flatness after the pour; monitoring and tracking oil and other contaminants in shipping lanes to identify leaking ships using chemical IDs; and aiding in the fight against and recovery from West Coast wildfires by tracking airborne chemicals, paint, oil and other hazards kicked up by the fires.
Intriguingly, recap data collected for a specific present-day purpose can provide answers to as-yet-defined challenges in the future. “The raw, unfiltered data provides the opportunity to discover information ‘hidden’ in the data,” says Fredericque. “Months or years later, a contractor, engineer, facility owner or someone yet to be identified might resolve challenges by extracting additional information that wasn’t originally needed.”
Returns on investment are notoriously difficult to quantify, usually due to the lack of a well-defined baseline. In the AEC/I industry, the benefits realized today can easily be measured in a couple of ways.
“You can look at recap investment ROI from the cost-leadership angle and the differentiation angle,” says Hesam Hamledari, associate at McKinsey & Company and recent PhD graduate from Stanford University’s CIFE. “For example, in a recent study, a general contractor in the United States used robotic data capture to reduce data-capture time by 90 percent while simultaneously collecting five to six different types of data over what was required. This can significantly reduce the cost of data capture as well as a project’s cost overruns due to increased situational awareness.
“The contractor now had a tangible differentiator from their competitors,” he explains. “They had better data and could share 3D models of the site with the owners instead of the 2D plans used by their competitors. They could more-rapidly verify work progress, issue payments and resolve conflicts.”
Reality the Day After Tomorrow
As these practical applications of recap are just gaining traction today, reality capture of tomorrow will enable more-spectacular results. In fact, we’re starting to glimpse future jobsites.
In one incredible mashup of recap, robotics, AI and block-chain technology, Stanford’s Fischer and Hamledari, along with their industry colleagues, used lidar-equipped unmanned ground vehicles and camera-equipped aerial drones to automatically scan a project site, document and verify construction progress, and, using a blockchain-enabled smart contract, automatically issue conditional lien waivers from the contractor to the owner and issue payments from the owner to the subcontractors. The full details are available in the paper “Construction Payment Automation Using Blockchain-Enabled Smart Contracts and Reality Capture Technologies” by Hesam Hamledari and Martin Fischer (found at bit.ly/31ro88c).
Hamledari sees the future of recap developing in three areas: capturing, analyzing and using the data. He explains devices used for capturing reality will continue to expand to collect multisensory data throughout a project’s lifecycle. Analyzing the data for useful information will improve with continued progress in AI and computing performance. And the way we use recap information will continue to develop to further document ROIs, integrate with other tech (such as blockchain-enabled smart contracts) and tie contract “success” to business objectives.
Expanding capture capabilities is the first step, and Esri’s Callough agrees. “Once we move past the stigma of surveillance, we can capture more data and move the industry forward. One example where we’ve accepted this is in vehicle tracking, which has been around for a while. Soon we’ll see constellations of sensors on most project sites, gathering reality data for air quality, personnel, tool location, activity, materials and more.” Before this happens, however, Callough says a big increase in security is needed to reduce risks of hacking. “Security is the responsibility of everyone.”
Fischer with Stanford expands the traditional concept of recap for AEC/I to include workflows and information itself. “Reality capture should extend into the supply chain to provide visibility of the reality of the material flows. For example, without a reliable sense of what materials are or will be needed, contractors must have more materials onsite and manage those materials onsite. By creating visibility into the material flow, its transit and delivery information flow, we can cut down on ‘just-in-case’ materials.”
Developments in capturing data are not limited to what is captured but also will include how frequently the data are captured. Capture frequency will increase in many ways, including the use of terrestrial robots.
“Applying drone automation techniques to terrestrial capture technologies will greatly expand the amount of data collected for projects not suitable for drones,” says Fredericque from Bentley.
Nearmap’s Weigel adds, “Data changes frequently, so capturing it frequently provides insights from months to days to hours. The time from capture to desktop will reduce dramatically. This will have immediate benefits to time-sensitive uses such as search and rescue, disaster response and mitigation, and more.”
The second area where recap will develop is in the analysis of the capture data. “AI will get better and help save more time, providing current, accurate and insightful analysis-ready information,” says Weigel. This is a needs-driven result in which engineers, planners, contractors and owners need to evaluate multiple scenarios and narrow their focus to a handful of good options.
“Customers will have more voice in training the AI models to answer specific questions,” he adds. “For example, we envision bundles of AI attributes and layers for specific use cases, such as assessing roof condition for the insurance industry, and bundles of layers for building footprints and elevations for the telecom industry.”
In addition to comprehensive recap models, Weigel also sees an increased use of “mini models” of reality. “Think traffic studies: a recap of roadway geometry provides a digital environment for modeling car and pedestrian flow.”
Analysis of recap data continues to benefit from cloud-based processing, storage and collaboration. “[The] cloud has proven its ability to break down a lot of walls and enable actions that were previously impossible, especially around collaboration and data-sharing activities,” says Fredericque. “Adoption of recap tools will continue to grow, with different recap technologies connecting in the cloud. As this happens, we’ll start to gain real-world benefits at scale.”
The capture, analysis and preparation of recap data culminates in their usage to solve real problems. The continued development of existing technologies (e.g., augmented reality, virtual reality and mixed reality (AR/VR/XR)) and the roll-out of new technology (e.g., 5G, optical fiber connections to the cloud and advanced edge computing) will further enable use of recap data whenever and wherever needed.
As the volume of data increases through time from many projects, their usefulness also increases, even if those uses currently are unclear. “Acquiring data requires investment with the idea that at some point in the future, the data will be involved. Currently, [the industry is] lacking enough historical data to prove the value of collecting data without knowing what you’ll use it for,” explains Bentley’s Fredericque. “Many countries and asset owners are taking the initiative to capture this data for future use. One example comes from France’s main natural-gas distribution system operator, GRDF. One of their initiatives is to capture all-new trenches before they are backfilled. This is owner-driven.”
Bonnafoux summarizes the advances in capturing, analyzing and using recap data with a drone example. “Reality capture technology is moving from a ‘toy’ to a standard jobsite tool. Soon, drones will be in the hands of everyone who needs them. This is enabled by more automation, both in capture (the ‘drone-in-a-box’ concept) and data extraction—using AI to assist in decision making such as detecting faults, making performance predictions and automating scheduling.” Widespread capture tools and advanced analysis algorithms will deliver ready-to-use recap information at an ever-increasing scale and scope.
Beyond the Day After Tomorrow
As recap tools continue to expand their capabilities, their impact on the industry also will expand, enabling workflows and outcomes heretofore impossible. These include specific advances such as improved design assistance from our applications to entirely new business models. Integrating feedback loops into the process will accelerate these future uses.
As Stanford’s Fischer explains, “Current recap methodologies lack situational awareness, because they lack feedback loops. Future recap will integrate feedback loops to inform and optimize design and operations. Once we have more data, we can optimize; for example, computer tools can design optimal steel structures in significantly less time. We have tools today which help design light structures, but what if by making it heavier you reduce the complexity of the connection designs? This is not possible today, but once we have more data, we can create these automations.”
The capture, analysis and use of data leads toward more-robust and accurate modeling, which enables real-time monitoring, preventive maintenance, and simulations to predict future operations and outcomes. To this end, Fischer points to an example of an industrial structure developer that simultaneously builds a digital twin along with the actual structure. “The developer requires weekly recap so the digital information can be digested into the digital twin over a longer period of time,” says Fischer. “The digital twin of the asset is then available at the time of handover of the physical asset.”
This type of tight integration between the physical and digital assets can change the way contracts are defined and projects delivered. Currently, contracts typically define success as the delivery of an asset such as a building, bridge or water-treatment facility. New business models will emerge, with a clearer definition between delivering a product vs. delivering an outcome. Those AEC/I players that can guarantee a desired outcome will have a distinct advantage. “In the future, success will be defined by outcomes, such as ‘the building will operate at 50-percent less energy consumption’ or ‘the roadway will require zero maintenance for X number of years,’” forecasts Fischer.
The consensus is that firms which learn to adapt to the rapidly evolving recap landscape will thrive, while those which do not will struggle. Capturing reality and using the information extracted from the data is simply a means to the end of addressing efficiency problems, labor and expertise shortages, getting everyone home safely, and ultimately delivering not just a project, but results.
As Fischer summarizes, “The current feeling is that any one individual [person or entity] sees too little of the whole. Digital reality capture enables business models, workflows and outcomes to work together, so we can accelerate the development and incorporation of these new methods for greater efficiency, less waste and better results.”