Modern plants have a large number of sensors, meters and cameras that provide data in real time during the production process. This data is recorded, sometimes immediately analyzed and flows into the manufacturing execution system (MES) to control production, prevent rejects and ensure quality. Another group of data, the machine data, monitors the production equipment and warns of wear, failure or malfunction. This data is also available in modern MES and supports processes like maintenance and servicing, both processes of predictive maintenance.
This data is usually displayed on charts, dashboards or as numerical values in schematic diagrams. The problem with this is that the place where the data is created, even though known, is not displayed or is only displayed schematically. For a lot of processes, however, it would be helpful to be able to locate the sensor, meter or camera in order to draw the correct conclusions from the values that are being displayed.
Our 3D data localization concept assumes that the installation has been visualized in a realistic way, because the user has to be able to immediately navigate through it. Experience has shown that this is possible only with the help of a photorealistic digital twin.
Therefore, a three-dimensional digital twin can not solely be represented by a CAD or point cloud model. Even if it is elaborately modeled, it will still never offer the same degree of detail a user gets while standing in front of the asset. In order to get close to reality, CAD or point cloud models must always be supplemented with high quality images or panoramic shots. In addition, an asset is often changed over the course of its lifecycle, and these changes must also be represented in the digital twin. CAD designs are, however, too expensive and too time-consuming for this. Here, the photographic documentation of new situations presents an alternative.
In a photorealistic digital twin that is based on a hybrid data structure, i.e., as a 3D and image model, most of the data-providing components are already directly visible or can be made visible through corresponding detailed images. The display of the measured values then takes place at exactly these points.
Each data point is located three-dimensionally in the digital twin and contains links to external data systems. Thus, it does not compete with existing data processing programs, databases or cloud storage – it only displays the contents of these systems. You could say that the digital twin is an interface that brings reality to the user’s screen and gives them easy access to all kinds of data. Of course, data access takes place in accordance with all the usual rights and authentication processes.
Navigating through the photorealistic digital twin from anywhere in the world and accessing all production and asset data in real time opens completely new possibilities for digitalization. This is especially true for cases in which production facilities cannot be easily visited. During the Covid pandemic, where travel was in some cases no longer possible, the photorealistic digital twin combined with MES data became an effective tool – and it will continue minimizing costs in terms of travel and time.
Since the digital twin is the carrier of the data points in three-dimensional space and the twin corresponds exactly to the reality from which it was created by photogrammetric processes, the data points can also be displayed on the real asset. This is where an AI-assisted augmented reality method comes into play. A tablet is used to capture the asset, onto which the virtual data points and objects are then automatically superimposed. All linked information can be accessed via these data points. The only requirement is an internet connection within the real factory, but this is no longer an issue nowadays.
In addition to retrieving information, it is also possible to work on the digital twin on-site. Virtual 3D objects can be placed at suitable locations and immediately checked against reality. Faults and alerts can be recorded and located in the right places so that there is no longer any confusion regarding which asset is affected by the malfunction. If the digital twin allows for intervention in the real asset, such as switching, activating, opening or closing some of its functions, they can also be executed within the AR environment.
Using digital twins for AR has significant advantages compared to the methods used today. Usually, the AR system itself records the spatial environment in the form of a mesh model and the virtual information is then entered into this model.
Because our digital twin is an exact, photorealistic representation of reality, the AR system already knows its environment and all the information stored in it. Another advantage is that entries and comments set from your office are immediately available on-site. This allows for a collaborative way of working that was not possible until now.
In one of our research projects, we are developing a method of sending a robot to a specific location inside the factory to collect real-time data in reality or to perform other tasks by using mobile sensors. The location is defined through the digital twin by an employee. The entire spatial geometry and thus the path is known inside the twin and communicated through the twin to the robot. Its autonomous characteristics ensure that it avoids unforeseen obstacles and collisions.
Controlling robots via the digital twin bypasses the need for time-consuming training in the real environment; this should make setting up robots much faster and cheaper.
Would you like to create a photorealistic digital twin of your industrial plant, saving time and money? Then send us an email at info@framence.com or call us at +49 6251 5840 – we will be happy to advise you.
