New directions for applied knowledge-based AI and machine learning

Ontologies are a major approach of knowledge-based AI and are in everyday use. A commonly cited definition of ontology is “an explicit specification of a conceptualization” [12]. Gruber furthermore explains conceptualization as “the objects, concepts, and other entities that are presumed to exist in some area of interest and the relationships that hold them”. This definition is broad enough to include formal upper-level ontologies but also knowledge graphs, thesauri and other forms of knowledge representation. (There are entirely different opinions on this, as we point out below.) Examples are domain-specific ontologies like Gene Ontology¹ or the Global Biodiversity Information Facility² for life sciences, but also Wikidata³ or the Google Knowledge Graph⁴ for general purposes.

Use cases for ontologies are semantic search, decision support systems or interactive systems like knowledge-based chatbots. In addition, conceptualizing a domain of interest is of value per se. Ontology search engines like the Basic Register of Thesauri, Ontologies & Classification (BARTOC)⁵ include more than 3000 ontologies, each with thousands or even hundreds of thousands of entities being specified.

When having a closer look at concrete ontology examples (in the broad meaning of Gruber’s definition cited above), different flavors can be distinguished; and with those flavors, entirely different communities are associated.

One flavor includes ontologies, for which the term knowledge graph has been used in recent years⁶. For example, for the Google Knowledge Graph, simple standardized ontology schemas like schema.org⁷ are used. schema.org specifies a limited number of classes (currently about 800) like persons, organizations, locations, products, events, etc. Simple standardized modeling languages like RDFa, Microdata and JSON-LD may be used to specify concrete entities. Those simple modeling languages provide restricted semantic expressiveness, including specifying classes, instances, attributes, and relationships, but excluding complex reasoning.

The following example⁸ shows the JSON-LD specification of an organization with its legal name, founding date and email address:

Because of the simplicity of modeling languages and schemas used, we call this flavor of ontologies lightweight. Lightweight ontologies are much used in industrial practice, e.g., by all major internet search engine providers and tech companies, including Google, Amazon, eBay, Facebook, IBM, LinkedIn and Microsoft [14]. They are relatively easy to use. Various tutorials and guidelines can be found to access such ontologies or provide data for them, e.g., jsonld.com⁹. However, their discussion in the academic community is rather sparse.

Another flavor of ontologies uses modeling languages with much higher semantic expressiveness, such as OWL¹⁰ or F‑logic [16]. Such ontology modeling languages allow knowledge engineers to express reasoning, logical operators, quantifiers, etc. Ontology examples are the BFO¹¹ and the Gene Ontology. Consider the following example from [15] formalizing the red color of strawberries using a set relation (subClassOf) and a quantifier (some):

Those ontologies are inherently complex, and, therefore, we call this flavor heavyweight. There is much attention being paid to this flavor in the academic community [13, 17‐19]. Due to the high expressivity and, thus, inherent complexity of ontology modeling languages, several guidelines for engineering ontologies exist, e.g., [15]. To model the simple statement that strawberries are red in color in the complicated way shown above is, in fact, a recommended modeling pattern from [15].

In those guidelines, lightweight ontologies (including languages for modeling them like JSON-LD) are usually not mentioned at all, or it is stated that this flavor cannot be considered in ontologies at all (e.g., [15], Section 2.5). Even though this viewpoint is rather common, it is still in contrast with Gruber’s broad ontology definition from 1993 cited above.

We notice that around those different ontology flavors, there are different communities dealing with them: rather developer-oriented communities for lightweight ontologies and academic communities for heavyweight ontologies. However, there seems to be minimal overlap and little communication between those communities. It seems like they live on different, disconnected islands. Why is this the case? Maybe there are fundamentally differing goals driving ontology engineering in those different communities.

One goal for developing lightweight ontologies is developing knowledge-based applications like semantic search engines. And the software engineering principle KISS (keep it short and simple) also applies to engineering ontologies to be used in such applications. Consequently, modeling languages, guidelines and tutorials are as simple as possible. The simpler, the better; this is the secret to their success.

One goal for developing heavyweight ontologies is formalizing an application domain as accurately as possible in order to draw valid logical conclusions. And, as reality is complex, the modeling languages and resulting ontologies reflect this complexity; the more complex, the more accurate, and therefore, the better.

What can we conclude from this observation? We think that it is most important to be absolutely clear about the goals before starting to develop an ontology. To be aware of the different ontology flavors and corresponding communities may help us choose the appropriate modeling languages and tools suiting those goals.

A Wordnet is a lexical database of semantic relationships between words in a specific language. The first Wordnet¹³ was created for the English language at Princeton University. As the usefulness of Wordnets as lexical resources became apparent, the Princeton Wordnet was expanded, and WordNets were constructed for other languages.

OMW (Open Multilingual Wordnet)¹⁴ is an open-source project that was launched with the goal to ease the use of Wordnets in multiple languages. OMW has the added benefit of connecting equivalent sets of synonyms (called “synsets” in Wordnet)¹⁵ in different languages by means of an interlingual ID called “ILI”.

OdeNet¹⁶ was constructed from open-source linguistic resources in combination with some manual and semi-automatic corrections. Since OdeNet was constructed independently of existing resources in OMW, it was difficult to connect equivalent synsets in OMW via ILI. As an initial implementation, Google Translate¹⁷ was used in combination with statistical methods as described by Siegel & Bond (2021 [31]). However, this implementation has some shortcomings, including:

A significant problem in using machine translation to connect equivalent synsets in different languages occurs when translating homographs (words with the same spelling but different meanings) and polysemes (words with the same spelling but different although related meanings). This is particularly noticeable, when a word translated from a source language is a homograph or polyseme in the target language. As an example, we take the German word “Unterlegscheibe” from OdeNet. The corresponding English translation is “washer”. Searching for “washer” in EWN, we find three synsets containing the word:

All synsets in EWN have a short, concise definition. We propose to use this definition to get more context for the disambiguation. First, we combine the word in the synset and the definition and do machine translations. Then, we extract the translated word from the machine translation and look for a corresponding match in OdeNet.

These are the results for the “washer” example:

As is evident, the machine translation of the second item now enables us to make the correct ILI classification (i60971) for the corresponding OdeNet synset.

Since there still could be multiple candidates in OdeNet synsets for ILIs in EWN synsets, it is necessary to write a classification function to assign weights to each of the candidates, so that the most optimal assignment can be made. Fortunately, OdeNet is very synonym-rich (much more so than other Wordnets), and we can use these synonyms in combination with a German Word2Vec [32] model to do the classification.

The content words in the translation of the definition are added to a vector. All the synonyms in the candidate synset are also added to a vector. For each value in both vectors, a similarity value is computed. These values are summed and normalized to a value between 0 and 1, which is the weighted value for the candidate synset in OdeNet competing for the ILI in a specific EWN synset.

As a result, there are no more duplicated ILIs in OdeNet, and we could correct more than 400 wrong POS tags (of about 36,000 synsets).

Semantic search is well understood and well documented, see [5]. It exploits the semantic relationship between concepts such as generic term, subordinate term, opposite, synonym, etc. However, relationships to partial terms and partial meanings of concepts are usually not exploited. How can semantic search be improved further? One idea is to expand the search to include partial terms that are contained in term definitions. Definitions of terms can be created according to the genus-differentia scheme (Gen-Dif) known from Aristotle [18]. In an innovative approach to semantic search [1], we use so-called Basic Linguistic Symbols (BLS) as linguistic identifiers, which can record the nouns of the terms in any language. For example, for >BLS-gauge, in German we have “Meßinstrument”, in English “gauge”, in French “jauge” and in Spanish “calibre” (see Fig. 1). Now we can compose terms using the Gen-Dif pattern as follows. In >BLS-barometer = (>BLS-gauge, >BLS-air_pressure) the left term gauge is the genus (◊Gen), and the right term air pressure is the differentia (◊Dif). According to the same scheme, we have >BLS-air_pressure = (>BLS-pressure, >BLS-air) with the triples (>BLS-air_pressure, ◊Gen, >BLS-pressure) and (>BLS-air_pressure, ◊Dif, >BLS-air). This definition can be inserted into the first: (>BLS-barometer = (>BLS-gauge, (>BLS-pressure, >BLS-air))). Since both the left ◊Gen and the right ◊Dif argument of a definition can be resolved by further compound definitions, a so-called conceptual binary tree (CBT) is created [1]. The leaves of the CBTs are always indecomposable, atomic concepts like >BLS-air, >BLS-pressure, >BLS-aid, >BLS-thing, and >BLS-measurement. The methodology presented here can also be used to more precisely define and disambiguate the terms used in ontologies and in WordNets. From (>BLS-Barometer, ◊Gen, >BLS-gauge), (>BLS-Barometer, ◊SymbolOf, ^Barometer) and (>BLS-Gauge, ◊SymbolOf, ^Gauge) e.g. automatically (^Barometer, ◊subClassOf, ^Gauge) can be inferred, i.e., that ^Barometer is a subclass of ^Gauge.

This allows a deep semantic search (DSS) to be implemented. Query processing then, in addition to the usual semantic relations such as superclass, subclass, synonym, etc., also makes use of the terms that occur at any level of the conceptual binary tree. A search radius can be set to the depth to which the partial terms should be included in the search. To find out whether Google also takes compound terms into account when searching, we evaluated Google search against DSS. As an example, various musical instruments were modeled with so-called Word sense definitions (WSD).

A Google search for “wind keyboard instrument”¹⁸ returns 24 million search results, most of which refer to pages with melodicas. Searching for pipe organ only returns the quiz question “What is the largest wind keyboard instrument called?” Our deep semantic search finds 67 instruments, 8 concepts for keyboard and 30 concepts for wind. In addition to melodica and pipe organ, the word accordion is also found. The term accordion does not even appear on the first Google results page. Thus, DSS can be used to determine additional suitable search terms, which can then be entered in the respective search engine. Also, our sample query “wind keyboard instrument” was evaluated against WordNet Search¹⁹. WordNet finds no result at all, only a search for “wind instrument” finds an entry that describes the term itself, but not the concepts that fall under it.

On the basis of the Longman Defining Vocabulary²⁰, we developed a Foundation Core Glossary (FCG) with 450 atomic concepts and approximately 2500 compound terms in eight languages. All new terms can be defined with the concepts of the FCG. This means that queries with any combination of terms from these languages deliver the same results as if they had only been made in one of the languages.

A first version of the deep semantic search has been used in the EnArgus research project for search optimization [2]. The multilingual news app rob.by²¹ also uses this technology. Furthermore, applications are being implemented where the technology is to be used in portal search engines to search for scientific publications. These applications also benefit from the fact that the number of occurrences in documents is noted for each term in the glossary. As a further parameter in the deep semantic search, it can thus be taken into account whether the term is very general or very specific.

Problems with the implementation of anomaly detection in an industrial setup result from the different industrial requirements that must be considered. Among time prerequisites to establish a fast reaction to an anomaly, a reasonable prediction quality and a configurable design to adapt to the dynamics of such a system are essential. Moreover, the communication and processing limitations of those systems must be considered.

In cooperation with Yaskawa²⁶, a leading mechatronics and robotics company, a fully automated, data-driven, and feasible anomaly detection for cyber-physical systems in manufacturing has been developed [30]. The cyber-physical system, shown in Fig. 5, demonstrates different drive units and their collaboration to realize complex processes. The overall process involves picking small items from a round table in the middle of the unit and placing them into several cups while they move on conveyor belts around the machine.

One example of an anomaly that can appear in the described system is an incorrect establishment of the vacuum generation for picking the cups from the conveyor belt to place them in a delivery position. This can be caused by broken cups, an incorrect picking position, or a worn gasket at the intake. As a result, the system can run into an undefined state, and the normal process can no longer be executed.

To realize anomaly detection while fulfilling industrial requirements, several reconstruction-based, one-dimensional convolutional autoencoders were applied to the cyber-physical systems. The models are trained without the need for domain knowledge. No anomalous data is provided in the training step to meet the data-driven requirements. A sliding window approach achieves the timing requirements and the prediction quality. A shallow structure and the decentralized integration of the separate models address the inherent processing limitations.

An example of a reconstructed data stream and the detection of the abnormal behavior can be seen in Fig. 6; red dots indicate the detected anomalies, and the marked areas display the abnormal sections.

Effective and feasible anomaly detection, considering industrial requirements, can significantly impact automated processes to realize modern and safe cyber-physical systems in production.

The recent Corona pandemic, the flooding of the German Ahr valley, the accident of the container vessel Evergiven stuck in the Suez Channel, the traffic jams caused by it at container terminals in China and Europe and the current Russian war against Ukraine have shown the vulnerability of our society and economy in the case of unforeseen disruptions. In 2020, the German Federal Ministry for Economics and Climate Protection (Bundesministerium für Wirtschaft und Klimaschutz, BMWK) started the funding of five projects addressing questions of economic resilience in pandemic and other economic crises. Within these projects, the ResKriVer-Project²⁷ addresses—besides other problems—some issues of modeling, documenting, analyzing and simulating supply networks of crises-relevant and potentially substitutable goods and resources, in order to strengthen the resilience of critical supply paths.

Fraunhofer FOKUS’²⁸ role within this project—besides its role as consortium leader—is the semantic modeling of supply networks as a knowledge graph, which, in contrast to simple supply chains, considers entire cross-organizational networks in order to perform preventive as well as reactive analysis. The approach chosen by FOKUS is the adoption of W3C’s provenance standard PROV‑O²⁹ as one of the top-level ontologies of a “ResKriVer common core ontology”. This ontology supports the modeling of supply networks on a fine level of granularity and at the same time should allow for (a) the extraction of simulation models for Fraunhofer IML’s ODE-Net system for discrete event simulation and (b) approaches for dependency analysis in order to determine critical nodes within a supply network and their associated probabilities.

During the initial modeling of some examples of supply networks, it soon became clear that this approach will face three major issues. First, the integration of probability information can be addressed by partial adoption of PR-OWL, an ontology for modeling probability information. Second, structural equivalent RDF subgraphs will frequently occur during the modeling of supply networks in order to capture alternative supply chains. Thus, some kind of efficient RDF templating mechanism would be helpful for efficient modeling. Consequently, mechanisms to partially instantiate these subgraphs with available information need to be developed. A basis for such templates could be SHACL³⁰ or—probably better suited—ShEx³¹, augmented by some mechanism to generate new unique anonymous resources and instances during template instantantiation. We consider this to be a new research topic. As a third issue, the non-availability of supply chain information, which is usually difficult to obtain for second and higher-order tier suppliers, needs to be addressed. Hence, coping with highly incomplete information about supply networks and acquiring them during the pre-crisis documentation phase and within some current crisis will be a major issue in this applied research project.

On 11 December 2019, the European Commission presented the Green Deal³². The Communication from the Commission also includes a detailed action plan, including an annex with a table of measures within a timeline. In March 2020, the Communication on “A new Circular Economy Action Plan” followed³³. The action plan contains an industrial strategy for a clean and circular economy, aiming at sharing, reusing, repairing and recycling existing products as long as possible. Within the framework of the Multi-Stakeholder Platform for Standardization (MSP)³⁴ W3C and GS1 gave birth to the idea of combining the Linked Data platform of W3C with the work on GS1 Digital Link. A report was handed to the Commission.

The idea contributed to the considerations around the works on the Digital Product Passport (DPP). Good ideas must be simple, and this is the case here. On a high level, the problem of circularity is seen from an information system perspective. Today, if a product hits the recycling plant, we do not know much about it, and the recycling plant has to do a lot of guesswork and research to find out about the physical object in their hands. This does not really work well and, consequently, recycling can still be improved. But how can we get information to the recycling plant? This question looks simple but has a variety of complex issues of system engineering.

Who can provide useful information? The producer of the tangible goods has lots of information, but not necessarily everything. First, this producer may only assemble components. So the composition of the components is not always known to them. Secondly, the producer has commercial secrets around producing certain goods. So it is not simple and easy to just make information public. The same goes for the producer of the components assembled into a consumer product. It may be produced with secret production techniques not known to the entity assembling the various components. And the producer of the component cannot know about all the products that their component is used for.

Finally, there is the user of the product. Users may be careful with a product or treat it badly. There may be accidents, lightning, repair, etc. in the lifetime of a product that may influence the way the product or its components can be recycled. This information is with the consumer or end user. But not only with end users, also with their agents and repair services. This is important information that needs to be collected.

While a central database is easier to establish, the large variety of actors or sources of information and the heterogeneity of the information collected makes monolithic systems see their limits quickly.

But what would such a distributed, product-centric system look like? The basic idea is to use GS1’s Digital Link Specification [20] together with the Linked Data platform [21] and create a system of annotations to the physical product identified by some IRI. Linked Data annotations [22] have the power to create a knowledge graph around a type of product or the instance of a product. Because the system uses the Linked Data paradigm, data from a variety of sources can be retrieved and easily merged into a full picture. Meanwhile, DL 1.1 makes sure that the annotations are reliably linked to the product instance at hand. A persistent link between the physical and the virtual world is established.

The overall architecture of the system is shown in Fig. 7. We are only at the beginning of a development.

GS1 Digital Link uses the Global Trade Identification Number (GTIN) that is often represented as a barcode, a QR code or an RFID tag [23]. Batch numbers, serial numbers and more can also be included. From the code, a link is created that is an entry point into the web. The simplest case is that a barcode on a product just points to a web page. The GS1 Digital Link Specification not only uses the simple web but allows for sophisticated operations using so-called “resolvers”, applications identifying that the URI at hand is a GS1 Digital Link URI. These resolvers have a set of typed URIs corresponding to the GS1 Digital Link requested, as indicated in [24]. Although designed for GTINs, the system would work with any other identity scheme (e.g., DNS plus URI) with its own resolver. Standardization work is very likely to begin at ISO shortly to expand the GS1 ideas to encompass any recognized identifier system.

Using the GS1 Digital Link standard, one type of link can now point from the product into an RDF knowledge graph. This graph may receive information from a large variety of sources. Having the product instance at hand, all this information is instantly accessible. If transformed back into HTML, this can be used for consumer information. If given in machine-readable form to a gateway in the recycling plant, it can help sort incoming goods. But the knowledge graph can also help with compliance and regulatory constraints. The minimum requirements for the DPP could be expressed in SHACL [25]. It is then easy to match the regulatory requirements with the existing graph. By including information about access, usage limitations and other constraints, the knowledge graph itself already contains the necessary administrative information. Both can also serve as a proof of compliance [26].

Much remains to be done. Vocabularies and ontologies for the industry will have to be created. The first milestone will be to address the new battery regulation³⁵.

This section deals with the semantic enrichment of technical information that shall ensure the safe operation of machinery put into service in the European Union. The scope of this technical information is defined by the European Machinery Directive 2006/42/EC [33]. Technical information becomes smart by being labeled with metadata addressing its issues and concepts as well as the different entities it relates to, for example the machinery and its components (e.g., battery), the actors and their roles (e.g., service technician), and the actions it describes regarding its use and maintenance (e.g., replacing a component). We call these data context-related metadata. Thus, smart technical information has been semantically enriched and put into a comprehensive context.

Making legally compliant technical information smart enables it to be retrieved within a semantics-based information system over a facetted or semantic search and to be semantically linked to other pieces of information. Making legally compliant technical information smart is a development that has increasingly taken place over the last few years, due to the change in information consumption, the requirements for context-aware information and the digitalization of information delivery in the context of industry 4.0 [34, 35].

The semantics-based hybrid information system SOLIS-Doc³⁶ intends to include a technological as well as methodologically sound framework for authoring and delivering such smart technical information.

Authoring technical information often occurs within a component content management system (CCMS), enabling technical writers to author and manage technical information in a modular way, meaning that each single issue of the technical information becomes a small information unit, a topic, or a fragment. Putting all the needed topics together, technical writers can, over the CCMS, automatically publish an operating manual, either as a PDF document or as an HTML snippet.

To facilitate the modular authoring of legally compliant technical information within a CCMS, SOLIS-Doc provides the semantic model of an operating manual according to the European Directive as an RDF/RDFS ontology. The semantic model builds a minimally valid, and thus transferable, reference template of the entire mandatory content of such a legally compliant operating manual. The SOLIS-Doc Reference Template (see Fig. 8) can be configured and imported into a CCMS and provides technical writers with a reference structure including all the topics and fragments that they can use as a basis within their projects.

Over the SOLIS-Doc reference template, all reference topics have already been pre-labeled with standardized context-related metadata of the new iiRDS³⁷ RDFS-based metadata model [36]. Consequently, technical writers only need to add those project-specific context-related metadata such as product metadata. These proprietary context-related metadata have been modeled within the SOLIS-Doc Twin of Customer (Product) World. Thanks to this pre-labeling, the amount of context-related metadata that need to be assigned to the information units for creating the intended smart technical information can be reduced.

The SOLIS-Doc metadata model not only uses iiRDS standardized vocabularies for labeling information units but also adds semantics to these vocabularies by modeling relations between them. Thus, SOLIS-Doc becomes an RDF knowledge graph that is a semantically enriched iiRDS twin. This enriched iiRDS twin enables an additional reduction of the number of context-related metadata technical writers must assign to information units. For example, being modeled over the iiRDS twin that the action replacing (e.g., replacing a battery) is related to the product life cycle phase (PLCP) service (customer:replacing solisdoc:related-PLCP solisdoc:service), technical writers only need to assign the action-related metadata to the topic, while the PLCP-related metadata is automatically assigned to it by the CCMS.

As shown in Fig. 8, SOLIS-Doc consists of three semantic models, each of them based on standard frameworks for the development of semantic web applications. It is the aim of SOLIS-Doc to be interoperable with different CCMS as well as different content delivery portals (CDP). Of course, it is a prerequisite that these CCMS and CDP have been (technologically speaking) prepared to integrate semantic web-based models. That is a development also taking place within the SOLIS-Doc project.