1 Introduction

Business Process Management (BPM) describes the discipline of bridging the gap between business, technology, and human workers in organizations (Rahimi et al. 2016). In more detail, modern information technologies (e.g., process-aware information systems; PAIS) are the enabler towards the automation of the processes in organizations and comprising the interaction between humans and the application of technology (i.e., human-driven processes) (Dumas et al. 2005). As a prerequisite for successfully utilizing PAIS, process model readers should be able to correctly interpret the many organizational processes. The latter can be ensured while respective model information, as well as knowledge about these processes are properly documented; either textually or visually (Zimoch et al. 2017a). Thereby, in order to sustain competitive advantage, an easy to comprehend and correct documentation of process information and knowledge is essential (Polyvyanyy et al. 2008). In this context, an established approach to document process information and knowledge relies on the process modeling technique, in which respective information and knowledge are visually documented in process models. More specifically, process models summarize the individual processes of organizations with their logical sequence of activities and functions, together with involved stakeholders or exchanged data. For this reason, one of the primary purposes of process models is to communicate information and knowledge about corresponding processes. As a result, process models should be created in a way that involved stakeholders do not encounter any challenges in the comprehension of such models in order to take full advantage of their benefit (Awadid and Nurcan 2019).

All stakeholders involved in working with process models can be assigned into a group consisting of process modelers, process readers, or a combination of both (Zimoch et al. 2017c). Initially, a process modeler consolidates required information and knowledge about a process and creates a corresponding process model based on it. The process modeler should be aware that the created process model reflects a high model quality to ensure a proper process model comprehension (de Oca et al. 2015). Accordingly, a process model should help readers understand the relationships between processes. However, a process model of high quality that is comprehensible for the initial model creator fails to ensure that even a process reader can comprehend the same model (Mendling et al. 2019). Usually, the two major reasons for this are, on the one hand, that there exists a gap of experience and expectations (i.e., different perspectives) in working with process models (Figl 2017). On the other hand, pitfalls (e.g., modeling errors) in the communication of process knowledge, as well as information between process modelers and readers, describe another reason (Haisjackl et al. 2018). To tackle these issues, specific frameworks and guidelines in the literature exist, emphasizing quality aspects (e.g., consistency in process models) to foster the comprehension of conceptual and process models.

One of the most influential frameworks for conceptual modeling constitutes the SEQUAL framework, also known as the framework of Lindland, Sindre, and Solvberg (LSS), introduced in Lindland et al. (1994). This framework considers three different quality dimensions (i.e., syntactic, semantic, and pragmatic quality). It provides means of improvement for each quality dimension in order to maintain a high quality in conceptual models, thus having a positive influence on the comprehension of such models.

Further, the authors in Krogstie et al. (2006) address shortcomings (e.g., static view upon semantic quality) of the SEQUAL framework and propose an adjusted framework. In addition, a significant enhancement of this work describes the consideration of as-is as well as to-be states (i.e., domain and knowledge). Based on semiotics, an integrative framework for information systems is discussed in Mingers and Willcocks (2014). Thereby, the authors consider the interaction of three worlds (i.e., material, personal, and social) derived from Sociomateriality Theory and use this kind of interaction to discuss deficits and improvements in model comprehension. Another framework for evaluating the quality and comprehension in conceptual models constitutes the Bung-Wand-Weber (BWW) framework (Gehlert et al. 2007). It comprises metrics to evaluate the quality in conceptual models. Thereby, a focus is set on the modeling process and the BWW framework considers how objects from the real world change during the transformation into a conceptual model and the impact on the model quality as well as comprehension during this transformation. The work presented in De Bock and Claes (2018b) introduces the Comprehensive Process Model Quality Framework (CPMQF). The CPMQF summarizes existing knowledge about process model quality and structures related knowledge based on six key questions, with an emphasis on the completeness and relevance of quality aspects in process models. Finally, another prominent framework addressing the improvement of the process model lifecycle (e.g., identification, modeling, optimization (Koliadis et al. 2006)) constitutes the Capability Maturity Model Integration (CMMI) (Constantinescu and Iacob 2007). The CMMI comprises a set of best practices for the identification of potential shortcomings in process models and provides specific recommendations for model improvement.

Furthermore, past works presented various rules that can be applied for the improvement of process models. For example, the Guidelines of Modeling (GoM) provides directives to measure the quality in process models from different viewpoints (e.g., user and purposes) in order to foster the comprehension of process models (Becker et al. 2000). The work presented in Mendling et al. (2010) describes a set of seven process modeling guidelines (7PMG) assisting process modelers in the creation of comprehensible models. Finally, good practices for the modeling of processes by addressing the syntactic, semantic and pragmatic quality are presented in Gabriel et al. (2022). Finally, the author in Moody (2009) presents fundamental principles ensuring that visual notations, from which such models are created, are used in an effective way to address especially human communication.

In the literature, numerous and significant works deal with, on the one hand, the improvement of process modeling and, on the other, the fostering of process model comprehension. However, the discussed works are mainly on a theoretical basis. Furthermore, emphasis is put primarily on conceptual models and none provide an applicable measurement and the quantification of the perspectives of process modelers and readers in process model comprehension. The BWW presents insights about how objects from reality (e.g., processes) change through transformations (e.g., modeling), whereas the 7PMG provides guidelines and recommendations for the definition of comprehensible process models. A final evaluation of the impact of the application of such frameworks and guidelines in terms of quantifiable and comparable parameters still needs to be included. As a consequence, the identification of aspects in a process model that are hard to comprehend (i.e., noise) is still tedious because the results presented in the discussed works might be too abstract (i.e., no clear directional guidance for process model improvement). In addition, especially novices or non-experts may need help recognizing their benefits in the context of process model comprehension.

For this reason, in line with prior conducted research and as a further contribution to improve our understanding of working with process models, we try to foster process model comprehension with a first approach that recapitulates and quantifies the specific perspectives of process modelers and readers as well as the interaction between both groups as main determinants in model comprehension. Therefore, this paper presents the Process Model Comprehension Framework (PMCF). The PMCF describes the first step towards a framework to measure the comprehensibility of process models from the perspective of process modelers, readers, and the interaction of both. Therefore, from interviews with domain experts from the field of BPM and insights obtained from the literature, an Evaluation Theory Tree (ETT) with 96 quality metrics was defined. The ETT was evaluated in a survey with 131 students and practitioners to determine the importance and degree of impact on process model comprehension of the quality criteria as well as metrics used in the ETT. In conclusion, the PMCF contains a pilot implementation that quantifies process model comprehension by considering both the perspective of process modelers and readers. To demonstrate the applicability of the PMCF, a case study with 33 participants from the industry was conducted. In general, the PMCF shall unravel prevalent pitfalls that need to be addressed in order to ensure proper comprehension of process models. Furthermore, uniform model comprehensibility is pursued by applying the PMCF between process modelers and readers. In the future, the PMCF is intended to provide additional assistance for organizations in efficiently and effectively utilizing information systems.

The structure of this paper is as follows: Sect. 2 provides theoretical fundamentals of the PMCF. The PMCF and the defined ETT are presented in Sect. 3. Section 4 describes the implementation of the PMCF. In Sect. 5, the PMCF is demonstrated in a case study. In addition, based on the case study, Sect. 5 presents how existing process models in a practical environment can be improved in terms of process model comprehension with the PMCF. Furthermore, current limitations as well as implications of the PMCF and future work are discussed in this section. Finally, Sect. 6 summarizes the paper.

2 Theoretical fundamentals

This section introduces the underlying theoretical fundamentals of the PMCF: the Communication Theory (see Sect. 2.1) and the Conceptual Modeling Quality Framework (CMQF) (see Sect. 2.2).

2.1 Communication theory

According to the Communication Theory (see Fig. 1), a process model constitutes an artifact utilized for the communication of information and knowledge about a process between two participants (Moody 2009). Thereby, the two participants involved in this kind of communication can be denoted as the transmitter (i.e., process modeler) and receiver (i.e., process reader) of information and knowledge. More specifically, the process modeler encodes respective information and knowledge about a process within a medium. In this context, the medium describes a process model. In general, a process model delineates a conceptual model that is used to transfer information and knowledge about a subject the model represents (e.g., order to cash process) (Da Silva 2015). Thereby, a process model is expressed in terms of a particular process modeling language (e.g., Business Process Model and Notation (BPMN)), which is used to communicate information and knowledge about events, activities, decisions, data, and involved participants (Kocbek et al. 2015). Thereby, a process modeling language is described by two components: (1) alphabet (i.e., set of graphemes) and (2) grammar (i.e., systematic description of the process modeling language). Hence, a process modeler must have an adequate understanding of the alphabet and the corresponding grammar for the proper documentation of process information and knowledge in a process model. In turn, captured information and knowledge in a process model are decoded by a process reader. In decoding, human perception constitutes the central information processing system and describes the two psychological processes (a) visual perception (i.e., processing of visual information and knowledge) and (b) comprehension (i.e., interpretation of information and knowledge). As a consequence, the encoding as well as the decoding of information and knowledge in a process model results in different perspectives for process modelers and readers. However, any kind of transmission and communication is susceptible to errors. For example, information is lost or not understood. Consequently, pitfalls (i.e., noise) may occur between the communication of process modelers and readers. In particularly, noise defines perturbations in the comprehension of process models. These perturbations cause ambiguities between process modelers as well as readers regarding the communicated information and knowledge in a process model, thus leading to a non-uniform process model comprehension. For example, the conception of the process modeler about the process or the used modeling language for the creation of a corresponding process model are potential noise factors in the encoding phase (Claes et al. 2017). Regarding the process model, the intention (e.g., process optimization), with which the process model (e.g., textual or visual) is perceived, denotes another noise factor (De Bock and Claes 2018a). Additionally, visual representation factors of a process model such as size and structure constitute further significant factors impeding process model comprehension (Winter et al. 2020). Finally, reasons for noise in the decoding phase are mainly the perceptual as well as cognitive processing (e.g., expertise in working with process models) of information and knowledge in the process model (Caivano et al. 2018). Generally, the occurrence of noise in this context depends on many additional factors (Trkman et al. 2019; Mendling et al. 2007). For this reason, a vast body of research emerged in the past studying the factors that influence process modeling as well as the comprehension of such models (Figl 2017).

Fig. 1
figure 1

Communication theory

Full size image

A significant reason for the occurrence of noise between the three aspects encoding, process model, and decoding is mainly due to the lack of the overall process model quality in this communication procedure (Krogstie 2016). Thereby, quality defines characteristics aspects (e.g., process modeling expertise, the correctness of a process model) that can be measured and compared with each other (e.g., degree of excellence) (Ghicajanu et al. 2015). In this context, the Conceptual Modeling Quality Framework (CMQF), therefore, defines a set of quality aspects in order to prevent shortcomings (i.e., noise) and, at the same time, to ensure high quality in the creation and comprehension of conceptual models (e.g., process model).

2.2 Conceptual modeling quality framework

The Conceptual Modeling Quality Framework (CMQF) presents a unified overview considering the quality of the conceptual modeling process as well as the quality of the corresponding final result (i.e., conceptual model) (Nelson et al. 2012). The CMQF aims at the creation and comprehension of conceptual models with a high external quality. It addresses the fact that high quality models can only be achieved when the complete development process (i.e., from the initial collection of information to the final definition of the model) is of high quality (i.e., free from errors or ambiguities). The CMQF has its origins in the two former frameworks comprising the Bung-Wand-Weber (BWW) framework (Gehlert et al. 2007) and the Lindland, Sindre, and Solvberg (LSS) framework (Lindland et al. 1994). The BWW framework introduces methods for the evaluation of reference models. The central concept of this framework describes the ontological normalization of a reference model. More specifically, it aims at achieving a uniform representation of facts by applying certain transformations. The LSS framework, in turn, comprises a set of different statements in order to evaluate the outcome from conceptual modeling. For example, semantic quality is determined by juxtaposing the statement sets of the modeling domain with the ones from process modeling. Figure 2 presents the CMQF with corresponding clusters, dimensions, and layers with related quality types (i.e., physical: red, knowledge: green, learning: purple, development: blue). In general, the CMQF attempts to address occurring noise known from the Communication Theory (see Sect. 2.1) without a concrete perspective of a process modeler and reader. Importantly, the CMQF defines two horizontal clusters describing the physical (i.e., real world) and the cognitive reality (i.e., cognitive perception), as known from the LSS and BWW framework. The physical reality refers to the domain of discourse (Regoczei and Plantinga 1987), whereas the cognitive reality describes the constructed representation of the perception from the real world. Moreover, for each horizontal cluster, the CMQF defines four vertical clusters: domain, model, language, and representation. These four vertical clusters represent the conceptual modeling process. In particular, the domain refers to the process environment, which can be depicted in a conceptual model. The conceptual model, in turn, is created in terms of a particular modeling language resulting in a specific representation of the conceptual model. Moreover, all clusters comprise eight different quality dimensions. These eight quality dimensions constitute either physical or cognitive artifacts in conceptual modeling. Furthermore, the quality dimensions are associated with quality types, summarized in four different layers: the physical (see Fig. 2, red), knowledge (see Fig. 2, green), learning (see Fig. 2, purple), and development (see Fig. 2, blue) layer. In the physical layer, the appropriateness of a conceptual model for depicting a process and its environment is evaluated. The knowledge layer states that for each physical representation, a cognitive equivalent representation in the perception exists. Furthermore, the learning layer explains how information and knowledge are acquired by interpreting the real world. Finally, the development layer describes that knowledge and information are used to create physical artifacts (e.g., conceptual model). The quality types define for each layer the relationship between a reference and a purpose of application. More specifically, the reference constitutes the chosen quality dimension, whereas the purpose of application depicts the quality dimension that is being considered across all quality dimensions. Moreover, to draw on the Communication Theory, the quality types are responsible for the prevention of noise. For example, the quality aspect between the physical domain (i.e., reference) and the domain knowledge (i.e., the purpose of application) depends strongly on the perception of a person. Hence, it is of importance to ensure that the person has a correct understanding of the domain. The four layers, as well as the quality types (i.e., seven in physical, seven in learning, four in learning, and six in development), depict the conceptual modeling process and, at the same time, preserve the completeness as well as the correctness of the final conceptual model.

Fig. 2
figure 2

Conceptual modeling quality framework (CMQF)

Full size image

3 Process model comprehension framework

The Process Model Comprehension Framework (PMCF) is an adaption of the CMQF and considers the comprehension of process models based on the fundamentals of the Communication Theory (see Sect. 2). The PMCF allows for the measurement of the perspectives of process modelers and readers as well as the interaction between both as main determinants in the context of process model comprehension. As a novelty, the PMCF firstly attempts to quantify process model comprehension for different perspectives (i.e., process modelers and readers) and facilitates the identification of noise in model comprehension. Note that noise in this context represents factors that influence process modeling as well as the comprehension of process models. Figure 4 delineates the PMCF. As known from the CMQF (see Fig. 3), the two vertical clusters (i.e., physical and cognitive reality), the four layers (physical (P1 - 7, red), knowledge (K1 - 7, green), learning (L1 - 4, purple), and development (D1 - 6, blue) remain unchanged in the PMCF. The four vertical clusters (i.e., domain, model, language, and representation), the inherent eight quality dimensions as well as the associated quality types have been adapted accordingly to fit to the requirements concerning process models. The first vertical cluster refers to the process and its environment as well as the process knowledge thereof. The second cluster, in turn, considers the process model and related model knowledge. Similarly, as in the second cluster, the third cluster correlates the used process modeling language with respective knowledge. Finally, the fourth cluster describes the representation of the process in the real world and in perception. In the PMCF, the same quality types from the CMQF are used and defined as follows:

Fig. 3
figure 3

Process model comprehension framework (PMCF)

Full size image

Physical layer

  • P1: How well can the context be represented in a process model?

  • P2: How well can the developed process model be represented by a specific modeling language?

  • P3: How high is the syntactic quality of the process model?

  • P4: How high is the semantic quality of the process model?

  • P5: How well can the process facts be represented with a specific modeling language?

  • P6: Does the process representation correspond to the process model in the minds of modelers and readers?

  • P7: How high is the pragmatic quality of the process model?

Knowledge layer

  • K1: Which knowledge is needed to determine whether a context can be represented in a process model?

  • K2: Which knowledge is needed to determine whether the process model can be represented with a specific modeling language?

  • K3: Which knowledge is needed to determine the syntactic quality of the process model?

  • K4: Which knowledge is needed to determine the semantic quality of the process model?

  • K5: Which knowledge is needed to determine whether the process can be represented with a specific modeling language?

  • K6: Which knowledge is needed to determine whether the process model in the minds of users and modelers corresponds to the process representation?

  • K7: Which knowledge is needed to determine the pragmatic quality of the process model?

Learning layer

  • L1: Which knowledge can be acquired by analyzing the process environment?

  • L2: Which knowledge can be acquired by analyzing the process model?

  • L3: Which knowledge can be acquired by analyzing a specific modeling language?

  • L4: Which knowledge can be acquired by analyzing the process representation?

Development layer

  • D1: How well can process knowledge be transfered into a process model?

  • D2: How well can process knowledge be transfered with the application of a specific modeling language?

  • D3: How well can process knowledge be transfered into a process representation?

  • D4: How well can process model knowledge be transfered into a specific modeling language?

  • D5: How well can process model knowledge be transfered into a process representation?

  • D6: How well can modeling language knowledge be transfered into a process representation?

Juxtaposed to the CMQF, the relationships between the quality types are bidirectional. The reason is that a quality type between two dimensions, on the one hand, leads to knowledge gain and, on the other hand, indicates the necessary knowledge level to assure a high model quality. For example, consider the quality type K1 in Fig. 4, this quality type between the Process Domain Knowledge (i.e., reference) and Process Model Knowledge (i.e., the purpose of application) addresses the fact that describes which knowledge level about the process is required (e.g., information about value-adding activities) to represent this process in a process model. On the contrary, changing reference with the purpose of the application describes that the comprehension of a process model consequently results in new insights (e.g., identification of bottlenecks) about the process.

The PMCF allows for the analysis of the process of process modeling as well as process model comprehension by addressing different aspects. For example, the semantic quality of a process model can be measured and statements about the representability of process models while using specific modeling languages can also be made.

3.1 Evaluation theory tree (ETT)

Based on the fundamentals discussed, an Evaluation Theory Tree (ETT) was defined (Christie and Alkin 2008). In general, the ETT represents a convolution of the PMCF. It serves as a foundation that is used in a first approach towards the quantification of process model comprehension. The roots of the ETT consider the perspectives of the process modelers and readers. The two roots in the ETT exist because aspects exist that cannot be mapped directly between both perspectives. For example, the creation of a process model is only relevant for process modelers. Therefore, process modelers and readers must be considered separately. Each perspective, in turn, consists of several aggregated quality criteria in the context of process model comprehension. These quality criteria are related to the eight quality dimensions from the PMCF (see Fig. 4). Furthermore, the quality criteria and metrics were obtained from existing literature in a related review and from interviews with domain experts from the field of BPM.

Narrative literature review In literature, there exist numerous works focusing on process model quality in order to, on one the hand, improve the creation and, on the other hand, to foster the comprehension of process models (Figl 2017; Dikici et al. 2018). Moreover, different frameworks and guidelines with an emphasis on process model quality were defined for this context as well (see Sect. 1). Regarding the PMCF, several data sources and publication libraries (e.g., Google Scholar, SpringerLink, IEEE Xplore Digital Library) were examined in the context of a narrative literature review (Jahan et al. 2016). Therefore, general search strings were elaborated with the combinations of keywords derived from our knowledge of the subject focus regarding works concerning process model quality during process model creation and comprehension. The narrative literature review allowed us for a more general search of relevant literature with no clear question to be answered. In the first place, it was not attempted to identify all relevant literature, but ensures the finding of pivotal papers in this context. However, we are aware of the fact that a narrative literature review, juxtaposed to a systematic one, bears potential risks that need to be discussed (e.g., search bias; see Sect. 5.2).

Structured interviews In the conducted structured interviews, eight domain experts (i.e., from different organizations (i.e., five in total) in academia (i.e., three experts) and industry (i.e., five experts)) with many years of expertise (i.e., ranging from five up to eleven years) in the field of BPM were personally consulted. The used catalog of questions was derived from the insights obtained from the conducted narrative literature review. The interview type was structured and all interviewees were asked the same questions in the same order to ensure comparability of the results. A set (i.e., 17 questions in total) of opinions (e.g., Which quality aspects in a process model are essential in the creation of a complete and correct process documentation, but also to ensure that the created model can be comprehended by all involved stakeholders?), behavioral (e.g., How can it be avoided that a process modeler creates an incorrect model?), and competency (e.g., When does the application of a process modeling language or the creation/comprehension of a process model become too complex?) questions in terms of preservation of quality in process modeling as well as process model comprehension (see Appendix E). All interviews were recorded and transcribed with the method of intelligent verbatim. From the given answers of the interviewed domain experts, quality criteria and metrics were identified referring on the perspective of process modelers, readers, and synergies covering both perspectives.

The categorization of the quality criteria and corresponding metrics was done within a consensus decision-making process, involving participants from industry (i.e., two BPM domain experts) and academia (i.e., three BPM domain experts)) as well as obtained insights from the literature review, including the interviews. In more detail, obtained insights from the narrative literature review were juxtaposed with the answers from the experts of the interview to identify the most relevant criteria and metrics. Frequently encountered and crucial aspects (i.e., process model complexity), both in the review and interviews, were emphasized and later used in the definition of the quality criteria as well as metrics.

As a result, the perspective of process modeler comprises the following six main quality criteria:

  • Process Modeling Language: This criterion covers crucial aspects (e.g., workflow patterns) that a process model language should support for the creation of high quality process models.

  • Process Modeling Tool: The quality of created process models are dependent of the used process modeling tool. Hence, this criterion summarizes vital aspects (e.g., process views) about tool-support in process modeling.

  • Information: This criterion is concerned with process information retrieval and addresses aspects like correctness and completeness of process information.

  • Errors: Semantic (e.g., logical errors) and syntactic (e.g., errors in modeling conventions) errors in a process model are subject of this criterion.

  • Person: Person-related characteristics (e.g., process modeling experience) are considered in this criterion.

  • Process Modeling Guidelines: This criterion covers guidelines and rules (i.e., from enterprise and academic) that were defined in order to create process models of high quality.

These quality criteria are subdivided into several sub-metrics, resulting in 54 different quality metrics.

Regarding the perspective of process reader, 42 quality metrics are summarized in a total of seven main quality criteria, which are defined as follows:

  • Process Modeling Language: This criterion addresses aspects (e.g., modeling language complexity) that define comprehensible process modeling language.

  • Medium: The subject of this criterion is the question with which medium (e.g., paper-based) is the process model comprehended.

  • Information: This criterion deals about which kind of process information (e.g., process participants) are included in the process model.

  • Person: Person-related characteristics (e.g., process modeling experience) are considered in this criterion.

  • Level of Detail: In this criterion, the level of detail (e.g., abstract or concrete) of the comprehended process model is addressed.

  • Representation Factors: Aspects about the process model representation (e.g., number of elements) and the model structure (e.g., block structure) are subject of this criterion.

  • Comprehension Questions: Process model comprehension performance analysis (e.g., comprehension questions) are considered in this criterion.

Altogether, the ETT contains 96 quality metrics (see Appendix F). Each of the 96 metrics can be assigned to one or more quality types in the PMCF (e.g., learnability of a modeling language refers to K1 in Fig. 4). Further, the evaluation of the quality types facilitates the identification of noise (e.g., the difference in process model knowledge) between process modelers and readers as known from the PMCF. Due to space limitations, Figure 5 only presents an excerpt of the ETT (see Appendix A). As a first approach to measure and quantify process model comprehension, please note that the ETT does not claim to be complete. Therefore, the ETT contains only a limited set of quality criteria and metrics obtained from the described procedure (i.e., narrative literature review and domain expert interviews). We are aware of the fact that numerous factors exist that need to be covered by the ETT (see Sect. 5.2).

Fig. 4
figure 4

Excerpt from the ETT

Full size image

In order to be able to measure and quantify process model comprehension, the importance and the impact on process model comprehension for each quality criterion and its related metric in the ETT had to be determined. For example, as shown in (Tallon et al. 2019), the number of elements in a process model constitutes a more critical factor having a more substantial impact on process model comprehension juxtaposed to the labeling of process model elements. For this reason, a survey with 131 participants from academia (i.e., 103) as well as industry (i.e., 28) was conducted (see Appendix G). The survey participants were asked to rate and place the quality criteria and related metrics for both considered perspectives (i.e., process modeler and reader) in an order from important to unimportant to determine the rank and the impact of each quality criterion and metric. For the determination of the rank, the results were analyzed with the weighted arithmetic mean \(\bar{X}\) that is defined as follows:

$$\begin{aligned} \bar{X} = \frac{ \sum _{k = 1}^{n}{ w_{k} }{p_{i,k}}}{\sum _{k = 1}^{n}{w_{k}}}, \end{aligned}$$
(1)

where k is the rank, n is the number of ranks, \(w_{k}\) is the weighting factor for the rank k, i is the quality metric, and \(p_{i,k}\) is the percentage choice of the quality metric i on the rank k.

The weighting factor \(w_{ k }\) had to be determined since each quality criterion and related metric exert a different impact on process model comprehension. For the calculation of the weighting factor \(w_{ k }\), the following evaluation methods for information retrieval were juxtaposed in a series of repeated measurements: Rank Sum, Reciprocal Rank, Rank Exponent, Discounted Cumulative Gain, and Distance Normalized Logarithm (Sakai 2007). Most considered methods showed a disparate differentiation between the quality criteria and metrics (i.e., Rank Sum, Rank Exponent, Discounted Cumulative Gain). Moreover, the methods presented no standardized distance between the highest and the lowest rank as well the ranks in between. In more detail, the provided weighting factors of those methods exhibited limited growth leading to an incorrect differentiation of the quality criteria and metrics. However, further analyses demonstrated that a cubic (i.e., Reciprocal Rank) or an exponential growth (i.e., Distance Normalized Logarithm (DNLog)) should be considered in this context. Comparing both methods revealed that the DNLog was more suitable for our purpose. The reason for this decision was that the DNLog ensures a more exact weighting of the quality criteria as well as metrics. For example, a vital criterion or metric (e.g., syntactical process model correctness (Leopold et al. 2015)) has a more significant impact on process model comprehension and should be given a greater weight compared to negligible ones (e.g., avoid OR routing process model elements (Mendling et al. 2010)). As a result, the DNLog was chosen for the calculation of the weighting factor \(w_{ k }\), which is defined as:

$$\begin{aligned} w_{ k } = 10^{(n - k) \frac{\log _{10}(d)}{(n - 1)}}, \end{aligned}$$
(2)

where n is the number of items, k is the rank, and d is the score in the survey.

Equations (1) and (2) were used to analyze the responses from the 131 participants of the survey and to determine the rank as well as the impact of all quality criteria and related metrics on process model comprehension for the perspective of process modelers, readers, and the interaction between both perspectives (see Appendix H).

4 Implementation of the process model comprehension framework

This section presents the pilot implementation of the PMCF in order to measure and quantify process model comprehension. For this purpose, the ETT has been implemented in a Microsoft Excel workbook template (see Appendix B). According to the Communication Theory, this workbook is used to evaluate process models regarding their comprehensibility to unravel noise (i.e., pitfalls) between the communication of process modelers and readers. Note that the workbook consists of nine sheets:

$$\begin{aligned} \begin{array}{ll} \textcircled {1}\ \text {Configuration} &{} \textcircled {2}\ \text {Modeling Language Complexity} \\ \textcircled {3}\ \text {Supported Patterns} &{} \textcircled {4}\ \text {Quality Metrics} \\ \textcircled {5}\ \text {Questionnaire for Process Modeler} &{} \textcircled {6}\ \text {Questionnaire for Process Reader} \\ \textcircled {7}\ \text {Perspective of Process Modeler} &{} \textcircled {8}\ \text {Perspective of Process Reader} \\ \textcircled {9}\ \text {Summary} &{} \\ \end{array} \end{aligned}$$

The nine sheets resulted from the creation process of the template in order to ensure a clear separation of the individual aspects (e.g., separation of the perspectives between the process modeler and the reader). For example, when the originator of a process model is unavailable, it is still possible to measure and quantify the reader’s perspective. Furthermore, the separation into different sheets allows for an accessible extension of additional process modeling languages. The workbook supports the evaluation of process models expressed in terms of the following process modeling languages: Business Process Modeling and Notation (BPMN) 2.0, Event-driven Process Chains (EPCs), and UML Activity Diagrams. Since the relevant sheets are predefined with the results obtained from the survey (i.e., the weighting of the quality criteria and metrics), only the sheets (4), (5), and (6) must be completed to quantify the perspectives (i.e., process modeler, reader, and both) in the context of process model comprehension. Changes in the remaining sheets are only necessary if factors (e.g., weighting factor \(w_k\)) shall be adjusted, further quality criteria and metrics need to be introduced, or the workbook shall be extended to support additional modeling languages. Regarding the latter, in all sheets, respective information for the newly introduced modeling language must be added.

(1) Configuration Process modeling languages have various impact on process model comprehension (e.g., in terms of expressiveness) (Zimoch et al. 2017c). Thus, the operationalization thereof is performed in the workbook in (2). Moreover, with respect to comparability, all results in the workbook are normalized within an interval between [1, 10]. Thereby, a 1 represents a worse outcome, whereas a 10 indicates the best outcome regarding process model comprehension. Hence, in this sheet, the complexity coefficient \(\Vert C_i \Vert\) calculated in (2) (see Eq. (5)) is normalized to \(\bar{C} _{i}\) for each process modeling language within an interval between [1, 10] , and is defined as follows:

$$\begin{aligned} \bar{C}_i = 10 - {\frac{(10 * \Vert C_i \Vert )-\Vert C_i \Vert }{MAX(C_n) * 10}}, \end{aligned}$$
(3)

where i is the process modeling language, \(\Vert C_i \Vert\) is the complexity score for the specific process modeling language i, and \(C_{n}\) is the set of all complexity scores.

Another factor having an impact on process model comprehension is the use of workflow patterns in a process model. These patterns play a crucial role in the creation of such models and are, therefore, especially for process modelers of importance (White 2004). The impact of workflow patterns, (i.e., \(P_i\) calculated in (3), currently just for control flow patterns) is shown in this sheet and is included in the determination of the process model comprehension score for the perspective of process modelers (see (7)).

(2) Process modeling language complexity In general, a process modeling language is composed of a number of modeling elements, their characteristics (e.g., different activity types), and their relations (e.g., different flow types such as sequence or data), which define the expressiveness of respective language (List and Korherr 2006). Based on this consideration, the workbook defines the complexity of a process modeling language \(C_{i}\) as a three-dimensional vector:

$$\begin{aligned} C_i = (x_i, y_i, z_i), \end{aligned}$$
(4)

where i is the process modeling language, \(x_i\) is the number of elements of the modeling language, \(y_i\) is the number of characteristics per element, and \(z_i\) is the number of relationships per element (Laue and Gruhn 2006).

Accordingly, the number of elements, their characteristics as well as their relations reflect the complexity of a modeling language. With the Euclidean norm, \(C_i\) can be converted to the complexity score \(\Vert C_i \Vert\) for a specific process modeling language i:

$$\begin{aligned} \Vert C_i \Vert = \sqrt{x_i^2 + y_i^2 + z_i^2} \end{aligned}$$
(5)

(3) Supported patterns The expressiveness as well as suitability of a process modeling language is not only determined by the number of elements, their characteristics, and their relations (see (2)), but also by the number of supported workflow patterns (White 2004). Workflow patterns describe specific mechanisms supporting stakeholders dealing with the complexity of process models (e.g., consideration of different perspectives such as control flow and data). For this reason, (van der Aalst et al. 2003) defined a set of workflow patterns, which are considered in the workbook. In this context, the workbook supports the following workflow pattern types: control flow, data, and resource patterns. Furthermore, for each workflow pattern type, the workbook considers which workflow patterns are fully, partially, or not supported in the respective modeling language. Based on this consideration, the score for supported patterns \(P_i\) is determined as follows:

$$\begin{aligned} P_i = m_i + n_i + o_i, \end{aligned}$$
(6)

where i is the process modeling language, \(m_i\) is the number of fully and partially supported control flow patterns, \(n_i\) is the number of fully and partially supported data patterns, and \(o_i\) is the number of fully and partially supported resource patterns.

\(P_i\) (in percentage, currently just for control flows patterns) is used in (1) for the determination of the process model comprehension score pertaining to the perspective of process modelers (see (7)).

(4) Quality metrics In this sheet, for the perspective of process modelers and readers, the quality metrics from the PMCF (i.e., ETT) related to the evaluated process model are determined. In particular, for each quality metric, an explanation of the respective metric is given as well as an instruction on how to determine the corresponding metric (e.g., number of in-/outgoing edges per process modeling element). The metrics are determined either as described in respective literature or must be determined manually considering the process model to be evaluated. If the determined result has not yet been normalized, the result will be normalized in an additional step within an interval between [1, 10]. Thereby, a result towards the right boundary (i.e., 10) describes a more positive impact on the comprehension of the process model.

(5) Questionnaire for process modeler The comprehension of a process model depends not only on factors of the respective process model (e.g., size of the process model), but also on the perception of the original creator (i.e., process modeler) of the model. Thereby, a process modeler has personal related characteristics (e.g., expertise in process modeling) in the context of process modeling, as well as an individual interpretation of the information and knowledge regarding the process and its related model. Furthermore, there exists a specific mental interpretation of the process and resulting process model in the mind of the process modeler. As described in Sect. 2.1, noise may occur in the communication of process information and knowledge between the process modeler as well as readers. For this reason, it is important to capture and know both personal related characteristics and the interpretation of the process modeler (i.e., the perspective of process modelers) to identify respective noise and initiate countersteps. Therefore, the original process modeler of a corresponding model has to answer a specific questionnaire capturing personal related characteristics as well as the related interpretation of the process and its resulting process model. The questionnaire consists of a set of 49 different questions addressing quality criteria and metrics from the ETT to capture the perspective of the process modeler. The question types are a set of true-or-false and Likert scale questions. The responses are compiled to a score within the interval [1, 10], whereas ten indicates a more positive impact on process model comprehension.

(6) Questionnaire for process reader Similar to the process modeler, a specific questionnaire to capture the perspective of process readers has to be answered. This questionnaire consists of 24 questions related to corresponding quality criteria as well as metrics from the ETT in order to gather the perception and interpretation of process readers about the comprehended process model. Equally to (5), the responses reflect a score within the interval [1, 10], whereas ten constitutes the best score regarding process model comprehension.

(7) Perspective of process modeler This sheet contains a multidimensional presentation of all the ranked as well as weighted quality cr metrics \(Q_c\) of the respective criterion is calculated:

$$\begin{aligned} Q_c = \sum _{i = 1}^{n}i, \end{aligned}$$
(7)

where c is the quality criterion, i is the quality metric, and n is the number of quality metrics.

The final process model comprehension score for process modelers \(S_m\), which represents a score within the interval [1, 10] (i.e., ten is the best), is built from the sum of all aggregated quality criterion \(Q_c\):

$$\begin{aligned} S_m = \sum _{c = 1}^{6} = Q_c \end{aligned}$$
(8)

Based on \(Q_c\), possible factors for noise can be identified by considering related quality metrics with a score towards the left boundary within the interval [1, 10] (see Sect. 5).

(8) Perspective of process reader Similar to (7), the process model comprehension score for process readers \(S_r\) is determined in this sheet. Hence, all ranked as well as weighted quality criteria and corresponding metrics from the ETT are shown here. The determination of the score is carried out in the same way as described in (7), only with relevant aspects for process readers. Therefore, no changes are required in this sheet. The process model comprehension score for process readers reflects a score within the interval [1, 10] (i.e., ten is the best), based on the sum of the aggregated quality criteria for process readers. As with the process modeler, factors for noise in the comprehension of a process model can be identified on the basis of the individual calculated scores for respective quality criterion and related metrics (see Sect. 5).

(9) Summary The final sheet in the workbook presents the quantified process model comprehension scores on the evaluated process model. Here, the scores for the perspective of process modelers \(S_m\) and readers \(S_r\) as well as the interaction of both \(S_b\) are presented. The single scores are within the interval [1, 10], whereas 1 indicates the worst score regarding process model comprehension and 10 the best. Thereby, the scores for process modelers and readers are determined in the sheets (7) and (8), respectively. The score for the interaction of both perspectives \(S_b\) is determined as follows:

$$\begin{aligned} S_b = (w_m * S_m) + (w_r * S_r), \end{aligned}$$
(9)

where \(w_m\) is the weight for process modelers and \(w_r\) is the weight for process readers.

The two weights \(w_m\) and \(w_r\) were determined within the survey with the specific question asking about which aspect in a process model is considered to be more significant, i.e., ease of creation (\(w_m\)) or ensuring proper comprehensibility (\(w_r\)). Hence, the percentage distribution was calculated from the responses given.

5 Case study and application of the process model comprehension framework

In order to demonstrate the applicability of the PMCF (i.e., workbook), a case study with 33 participants from industry (i.e., 19 from automotive and 14 from business consulting) was conducted (see Appendix C). According to collected demographic data, all participants stated they had already worked with process models. Hence, we determined that the experience in process modeling and model comprehension was between a novice and intermediate level. The participants were asked to comprehend five real-world scenarios (i.e., purchasing, training, goal agreement, ordering, travel & expense) from a business consultant company. Each scenario was documented in two different process model variants (i.e., ten in total), emphasizing the following process modeling aspects: start events, end events, loops, parallelism, and decomposition. In one variant, mentioned aspects were explicitly documented in a process model, while in the other models, they were only implicitly (i.e., described in an activity) documented. In addition, for each process model, participants needed to answer a set of four true-or-false comprehension questions about the semantic aspects in the models. Regarding the PMCF, the workbook sheets (4) Quality Metrics, (5) Questionnaire for Process Modeler, and (6) Questionnaire for Process Reader were completed accordingly. Thereby, the sheet (4) was completed by considering the characteristics of the process models (e.g., the number of modeling elements). The sheet (5) was answered by the original creator of the process models. Thereby, there was only one original creator for each process model. Finally, the sheet (6) was answered by the 33 participants of the study after each comprehended process model. Table 1 presents the results from the case study. In detail, the table shows, for each process model and respective variant, the mean of the results from the comprehension questions (i.e., max is four) as well as the determined process model comprehension scores with the workbook for the process modeler, reader (i.e., average), and both (i.e., average).

Table 1 Demonstration of the applicability of the PMCF
Full size table

According to the results from the comprehension questions, the variants with explicitly documented process modeling aspects had a more positive impact on process model comprehension (i.e., higher comprehension scores). Considering both perspectives, the process model comprehension scores mainly confirm this observation (i.e., higher perspective scores). The score is slightly higher for the second implicit process model (i.e., end event) variant. Consider Table 1, there are only minor differences in the comprehension scores between the perspectives compared to the differences in the comprehension questions. A reason is that the use of questions represents a simple metric, which is susceptible to deviations (e.g., guessing or heterogeneous distribution of expertise). The PMCF, in turn, considers the performance in model comprehension (i.e., answering the questions) and a variety of quality metrics, each having a different strong impact on process model comprehension, leading to a more fine-grained result. Furthermore, which is not apparent from the consideration of the comprehension question results only, there are differences between the process modelers and readers. Regarding the process readers, the PMCF workbook results in a comprehension score of about 6. Since the comprehension score is within the interval [1, 10] (i.e., 10 is the best), it indicates that the process models are slightly above the average in terms of process model comprehension. Furthermore, it is remarkable that the original creator of the process models evaluated their own created process models as less comprehensible in the retrospect compared to respective readers. The comprehension scores for process modelers are approximately between 4 and 6. A reason could be that the process modelers, during answering the PMCF worksheet (5) Questionnaire for Process Modeler, have critically recapitulated their own process model. More specifically, single items from the PMCF worksheet (5) (e.g., knowledge about process domain, correctness of process information) may have drawn attention to possible deficits in the process model. Since process modelers and readers have different perspectives, a uniform comprehension of the process models used in the study was not given, due to occurring noise in the communication of process information and knowledge.

5.1 Application of the process model comprehension framework

The PMCF allows for the identification of reasons for difficulties in order to prevent noise (e.g., discrepancies in process domain knowledge) during the comprehension of the presented process models in order to initiate steps to improve respective models. Therefore, the workbook sheets (7) Perspective of Process Modeler and (8) Perspective of Process Reader may be considered. As described in Sect. 4, these sheets containing a multidimensional valuation of different aspects presented in the ETT (see Sect. 3.1). The latter are aggregated and used for calculating the process model comprehension scores for respective perspectives. For this purpose, the sum of all quality metrics for each quality criterion are calculated (i.e., six for process modeler and seven for process reader; see Sect. 4). Afterwards, the final comprehension score is determined from the sum of the aggregated quality criteria and compiled to a score within the interval [1, 10], whereas 10 indicates the best score regarding process model comprehension. In the optimum case, the final comprehension score is 10, which means that the quality criteria and metrics have also been aggregated to a value of 10.

In the following, an example is presented of how the PMCF worksheets’ insights may be used to foster process model comprehension. Therefore, the results regarding the third process model (i.e., explicit loop) from the process modeler and a reader are considered (see Appendix D).

Perspective of process modeler We are considering the individual scores in the multidimensional valuation we obtained from the case study. For example, we noticed from the perspective of the process modeler that the score regarding the quality criterion Information (see Sect. 3) is 5.01. Thereby, the quality criterion Information is concerned with process information retrieval and consists of the following metrics: completeness (i.e., Is the process information complete?), correctness (i.e., Is the process information correct?), availability (i.e., What availability does the process information have?), and method (i.e., Which methods are available for process information retrieval?). Regarding the two latter metrics, in our example, the score is 2.1 for availability and 1.6 for method. As a direct consequence, the original process modeler had difficulties with the availability of process information (i.e., only textual process documentations were available) and in the choice of methods for process information retrieval (i.e., only the study of the textual process information was available). Therefore, an increase in the availability of process information and methods for information retrieval would, on the one hand, lead to the creation of a better comprehensible process model because process information can be collected more effectively. On the other hand, as a result, an increase in these two metrics would positively affect the score regarding the quality criterion Information, thus leading to a rise in the final process model comprehension score for the process modeler, reader, and the interaction between both perspectives.

Perspective of process reader Considering the perspective of process readers and their individual multidimensional scores obtained from the case study. For example, the score regarding the quality criterion Person (see Sect. 3) is 5.34 in our example. In more detail, the process model readers have stated that their experience working with process models (e.g., the number of analyzed process models) is maximum at an intermediate level. Moreover, contemplating the quality criterion Representation Factors and related metrics that are concerned with structural factors of the process models (e.g., block structure), the score is 3.68. These scores indicate how to increase the final comprehension score for the perspective of the process reader. On the one hand, process readers should be more concerned with different kinds of process models to increase their experience working with such models. Further, on the other, the process models used in the study could be adjusted by respecting a consistent block structure. These steps would then positively impact the final comprehension score of process readers and the comprehension score of process modeler as well as the interaction between both.

However, the first case study was confronted with some limitations that needed to be considered. Although the process models documented real world scenarios, their generalizability is limited. More specifically, the scenarios originated from the administration field. Scenarios from other domains might have a different impact on the outcome. Additionally, the comprehension of more complex process models might have a different influence on the resulting scores (i.e., inferior). The study participants were employees in the respective company from which the process models have been derived. As a result, process model comprehension scores might have been biased due to prior knowledge of the used processes.

In summary, the conducted case study demonstrated the successful application of the PMCF in a practical environment. The results indicated that there needs to be a more uniform comprehension of process models between process modelers and readers. Moreover, the PMCF revealed that process modelers and readers are confronted with different challenges (i.e., noise) in process model comprehension. Generally, although the PMCF constitutes a pilot implementation and is still in the early stage of development, it can already be applied on real-world process models of organizations for identifying potentials for process model improvements, i.e., determining noise in the communication of process information and knowledge, thus unraveling potentials to foster the general comprehension of such models.

5.2 Limitations

Although the PMCF demonstrated that it is applicable in practical environments, we want to emphasize that the framework is still in an early stage of development. Furthermore, the PMCF constitutes a first prototypical approach towards measuring and quantifying process model comprehension. Hence, the PMCF and the developed workbook are currently confronted with limitations that need to be discussed and will be the subject of future work. First, we need to evaluate the interpretation of the calculated process model comprehension scores must be considered. In detail, the results range in the interval between [1, 10], whereas 10 indicates the best score regarding process model comprehension. The first applications of the workbook demonstrated that the calculated scores are reliable (see Sect. 5). However, the workbook must be applied on many more process models in order to be able to interpret differences in the scores accurately and to define a score threshold from which process models are well comprehensible for the general public. Second, the aggregation of the final process model comprehension score might need to be more sophisticated. Although it may serve as an indicator regarding the general comprehensibility of process models. However, this does not allow any explicit statement as to which factors in the process model are responsible for this score. Is the reason the complexity of the process model or the lack of expertise in working with such models? The single multidimensional scores from the specific quality criteria might be more appropriate. These scores allow for a more fine-grained interpretation of the comprehensibility of process models. For example, considering the perspective of a process reader with a high score in the criteria referring to the process modeling language and a low score in the criteria person. These scores indicate that the complexity of the modeling language is manageable, but, in turn, specific character traits (e.g., expertise) need to be sufficiently pronounced and might be in need for additional training. Third, the use of the DNLog to determine the weights (i.e., \(w_k\)) for the quality criteria and metrics should be further evaluated. Fourth, the normalization of the results from the quality criteria and related metrics and the meaningful applicability of the normalization approach needs to be further evaluated. Fifth, the implementation of several sheets in the workbook might need to be thought through more complexly. It is arguable whether the expressiveness (e.g., supported workflow patterns) or the complexity (e.g., number of elements) might be the sole or appropriate parameters for an initial rating of process model comprehension. A consequence of this approach is that specific modeling languages might be favored while others are penalized. Sixth, in general, the completeness, accuracy, and validity of the PMCF and the workbook need to be scrutinized in detail as the PMCF contains many quality criteria and metrics. Especially the application of a narrative literature review identified only a set of criteria and metrics. These quality aspects are derived from the process of adaption of the CMQF, a narrative literature review, and conducted expert interviews. However, myriads of quality aspects exist that currently fall outside the scope of the PMCF (Dikici et al. 2018; Borthick and Schneider 2016). These include, in particular, cognitive aspects, which, as known from recent studies in this context, exert a significant impact on the comprehension of process models (Haisjackl et al. 2018; Zimoch et al. 2018). Moreover, cognitive aspects may be the critical mediator in the communication of process information as well as knowledge between process modelers and readers.

5.3 Implications

The provided insights have implications for research as well as for practice.

For practice With this paper, we highlight the important implications of the PMCF and the ability to measure and to quantify process model comprehension for practice. Process models constitute vital artifacts in the application of information technologies (e.g., PAIS). In particular, during the utilization of information systems, undiscovered errors made (e.g., incorrect process documentation due to noise in the communication of process information and knowledge) may have critical impacts in the later utilization and, hence, projects might not deliver the required results or even fail. For this reason, it is of importance that process models are created correctly as well as accurately. At the same time, it should be ensured that these models are comprehensible for all involved stakeholders. In this context, a process model is an artifact used for the communication of process information and knowledge between participants (see Sect. 2.1). During this kind of communication, noise may occur that may impairs the comprehension of process models. Therefore, during process model comprehension, the PMCF allows for the measurement and quantification of the perspectives of process modelers, readers, and the interaction between both. This allows for the identification of noise, which could therefore be addressed ensuring a proper process model comprehension. Moreover, since the PMCF covers different quality criteria and metrics covering various aspects (e.g., process modeling tools, medium (see Sect. 3.1)) for respective perspectives, organizations are able to identify concrete deficiencies in the context of process models with the provided scores of the PMCF. With the support and extensibility of additional process modeling languages, the PMCF assists organizations in the selection of an appropriate modeling language. This is applicable when a process modeling language is selected for the first time (e.g., in the early phases of the information systems development process), or in case of a modeling language change, which is, for example, pursued due to a process model redesign.

For research With the results from this work as theoretical foundation, research may focus on the execution of additional studies in order to foster our interpretation of the determined scores for process model comprehension. Further, the findings highlight the different perspectives of a process modeler as well as reader during process model comprehension. The modeler of a process emphasizes the completeness, availability, and collection methods from information, while the reader of a model emphasizes representation factors of a process model. Research should considers this fact that a model originator refers on the semantic dimension of a process model and the reader on the pragmatic dimension. Considering expertise in working with process models, we assume that the comprehension of process model information referring on the syntactical, semantic, and pragmatic dimension depend more on individual character traits. However, expertise might constitute a catalyser regarding the comprehension of syntax, semantics, and pragmatics. Further, despite the preliminary focus on the comprehension of process models, the insights obtained with the PMCF also affect the creation of process models (see Sect. 5.4). Thus, in the creation or optimization of process models, factors for noise in the communication of process information and knowledge can be avoided paving the way for process models of high quality. Finally, the results confirm prior obtained findings that the BPMN 2.0 constitutes a well-comprehensible notation for process modeling (Zimoch et al. 2017c). Hence, it is recommended to put a focus in tertiary education on this notation in the training of future business analysts. However, other notations such es EPC should not be neglected, since these more simple notations seem to be more in favor when comprehending process models of a simple nature. A reason could be that the BPMN 2.0, juxtaposed to other notations, reflects a high notational complexity due to high number of different modeling elements.

5.4 Future work

In general, discussed limitations (see Sect. 5.2) will be addressed in future work. This includes, among others, that the used approach of definition for the PMCF allows for appropriate extensibility. More specifically, novel quality criteria or metrics can be added to the ETT. Therefore, to address the discussed limitations (see Sect. 5.2), the PMCF is currently used in ongoing studies that evaluate different process models from theory as well as practice with heterogeneous participant groups. The objective is, on the one hand, to improve our general interpretation of the calculated process model comprehension scores and, on the other hand, to identify additional noise factors in the communication of process knowledge and information between the process modeler as well as reader. The unraveled insights allow for the definition of directives towards creating better comprehensible process models with high quality. In this context, the results for the different perspectives obtained from the PMCF are juxtaposed with existing rules and guidelines (e.g., Guidelines of Modeling (Becker et al. 2000), Seven Process Modeling Guidelines (Mendling et al. 2010), which are intended to ensure a proper comprehension of process models, in order to evaluate their contribution regarding process model comprehension. Moreover, the weighting factor \(w_k\) is examined in detail and will be adjusted as well as refined accordingly when, for example, new quality criteria and metrics are added. In addition, other approaches, in addition to the already evaluated one (see Sect. 3) to determine the weighting factor \(w_k\) are juxtaposed to the DNLog in order to evaluate their appropriateness. Furthermore, the PMCF will be extended and enriched with further quality criteria and metrics to obtain more fine-grained scores. In this context, we are currently augmenting the PMCF with additional criteria and metrics to include the creation of process models (i.e., the process of process modeling) (Burattin et al. 2019). This augmentation should ensure that process models are created in a high quality and in a comprehensible form from the very beginning and, thus, should prevent the occurrence of noise in the communication of process knowledge as well as information. Support for additional process modeling languages and workflow patterns are subject of future work. Finally, to pave the way for cognitive aspects, the PMCF will be integrated into the conceptual framework of the authors that incorporates concepts from cognitive neuroscience and psychology introduced in Zimoch et al. (2017b).

6 Summary and conclusion

This paper presented the Process Model Comprehension Framework (PMCF) as a first step toward measuring and quantifying the comprehensibility of process models. Based on the Communication Theory and the CMQF, the PMCF considers the perspectives of process modelers and readers and the interaction between them as the main determinants in process model comprehension. Therefore, in order to identify and prevent noise (i.e., misinterpretation in the communication of process information and knowledge due to person- and model-related characteristics) in model comprehension, an ETT was defined composed of a set of quality criteria and 96 metrics in total. A narrative literature review and interviews with different experts in the field of Business Process Management were conducted to identify the used quality criteria and metrics. Further, these quality aspects are ranked and weighted concerning their importance and impact on process model comprehension in a survey with 131 participants from academia and industry. The ETT and the results from the survey have been implemented in an Excel workbook, which allows us to measure and quantify process model comprehension on existing process models. The application of the workbook and how to improve process models based on the results obtained was demonstrated successfully in a case study with 33 participants from the industry. Accordingly, the PMCF and its corresponding workbook shall contribute in identifying and avoiding pitfalls (i.e., noise) in the communication of process knowledge and information between process modelers and readers. As a result, the PMCF allows for the measurement and quantification of the process of process model comprehension. The insights obtained from the PMCF shall ensure that all stakeholders can properly comprehend process models. In addition, the PMCF wants to ensure those process models implemented in information systems are of high quality for proper model comprehensibility. Therefore, the calculated process model comprehension scores with the PMCF and related workbook serve as a signpost in order to foster and ensure a correct comprehension of process models. Further, the initial creation of comprehensible process models or the optimization of existing models is supported by the PMCF. In general, the PMCF and the future work thereof shall assist organizations in all phases (i.e., design, implementation, and management) in the utilization of information systems. As the initial work, the PMCF faces several challenges (e.g., incompleteness) that limit its application. Therefore, future work will address these limitations to provide a practical framework that organizations can use to foster communication, collaboration, and information as well as knowledge conveyance in terms of process models.