Requirements for Educational Objectives and Educational Opportunities of Didactic Digital Media Products

The pedagogical-content evaluation deals with the fundamental categories of education, with goals, content, and competencies, and analyzes the educational objectives and educational opportunities of didactic digital media products. Setting and realizing goals and sub-goals are fundamental prerequisites and guidelines for successful learning. Therefore, determining the knowledge, values, and competencies that should be acquired on different learning paths in connection with the target audience is a fundamental aspect of a pedagogical-content evaluation of didactic digital media products. closely related is the question of which content or materials, such as facts, rules, concepts, laws, methods, relations, are to be practiced, learned, and acquired. The evaluation for assessing the educational objectives and educational opportunities of didactic digital media products is thus a comprehensive approach that constitutes the evaluation area. The pedagogical-content evaluation takes precedence in the overall evaluation and deals with the following quality criteria.

Quality criteria of this evaluation area

1. Learning objective
2. Learning content
3. Target audience
4. Innovation
5. Action competencies
6. Values

Quality criteria and examination aspects

1. Learning objective

• In the didactic digital media product, the learning objectives for the user are identifiable and are implemented in achievable, logically structured, and didactically appropriate sub-goals and work steps.
• The learning objectives must be oriented towards acquiring qualified action competencies and align with the respective educational paths.
• All goal and content components (cognitive, affective, psychomotor, socio-communicative) are aligned with the overall concept.

2. Learning content

• The learning content enables the achievement of the intended learning objectives of the didactic digital media product.
• The subject matter is presented in a factual and scientifically accurate manner (structure, selection, quantity and density, as well as the connection of information, essential statements with reference to the degree of generality and abstraction level).
• The selection and communication of the learning content are appropriate from a pedagogical perspective.
• The learning content is aligned with corresponding educational programs.
• Terms and terminology are used consistently, appropriately, and logically correctly.

3. Target audience

• Learning content and learning objectives are tailored to the target audience.
• Learning content and learning objectives can be chosen by the learners and align with their prerequisites and interests.
• Necessary prior knowledge and skills of the target audience are taken into account (knowledge and abilities, emotions and attitudes, attention and concentration capabilities, socio-cultural environment).
• Opportunities for individual and cooperative learning are designed to be audience-appropriate.

4. Innovation

• The theme or the way it is implemented in the didactic digital media product is novel and progressive.
• Learning content and learning objectives correspond to the current state of research, development, and professional discussion.
• There are pedagogical advantages of the media product compared to other implementation forms.
• The content focus is primarily on a specific subject area or topic or is interdisciplinary in nature.
• The product can be characterized as a well-executed didactic digital media product or digital interactive educational medium, teaching aid, learning resource, work tool, or edutainment or infotainment program.

5. Action competencies

• Working with the didactic digital media product promotes independent, critical, multiperspective, and flexible thinking and action in social, ethical, and cultural contexts.
• The media product enables independent decision-making to handle the given task.
• The media product incorporates opportunities for creative design and interactivity.

6. Values

• Working with the didactic digital media product promotes humane thoughts and values.
• The targeted values and norms promote solidarity behavior.
• The targeted values and norms are free from glorification of violence, radical or obscene representations, ideological influence, negative prejudices, and deliberate manipulation.
• The content is free from narrow gender-specific role thinking and prejudices.
• The digital media product promotes ethical education.

Requirements for Learning Arrangements and Learning Opportunities of Didactic Digital Media Products

The didactic-methodical evaluation deals with essential aspects of teaching and learning, analyzing which learning arrangements and opportunities are pursued with the didactic digital media product. Didactics, as a scientific discipline of pedagogy, concerns itself with the rules of learning and the connections between learning and teaching. Under a didactic-methodical question, the method and manner of conveying and acquiring knowledge and competencies are examined. The didactic-methodical evaluation of didactic digital media products, therefore, constitutes a second essential evaluation area and structures the responses and criteria to the question of which learning arrangements and opportunities are pursued with the didactic digital media product. The following quality criteria can be assigned to this evaluation area.

Quality criteria for this evaluation area

1. Didactic principles
2. Didactic rules and procedures
3. Conveyance and learning forms
4. Didactic steps
5. Learning control
6. Interaction structures

Quality criteria and examination aspects

1. Didactic principles

• The didactic digital media product is based on a recognizable learning theoretical approach, such as an objectivist, constructivist, traditionalist, science-oriented, or action-oriented approach.
• The learning theoretical approach is implemented appropriately.
• From didactic perspectives, the educational content is meaningfully selected and justified.
• In accordance with the educational objective, a suitable didactic reduction may have been carried out if necessary.

2. Didactic rules and procedures

• Fundamental didactic rules and procedures are recognizable in the digital didactic media product and have been adhered to, such as:
  – Clarity
  – Scientific validity
  – Logical consistency
  – Illustrative quality
  – From general to specific
  – From simple to complex
  – From easy to difficult
  – From near to far
  – From familiar to unfamiliar
  – Connecting the concrete with the abstract.
• Logical learning processes such as analyzing, synthesizing, comparing, differentiating, generalizing, abstracting, generalizing, ordering, and concretizing are incorporated into the media product and are promoted.

3. Conveyance and learning forms

• Methodical basic forms of instruction (presentation, assignment, elaboration) are applied in the digital didactic media product.
• Possible and meaningful forms of cooperative instruction, such as frontal instruction, peer learning, group learning, or individual learning, have been considered.
• The media product allows for individual and cooperative learning in terms of learning forms. Individual learning can be combined with cooperative learning. Individual learning is meaningfully connected with entertaining forms or games.
• The media product is primarily suitable for one or more application areas, such as:
  – Individual users or for use in groups, including online groups
  – Afternoon activities / project-based learning / subject-based learning / independent work / substitute lessons / individual work.
  – Training, further education, lifelong learning.
• The application areas are recognizable and feasible.

4. Didactic steps

• Essential didactic steps that enable an optimal learning process are consistently applied in the digital didactic media product:
  – Introduction (goal setting and orientation, motivation, reactivation)
  – Working on new material / initial instruction / introduction
  – Reinforcement (memorization, repetition, practice)
  – Systematization, application
  – Control, evaluation.
• The didactic steps can effectively and appropriately achieve the educational objectives.
• The planned didactic steps enable users to work at different difficulty levels and paces.

5. Learning control

• The control of the learning process in the digital didactic media product is clear and self-explanatory.
• The execution of learning steps is emotionally stimulating and motivating in the digital didactic media product.
• Task formulations, response formats, and other learning activities are designed in a factually correct and meaningful manner corresponding to the objective, and are presented in a combination of text and images that is clear and understandable.
• The task processing is variable and not limited to mechanical execution. The learning path can be self-determined. The response format is flexible and can be optionally supported by acoustic or graphic cues and corrections.
• The exercises and repetitions are diverse and carried out in a variable manner.
• Branching occurs based on didactic considerations and corresponds to the requirements of the target audience.
• Games and other entertainment elements are in recognizable connection with the educational concept. The media product enables diverse learning and is not limited to entertainment alone.

6. Interaction structures

• The didactic digital media product enables interactive work, modification of task formulations, and flexible adaptation according to different learning needs and prerequisites. Feedback is provided in variable forms, motivating and effective.
• The media product responds to the learning process by analyzing the individual performance level and recommending appropriate branching.
• Branching paths are automatically taken based on response and learning process analysis and can be freely chosen. There is an appropriate and manageable number of branching paths. Branching provides variously difficult and varied task forms.
• Interactivity between the user and the media product is facilitated by presenting tasks and assignments, demanding solutions, and promoting the development of solution strategies.
• Interactivity is supported by:
  – The progression of the program depending on the user’s contributions and activities.
  – Triggering user activities, such as collecting data or expanding information.
  – Providing data for further processing.
  – Error messages with substance.
  – Objective and variable confirmation of work results.
  – Implementation of links to other media or through reward systems (leaderboard, games, etc.).
• The user’s performance level and learning progress are determined during the exercise and communicated appropriately, motivatively, and encouragingly. The evaluation of performance results is professionally and didactically correct and meaningful.
• The performance evaluations in the media product are professionally and pedagogically meaningful. The determination of performance results is statistically correct.
• Suitable options (such as text, sound, graphics, animation) are provided for performance evaluations. Incorrect solutions are indicated in different and variable ways. Feedback on incorrect solutions is provided in a motivating manner and assesses the answer, not the person.

Requirements for the Design and Layout of Didactic Digital Media Products

the medial requirements for assessing the design and layout of didactic digital media products focus on the question of how well the transformation of an idea into an aesthetically and functionally sophisticated result has been achieved. This involves evaluating the form-appropriate and functional design. The medial requirements for assessing design and layout are closely related to media pedagogical and media didactic questions but constitute an independent third group of criteria. Design and layout of didactic digital media products can have a significant impact on important learning skills, such as perceptual ability, imagination, constructive-productive thinking, sensitive grasp of aesthetic values, and restructuring ability. The use of various medial elements for the medial presentation of learning content must be considered as a whole, and the individual elements should be reviewed in terms of their function and interaction with other forms (cf. Zimmer, G.: E-Learning, BW Bildung und Wissen 2004, p. 103). The following quality criteria are applied to the medial-design evaluation.

Quality criteria for this evaluation area

1. Content-appropriate design
2. Audience-oriented design
3. Screen design
4. Visual design
5. Auditory design
6. Linguistic design

Quality criteria and examination aspects

1. Content-appropriate design

• The design (colors, typography, non-textual elements, etc.) of the didactic digital media product was content-appropriate (e.g., drawings for children, “cool” colors for winter, etc.).
• The type of media (video, images, text, etc.) was chosen purposefully according to the content (videos for movement sequences, audio recordings for music and language).
• The learning content is aligned with the capabilities of the media type (media and graphic design).
• The appropriate multisymbolic representation form was chosen for the content. The multisymbolic representation forms (texts, graphics, images, videos, audios, etc.) are correct and meet aesthetic considerations.

2. Audience-oriented design

• The didactic digital media product has been designed in a user-oriented manner.
• Different user groups are assigned various graphic and media design concepts.
• The design is adapted to the needs of the user (font size, contrasts, subtitles, etc.).
• Presentation forms of content such as language, sound, image, and animation are tailored to the target audience. Users can structure content themselves (emphasize, skip, etc.) and add to it.
• Accessibility was taken into account in the design.

3. Screen design

• The user interface of the didactic digital media product is clear, organized, accurate, and understandable. The amount of information per screen is appropriate for the target audience. The screen layout has an appropriate level of detail.
• The technical quality of the screen pages is characterized by clear resolution, uniform brightness, and good contrasts.
• Text and image elements on the screen pages are in a functional and aesthetic relationship.
• Each screen page is self-contained with a coherent content structure.
• The viewing time and processing time for a screen page can be freely chosen.
• The screen design appeals to the user both rationally and emotionally. It is a cohesive unit in both detail and entirety.

4. Visual design

• The text design of the didactic digital media product is clear, easily recognizable, and readable. Text representations and links are closely tied to educational content in functional relationships. Text presentations are well-structured, highlighting essential information.
• Graphics, images, symbols, and colors are understandable, meaningful, aesthetically pleasing, motivating, and produced with quality. They are closely linked to educational objectives in a functional context. They are characterized by clear lines, shapes, contrasts, and clarity.
• Through the thoughtful use of visual elements such as color design, learning content is emphasized, learning processes are facilitated, and the target audience is motivated.
• Animations and videos are understandable, meaningful, and motivating. The used animation and video sequences are necessary for the presentation and understanding of the learning content or provide sustained support. Animations and videos have a level suitable for the target audience and motivate the recipients.

5. Auditory design

• The acoustic elements of the didactic digital media product, such as speech, music, sounds, noises, etc., are meaningful, understandable, and motivating.
• The auditory design supports the acquisition of learning content and interaction.
• Auditory elements have impeccable quality and are used appropriately. Sound and volume can be adjusted and have a motivating effect on the target audience.

6. Linguistic design

• The language, both in its spoken and written form, is norm-compliant and correctly implemented in the didactic digital media product.
• The linguistic expressions and the style of the language are appropriate and motivating.
• The textual expression (spelling, grammar, and punctuation) is error-free.
• Texts are clearly structured and emphasize important information.

Requirements for Operation and Usability of Didactic Digital Media Products

The operational and organizational requirements deal with fundamental organizational aspects when using didactic digital media products. These are issues related to the humane design of user interfaces or human-computer interfaces. Operation and usability are of essential importance for the effectiveness of didactic digital media products and are therefore summarized in a fourth group of criteria. In the evaluation of operation, especially organizational aspects such as operating characteristics, handling considerations, usage characteristics, organization, and technology are crucial. The evaluation focuses on the working conditions for educators and learners with the computer or with the didactic digital media product. The organizational-technical evaluation includes the following quality criteria.

Quality criteria for this evaluation area

1. Self-explanation and Reliability
2. Clarity and Flexibility
3. Navigation and Control
4. Adaptability
5. Technical Functionality
6. Product Information and Help

Quality criteria and examination aspects

1. Self-explanation and Reliability

• The didactic digital media product largely explains itself through concrete visual and auditory aids.
• The media product operates reliably, without errors, quickly, and without interruptions.
• All specified functions work seamlessly, especially loading, saving, printing, and exiting.
• The media product exhibits a high degree of resistance to user errors. Operating mistakes are corrected through visual or auditory cues. Operating errors are largely ignored.

2. Clarity and Flexibility

• The didactic digital media product is designed to be clear, straightforward, and easy to use.
• The content menu is clearly and logically structured.
• The control elements are consistently and uniformly used. The user can always discern in which section they are located.
• User instructions and explanations for beginners and novices can be canceled and skipped at any time.
• The media product allows for diverse choices and forms of application.
• The command scope, terms, and symbols are clear, manageable, and suitable for the target audience.
• The user can customize the media product according to their preferences and interests regarding content, difficulty, and assistance. There are a sufficient number of selectable areas, and they are easily accessible.

3. Navigation and Control

• The control options of the didactic digital media product are characterized by changing input forms, facilitating input, options for control (e.g., keyboard, mouse), availability of all control elements, access to additional information, and communication possibilities over networks.
• The learning control options are flexible, such as influencing the speed of processes, selecting and sequencing steps, managing task complexity and difficulty, and determining the learning time.
• The navigation and orientation options are easy to handle and clear.
• The user can effortlessly switch between different views using familiar or recognizable control symbols.

4. Adaptability

• The didactic digital media product allows for adaptation to the user’s capabilities through changes in the default settings (e.g., turning off sound, switching between text and audio output) and adjusting the difficulty level (e.g., tasks with various difficulty levels).
• Adjusting the time behavior (e.g., setting reaction times according to the user’s needs) is ensured by the media product.
• The media product enables adaptation to the user’s capabilities by the type and extent of information (e.g., separate and combined selection of text or audio information).
• Adaptation of the help system (e.g., a variable offering of assistance) is facilitated by the media product.

5. Technical Functionality

• Installation and uninstallation or access and activation for the didactic digital media product occur without problems.
• The media product is compatible with various hardware and software.
• Mouse and keyboard usage are comfortable, meaningful, and self-explanatory.
• Saving and printing all important results is simple, comfortable, and done in a form suitable for the respective target audience.

6. Product Information and Help

• The necessary information for using the didactic digital media product is user-friendly for the intended users such as children, teenagers, apprentices, adults, parents, and teachers.
• Product description and user manual encompass all information required for understanding and utilizing the media product. They are clear, unambiguous, logically structured, and easy to follow.
• Essential information for initializing or installing the media product is accurately and correctly provided.
• Required hardware information and necessary system requirements are presented in a factual and user-friendly manner.
• The media product has a well-functioning support and advisory service (hotline, email, tutorial, etc.).

Firstly, the value of what DMBs/AMPs aim to achieve, i.e., informing and providing information, is examined.
(1) Information must first be verified to relate to existing phenomena (in any form of things, facts, topics, etc.) and not be based on speculation (commonly referred to as Fake News). This applies to all presented information, not just the product theme.
(2) Central information (or the overall product theme) should be presented impartially, not one-sided, ideally evident from the product’s structure.
(3) Presented information (or the overall product theme) should be up-to-date or provide sources for users to track the development of the topic (referenced by product producers during information gathering).
(4) The information should be better researched and more detailed than what is typically encountered daily in mass media. This criterion depends on the target audience and their potential to absorb (specialized) details.
(5) The product collects and presents information that is not “everywhere.” It offers new information or presents known information in a new context, with a new focus, etc. It presents less-known information and does not repeatedly convey information already presented in (other) mass media.
(6) Finally, an assessment is made of how presented information in all the above criteria relates to other media and products, or how it fits into the discourse on the product theme. The following quality criteria can be assigned to the Evaluation Area of Information Value.

Quality criteria for this evaluation area

1. World Relevance
2. Holistic Presentation
3. Currency
4. Expertise
5. Originality
6. Context

Quality criteria and examination aspects

1. World  Relevance

• Information deals with existing phenomena (in any form such as things, facts, topics, etc.), and it is not speculative (speculations are communicated as such in the information).

2. Holistic Presentation

• Information is presented holistically and impartially, not one-sided.
• The product has a structure in which the holistic approach is evident (thoughtfully chosen chapters/menu entries and completed units).

3. Currency

• The theme and information are updated and presented in a current manner.

4. Expertise

• The producer/submitter is considered an expert in the product’s thematic field.
• Sources are explicitly or implicitly provided in areas outside the producer’s expertise.

5. Originality

• Presented information is new or presented in a new context, detail, etc.

6. Context

• Content is generally acknowledged as accurate and aligns with recognized knowledge or accepted argumentation methods.
• Contents can be connected to existing knowledge/information (on the internet) (associations).
• It is indicated or implied where and how further information on the product theme can be found, including contact with producers.

In this section, product characteristics are examined based on which the product can also be used in education.
(1) The primary pedagogical property is examined, easily verifiable. Besides the most crucial aspect, content (thoroughly examined in the first part of this criteria catalog), it is checked if the goal is disclosed or easily recognizable, specifically for informing, enlightening, etc., and not for sales, advertising, etc. It is also assessed whether the product has been adapted to a target audience or multiple target audiences.
(2) In addition to the pedagogical aspect, the didactic potential is also examined. This includes how information follows each other (inductive, deductive), whether information is clarified through examples, whether motivation is provided, and whether information encourages further exploration.
(3) In times of automation and artificial intelligence, didactics that promote critical thinking and creativity are particularly important and valued.
(4) From an educational perspective, it is always important to measure the efficiency of learning. There are ways in which the learner can demonstrate what they have learned, mostly in the form of tests or quizzes. However, there can also be “open formats,” such as submitting an essay, a drawing, etc. The following quality criteria can be assigned to this evaluation area:

Quality criteria for this evaluation area

1. Recognizable pedagogical potential
2. Recognizable didactic potential
3. Critical thinking and creativity
4. Pedagogical efficiency (tests)

Quality criteria and examination aspects

1. Recognizable pedagogical potential

• The goal of the product is disclosed or evident, and it is to disseminate knowledge (rather than encouraging purchases of the depicted or providing entertainment, etc.).
• The product specifies the target audience and provides information on how to best utilize the product for learning.
• It encourages the educational use of the product, i.e., for learning – motivation being a significant factor in learning.

2. Recognizable didactic potential

• Didactic methods (analysis, synthesis, comparison, ascending/descending quantity and quality of complexity, etc.) are recognizable or built into the product.
• The application of didactic activities such as recognition, completion, connection, etc., is encouraged.
• To the extent that didactic activities are present, users are motivated to use them (through instructions, interactivity, etc.).

3. Critical thinking and creativity

• The product stimulates or promotes independent thinking (and working).
• The product allows interventions, and these stimulate the user’s own considerations and decisions.

4. Pedagogical efficiency (tests)

• The user/learner can assess their (newly acquired) knowledge (tests).
• The user receives feedback on the tests/knowledge assessments conducted.
• Motivation for applying the acquired knowledge is provided, and the product offers suggestions (in a simple form of further connections or through specific instructions on where, when, and how to apply the knowledge).

Every product is currently designed to be as user-friendly as possible, making it easy for the user to achieve the intended goal with the product. In this context, these products, DMBs (AMPs), pose a significant challenge as their primary goal is to inform, but they are also assessed for their suitability for teaching or learning. In this evaluation area, general criteria for user experience are combined with specific criteria for educational media. The following quality criteria can be assigned to this evaluation area:

Quality criteria for this evaluation area

1. Intuitive use and navigation
2. Visualization and expressive means
3. Personalization and customization
4. Scope – Detail
5. Interaction and communication

Quality criteria and examination aspects

1. Intuitive use and navigation

• It is apparent that the type and quantity of intended user actions are tailored to a more or less defined target audience or can be adjusted.
• It is always clear what is expected from the user and what actions they need to take (except for tasks where they are supposed to figure it out themselves).
• Page layout and controls are consistent and simple, allowing the user to focus on content and tasks without disruption from complicated, multi-step processes.
• The function of each icon is intuitively understandable for the user and does not require repeated explanations (e.g., when interacting with the mouse).
• The product always indicates the user’s position within the product.

2. Visualization and expressive means

• The product design aligns with the product theme – for example, themes like art, IT, social work are reflected in the design.
• It is evident that the graphic design is adapted to a more or less defined target audience.
• Information is structured through visualization, emphasizing its parts, and facilitating navigation (e.g., colors indicating similar levels/themes, function buttons consistently placed).

3. Personalization and customization

• Users can adjust settings to suit their needs (e.g., turn sound on/off, change font size and contrast, choose between text and audio playback).

4. Scope – Detail

• The scope and level of detail of the product are adapted to goals, target audiences, and presentation methods.
• There is a consistent relationship between scope and details – the product as a whole adheres to this relationship, not significantly delving deeper into some parts/chapters than others.

5. Interaction and communication

• The product includes interactive elements, and users receive feedback for their interaction with the product.
• It is evident that communication is considered important. Users are motivated to use communication, and the product provides tools for this whenever possible.

Technical quality could be summarized as follows: The product functions smoothly and in good quality on all devices. In this evaluation area, the potential innovative character of the product and marketing have also been considered. The following quality criteria constitute this evaluation area:

Quality criteria for this evaluation area

1. Adaptive and Responsive
2. Media Variety and Quality
3. Innovation
4. Marketing

Quality criteria and examination aspects

1. Adaptive and Responsive

• The product can be used without restrictions on all modern operating systems and/or browsers.
• The product is adapted for various devices, their screens, cameras, speakers, and microphones (if needed).

2. Media Variety and Quality

• Various types of media (text, image, video, audio) are utilized depending on the content and, most importantly, the target audience.
• The employed media are of high quality and maintain this quality in various presentation situations (screens of different resolutions and sizes, projections, with/without external speakers, headphones, etc.).
• Users are warned (and given instructions) if the quality of the used/played media does not meet the intended standards.
• Language is in line with goals and target audiences and is implemented at a high-quality level.

3. Innovation

• The product, as a whole, introduces something new.
• The product includes new and/or forward-thinking elements that motivate the user.
• The product includes new and/or forward-thinking elements that promote its didactic use.

4. Marketing

• In the event that the product or its producer has already received a Comenius EduMedia award, this has been properly publicized (alongside the awarded product).
• The product is promoted with arguments contributing to the quality of education (not with arguments expressing other qualities, e.g., simplicity, speed, fun, etc.).

In the first evaluation area, the focus is on examining the possibilities available to the teacher/author to guide and shape the learning process of the learner (the recipient of these possibilities/work). Quality criteria concentrate more on the content than on the didactics, as supporting the teacher in content creation is much easier to assess than supporting in didactics, i.e., in the implementation/teaching of the prepared content.

Quality criteria for this evaluation area

1. Quality of content
2. Content creation
3. Variety of supported media types
4. Integrated/connected media libraries
5. Didactic specifications and possibilities
6. User-friendly design and instructions for authors

Quality criteria and examination aspects

1. Quality of content

• In Learning Management Systems (LMS), content can be published by individual authors (e.g., teachers) as well as by professional productions.
• Before release, the LMS owner/tenant reviews content for its quality (content, didactics, technical aspects).
• External authorship is clearly indicated (with proper source attribution).

2. Content creation

• Content can be created directly within the LMS.
• LMS-native content creation tools adhere to the standards of common content creation tools (e.g., Word, PowerPoint, photo and video editing).
• Integration of pre-produced content is straightforward and can be done by the teacher.

3. Variety of supported media types

• During content creation, various media types (text, image, audio, video) can be integrated.
• For learners (and instructors), the type (mentioned above) and size (in bytes, duration, etc.) of the media are easily recognizable.

4. Integrated/connected media libraries

• Authors have access to one or more integrated or linked media libraries for incorporating into their own content/courses.
• Content from the media library can be easily integrated into the course and is automatically clearly labeled (source, media type, size).
• Content from the media library can be edited, preferably directly within the LMS/media library.

5. Didactic specifications and possibilities

• The LMS includes built-in didactic methods (e.g., learning materials, discussions, quizzes) that authors can choose from.
• The LMS is limited to certain didactic methods that are integrated into all content/lessons/courses (e.g., intermediate and final tests).

6. User-friendly design and instructions for authors

• Authors can easily identify their location within the LMS structure (overview).
• LMS-native icons adhere to common icons for the same elements (e.g., for web page/link, video, etc.).

The second evaluation area focuses on the learner’s perspective, addressing the opportunities available to them for effective, efficient, and enjoyable learning. Adaptation and user experience are crucial for users in all roles within an LMS (teachers, learners, administration), but they are particularly vital in the learner’s role. This is because learners constitute the largest user group, and they need to be motivated to use the system since they are not compensated like other roles.

Quality criteria for this evaluation area

1. Motivating user experience
2. Organization of learning
3. Learning progress
4. Tasks and assessments
5. Feedback
6. Personalization

Quality criteria and examination aspects

1. Motivating user experience

• It is evident that graphic design can be adapted to a more or less defined target audience.
• It is evident that the type and quantity of intended user actions can be adapted to a more or less defined target audience.
  angepasst ist oder werden kann.
• Page layout and controls are consistent and simple, allowing the learner to focus on content and tasks without disruption from complex, multi-step processes.
• The function of individual icons is intuitively understandable for learners and does not always require repeated explanations (e.g., when interacting with the mouse).

2. Organization of learning

• The LMS displays the navigation bar (depending on the target age group) or uses other signals/symbols (colors, icons, etc.) to indicate where the learner is in the LMS or what they are currently doing.
• Materials created or submitted (tasks) are stored in the learner’s profile and can be sorted based on multiple properties (filters).

3. Learning progress

• Learners can track their own progress, including information about tasks they have already completed (to what extent and how successfully) and those that are still pending (along with estimated time requirements).
• The LMS includes built-in tools through which teachers (or the LMS automatically) can alert learners to their progress or any deficiencies.
• Learning pace (deadlines) and the option to revisit (individual) sections can be configured for all, groups, or individual learners.

4. Tasks and assessments

• The LMS includes built-in tools that enable learners to fulfill assigned tasks or create content within the LMS.
• Submitting tasks in the LMS is straightforward and requires few steps.
• It is planned for learners to conclude content/units with (self)tests (in the form of quizzes or similar), for example, in the listing of didactic methods where tests (quizzes) are mentioned at the end.
• (Self)tests can be repeated, depending on the target audience or learning purpose.

5. Feedback

• The LMS includes built-in tools that allow the teacher or the LMS to provide automatic feedback/assessment.
• The LMS incorporates input fields where the teacher can explain how feedback is given and how it should be interpreted, or the LMS automatically provides this information during the evaluation.

6. Personalization

• Learners can customize the LMS/interface (settings such as color, contrast, turning off unnecessary features, notifications, etc.).
• The LMS enables learners to create profiles/portfolios with various media formats, such as text, images, videos, audio files, etc.
• Learners can connect their LMS profile with their profiles on other platforms (e.g., Instagram, Facebook, Twitter, etc.) or provide this information in the designated place within the profile form.

Initially, LMS were defined as the link between the teacher and the learner. In this context, this “binding function” of the LMS is examined, but in an even more expanded form, encompassing not only users but also interactions with the content.

Quality criteria for this evaluation area

1. Interaction with learners
2. Interaction with content
3. Tools for communication
4. Tools for collaboration
5. Structure, filters, and search
6. Guidelines

Quality criteria and examination aspects

1. Interaction with learners

• Learners are treated as individuals (registration with names, individual enrollment in the course is possible).
• Learners can be contacted individually by the teacher.
• Content/courses can be individually assigned to learners.

2. Interaction with content

• The instructor can still (easily) modify content (learning material) after its publication.
• Learners can still (easily) modify their assignments after submission (until the deadline).
• Content/assignments can be (easily) assigned in the LMS (by the teacher to learners) or opened for discussion (initiated by learners – among themselves or directed to the teacher).

3. Tools for communication

• The LMS has built-in tools for asynchronous communication between the teacher and learners, such as forums, group emails, etc.
• The LMS has built-in tools for synchronous communication between the teacher and learners, as well as among learners, for example, chat.
• In communication, teachers and learners have equal rights (e.g., learners can create forums and topics within them, upon request they have access to group email, etc.).

4. Tools for collaboration

• The LMS has built-in tools for asynchronous collaborative learning/working, such as shared folders, the ability to view assignments submitted by other learners, etc.
• The LMS has built-in tools for synchronous collaborative learning/working, such as video meetings, a shared whiteboard, documents, etc.
• The LMS has built-in tools that allow learners to organize group work, such as shared calendars, management tools (e.g., like Trello).

5. Structure, filters, and search

• The LMS is user-friendly with clear categories (menu elements) that are hierarchically structured and presented in an understandable manner.
• For multiple entries (e.g., courses, content, learners, etc.), users can filter and arrange them.
• The LMS has internal search function(s) that users can utilize and customize on different levels (e.g., platform-wide, my courses, my content, etc.).
• The above functions are accessible to all levels of LMS users (administrator, teacher, learner).

6. Guidelines

• Instructions for (all or the most important) functions are accessible to teachers, learners, and other users at any time.
• Instructions are logically divided based on functions, amount of information, scope, and duration.
• Instructions are adapted to the presentation format and complexity suitable for the target audience.
• Instructions are linked to the specific location in the LMS where they are needed, so users do not need to independently search for instructions.

The last evaluation category encompasses several perspectives, with technical and innovative aspects taking precedence over marketing. High-level technical implementation has become the standard today, so the focus is on verifying that “everything works.” There are then two criteria that ensure the LMS functions on a wide variety of different devices, operating systems, or browsers (Adaptive and Responsive) and that it provides data that helps the teacher optimize their teaching (Analytical Data). Finally, there are two criteria for innovation, allowing (very) innovative products to perform better in the overall assessment/points.

Quality criteria for this evaluation area

1. Flawless Functionality
2. Adaptive and Responsive
3. Analytical Data
4. Marketing
5. Major Innovation
6. Minor Innovations

Quality criteria and examination aspects

1. Flawless Functionality

• All intended/stated and built-in functions in the product are quickly accessible.
• In the case of technical issues, users can easily and quickly find help instructions that guide them in resolving the problem themselves.

2. Adaptive and Responsive

• The LMS can be used without restrictions on all modern operating systems and/or browsers.
• The LMS is adapted to various devices, including their screens, cameras, speakers, and microphones.

3. Analytical Data

• During registration, users are made aware of the terms and conditions as well as data collection practices (compliant with GDPR).
• In accordance with GDPR, the LMS collects data on the behavior of platform users.
• Teachers have access to the aforementioned data of learners enrolled in their courses and access to the usage of their content.
• The above data is automatically prepared in various formats that are useful for users, especially teachers.

4. Marketing

• In case the product or its producer has received a Comenius EduMedia award, it has been properly publicized alongside the awarded product.
• Promotional efforts for the LMS focus on arguments contributing to the quality of education (not arguments easily interchangeable with products in other sectors, e.g., entertainment).

5. Major Innovation

• The product, as a whole, is an outstanding innovation in the field of LMS.

6. Minor Innovations

• The LMS incorporates innovative elements that have a motivating effect on users.

In the ongoing refinement of the criteria for this educational media category/type, it is being considered whether the designation should be updated and modified:
– Replace “Computer games with competency-enhancing potentials” (Game-Based Learning with Competency-Enhancing Potentials) (GBLEM (CKP)) with “Digital game-based educational media with competency-enhancing potentials” (Digital Game-Based Educational Media with Competency-Enhancing Potentials) (DGBEM). This version incorporates both terminologies.

Introductory Notes on Redesign

A The difference from today’s SDB (CKP) evaluation model

(1) In this proposal, the conceptual delimitation of SDB (CKP) is further defined compared to before, to clarify that the category includes educational games, games, digital as well as playful educational media, and is focused on education. Thus, they are distinguished from:
(a) Entertainment games where one can learn something incidentally, and
(b) Educational media that, with a different priority in educational focus, also incorporate playful elements.”

(2) Based on practical experiences with the appropriate evaluation of current games and the literature in the field of ‘Serious Game-based Learning’ evaluation, the SDB (CKP) evaluation has been modified to achieve a balanced assessment of didactic applications of game mechanics (game methods and material/technical implementation) and increased educational support.

B Considerations for the redesign of evaluation criteria.

For ‘educational games (playful educational media) with competence-promoting potentials’:

(1) The goal of adapting the SDB (CKP) criteria is to enable jury members to assess the quality of Comenius submissions based on current pedagogical knowledge of learning heuristics, educational quality, and the functional use of ‘playful methods.’
(2) The defined set of ‘new’ criteria must be capable of appropriately evaluating not only current SDB (CKP)s but also future (entirely) new technological educational game applications with regard to their integral educational qualities.
(3) This entails explicit attention to learning and game elements and their balanced integration in context.
(4) According to our opinion, a crucial aspect of evaluating SDB (CKP)s is how competence development is realized in a SDB (CKP) through the unique integral combination of pedagogical learning support and the integration of game mechanisms to maximize learning experiences, where playful learning is better able to constantly engage, motivate, and actively educate educators through continuous complex challenges.
(5) In the proposed evaluation areas for SDB (CKP)s
(a) Categories that focus on competence-promoting qualities (areas 1 and 2) and
(b) categories that focus on the implementation qualities of the ‘product’ or submission.

For the evaluation rationale of SDB (CKP)s:

Digital playful educational media (or digital educational games) are educational resources that purposefully utilize playful learning (game-based learning) for the development of educational competencies.”

The term ‘computer games with competence-promoting potentials’ refers to all types of digital games that exhibit features conducive to the acquisition or promotion of users’ competencies. These games are platform-independent (e.g., PC, console, smartphone), can be used both offline and online, and can be played individually or with multiple persons (Grün & Rosenberger, 2017). This category encompasses digital serious games, educational games pursuing educational purposes for imparting knowledge for various forms of competency development.

The key characteristic of SDB (CKP)s is the pursuit of educational purposes through playful methods, activities that realize fun and motivation in delightful, innovative environments—always oriented towards learning processes and the achievement of intended learning objectives (Bitter & Giannoulis, 2021), regardless of the technical implementation (laptop, desktop, mobile phone, game console, cloud-based connected platforms or apps, single-multiplayer, synchronous-asynchronous AR-VR, or mixed reality settings).

The manifestations of games that facilitate learning are diverse: from serious/educational games, open-world/sandbox, role-playing games (RPG), virtual world games, epistemic games to entertainment games, adventure games, action games, and strategy games, up to pure entertainment games. Crucially, Serious Game/Educational Game submissions for the Comenius Award must directly relate to an educational context (Bitter & Giannoulis, 2021).

Evaluations in the SDB (CKP) category require a focused pursuit of educational goals. This means that purely entertainment games are not evaluated in this category. Educational games are particularly well-suited for the realization of various competencies, effectively acquiring complex skills in context through engaged and playful learning processes.

The holistic and dynamic integration and fluid media realization of learning during play necessitate a separate SDB (CKP) evaluation category. In the context of Comenius evaluations, the goal is to define quality requirements and evaluation criteria that can be easily applied in educational practice. They must be well-structured and clearly conceived, emphasizing the essential and omitting the incidental. Therefore, following the analogy proposed by Mikuszeit, the competency areas are categorized into the following four evaluation areas (Bitter & Giannoulis, 2021).

Requirements for educational intentions of SDB (CKP)

Digital educational games (SDB (CKP)s) are characterized by promoting educational intentions through playful learning. The first evaluation area with quality criteria deals with the overarching question of how the intended learning objectives can be achieved with an SDB (CKP)-Comenius submission. The conceptual learning foundation of the educational game is assessed. Game-based learning is attributed special potentials in studies, for example, for engagement and motivation. Various cognitive competencies can be acquired through in-game challenges.

SDB (CKP)s can pursue a wide variety of learning objectives. Benjamin Bloom has defined a taxonomy of cognitive activities that is well-suited for evaluating the learning goals and activities in an SDB (CKP). Bloom divides cognitive activities into six levels, each becoming increasingly complex. This hierarchical structure of cognitive activities ranges from Knowledge (1), Understanding (remember/understand) (2), Application (apply) (3), Analysis (analyze) (4), Synthesis (evaluate/synthesis) (5) to Exploration/Creation (6). (see Appendix 1).

This subdivision aligns with learning game scenarios where players must master and apply the skills, knowledge, and strategies learned in the preceding phases to become more competent and reach the (next) higher level.

With the quality criteria and examination aspects for pedagogical-content evaluation, jury members assess the learning objectives. In the second evaluation area, jury members evaluate how the learning objectives are aligned with suitable game mechanics. The method of competence development is assessed in the third evaluation area, and the media realization in the fourth evaluation area.

The quality criteria for pedagogical-content evaluation include the following:

Quality criteria for this evaluation area

1. Learning Objectives
2. Learning Objective Innovation

Quality criteria and examination aspects

1. Learning Objectives

• Are the learning objectives set, explicitly articulated, and successfully realized through the SDB (CKP)?
• Do the integrated learning activities and game-based learning methods align effectively with the achievement of educational goals (see Appendix 1)?
• Does the focus remain on the learning objective throughout the game?
• Is the didactic support for achieving the learning objective effectively integrated? Are players adequately supported in reaching the learning objective?
• Do the game scenarios and methods allow for the pedagogically appropriate achievement of the intended learning objectives?
• In the educational game, is targeted feedback and information provided on progress ‘on the go’ to support the learning process?
• Throughout the duration of the game and activities, is support and pedagogically appropriate challenge effectively provided?
• Is consideration given to cognitive overload, action, and concentration demands? And are the necessary positive stimuli and feedback utilized for learning success?
• Does the educational game achieve a depth of user interaction to attain specific learning objectives and ultimately anchor them permanently in the learner?

2. Learning Objective Innovation

• Is the thematic and educational game world novel in terms of learning content?
• Is the method of achieving learning objectives innovative?
• Is the educational game innovative due to the educational content of the learning objectives?

Requirements for Playful Learning in SDB (CKP)

Playful learning can involve an individual game, a single-player game, but most often, it occurs in social contexts with communication. Collaboration among participants in different roles, into which players immerse themselves, is crucial. In a virtual (possibly fantasy) world, players need clarity about their identity (role) in the game, the learning goal of the adventure/game, the interaction possibilities, and the feedback information to master increasingly complex challenges (Annetta, 2010, Malone 1981).

Learning increasingly complex things in educational games is facilitated by “levels.” The “levels” as a game mechanism allow players to stay in the playful (learning) “flow” (Csikszentmihalyi, 1988) and enter the next zone of competence acquisition (Vygotsky’s 1978 ZPD/Zone of Proximal Development) while remaining continuously engaged and “curious” about the next “challenges” (Malone 1981).

This evaluation area describes how we can assess the integrated application of game elements in educational games and how their quality can be evaluated in acquiring competencies (such as motivation, concentration, etc.) through playful learning in the learning process to achieve the learning objective.

In this evaluation area, we focus on the competencies of playful learning that are specifically important for engaged and successful learning with SDB (CKP)s.

The following quality criteria are included in this evaluation area:

Quality criteria for this evaluation area

1. Use of appropriate game elements (game mechanics).
2. Nature and support of the development of playful competencies.
3. Innovation in playful learning.

Quality criteria and examination aspects

1. Use of appropriate game elements (game mechanics)

• Are optimal forms of play designed to match the learning objectives, learning processes, and the target audience?
• Are appropriate game elements employed for successful playful learning?
• Does the mix of game elements (such as game rewards, professional/user ratings, leaderboards, recommendations from domain experts, etc.) support the teaching-learning process?

2. Nature and support of the development of playful competencies

• Do the game worlds/game scenarios enable the achievement of the intended learning goals while adhering to didactic game quality standards such as engagement, motivation, concentration, and empowerment?
• Are the game elements integrated in a way that does not hinder the learning/training process?
• Are learning and game objectives clear and appropriate, allowing instructors to work towards the defining goals? Are learning objectives mandatory?
• Are appropriate feedback mechanisms provided to players regarding their progress and performance?

3. Innovation in playful learning

• The game mechanics/playful learning methods of the didactic digital media product are innovative (novel).

Requirements for Competence Development in SDB (CKP)

In evaluating entries in the Comenius Competition, it is meaningful for the jury to assess educational games based on the goals set by players/learners. This means evaluating the extent to which the educational game, through its pedagogical heuristics, standards, teaching methods, and technical means, is capable of delivering the knowledge and skills it promises. Grounded in Bloom’s Taxonomy (1956) of the hierarchy of educational goals, this evaluation area focuses on assessing competence development:
– On the one hand, in the realm of individual competencies: cognitive, affective, psychomotor skills.
– On the other hand, in the realm of social competencies.

Empowerment through playful learning

Recognized didactic qualities of educational games include motivation, mechanism, and the empowerment of personal and socio-emotional competencies for individuals and teams. The related competencies are specified in this third evaluation area.

Cognitive competencies

Cognitive competencies are learnable, cognitively anchored, knowledge-based abilities and skills aimed at successfully coping with the demands of everyday and professional situations, and can be acquired through educational games. Such demands involve competencies that are functionally determined, learnable, and verifiable (Stangl, 2015). Some cognitive competencies identified by Stangl (2015), such as knowledge, creativity, critical thinking, observation, comparison, confrontation, evaluation, interpretation, analysis, argumentation, control, doubt, and problem-solving, should be assessed in the competency development pursued by an educational game. The relevance of these competencies depends on the specific educational game. As an example, we list some frequently relevant competencies as quality criteria for educational games.

Social-emotional and ethical competencies

Kompetenz

According to Jansen, Melchers & Kleinmann (2012), social competence is the ability to pursue one’s own goals in different situations while acting socially appropriately. Socially appropriate behavior involves considering the needs and interests of others or clearly explaining one’s own behavioral intentions (Stangl, 2015). Emotional competence refers to the acquisition of emotion-related skills (expression of emotions, understanding emotions, emotional regulation) and empathy. Social-emotional competence denotes the ability to express, regulate, and understand one’s own emotions and to show understanding for the feelings of others (Stangl, 2021). In this context, games can serve the function of building relationships.

Innovation in playful learning – Involvement

Recognized didactic qualities, in addition to strengthening personality-related and socio-emotional competencies, include the qualities of educational games for sustained engagement and motivation. For this, we need separate attention to assess the involvement and empowerment qualities of an educational game.

Technical-multimedia competencies

In this evaluation area, we distinguish between two aspects:
– On one hand, the necessary technological proficiency required by educators and learners in the educational game.
– On the other hand, the technical-multimedia competencies and knowledge that they acquire in the game as skills crucial in today’s or future professional life. Many of the educational resources submitted under the Comenius competition so far have focused on specific knowledge, skills, and attitudes in particular industries or companies. In preparation for the near future, generally useful media literacy competencies, broadly applicable multimedia action competencies that can be acquired in educational games, are important. Therefore, we consider it appropriate to include an additional criterion for specialized media-technical skills and knowledge within the evaluation area ‘Competency-oriented evaluation.’ The following quality criteria are part of this evaluation area.

Quality criteria for this evaluation area

1. Cognitive Competencies
2. Social-Emotional and Ethical Competencies
3. Innovation in Playful Learning – Involvement
4. Innovation in Playful Learning – Enjoyment of Play
5. Technical-Multimedia Competencies

Quality criteria and examination aspects

1. Cognitive Competencies

• Problem-solving ability: Many tasks (challenges) in the educational game promote cognitive competencies, such as cognitive analytical information processing, research, and action and planning skills when solving problems.
• Creativity: Enhancing imagination.
• Personal development: The challenges in the game trigger the competence growth of participants and strengthen the “self.” This includes the development of self-control, critical thinking, argumentation, doubt, teamwork, moral judgment competence, dealing with social (game) rules, coping with defeats, and more.

2. Social-Emotional and Ethical Competencies

• Personality-related: The game supports the development of self-awareness and self-control. Do players have control over their actions in the game?
• Self-awareness: Players’ self-observation and self-efficacy are stimulated and promoted through in-game feedback.
• Consequences: Players’ actions are made clear by providing statements about their gameplay, success and failure, as well as indications of disregarded rules.
• Team Skills: How are team skills developed in multiplayer games (competition, collaboration, interactions, and feedback from other players)?
• Decision and Judgment Competence: How does gameplay succeed in virtual simulated worlds of the educational game, including dealing with moral viewpoints and ethical foundations?

3. Innovation in Playful Learning – Involvement

• Engagement: Are positive experiences and sustained motivation ensured during gameplay?
• Flow (gameflow): Does the educational game remain continuously engaging and entertaining?
• The game pursues a “game-flow”: Is there a balance achieved between the player’s skills and the challenges presented (Csikszentmihalyi)?

4. Innovation in Playful Learning – Enjoyment of Play

• The game motivates: It provides varied gaming enjoyment.
• The game engages: It offers appealing experiences for different types of players.
• The game is adaptive: It dynamically adjusts to the difficulty level and performance of the current player.
• The game adapts to the players to enhance effectiveness (exercises/explorations repetitions)? Increasing complexity as players improve.

5. Technical-Multimedia Competencies

• Coordination: The educational game assesses different sensorimotor processes and coordinates of multimedia actions.
• Control and responsiveness: Does the game provide clear and easily understandable orientation options? Does the required flexibility of response meet current standards in the future professional world?
• Introduction to multimedia mixed-reality environments: In the learning game, players learn to navigate with the newest digital multi/mixed media environments.
• Reflection: The game provides starting points for media and play.

Media Requirements for SDB (CKP)

This category deals with the assessment of media-related requirements imposed on educators when using the SDB (CKP). This includes technological implementation and the requirements it poses during the use of the SDB (CKP). Enabling engaging learning processes that lead to learning success and a rich learning experience requires not only a good balance in learning games but also excellent (“coherent overall”) media design. In this context, not only visual and auditory design plays a crucial role, but also the possibilities of active design by users within the computer game. Examining the aspects of media requirements includes the creative realization of visual, auditory, and aesthetic interaction structures. This involves technological design that is tailored to the target audience and context.

The evaluation area of media-technological design includes the examination of the following quality criteria:

Quality criteria for this evaluation area

1. Media-Technological Implementation
2. Elaboration in Context – Balance Between Playing and Learning
3. Elaboration in Context – Abstraction & Transfer Potential
4. Innovation

Quality criteria and examination aspects

1. Media-Technological Implementation

• Is the orientation in the “learning game” and learning in a playful “flow” guaranteed by the media realization of the game?
• Where/how does technological design hinder or support learners in their learning game activities in the educational game?
• How do players navigate within the media implementation (in the virtual world, 3D/AR-VR Mixed Realities)?
• How consistent is the virtual game world designed for educators during the educational game in the virtual world? How comfortable is the representation of the imaginary world? And how appropriately is the “High-Fidelity game environment” designed, for example, when the simulation requires a realistic virtual world depiction?
• How natural and effective is virtual communication implemented in the educational game?
• The media design is at the current state of technology with recognizable effects (visual, auditory, haptic; sensory, multimedia). It is qualitatively well implemented and adaptable. Ensure a comprehensive experience so that players feel like they are “there.”
• The design provides a clear ecosystem where unnecessary information, media presentations, opportunities, and elements do not distract players.
• Sensorimotor and technical requirements should not hinder or restrict players.

2. Elaboration in Context – Balance Between Playing and Learning

• The educational game achieves a successful balance and integration between learning and playful elements.

3. Elaboration in Context – Abstraction & Transfer Potential

• The developed competencies in the educational game can be applied to other areas and applications.
• Generic knowledge gained is transferable to other learning domains.
• The game provides sufficient assistance in abstraction, allowing players to clearly distinguish virtual game worlds from real ones.
• The game also contributes to the development of playful learning competence (game-media literacy).
• Playful action patterns can be applied to other games of similar genres.

4. Innovation

• The implementation of the didactic digital media product is technologically innovative and novel.

Pedagogical and Content-related Requirements for Educational Intentions and Possibilities for Blended Learning Programs (BLEP)

The competency requirements focus on the fundamental categories of education, including goals, content, and skills. They analyze the educational intentions and possibilities for Blended Learning courses. Setting and achieving goals and sub-goals are fundamental prerequisites and guidelines for successful learning. Identifying the competencies and values that should be acquired in various teaching and learning scenarios in connection with the target audience is, therefore, a fundamental consideration in the design of Blended Learning courses. Closely related to this is the question of which content or materials, such as facts, rules, concepts, laws, methods, and relationships, should be practiced, learned, and acquired. The pedagogical and content-related requirements for the design of Blended Learning courses constitute a comprehensive approach that defines this evaluation area. This evaluation area addresses the following quality criteria in a Blended Learning scenario.

Quality criteria for this evaluation area

1. Learning Objective
2. Learning Content
3. Target Audience
4. Values

Quality criteria and examination aspects

1. Learning Objective

• All goal components of the Blended Learning program (cognitive, affective, psychomotor, social-communicative) are clearly defined and recognizable.
• The program’s learning objectives address a morally relevant issue for the users.

2. Learning Content

• The learning content is presented in a factual and scientifically correct manner (structure, selection, quantity and density, as well as the integration of information, essential statements with reference to the level of generality and abstraction).
• The learning content is aligned with the corresponding educational programs.
• The educational content is sensibly selected and justified based on pedagogical principles, addressing current topics or being current.

3. Target Audience

• The program concept is tailored to the target audience.
• Existing experiences, necessary prior knowledge, and skills of the target audience are taken into account (knowledge and abilities, emotions and attitudes, attention and concentration, environment).
• The program promotes (supports) the integration and inclusion of the target audience.

4. Values

• The program promotes ethical orientations and guiding principles.
• The program fosters humane thoughts and values.
• The targeted values and norms promote human dignity and solidarity.
• The targeted values and norms are free from glorification of violence, radical or obscene depictions, ideological influence, negative prejudices, and deliberate manipulation.
• The content of the program is free from narrow gender-specific role thinking and prejudices towards individuals and societal groups.
• The designed learning content is suitable for conveying insights (awareness raising).

Didactic-Methodological Requirements for Face-to-Face Phases in Blended Learning Programs

The didactic-methodological requirements for face-to-face phases in blended learning courses address essential aspects of teaching and learning, particularly focusing on the teaching and learning scenarios pursued in blended learning courses. Didactics, as a scientific discipline within education, deals with the rules of learning and the relationships between learning and teaching. Under a didactic-methodological perspective, questions are raised about the method, as well as the manner of conveying and acquiring knowledge and competencies. The didactic-methodological requirements for face-to-face phases thus constitute a second essential area of evaluation, outlining the requirements that must be met for high-quality blended learning courses.

Quality criteria for this evaluation area

1. Didactic Blended Learning Approaches and Phases
2. Conveyance and Learning Formats in Face-to-Face Phases
3. Didactic Steps in Face-to-Face Phases
4. Didactic Rules and Logical Learning Processes
5. Didactic Emphases in Face-to-Face Phases

Quality criteria and examination aspects

1. Didactic Blended Learning Approaches and Phases

• The Blended Learning program is based on a clearly identifiable learning-theoretical approach, such as an objectivist, constructivist, traditionalist, science-oriented, or action-oriented approach. The learning-theoretical approach is appropriately implemented, and the educational content is purposefully structured from a didactic perspective.
• The Blended Learning concept includes one or more face-to-face phases.
• The face-to-face phases are didactically aligned appropriately within the overall course concept and are effectively coordinated and connected with e-learning phases.
• The instructional scenario is appropriately presented in the form of “MOOCs (Massive Open Online Courses).”

2. Conveyance and Learning FOrmats in Face-to-Face Phases

• Methodical basic forms for face-to-face phases, such as presenting, instructing, and elaborating forms, are applied.
• Possible and meaningful forms of collaboration in face-to-face events, such as frontal presentation, partner learning, group learning, or individual learning, have been taken into account.
• In face-to-face phases, meaningful forms of collaboration and learning are applied. Frontal, cooperative, and individual learning forms are appropriately combined.
• The planned face-to-face phases can easily be transformed into e-learning phases.
• P2P (peer-to-peer) / group projects are encouraged and can be carried out and presented in both face-to-face and e-learning phases.

3. Didactic Steps in Face-to-Face Phases

• Essential didactic steps that enable an effective learning process are logically applied in the face-to-face phases:
  – Introduction and activation, leading and reactivating, group dynamics – strengthening diversity
  – Conveyance and processing, deepening and generalization
  – Consolidation and application, repetition and application, systematization
  – Evaluation and analysis, making agreements.
• The planned didactic steps allow users to work at different difficulty levels and paces.
• The execution of the learning steps is emotionally effective and motivating.
• Guidance/support should be present/accessible at fixed times as well.

4. Didactic Rules and Logical Learning Processes

• Basic didactic rules and principles were adhered to in the course design for the face-to-face phases, such as:
  – Clarity
  – Scientific basis
  – Logical consistency
  – Illustrative nature
  – From the general to the specific
  – From simple to complex
  – From easy to difficult
  – From near to far
  – From familiar to unfamiliar
  – Connecting the concrete with the abstract.
• Logical learning processes, such as analyzing, synthesizing, comparing, differentiating, generalizing, abstracting, summarizing, organizing, concretizing, are embedded in the course and promoted.

5. Didactic Emphases in Face-to-Face Phases

The Blended Learning concept includes one or more face-to-face phases with some of the mentioned didactic-methodological steps.

• Face-to-Face Phase A: Presentation, Initiating, Leading
  – Topic presentation
  – Problem situation
  – Impulse (lecture or media)
  – Introduction to the Blended Learning concept
  – Introduction to the eLearning phase (handling the internet, media, and communication methods)
  – Introduction to the utilized media
  – Assignment for the eLearning phase (individual or in groups)
• Face-to-Face Phase B: Continuing, Deepening
  – Presentation of participant results (individual or in groups)
  – Establishing feedback rules
  – Discussion of learning outcomes
  – Systematization of the topic, possible incorporation of media offerings
  – Assignment for the continuation phase on the topic (forum, multimedia)
• Additional Face-to-Face Phases: Continuing, Deepening
  – Similar to Face-to-Face Phase B, depending on the topic

Didactic-methodical requirements for e-learning phases of blended learning programs

The didactic-methodical requirements for e-learning phases/self-learning phases of blended learning courses deal with essential aspects, especially those related to learning scenarios pursued in blended education courses. Didactics, as a scientific discipline within pedagogy, focuses on the rules of learning and the relationships between learning and teaching. Under the didactic-methodical questioning, inquiries are made regarding the method, as well as the manner of conveying and acquiring knowledge and competencies. The didactic-methodical requirements for e-learning phases of blended learning courses therefore constitute a third essential evaluation area and structure the criteria that must be met for didactic-methodical requirements in e-learning phases.

Quality criteria for this evaluation area

1. Didactic Blended Learning Approaches and Phases
2. Didactic-Content Aspects of E-Learning Phases
3. General Requirements for E-Learning Phases
4. Didactic Rules and Logical Learning Processes
5. Didactic Emphases of E-Learning Phases

Quality criteria and examination aspects

1. Didactic Blended Learning Approaches and Phases

• The Blended Learning program is based on a recognizable learning theoretical approach. The learning theoretical approach is implemented appropriately, and the educational content is structured purposefully from a didactic perspective.
• The Blended Learning concept includes one or more E-Learning phases.
• The E-Learning phases are didactically integrated into the overall course concept in a purposeful manner and are appropriately coordinated and connected with the face-to-face phases.
• The planned face-to-face phases can easily be transformed into E-Learning phases.
• The learning scenario is offered appropriately in the form of a “Flipped Classroom” (also known as Inverted Classroom).

2. Didactic-Content Aspects of E-Learning Phases

The E-Learning phases and offerings are characterized by the following didactic-content aspects:

• Curricular Integration – The E-Learning offerings are didactically purposefully integrated into the Blended Learning concept and are necessary for achieving the course objectives.
• Process-Oriented Learning – The E-Learning offerings involve the active acquisition of new knowledge through a process of engagement, rather than the transmission of a set of finished and unambiguous information units.
• Competence Development – The E-Learning offerings promote and enhance the subject-specific, methodological, and social competence of all participants.
• Promotion of Cooperative Work and Learning Forms – The E-Learning offerings encourage cooperative work and learning forms, as well as models of distributed knowledge communities.
• P2P (peer-to-peer) / Group Projects are promoted and can be conducted and presented in both face-to-face and E-Learning phases.
• Contextualization – The E-Learning offerings promote the connection of the content with other societal fields (especially over the internet).

3. General Requirements for E-Learning Phases

• Learning can take place independently of time and location.
• Participants can determine their own learning pace.
• Various media offerings (apps, images, videos, audio, animations, text) and different tasks (practical, theoretical approaches, etc.) address different learning types.
• The course content is well-prepared and flexible in terms of didactics and methodology.
• Tutor support is ensured, and assistance/guidance is available at scheduled times.

4. Didactic Rules and Logical Learning Processes

• Fundamental didactic rules and principles have been adhered to in the course design of the E-Learning phases, such as:
  – Comprehensibility
  – Scientificity
  – Coherence
  – Illustrative nature
  – From the general to the specific
  – From the simple to the complex
  – From the easy to the difficult
  – From the near to the distant
  – From the familiar to the unfamiliar
  – Connection of the concrete with the abstract.
• Logical learning methods, such as analyzing, synthesizing, comparing, differentiating, generalizing, abstracting, generalizing, ordering, concretizing, are embedded in the course and are encouraged.

5. Didactic Emphases of E-Learning Phases

The Blended Learning concept includes one or more E-Learning phases with some of the mentioned didactic-methodical steps.

• E-Learning Phase A: Guidance, Working, Independent Processing
  – Processing the topic using a medium and following a task (self-directed learning)
  – Communication with fellow learners and instructors (forums, chat, tutorial, P2P groups)
  – Online collaboration (workspace)
• E-Learning Phase B: Guidance, Independent Processing
  – Continued participant-led forum for discourse among participants
  – Inclusion of additional media products
• Additional E-Learning Phases: Guidance, Independent Processing
  – Similar to E-Learning Phase B, depending on the topic and face-to-face phases

Requirements for Organization and Media in Blended Learning Programs

The organizational requirements address fundamental work organizational aspects in the implementation of blended learning courses and the use of didactic digital media products. These are questions related to user-friendly design of teaching and learning scenarios or human-computer interfaces. The framework conditions of the blended learning project include not only curricular integration but also the available budget as well as the consulting and support capacities that can be accessed. Additionally, the user-friendliness and handling of didactic digital media products are crucial for a successful blended learning course. Therefore, these criteria are grouped together in a fourth criterion group as requirements for organization and media. In these requirements, especially work organizational aspects are important, such as framework conditions, operating characteristics, handling aspects, and usage properties. Working conditions for teachers and learners are often the key to success or failure in a blended learning course. Training events can be seen as institutionalized communicative actions, focusing on the creation of learning situations. Instructors and participants must communicate within a framework of institutional conditions about their starting point, course objectives, mediating variables such as behavior, teaching and learning methods, organizational measures, and possible success controls. Within these mutually interacting aspects of action, the organization of a blended learning course becomes an important mediating variable. Further education organization also refers to the organization of the social forms of further education. The face-to-face phases conducted frontally, both offline and online, particularly favor receptive learning behavior. In group work, the entire teaching and learning group is involved in the planning and implementation of instruction. Self-learning phases most readily allow for adaptation to individual learning progress.

The following quality criteria are among the organizational, communicative, and medial requirements:

Quality criteria for this evaluation area

1. Framework Conditions
2. Content-appropriate and audience-specific selection and design of didactic digital media products
3. Multimedia
4. Interactivity
5. Adaptivity
6. Information about the Blended Learning Program

Quality criteria and examination aspects

1. Framework Conditions

• For the implementation of the Blended Learning Program, technical consultation and support capacity are available at scheduled times.
• It is ensured that course participants can independently acquire the necessary technical and media competencies.
• The required hardware and software for the course are generally available or can be easily procured.
• The course’s requirement for a appropriately equipped multimedia room as the venue (e.g., a circular arrangement of computer workstations) is outlined.
• The requirements for the planned teaching staff, their preparation, and training are clearly defined.
• Various didactic variables in the teaching and learning process for the development of learning motivations are considered, particularly intrinsic motivation, for example, in gaining emotional security, participant involvement in the organization and implementation of events, the number of participants in events, and the abundance and structure of the information to be conveyed with respect to the target audience.

2. Content-appropriate and audience-specific selection and design of didactic digital media products

• The selection of didactic digital media products for the Blended Learning course is done appropriately for both the face-to-face and e-learning phases.
• The didactic media products and the media category have been chosen purposefully based on the content. The learning content is aligned with the capabilities of the media category.
• The multisymbolic representation form suitable for the content has been chosen. The multisymbolic representation forms (texts, graphics, images, videos, audios, etc.) are correct and align with aesthetic considerations.
• The media product has been designed to be audience-specific.
• Representation forms of content such as language, sound, image, animation are appropriate for the target audience.

3. Multimedia

• The integrated media products in the Blended Learning Program can be functionally, meaningfully, and supportively utilized for learning.
• Learning channels are sensibly activated to support learning (visual, auditory, tactile, motor).
• Various forms of instruction and communication are provided.
• The selected programs and program elements (tutorial programs, simulations, exercises, games, reference materials, etc.) are justified, complement each other, and are appropriate for the content.
• The media products are compatible with mobile phones.

4. Interactivity

• The integrated media products in the Blended Learning Program enable interactive work, the modification of tasks, and flexible responses according to different learning needs and prerequisites. Feedback is provided in variable forms, motivating, and effectively presented.
• For performance evaluations, appropriate options (such as text, sound, graphics, animation) are offered. Incorrect solutions are identified in different and variable ways. Feedback on incorrect solutions is given in a motivating manner and assesses the response, not the person.
• The media products respond to the learning progress by analyzing individual performance levels and recommending corresponding branches.
• Branches are automatically taken based on response and learning progress analysis and can be freely chosen. There is an appropriate and manageable number of branches. Branching provides different difficulty levels and varied task forms.
• Interactivity between the user and the media products is facilitated by presenting tasks and assignments, demanding solutions, and encouraging the development of solution strategies.
• Interactivity is supported by the dependence of program progress on user contributions and activities, triggering user activities, such as collecting data, expanding information, providing data for further processing, delivering error messages with content relevance, providing factual and variable confirmation of work results, implementing LINKS to other media, or through reward systems (leaderboards, games, etc.).
• Motivation systems support interactivity (bonus points, rewards, level skipping).

5. Adaptivity

• The media products of the Blended Learning course allow adaptation to the user’s performance by making changes to the basic settings (e.g., turning off sound, switching between text and audio output) and adjusting the difficulty level (e.g., tasks with various difficulty levels).
• Adjusting time behavior (e.g., setting reaction times according to user requirements) is ensured by the media products.
• The media products enable adaptation to the user’s performance through the type and extent of information (e.g., separate and combined selection of text or audio information).
• An adjustment of the help system (e.g., variable offering of assistance) is facilitated by the media products.

6. Information about the Blended Learning Program

• The Blended Learning course has been adequately publicized through various channels such as flyers, the internet, and the press.
• Participants in the Blended Learning course are provided with study materials or suitable literature.

To arrive at an approximate quality judgment relatively quickly, it is expedient to conduct an arithmetic criteria assessment. This assessment can be carried out using the Comenius evaluation sheet, which includes all quality criteria and testing aspects/indicators. The central question in this assessment is the extent to which the quality criterion has been conceived or can be realized. The assessment is straightforward based on various testing aspects/indicators. Quality criteria that do not apply to an educational medium or multimedia product are not assessed (0 points). The following overview summarizes the short assessment for a testing aspect:

After evaluating all testing aspects for the quality criteria within one of the four evaluation areas of a media group (e.g., for the product group DMP: pedagogical-content, didactic-methodical, medial-design, and user-oriented-technical evaluation), an arithmetic mean of the points assigned to the quality criteria is calculated. The arithmetic means obtained from the short assessment of a rating group should be interpreted as follows:

The overall rating is the sum of the respective arithmetic means of the four evaluation areas. With this rating, it is ensured that the four evaluation areas are treated equally in the overall assessment, but, for example, in the case of DMP, pedagogical and didactic aspects dominate in two evaluation areas and make up 50% of the assessment. This reflects the fundamental intention of the Comenius evaluation and testing procedure for educational media, aiming to determine up to 50% of all assessment aspects pedagogically and didactically. In this way, a maximum of 20 points can already be achieved, with 10 points being attainable.

The achieved total score can be interpreted as follows:

In the Comenius EduMedia Competition, products with a rating of more than 18 points are proposed for the award of a Comenius Medal, while products with a rating between 10 and 18 points are suggested for the award of a Comenius Seal.