1. Introduction

While foundational knowledge is essential, modern education increasingly emphasizes the cultivation of higher-order thinking skills: the ability to analyze, evaluate, create, think critically, and solve complex problems. Teaching these skills effectively demands more than rote instruction; it requires educators to dynamically adapt their strategies, pose challenging questions, facilitate exploration, and provide targeted feedback based on individual student needs and cognitive states.

Developing this adaptive expertise is challenging. Real-world classroom experimentation is constrained by time, resources, ethical considerations, and the difficulty of isolating the impact of specific interventions. Existing tools, including traditional Intelligent Tutoring Systems (ITS) [1, 2], often fall short in capturing the complexity required to foster skills like creativity or critical thinking, sometimes lacking the flexibility to discover novel strategies based on nuanced student feedback. Recent work continues to explore data-driven approaches within ITS, but simulating complex skill development remains an open area [10].

This paper builds upon the concept of using Reinforcement Learning (RL) [3] in a simulated classroom, significantly enhancing the model's fidelity and pedagogical depth by incorporating established educational frameworks like Bloom's Taxonomy [4], explicit measures of creativity, defined problem-solving methodologies, and detailed student profiles. The aim is to create a testbed where RL agents can learn complex, adaptive teaching policies specifically targeting higher-order skills.

2. Background and Related Work

2.1 Reinforcement Learning (RL)

RL [3] involves an Agent interacting with an Environment. The agent observes the environment's State, takes an Action, receives a Reward, and transitions to a new state. The goal is to learn a Policy (a state-to-action mapping) that maximizes cumulative reward. Key components are the agent, environment, state, action, reward, and policy. Algorithms like Q-learning, PPO, or SAC enable policy learning. RL has seen application in various educational contexts, often within ITS or game-based learning, for tasks like optimizing problem sequencing, feedback generation, or adapting difficulty [5, 6, 11].

2.2 Group Relative Policy Optimization (GRPO)

Our approach builds on recent advances in RL optimization techniques, specifically the Group Relative Policy Optimization (GRPO) methodology introduced by Shao et al. [9]. GRPO has demonstrated exceptional capability in mathematical reasoning tasks, achieving a 51.7% success rate on competition-level problems. We adapt this approach to the educational context, applying its group-based optimization to enhance the learning of pedagogical strategies. Our implementation leverages standardized benchmarking techniques inspired by the OpenAI Gym framework [18] to ensure consistent evaluation and comparison of different optimization strategies.

2.3 Bloom's Taxonomy (Cognitive Domain)

Bloom's revised taxonomy [4] provides a hierarchy for cognitive skills: 1. Remembering, 2. Understanding, 3. Applying, 4. Analyzing, 5. Evaluating, 6. Creating. This framework allows modeling student cognitive states and targeting teacher actions.

2.4 Creativity Matrix

We adopt a matrix to quantify creative output levels: 1. Imitation, 2. Variation, 3. Combination, 4. Transformation, 5. Original Creation.

2.5 Problem-Solving Techniques

A repertoire of techniques (e.g., Direct Analogy, Attribute Listing, SCAMPER, Morphological Synthesis, Fantasy Analogy, Reverse Brainstorming, etc.) serve as potential pedagogical actions for the teacher agent to facilitate problem-solving [cf. 7].

2.6 Detailed Student Attributes

Student modeling is crucial [8]. We incorporate richer state vectors than typical ITS, including:

• Static/Semi-Static: Personality (introvert/extrovert), Learning Style (Kinesthetic, Audio, Visual), Process Style (Doer, Thinker, Talker), Grade Level, IEP/504 status.
• Dynamic: Mastery Score (0-100), Content Knowledge (assessment history), Behavior state, Current Bloom's Level (1-6), Current Creativity Level (1-5), Critical Thinking Score, Problem Solving Score, Engagement Level. Recent approaches also explore inferring latent student states using techniques like inverse RL or deep learning [12].
• Curiosity Metrics: Following the Morpheus-RL Protocol, we track specific indicators of student curiosity, including Exploration Tendency, Persistence, Question Frequency, and Knowledge Gap Awareness, which provide valuable insights into intrinsic motivation levels.

3. Methodology: The Enhanced RL Simulation

We propose an RL environment simulating learning sessions focused on higher-order tasks.

3.1 The Environment

Represents the learning context: current task/problem, time constraints, available resources, and aggregated student metrics. This aligns with the need for structured environments for RL training [3]. Our environment implementation follows the Morpheus-RL Protocol, which defines standardized interfaces and data formats for educational reinforcement learning focused on personalized tutoring. This protocol-first approach ensures transparency, interoperability, and consistent tracking of cognitive states across different implementations.

3.2 Simulated Student Agent(s)

• State Vector: Possesses the detailed static and dynamic attributes (Sec 2.6).
• State Dynamics: Internal states evolve based on teacher actions, task interactions, and internal logic (e.g., learning gain/decay, shifts in Bloom's level based on success/failure, creativity score based on task output, engagement fluctuations). Modeling these dynamics realistically is a key challenge [8].
• Response Generation: Students attempt tasks or answer questions probabilistically based on their state relative to task demands. Output is evaluated (e.g., correctness, creativity level).
• Memory and Reflection: Drawing on recent advances in generative agent simulation [19], our student agents maintain memory records of past interactions, synthesize these memories into reflections on their learning progress, and dynamically retrieve relevant memories when generating responses.
• Reward Contribution: Changes in student state (knowledge, Bloom's level, creativity, etc.) and task performance contribute components to the teacher's reward signal.

where σ is the sigmoid function, and βᵢ are weighting parameters.

3.3 Real-Time Dialogue Architecture

Our implementation builds upon recent advances in speech-text foundation models, particularly the architecture principles demonstrated in Moshi [23]. Moshi's full-duplex spoken dialogue framework addresses critical limitations in traditional pipeline-based systems: latency, textual information bottlenecks, and turn-based modeling constraints. While Moshi focuses on general conversational AI, our educational application adapts its core architectural insights for pedagogical purposes.

The integration of Moshi-inspired architecture in our system enables:

• Low-Latency Interaction: Real-time speech-to-speech generation with theoretical latencies below 200ms, crucial for maintaining natural conversational flow in tutoring sessions.
• Multi-Stream Processing: Simultaneous handling of student speech input and teacher speech output, allowing for natural interruptions and turn-taking behaviors that mirror human tutoring interactions.
• Preserved Paralinguistic Information: Unlike traditional text-based intermediary systems, the speech-to-speech approach retains emotional cues, prosody, and non-linguistic information that significantly impact learning effectiveness and student engagement.
• Adaptive Response Generation: The foundation model backbone enables contextual awareness that improves pedagogical decision-making within the RL framework.

This architectural approach complements our GRPO-based teacher agent by providing a more natural and responsive interaction modality, reducing cognitive load on students and enabling the capture of richer behavioral signals for reward computation.

3.4 Simulated Teacher Agent

• Goal: Maximize a weighted, multi-objective reward function reflecting holistic student progress (Sec 3.5). Multi-objective RL is an active research area [13].
• State Observation: Perceives the environment and student states (potentially partially observable).
• Action Space: Includes diverse pedagogical actions:
  • Instructional: Explain, Example, Feedback.
  • Questioning: Targeted at specific Bloom's levels [4].
  • Problem-Solving Facilitation: Applying techniques from Sec 2.5 [7].
  • Creative Task Assignment: Tasks designed for specific Creativity Matrix levels.
  • Assessment/Motivational Actions.
• Learning Mechanism: Utilizes GRPO [9] and other advanced RL algorithms to optimize its policy.
• Parameter-Efficient Adaptation: Employs Low-Rank Adaptation (LoRA) techniques [20] to efficiently fine-tune foundation models for different educational domains and individual student needs, enabling rapid personalization while maintaining computational efficiency.

3.5 Interaction Loop & Multi-Objective Reward

The simulation proceeds in discrete time steps:

1. Teacher observes state.
2. Teacher selects action based on policy.
3. Action impacts student states.
4. Students respond/perform task.
5. Student output/response is evaluated.
6. A multi-component reward is calculated:

(Weights w_i allow tuning pedagogical focus). Defining effective reward functions is critical in applied RL [3, 11]. For mathematical tutoring contexts, we employ specialized reward functions including Taxonomy Format Reward, Reasoning Quality Reward, and Student Adaptation Reward that assess how well the model adapts to different student knowledge states.

7. Teacher receives R_total and observes the new state.
8. Teacher updates policy.
9. Loop continues until termination condition.

All interactions are logged following the Morpheus-RL Protocol's standardized data format, facilitating transparent analysis and reproducibility of results. The protocol enables structured tracking of cognitive state transitions, curiosity metrics, and learning breakthroughs across sessions.

4. Key Findings

Our research has yielded several significant findings:

These results demonstrate the potential of GRPO-enhanced reinforcement learning to discover effective teaching strategies that can be applied in real educational settings.

5. Potential Applications and Benefits

• Training for Higher-Order Skills: Safe practice environment for educators [cf. 14 for simulation in training].
• Curriculum Design: Testing sequences for fostering creativity/critical thinking.
• Personalized Learning Pathways: Discovering optimal interventions for diverse student profiles [11, 15].
• Advanced AI Tutors: Informing tutors capable of guiding complex skill development [1, 2].
• Assessment Research: Simulating relationships between different assessment forms.

6. Challenges and Considerations

• Modeling & Assessing Higher-Order Skills: High fidelity simulation and assessment are difficult [8].
• Reward Function Design: Balancing multiple objectives via weighting (w_i) is complex and critical [13].
• Action Space Complexity: Large action space requires efficient RL algorithms and potentially longer training [9].
• Student Model Validity: Ensuring simulated behavior plausibly reflects real students is paramount [8, 12].
• Transferability (Sim-to-Real): Bridging the gap between simulation-optimized strategies and real-world effectiveness requires careful validation [16].

7. Future Directions

• Richer Cognitive/Affective Models: Incorporating metacognition, motivation, detailed emotions [8].
• Natural Language Interaction: Enabling more naturalistic communication, potentially using large language models within the loop.
• Collaborative Learning Simulation: Modeling student-student interactions using multi-agent RL [17].
• Explainable AI (XAI): Making the teacher agent's complex decision-making process transparent and understandable to educators.
• Real-World Data Integration: Using empirical data for model refinement and validation [10, 12].
• Protocol Enhancements: Expanding the Morpheus-RL Protocol to support distributed reward mechanisms and federated learning across educational environments.
• Multi-objective Reasoning: Refining GRPO-based training to better handle complex mathematical reasoning tasks with step-by-step solution generation.
• Generative Memory Systems: Further developing memory and reflection capabilities in student models based on generative agent frameworks [19], enabling more realistic long-term learning patterns and individual differences.
• Standardized Evaluation Environments: Creating comprehensive benchmark tasks inspired by OpenAI Gym [18] for systematic comparison of adaptive teaching strategies across different educational domains.
• Parameter-Efficient Fine-tuning: Expanding the use of Low-Rank Adaptation (LoRA) [20] for rapid adaptation of large foundation models to specific educational contexts and individual student profiles without extensive retraining.

8. Conclusion

Integrating established educational frameworks within a GRPO-enhanced RL-driven simulation provides a powerful paradigm for advancing our understanding and practice of teaching higher-order thinking skills. By enabling a simulated teacher to learn adaptive strategies based on nuanced student models incorporating Bloom's Taxonomy, creativity metrics, and problem-solving techniques, we can develop innovative tools for teacher training, curriculum design, and next-generation AI educational systems. While significant challenges remain, particularly in high-fidelity modeling and reward engineering, the potential benefits justify continued research at this intersection of AI and education.