Difference between revisions of "Just Enough Practice"

From Open Pattern Repository for Online Learning Systems
Jump to navigation Jump to search
(Add evaluation)
m (Edited source)
 
(17 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{Infobox_designpattern
{{Infobox_designpattern
|image=Just_enough_practice.png
|image=Just_enough_practice.png
|author= [[User:Pinventado|Paul Inventado]], Peter Scupelli
|contributor= [[Paul Salvador Inventado]], [[Peter Scupelli]]
|contributor=  
|source= Inventado and Scupelli (in press 2015)<ref name="Inventadoip">Inventado, P.S. & Scupelli, P. (in press 2015). [https://cmu.box.com/shared/static/m6qfs01z71gt38a7tf85gcgl8t84iw50.pdf A Data-driven Methodology for Producing Online Learning System Design Patterns]. In ''Proceedings of the 22nd Conference on Pattern Languages of Programs (PLoP 2015)''. New York:ACM.</ref>; Inventado and Scupelli (2015)<ref name="Inventado2015">Inventado, P.S. & Scupelli, P. (2015). [http://dl.acm.org/authorize?N09846 Data-Driven Design Pattern Production: A Case Study on the ASSISTments Online Learning System]. In ''Proceedings of the 20th European Conference on Pattern Languages of Programs (EuroPLoP 2015)''. New York:ACM.</ref>
|dataanalysis=[[Analysis:Student_affect_and_interaction_behavior_in_ASSISTments|Student affect and interaction behavior in ASSISTments]]
|dataanalysis=[[Analysis:Student_affect_and_interaction_behavior_in_ASSISTments|Student affect and interaction behavior in ASSISTments]]
|domain= General
|domain= General
|stakeholders= Students, Teachers, System developers
|stakeholders= Students, Teachers, System developers
|evaluation = [http://www.europlop.net/content/cfp-2015 EuroPLoP 2015] writing workshop, [http://www.hillside.net/plop/2015/ PLoP 2015] writing workshop, [[Talk:ASSISTments]]
|evaluation = [http://www.europlop.net/content/cfp-2015 EuroPLoP 2015] shepherding and writing workshop<br/> [http://www.hillside.net/plop/2015/ PLoP 2015] shepherding and writing workshop<br/> [[Talk:ASSISTments]]
|application =  [[ASSISTments]]
|application =  [[ASSISTments]]
|appliedevaluation =  [[ASSISTments]]
|appliedevaluation =  [[ASSISTments]]
}}
}}


If academic risk takers become frustrated when they are asked to repeatedly answer problems that test skills they have already mastered, then change the problem type and/or topic after students master it.
Allow students to practice a skill until they master it then switch to another skill in order to avoid over practice<ref name="Inventadoip"/><ref name="Inventado2015"/>.  


==Context==
==Context==
Students are asked to practice a particular skill through exercises in an online learning system. Teachers design the problems for the exercise in the online learning system. They also provide corresponding answers and feedback for each problem, and design their presentation sequence. Problems may vary in type (e.g., multiple choice, true or false), topic (e.g., addition, subtraction), and difficulty.  
Content creators for Skill Builders design problem-solving activities that facilitate student mastery of a particular skill. Skill Builder problem sets require a student to achieve three correct answers consecutively in order to move on to new assignments while continuing to provide struggling students with extended practice.


==Problem==
==Problem==
Academic risk takers become frustrated when they are asked to repeatedly answer problems that test skills they have already mastered.
Students cannot maximize their learning time if they are asked to practice skills they already mastered.


==Forces==
==Forces==
#'''Practice.''' Students need practice to learn a skill<ref>Clark, R. C., and Mayer, R. E. (2011). [http://www.wiley.com/WileyCDA/WileyTitle/productCd-0470874309.html E-learning and the science of instruction: Proven guidelines for consumers and designers of multimedia learning]. John Wiley & Sons.</ref><ref>Sloboda, J. A., Davidson, J. W., Howe, M. J., and Moore, D. G. (1996). [http://onlinelibrary.wiley.com/resolve/doi?DOI=10.1111/j.2044-8295.1996.tb02591.x The role of practice in the development of performing musicians]. British journal of psychology, 87(2), 287-310.Sweller, J., & Cooper, G. A. 1985. The use of worked examples as a substitute for problem solving in learning algebra. Cognition and Instruction, 2(1), 59--89.</ref><ref>Tuffiash, M., Roring, R. W., and Ericsson, K. A. (2007). [http://psycnet.apa.org/psycinfo/2007-14487-002 Expert performance in SCRABBLE: implications for the study of the structure and acquisition of complex skills]. Journal of Experimental Psychology: Applied, 13(3), 124.</ref>. It leads to greater improvements in performance during early sessions, but additional practice sessions lead to smaller improvement gains over time<ref>Rohrer, D. and Taylor, K. (2006). [http://eric.ed.gov/?id=ED505642 The effects of over-learning and distributed practice on the retention of mathematics knowledge]. Applied Cognitive Psychology, 20, 1209--1224.</ref>.
#'''Diminishing returns.''' Students learn more when they initially practice a skill, but eventually learn less as they master the skill through continued practice<ref>Rohrer, D. and Taylor, K. (2006). [http://eric.ed.gov/?id=ED505642 The effects of over-learning and distributed practice on the retention of mathematics knowledge]. ''Applied Cognitive Psychology, 20'', 1209--1224.</ref><ref>Sweller, J. (2004). [http://link.springer.com/article/10.1023%2FB%3ATRUC.0000021808.72598.4d Instructional design consequences of an analogy between evolution by natural selection and human cognitive architecture]. ''Instructional science, 32''(1-2), 9-31.</ref>.
#'''Expertise reversal.''' Presenting students information they already know can impose extraneous cognitive load and interfere with additional learning<ref>Sweller, J. (2004). [http://link.springer.com/article/10.1023%2FB%3ATRUC.0000021808.72598.4d Instructional design consequences of an analogy between evolution by natural selection and human cognitive architecture]. Instructional science, 32(1-2), 9-31.</ref>.
#'''Over practice.''' Students’ learning gains are almost the same when they either stop practicing a skill after mastery or over practice it<ref name="Cen2007">Cen, H., Koedinger, K. R., and Junker, B. (2007). [http://dl.acm.org/citation.cfm?id=1563681 Is Over Practice Necessary?-Improving Learning Efficiency with the Cognitive Tutor through Educational Data Mining]. ''Frontiers in Artificial Intelligence and Applications, 158'', 511.</ref>.
#'''Risk taking.''' Academic risk takers are students who prefer challenging tasks because they want to maximize learning and feedback<ref>Clifford, M. M. (1988). [http://onlinelibrary.wiley.com/resolve/doi?DOI=10.1111/j.2044-8279.1988.tb00875.x Failure tolerance and academic risk‐taking in ten‐to twelve‐year‐old students]. British Journal of Educational Psychology, 58(1), 15-27.</ref><ref>Clifford, M. M. (1991). [http://www.tandfonline.com/doi/abs/10.1080/00461520.1991.9653135#.VZwH65NVhBc Risk taking: Theoretical, empirical, and educational considerations]. Educational Psychologist, 26(3-4), 263-297.</ref><ref>Meyer, D. K., and Turner, J. C. (2002). [http://www.tandfonline.com/doi/abs/10.1207/S15326985EP3702_5#.VZwIHZNVhBc Discovering emotion in classroom motivation research]. Educational psychologist, 37(2), 107-114.</ref>. They are often intrinsically motivated, explore concepts they do not understand, and can cope with negative emotions resulting from failure<ref>Boekaerts, M. (1993). [http://www.tandfonline.com/doi/abs/10.1207/s15326985ep2802_4#.VZwIZJNVhBc Being concerned with well-being and with learning]. Educational Psychologist, 28(2), 149-167.</ref>.
#'''Limited resources.''' Student attention and patience is limited so they may switch to other tasks if they feel they are no longer learning from an activity<ref>Arnold, A., Scheines, R., Beck, J. E., and Jerome, B. (2005). [https://oli.cmu.edu/wp-oli/wp-content/uploads/2012/05/Arnold_2005_Time_and_Attention.pdf Time and attention: Students, sessions, and tasks]. In ''Proceedings of the AAAI 2005 Workshop Educational Data Mining'' (pp. 62-66).</ref><ref>Bloom, B. S. (1974). [http://psycnet.apa.org/journals/amp/29/9/682/ Time and learning]. ''American psychologist, 29''(9), 682.</ref>.
#'''Limited resources.''' Student attention and patience is a limited resource possibly affected by pending deadlines, upcoming tests, achievement in previous learning experiences, motivation, personal interest, quality of instruction, and others<ref>Arnold, A., Scheines, R., Beck, J. E., and Jerome, B. (2005). [https://oli.cmu.edu/wp-oli/wp-content/uploads/2012/05/Arnold_2005_Time_and_Attention.pdf Time and attention: Students, sessions, and tasks]. In Proceedings of the AAAI 2005 Workshop Educational Data Mining (pp. 62-66).</ref><ref>Bloom, B. S. (1974). [http://psycnet.apa.org/journals/amp/29/9/682/ Time and learning]. American psychologist, 29(9), 682.</ref>.  
 


==Solution==
==Solution==
Therefore, change the problem type and/or topic after students master it.
Therefore, give students problems to practice a skill until they master it then give them new problems to practice a different skill. There are different ways to assess mastery such as students’ performance on a skill-mastery test, or a statistical model’s prediction of student mastery<ref>Yudelson, M. V., Koedinger, K. R., and Gordon, G. J. (2013). [http://link.springer.com/chapter/10.1007/978-3-642-39112-5_18 Individualized bayesian knowledge tracing models]. In ''Artificial Intelligence in Education'' (pp. 171-180). Springer Berlin Heidelberg.</ref>.
 
Student mastery can be assessed in different ways such as, counting the number of times a student correctly answered a problem type and/or topic, or using individualized statistical models for predicting student knowledge<ref>Yudelson, M. V., Koedinger, K. R., and Gordon, G. J. (2013). [http://link.springer.com/chapter/10.1007/978-3-642-39112-5_18 Individualized bayesian knowledge tracing models]. In Artificial Intelligence in Education (pp. 171-180). Springer Berlin Heidelberg.</ref>.


==Consequences==
==Consequences==


===Benefits===
===Benefits===
#Students get enough practice to learn the skill, but not too much to over-practice it.
#Students get enough practice to master a skill.
#Students do not spend unnecessary time practicing skills they already mastered.
#Students do not spend unnecessary time over-practicing a skill when it does not contribute to learning gains.
#Students practice on problems that challenge them.
#Students make better use of their time by learning more skills in the allotted time
#Students with better learning experiences are more inclined to continue learning.


===Liabilities===
===Liabilities===
#If skill mastery is incorrectly predicted, the system can still cause over-practice on a skill or worse, prevent students from practicing a skill enough before it is mastered.
#The online learning system needs to support the measurement of skill mastery before the pattern can be applied.
#If skill mastery is incorrectly predicted, the learning system can cause over-practice on a skill or worse, prevent students from practicing a skill enough before mastery.
#Aside from creating problems to practice a particular skill, content creators will also need to prepare problems that target other skills students are asked to learn.


==Evidence==
==Evidence==


===Literature===
===Literature===
Cen, Koedinger and Junker<ref>Cen, H., Koedinger, K. R., and Junker, B. 2007. [http://dl.acm.org/citation.cfm?id=1563681 Is Over Practice Necessary?-Improving Learning Efficiency with the Cognitive Tutor through Educational Data Mining]. Frontiers in Artificial Intelligence and Applications, 158, 511.</ref> used data mining approaches to show that students had similar learning gains when they over-practiced a skill and stopped practicing a skill after mastery. However, it took less time when students stopped practicing after mastery. Instead of over-practicing, they suggested that students switch to learning other skills.  
An experiment conducted by Cen and colleagues<ref name="Cen2007"/> revealed that students had similar learning gains regardless if they over-practiced a skill or stopped practice after mastery. However, it took less time when students stopped practice after mastery. They suggest that students should switch to learning new skills instead of over practicing already mastered skills.


===Discussion===
===Discussion===
In a meeting with Ryan Baker and his team at Teacher's College in Columbia University, Neil Heffernan and his team at Worcester Polytechnic Institute, and Peter Scupelli and his team at the School of Design in Carnegie Mellon University (i.e., ASSISTments stakeholders), the team agreed that too much repetition can be problematic and controlling the number of times a problem type is presented to students can address the problem.
Shepherds, writing workshop participants, and learning system stakeholders (i.e., data mining experts, learning scientists, and educators) agreed that over-practice could be common among online learning systems, and adapting problems to student mastery could address this problem.


David West, the pattern's shepherd at PLoP 2015, also considered the pattern definition acceptable.


===Data===
===Data===
According to ASSISTments math online learning system data, frustration correlated with students repeatedly answering problems they have mastered.  
According to [[Analysis:Student_affect_and_interaction_behavior_in_ASSISTments#hintusage | ASSISTments math online learning system data]], frustration correlated with students repeatedly answering problems they already mastered.
<!--===Applied evaluation===
<!--===Applied evaluation===
  Results from randomized controlled trials (RCTs) or similar tests that measures the pattern's effectiveness in an actual application. For example, compare student learning gains in an online learning system with and without applying the pattern. -->
  Results from randomized controlled trials (RCTs) or similar tests that measures the pattern's effectiveness in an actual application. For example, compare student learning gains in an online learning system with and without applying the pattern. -->
==Example==
Some online learning systems measure students’ skill mastery to help control the amount of practice provided. For example, [[Cognitive_tutor_algebra | Cognitive Tutor Algebra]] and [[Cognitive_tutor_geometry | Cognitive Tutor Geometry]] are both online learning systems that track student mastery on a particular skill and provide students with problems that help them master that skill<ref>Aleven, V., Mclaren, B., Roll, I., and Koedinger, K. (2006). [http://content.iospress.com/articles/international-journal-of-artificial-intelligence-in-education/jai16-2-02 Toward meta-cognitive tutoring: A model of help seeking with a Cognitive Tutor]. ''International Journal of Artificial Intelligence in Education, 16''(2), 101-128.</ref><ref>Koedinger, K. R., and Aleven, V.  (2007). [http://link.springer.com/article/10.1007/s10648-007-9049-0 Exploring the assistance dilemma in experiments with cognitive tutors]. ''Educational Psychology Review, 19''(3), 239-264.</ref>. After the system detects that the student mastered a skill, it selects a different skill for the student to practice. The [[ASSISTments]] online learning system provides an IF-THEN-ELSE functionality that allows content creators to control the problems assigned to a student according to student performance <ref>Donnelly, C.J.  (2015). [https://www.wpi.edu/Pubs/ETD/Available/etd-042315-135723/unrestricted/cdonnelly_thesis.pdf Enhancing Personalization Within ASSISTments (Doctoral dissertation)].</ref>. This functionality allows students to be assigned problems that practice a particular skill and switch to another problem set after mastering the prior skill.
A concrete example of applying the pattern would be a teacher designing homework for her class. She can design a problem set that helps students practice decimal addition and another set for decimal subtraction. When students answer these problems, an online learning system may track the number of problems the student answers correctly. When a student answers three problems right in a row for example, then the student can advance to decimal subtraction problems. Otherwise, the student continues practicing decimal addition problems.


==Related patterns==
==Related patterns==
The pattern can be used in conjunction with '''[http://csis.pace.edu/~bergin/PedPat1.3.html#spiral Spiral]''' to help students master a subset of the larger topic through practice, before moving on to the next subtopic.
Make sure the system implements the {{Patternlink|Just Enough Practice}} design pattern when students are asked to use the {{Patternlink|Try It Yourself}} or {{Patternlink|Build and Maintain Confidence}} design patterns. When implementing the {{Patternlink|Just Enough Practice}} design pattern, move on to more challenging problems after mastery using the {{Patternlink|Personalized Problems}} design pattern. The {{Patternlink|Differentiated Feedback}} or {{Patternlink|Worked Examples}} design patterns may be used to facilitate learning.


==Example==
A teacher designs homework with different types of math problems (e.g., decimal addition, subtraction, multiplication, and division). He/she can use the online learning system’s control mechanism to switch between problem types whenever a student shows mastery on a particular type. The number of times a student answered each problem type correctly can be used to identify mastery. For example, if the student correctly answers 3 decimal-addition problems, then the student will be asked to advance to decimal-subtraction problems. Otherwise, the student will continue answering decimal-addition problems.


==References==
==References==
<references/>
<references/>


[[Category:Design_patterns]] [[Category:ASSISTments]]
==External Links==
*[http://assistments.org ASSISTments]
* [http://www.carnegielearning.com/learning-solutions/software/cognitive-tutor/ Cognitive Tutor Software]
 
 
[[Category:Design_patterns]] [[Category:ASSISTments]] [[Category:Full_Pattern]]  [[Category:Pattern Language for Math problems and Learning Support in Online Learning Systems]] [[Category:Online Learning System]] [[Category:Intelligent Tutoring System]]

Latest revision as of 14:42, 5 June 2017

Just Enough Practice
Just enough practice.png
Contributors Paul Salvador Inventado, Peter Scupelli
Last modification June 5, 2017
Source Inventado and Scupelli (in press 2015)[1]; Inventado and Scupelli (2015)[2]
Pattern formats OPR Alexandrian
Usability
Learning domain General
Stakeholders Students, Teachers, System developers
Production
Data analysis Student affect and interaction behavior in ASSISTments
Confidence
Evaluation EuroPLoP 2015 shepherding and writing workshop
PLoP 2015 shepherding and writing workshop
Talk:ASSISTments
Application ASSISTments
Applied evaluation ASSISTments

Allow students to practice a skill until they master it then switch to another skill in order to avoid over practice[1][2].

Context

Content creators for Skill Builders design problem-solving activities that facilitate student mastery of a particular skill. Skill Builder problem sets require a student to achieve three correct answers consecutively in order to move on to new assignments while continuing to provide struggling students with extended practice.

Problem

Students cannot maximize their learning time if they are asked to practice skills they already mastered.

Forces

  1. Diminishing returns. Students learn more when they initially practice a skill, but eventually learn less as they master the skill through continued practice[3][4].
  2. Over practice. Students’ learning gains are almost the same when they either stop practicing a skill after mastery or over practice it[5].
  3. Limited resources. Student attention and patience is limited so they may switch to other tasks if they feel they are no longer learning from an activity[6][7].


Solution

Therefore, give students problems to practice a skill until they master it then give them new problems to practice a different skill. There are different ways to assess mastery such as students’ performance on a skill-mastery test, or a statistical model’s prediction of student mastery[8].

Consequences

Benefits

  1. Students get enough practice to master a skill.
  2. Students do not spend unnecessary time over-practicing a skill when it does not contribute to learning gains.
  3. Students make better use of their time by learning more skills in the allotted time

Liabilities

  1. The online learning system needs to support the measurement of skill mastery before the pattern can be applied.
  2. If skill mastery is incorrectly predicted, the learning system can cause over-practice on a skill or worse, prevent students from practicing a skill enough before mastery.
  3. Aside from creating problems to practice a particular skill, content creators will also need to prepare problems that target other skills students are asked to learn.

Evidence

Literature

An experiment conducted by Cen and colleagues[5] revealed that students had similar learning gains regardless if they over-practiced a skill or stopped practice after mastery. However, it took less time when students stopped practice after mastery. They suggest that students should switch to learning new skills instead of over practicing already mastered skills.

Discussion

Shepherds, writing workshop participants, and learning system stakeholders (i.e., data mining experts, learning scientists, and educators) agreed that over-practice could be common among online learning systems, and adapting problems to student mastery could address this problem.


Data

According to ASSISTments math online learning system data, frustration correlated with students repeatedly answering problems they already mastered.

Example

Some online learning systems measure students’ skill mastery to help control the amount of practice provided. For example, Cognitive Tutor Algebra and Cognitive Tutor Geometry are both online learning systems that track student mastery on a particular skill and provide students with problems that help them master that skill[9][10]. After the system detects that the student mastered a skill, it selects a different skill for the student to practice. The ASSISTments online learning system provides an IF-THEN-ELSE functionality that allows content creators to control the problems assigned to a student according to student performance [11]. This functionality allows students to be assigned problems that practice a particular skill and switch to another problem set after mastering the prior skill.

A concrete example of applying the pattern would be a teacher designing homework for her class. She can design a problem set that helps students practice decimal addition and another set for decimal subtraction. When students answer these problems, an online learning system may track the number of problems the student answers correctly. When a student answers three problems right in a row for example, then the student can advance to decimal subtraction problems. Otherwise, the student continues practicing decimal addition problems.

Related patterns

Make sure the system implements the Just Enough Practice design pattern when students are asked to use the Try It Yourself or Build and Maintain Confidence design patterns. When implementing the Just Enough Practice design pattern, move on to more challenging problems after mastery using the Personalized Problems design pattern. The Differentiated Feedback or Worked Examples design patterns may be used to facilitate learning.


References

  1. ↑ 1.0 1.1 Inventado, P.S. & Scupelli, P. (in press 2015). A Data-driven Methodology for Producing Online Learning System Design Patterns. In Proceedings of the 22nd Conference on Pattern Languages of Programs (PLoP 2015). New York:ACM.
  2. ↑ 2.0 2.1 Inventado, P.S. & Scupelli, P. (2015). Data-Driven Design Pattern Production: A Case Study on the ASSISTments Online Learning System. In Proceedings of the 20th European Conference on Pattern Languages of Programs (EuroPLoP 2015). New York:ACM.
  3. ↑ Rohrer, D. and Taylor, K. (2006). The effects of over-learning and distributed practice on the retention of mathematics knowledge. Applied Cognitive Psychology, 20, 1209--1224.
  4. ↑ Sweller, J. (2004). Instructional design consequences of an analogy between evolution by natural selection and human cognitive architecture. Instructional science, 32(1-2), 9-31.
  5. ↑ 5.0 5.1 Cen, H., Koedinger, K. R., and Junker, B. (2007). Is Over Practice Necessary?-Improving Learning Efficiency with the Cognitive Tutor through Educational Data Mining. Frontiers in Artificial Intelligence and Applications, 158, 511.
  6. ↑ Arnold, A., Scheines, R., Beck, J. E., and Jerome, B. (2005). Time and attention: Students, sessions, and tasks. In Proceedings of the AAAI 2005 Workshop Educational Data Mining (pp. 62-66).
  7. ↑ Bloom, B. S. (1974). Time and learning. American psychologist, 29(9), 682.
  8. ↑ Yudelson, M. V., Koedinger, K. R., and Gordon, G. J. (2013). Individualized bayesian knowledge tracing models. In Artificial Intelligence in Education (pp. 171-180). Springer Berlin Heidelberg.
  9. ↑ Aleven, V., Mclaren, B., Roll, I., and Koedinger, K. (2006). Toward meta-cognitive tutoring: A model of help seeking with a Cognitive Tutor. International Journal of Artificial Intelligence in Education, 16(2), 101-128.
  10. ↑ Koedinger, K. R., and Aleven, V. (2007). Exploring the assistance dilemma in experiments with cognitive tutors. Educational Psychology Review, 19(3), 239-264.
  11. ↑ Donnelly, C.J. (2015). Enhancing Personalization Within ASSISTments (Doctoral dissertation).

External Links

Retrieved from "https://www.learningenvironmentslab.org/openpatternrepository/index.php?title=Just_Enough_Practice&oldid=3917"