What are the differences between formative evaluations and summative evaluations? At which stage in product development do you believe that evaluations should use controlled settings instead of natural settings? Why?
Be sure to respond to at least one of your classmates posts.
student post
Christopher Owens
Greetings Dr. D and Class,
Formative evaluations are conducted during design or redesign phases while summative evaluations are conducted to assess the success of a finished product (1). For example, when developing the first iPad, formative evaluations were conducted to ascertain the usability specifically focusing on apps over browser. Summative evaluations were completed a year later to help identify usability issues with apps and websites and have more interaction between developers and the Apple iPad development team as there was little previous interaction due to the required secrecy about the products launch.
In regards to the stage controlled settings should be used versus natural settings, I believe the answer depends on what question the test is attempting to answer. For example, field studies or in-the-wild studies can help identify opportunities for new technology, help establish requirements for a new design, or facilitate the introduction of technology (1). First, this is because controlled settings cannot account for how people will react or use a product in their daily lives. Second, behaviors can change as people interact with a product such as problem-solving strategies as they become more familiar and comfortable with a tool (1). Controlled settings are good at discovering usability problems (1). Returning to my opening comment relating to what question the test is attempting to answer, there are issues that can also influence the method of evaluation:
- Controlled settings have a high reliability of producing the same results on different occasions while natural settings may not.
- Ecological validity is concerned with how an environment can influence the results such as the Hawthorne effect of when participants are aware that they are being studied. Lab experiments have a low ecological validity while ethnographic studies have a high ecological validity.
- Bias can interfere with the results which can effect controlled and natural settings.
- Time can influence the evaluation method.
- Funding can influence the evaluation method.
Overall, using a combination of methods across categories can help designers better understand issues and opportunities such as usability tests combined with observation in natural settings
Ulrich and Yang describe the following product development process: planning, concept development, system level design, detail design, testing and refinement, and production ramp-up (2). I believe natural settings can be useful in answering questions in planning (trying to determine opportunities), concept development (developing opportunities), and production (observing how users naturally use a product). While controlled settings my be better at evaluating products in the conceptualization, system level design, detail design, and testing and refinement phases due to the specificity of goals as they relate to features and functionality more centered around the usability of the product being developed in its design stages (2). All in all, there are stages that can benefit from using a combination of methods to help designers get a fuller understanding of the usability issues and opportunities. However, what method or methods that are used will be determined by the questions that need to be answered (1).
1. Helen Sharp, Jennifer Preece, Yvonne Rogers. 2019. Interaction Design, 5th Edition. Strayer University Bookshelf, (8th Edition). Wiley Professional Development (P&T).
2. Karl T. Ulrich, Maria C. Yang. 2020. Product Design and Development, Seventh Edition. McGraw-Hill.