Subjective and Objective Reviews of Instructional Materials

Alan Griffin, SIMRA President, July 2013

The Association of American Publishers (AAP), in cooperation with Association of Educational Publishers (AEP), recently posted a video ( ) about the emerging collection of metadata, and how the Learning Resource Metadata Initiative (LRMI) is developing standards for it.  A stated goal of the initiative is to “make it easier and more convenient to find learning resources that meet specific student and class needs.”   The use of metadata in the search for these materials is predominantly based on objective data which is fundamental factual information about the items themselves, such as author name, publisher name, number of pages, copyright date, etc.  In the use of metadata, common descriptors are sought to provide selection criteria.  LRMI seeks to streamline the search process, and greatly shorten the time spent by all patrons of education in locating appropriate instructional materials.

Search engines that utilize metadata have become a fixture for society generally, and for the educational community specifically.  It has become increasingly easy to locate materials based on carefully selected descriptors.  While these descriptors are extremely helpful in locating materials in a particular category, and even can help in determining usage statistics, they tend to focus on very generic and objective information that does not provide personal, subjective opinions about the resources.   Teachers are interested in the number of pages in a resource and its chapter designations, publishing date, authors, etc., but more important to them is narrative information from their peers, those who understand pedagogy and best practices in instruction.

A key feature of the instructional materials review process is checking on alignment of content to core standards and objectives.   Publishers may submit alignment data, designating particular page numbers, segment designators, or URLs where subject matter for particular course objectives is presented.  Such data is subject to review by curriculum specialists to insure that the objectives are completely and adequately addressed.  A highly qualified classroom teacher’s expertise is imperative in verifying this alignment.  They provide reliable, subjective judgments from professionals in the field that certify that the content is appropriate and complete enough for the specified indicator, objective or standard3.  This professional opinion is stated in a narrative that is the integral part of the review.  It creates meaning from the objective descriptors and describes how the materials are designed to be used in practical terms.  It provides a human face for the data.  Subjective reviews submitted from highly qualified educators as well as independent reviewers are valuable elements that could be, and should be, included as unique data that can then be characterized with metadata descriptors.

The rapid development of metadata resources has led to much interesting discussion regarding future review and selection trends.  There are sometimes fine lines drawn between subjective and objective data.  Objective data are increasingly being used to measure subjective elements.  A simple Google search may quickly identify the popularity of a given text by the number of items returned in a search.  Google analytics can return the number of hits on a website, and where they are coming from.  Searchers often draw conclusions based on the data requested.  This paradata3 is oftentimes used to formulate or back up subjective arguments.  It is, of course, desirable that such arguments are well documented with accurate source materials and statistics.  The real question is the extent of human intervention required to develop such arguments.  Programmed algorithms are developed by humans after all, and mechanisms that put them into action reflect the design of the individuals and groups that created them.   The question then arises as to what true objectivity really is.

Searching often becomes a popularity contest.  It is hard not to make choices based on what the vast majority is choosing.  We all have a tendency to believe that what is good for the greatest number is good for all.  Data driven decision making may seemingly ignore the rights of the minority.  Perhaps it is just a matter of getting enough data.  It seems that the lack of data constitutes a minority.  Facebook has certainly well portrayed the advantages of a vast array of connections.  Linked-In, Yahoo, Twitter and a host of others are constantly touting the advantages of multitudinous connections.  More is apparently always better.  Automated is better than manual.  Many are better than one.  In selection procedures numerous uses of resources imply quality.  In this age of spiders and crawlers that search for data, it is important not to confuse quantity with quality, even though paradata can portray how resources are being used.

Once the metadata has helped us reveal candidates for the instructional materials review process, subjective review by highly qualified subject matter specialists is in order.  Three general areas of concern should be addressed by them: (1) the alignment of content with core objectives, (2) the accessibility of the resource to all, (3) the ease of adapting the resource to appropriate teaching pedagogy.  Careful consideration in these three areas will create the foundation for a quality review.

Aligning content with core objectives would seem to be a fairly straight forward process.  All that is needed is an understanding of the indicators, standards, and objectives and knowledge about the content and how it addresses them.  Unfortunately, a casual reading of the standards is rarely enough to arrive at a thorough understanding of them.  Examples and references often need to be provided so that there is little room for misinterpretation.  As a case in point, the Common Core standards for Language Arts require students to become skilled in argumentation.   Publishers have responded by producing materials that reflect varying perspectives of what argumentation is.  Some have encouraged the development of emotional, persuasive narratives without understanding that the standard requires that pure argumentation relies on the presentation of facts on both sides of an issue resulting in a student’s ability to back up a point of view with data, rather than emotion.  Alignment is best accomplished by highly qualified teachers who understand the core thoroughly.

Accessibility guarantees are important to insure that students from all sectors of society have equal opportunity to learn and achieve.  With the advent of high stakes testing, it is imperative that all students have the best possible instructional materials to learn from in order for them to perform well.  Teachers too benefit significantly as their performance is related to student achievement.  Materials, then, should be available on a wide variety of platforms to allow all students the opportunity to interact with them.  Materials must be free of bias and be sensitive to political or religious issues that may alienate segments of the population3.  Local, experienced reviewers are essential in this process.

Once alignment with standards has been verified and materials have been made freely accessible, they need to be evaluated in terms of their ability to fit into effective pedagogy.   Organization and design should be considered so that materials are easily adaptable for individual teachers and individual learners.  Clear, engaging presentation is also an important consideration that should be directed toward the core objectives and standards.  Technology should not distract from lesson objectives and content meaning.  Suggestions for teaching strategies should be included.  Teacher reviewers are best qualified to render subjective judgment, based on objective data in these areas.  Online searching should help them differentiate instruction and expand on learning experiences for individual students, enabling them to reach higher levels of achievement.

Metadata3 is treated as being objective, without being tainted by opinion.  In truth, the authors of the algorithms structure them to produce a desired result, and those who provide the tags for metadata supply subjective judgments in the words they choose as tags.  It could be argued that the whole process is every bit as much subjective as objective.  Perhaps the teacher/evaluator role will become that of the tagger.  Certainly as tagging becomes more collaborative, with many evaluators agreeing on terms to describe materials, it also will become more objective, at the very least providing a group opinion.  But at its base level, subjective opinions still have their place, and are the essence of the instructional materials review process.


Works Cited

  1. Allington, Richard L. “What I’ve Learned About Effective Reading Instruction From a Decade of Studying Exemplary Elementary Classroom Teachers.” Richard L. Allington, 2002. Web. 26 June 2013. <>.
  2. “Paradata in 20 Minutes or Less – Google Drive.” Paradata in Twenty Minutes or Less 1.1 (2011): 1-8. Paradata in 20 Minutes or Less – Google Drive. U.S. Department of Education, 2011. Web. 27 June 2013. <>.
  3. United States. Equity and Civil Rights. WASHINGTON MODELS FOR THE EVALUATION OF BIAS. N.p., Sept. 2009. Web. 19 June 2013. < >.
  4. “Dublin Core Metadata Element Set, Version 1.1.” Dublin Core Metadata Element Set, Version 1.1. N.p., 1995. Web. 26 June 2013. <>.