The most important thing one wants to know about an educational program is whether or not it is producing the kind of learning its designers are aiming for. Yet, most evaluation methods fall short of providing this information.

There are three important components to learning:

acquiring knowledge;
learning to reason well with that knowledge; and
using that knowledge effectively in real-world contexts.

We document changes in knowledge and reasoning

If you provide or purchase training for adults, you are almost certainly familiar with Kirkpatrick's requirements for program evaluation. Many educational program evaluations satisfy one of the requirements of level two in Kirkpatrick's model by including assessments of factual knowledge, but few examine how well students learn to reason with newly acquired knowledge or how students apply that knowledge. Our cognitive developmental assessments help to fill this gap. In fact, they not only satisfy assessment requirements for Kirkpatrick's level 2—learning—but also satisfy the requirements for level 3—transfer, because they focus on the application of knowledge to real-world problems.

If you develop curricula for K-12 (or higher education), you know how difficult is tis to build assessments that get at the kind of learning you know is most valuable—the kind that translates into real-world skills and deep understanding. Our assessments fill this gap. To learn more about how they work, visit DiscoTest.

Determining how well students think about what they know

Cognitive scientists have known about the importance of good thinking skills for decades, and numerous developmental researchers, including Baldwin, Kitchener & King, Piaget, Fischer, and Commons, have shown how reasoning skills develop over time. But until recently, cognitive scientists were unable to devise an accurate and affordable way to measure the development of these skills.

In the mid-nineties, Dr. Dawson decided it was time to make developmental assessment practical and affordable. Several years of research yelded a single, accurate measure—the Lectical™ Assessment System (LAS)—based upon the developmental levels described by Fischer. The system can be used to evaluate thinking skills in any area of knowledge. In fact, its psychometrics have been the subject of several peer reviewed articles and it has been used in numerous research and applied settings.

Our approach to assessment in evaluation contexts

Working closely with curriculum developers, we design educative assessments that can be embedded in curricula and employed by students, teachers, and evaluators as indictors of student learning. All assessments require students to show their reasoning within the particular subject areas targeted by the curriculum. They also ask students to use their knowledge to reason through relevant situations or problems encountered in real-world settings.

Students take these assessments at least twice during (and sometimes after) instruction. The assessments are analyzed for their (1) developmental phase and (2) conceptual content. Results can be subjected to numerous forms of psychometric and statistical analysis.

Beyond evaluation: Describing learning sequences

The same tools that we use to assess current skills have been used to investigate how people learn these skills. Our assessments double as data collection devices that can be employed to study how concepts are actually acquired. When we put together assessments of current skills with knowledge about the pathways through which these skills are learned, we can tell individuals what they are ready to learn next and point them to suitable materials, coursework, and activities. Contact us if you would like to view some sample reports.

We can also use this knowledge to contribute to curriculum design. In conventional approaches to curriculum design, content experts work together to determine the sequence in which material should be learned. We think that content expertise needs to be supplemented with solid research into how students actually learn concepts. By combining the knowledge of content experts with knowledge gleaned from research on conceptual development, we can identify the learning tasks and the optimal order(s) of instruction for any knowledge domain.

Beyond evaluation: Contributing to curriculum design

Contributing to curriculum design involves the following:

Working with clients to identify an appropriate base of expert knowledge in a given domain;
Employing the expert knowledge-base to identify the skills that make up a given domain;
Organizing these skills into a skill hierarchy;
Researching the development of each of the skills and sub-skills identified in the skill hierarchy; and
Describing the developmental sequences for building each of the skills; and
Refining the skill hierarchy and adding descriptions of each level of skill.

Once these staps have been taken, we can work closely with curriculum developers to design lesson plans and assessments that both promote and capture optimal learing.

Example: Evaluating a critical thinking curriculum

A few years ago we evaluated a very popular and expensive adult critical thinking course for a government client and found no difference in the way employees approached common workplace decision making situations before and after participating. The client was surprised, because the exit surveys given to students who had taken this course showed that they thought participating in the course had improved their decision making skills.

Our client had been relying on exit surveys for many years, believing they were a useful way to assess adult learning. Perhaps more important, they had not had alternative affordable ways to asses real skill development. Our tool made it possible for that client to assess the real “value added” of an expensive course.

Example: Evaluating a decision making curriculum

Concerned that first line, mid-level, and upper level managers were not demonstrating adequate decision making skills, one of our clients asked us to help design a new decision making course and examine how it performed relative to an existing course. The new course was based on research into (1) how particular workplace decision making skills develop, and (2) how people build new skills over time. The decision making skills of students who took the new course grew twice as much as the skills of students who took the conventional course.

But that’s not all. Students who started out with skills that were at the level targeted by both courses were more likely to learn the skills as intended than students who started out with skills that were not as well developed. We have shown the same pattern in other learning situations. The bottom line is that an educational investment is most likely to pay off when students are ready for the material covered by a course. Our assessments can help determine who is ready.

Research partnerships

If you are conducting a program evaluation in an area that happens to coincide with questions we're particularly interested in addressing, we may be able to provide our services at a reduced rate. If you are working on a particularly appealing developmental question or issue, contact us.

Working with DTS

If you are interested our program evaluation services, would like to follow up on a special offer, or would like more information, email us anytime.

We offer a 20% discount to all non-profit organizations that can provide documentation of their status.

If you would like to keep up with program evaluation at DTS, please join our mailing list.