News
Increasing Academic Efficiency
by Ryan Wisnesky
News Staff Writer

On Wednesday, November 18, Computer Science Professor Emeritus Jeffrey Ullman spoke to a crowd of mostly faculty and staff in the Packard Enginerring Building. Professor Ullman addressed the "inefficiency" of modern teaching methods. During the lecture he presented some of his research toward developing tools to better leverage computer technology in undergraduate courses.

Professor Ullman joined the Staford faculty in 1979, and has since then both retired and later been recalled to active duty. He gave a short description of his research project before discussing why it is needed. His project, dubbed the OTC -- Online Testing Center -- is currently used for several classes at Stanford, including CS 145: Databases, and is used by several other universities worldwide. The software is essentially an online database of "root questions" and problem hints. "Using 'root questions', we can turn the ordinary long answer question into a multiple choice question," said Ullman. The goal of the OTC is to automate grading of long answer questions by posing them in a manner that a machine can grade; essentially, to answer a root question, a student must work out a long answer question behind it. Additionally, the OTC also automates other aspects of grading, for instance, grading of student programs.

Before delving into just how a root question is written or how the OTC uses them, Professor Ullman described what he saw as a crisis in educational methods. He asserted that "it's hard to find an industry that has a worse record in education with exploiting technology." Comparing the education industry to the telecommunications industry, he noted that his education at an Ivy-Leauge school in 1979 cost the equivalent of 400 phone calls across the country (in 1959 a call cost $3 and his education cost $1,200). Now, in 2003, the education is equivalent to 200,000 cross country phone calls (a call is $.15 and an education $30,000). "It's 5000 times more expensive," said Ullman.

Professor Ullman also addressed the objection that the Telecommunications industry is one of the best exploiters of technology. He proceeded to perform the same comparison with the US Postal Service. In 1959, his education cost the equivalent of 15,000 8-cent stamps. In 2003, it costs as much as 81,000 37-cent stamps. His education is thus more than five times as expensive relative to a stamp as it was in 1957.

Having finished justifying the need for improvement in education efficiency, Professor Ullman began explaining exactly what is currently wrong with the education process. Speaking about his course on databases, Professor Ullman described the grading process as one where "TAs try to guess what would happen if we passed this [a student's homework] to a database". This type of grading tends to be difficult, and the rate of regrades is quite high. In fact, some TAs have, in the past, created automated systems for grading. Taking this idea further, Professor Ullman has constructed the OTC to be able to accept student programs and actually provide feedback about what in particular is good or bad about the program. If a program is incorrect, the student receives an error message, along with the output of their program, and the correct program, on a different database. In this manner, the student receives good feedback without being able to use the feedback to cheat when trying the program again. The system has been generalized to be able to grade student submissions in a variety of fields, including some as esoteric as Abstract Relational Algebra.

With this aspect of the OTC system, students are able to resubmit their programs as many times as they like, each time receiving feedback. "The whole philosophy behind the OTC is that it is not to grade them as to help them learn. They can submit as many times until they get it right," said Professor Ullman. There are, however, some constraints built into the system to prevent abuse. For instance, in CS145, students cannot submit homework more often than once every ten minutes. Instructors also have the option of imposing a limit on how many times homework can be submitted.

Having described this aspect of educational automation, Professor Ullman finally turned to the issue of "root questions" and how to grade long answer questions automatically. In a traditional class, a question might be posed as "Compute x", where the computation is quite long and complex. The root question version of this same problem would be "which of these values are in x", where to determine if a value is in x would essentially require computing x from scratch. The question then becomes a multiple choice one. In keeping with the education mission of the OTC, students can submit their work multiple times; however, each time a question is presented to the student, the answer choices are different. That way, a student cannot simply try all of the answers to determine which is correct; moreover, each time a student redoes a homework, the answers to all of the questions are different. If desired, the student may also receive different questions.

Since each time the answer choices are different, there is a great incentive to actually perform the calculation; if the student has performed the calculation, then it is easy for them to answer the same question with different answer choices. Each time a student submits an incorrect answer, they receive a hint about why their answer is wrong. When the student finally submits a correct answer, they receive an explanation about what the problem is trying to test and an explanation of how the instructor envisioned the students solving it.

Professor Ullman said that he has found that homework of 4-6 questions works best. With less than four questions, students ask "Can I get the right answers without understanding the subject matter?", he said. With more than six questions, students tend to miss questions from sheer carelessness, and thus have to repeat homework that they actually understand. Students are encouraged to get perfect scores.

Last fall, the OTC was used by CS145 for 11 assignments. This fall, eight universities have begun to use the software.

Page last modified on Thursday, 02-Mar-2006 00:27:12 MST.