Put to the test

  • CMU researchers have gone a long way to create a new software certification--7,670 miles to be exact
By Jason Togyer

An athlete can train to run a five-minute mile, says Phil Miller, but the ultimate test is running a mile in five minutes, "and there's really no way to fake it."

The same thing applies to airplane pilots, says Miller, a project scientist at the School of Computer Science and a co-founder of distance-learning pioneer iCarnegie. Testing a potential pilot on a simulator may demonstrate mastery of certain skills, he says, but until they've successfully taken off in a plane, flown it and landed it safely, they don't get a pilot's license.

While software development holds less immediate potential for danger than flying a plane, hiring a software developer does cost money, and companies that recruit large numbers of entry-level developers need to know they can actually produce code. They're not getting that assurance from current software certification programs, argues Miller, who's been involved in IT workforce training for more than a quarter-century, including development of the master's degree program in software engineering management offered jointly by SCS, the Heinz College and the Software Engineering Institute. Miller also chaired the committee that created the College Board's Advanced Placement Computer Science offering.

At best, Miller says, existing certifications are rough assessments of ability. At worst, "someone can pass them and not be able to write software," he says. "Companies don't use them."

To create a more authentic certification--one that has developers actually demonstrating their ability to write software on deadline--Miller is going all the way to Kenya. With funding from the World Bank, that East African nation is sponsoring the creation of a new software developer certification program that Kenya's scientific and business leaders think will help market the country as a hub for information technology. The project has been named "Chipuka," the Swahili word meaning "emerge" or "spring forth." Kenya's partners are Miller and other CMU researchers, including SCS Dean Randy Bryant; Roger Dannenberg, professor of computer science; Robert Seacord, who leads SEI's secure coding initiative; Jefferson Welch, SEI's manager of professional certifications; Marsha Pomeroy-Huff and Mary Ellen Rich, also of the SEI certification team; and SCS staffers Chad Dougherty and Bill Reier.

The Kenyan partnership with CMU was something of a lucky break. Miller was already looking for help creating a new international certification in software development. Kenyan officials were marketing the country as a software development destination, but finding few takers; companies said they had no proof that IT grads from Kenyan universities were ready for the international workforce. Mutual contacts at international development agencies connected Kenya's Information & Communications Technology Board with Miller and CMU.

"Kenya's ambitions with this project are to be the leading software development center in Africa and a significant global software player by engaging our latent intellectual capital," Paul Kukubo, CEO of the Kenya ICT Board, said earlier this year. Software engineering holds "great promise for our economy," Kukubo said, "hence our choice to invest in certifying our youth."

But a test that certifies only Kenyan software developers is no test at all, Miller says, any more than a foot race with only one runner is really a race. "Kenya is trying to get into a competition where the eyes of the world are focused on the results, and it plans on winning," he says. "If only Kenya is in the competition, it's a waste of time. If this was to have been a Kenyan exam, or an East African exam or even a sub-Saharan Africa exam, the Kenyans would have spent their money poorly."

On the other hand, if the certification is accepted by multinational software companies, "they will have spent their money wisely," Miller says.

Kenya has every intention of seeing Chipuka develop an international standard for software development certification. "We want to lead from the front and be the technology partner of choice on the African continent," said Bitange Ndemo, permanent secretary of the Kenyan Ministry of Information and Communication.

Kenya's capital city, Nairobi, has long been a center for international trade and finance, and the nation's gross domestic product of U.S. $72 billion is the largest in East Africa. Despite that, Kenya has struggled over the past decade. On several occasions, the International Monetary Fund and the World Bank have denied loans to Kenya, alleging high-level corruption and graft. In 2007, political unrest following a disputed national election turned into two months of sporadic violence in which 1,500 people were killed. Kenya's economy is mostly driven by agriculture and tourism, leaving the country vulnerable to droughts and oil price fluctuations; according to one estimate, Kenya's unemployment rate is near 40 percent. 

In an attempt to change the country's direction, Kenya has invested heavily in education. In 2005, Kenya began a national system of school assessments to monitor educational progress. The country now spends about 7 percent of its GDP on education, more than all but 10 or 20 nations, and its literacy rate, though lower than those of North America and Europe, is among the highest in Africa. About 60 percent of Kenyans who complete secondary school attend one of the nation's seven public or 23 private universities.

It would be "presumptuous," Miller says, for him to try and evaluate the quality of Kenya's universities. "There is a tradition of higher education which includes computer science education, and it seems recognizably good," he says. "I would not be at all surprised that the typical test-taker from Kenya ends up performing quite well" on traditional IT certification tests.

The problem is not necessarily with the test-takers, but the tests. One popular, ISO-accredited software certification quizzes developers on 25 different areas, Miller says. "They ask questions, rather than have you practice them," he says. "They ask some questions about software requirements, for instance, which is nice, but it would be more authentic if people actually had to gather and manage and otherwise deal with requirements." Someone with proper training in software development should do well on such an exam, Miller says, but it's "at least a couple of orders of magnitude" away from actually writing software.

To develop an authentic test, Miller and his team spent more than three months interviewing more than 100 software developers in Kenya, India and the U.S., asking them such questions as which languages do you use? What data structures and algorithms have you worked with? What's your day-to-day workflow? From this role delineation study, the Chipuka team developed a list of typical tasks that an entry-level software developer should be able to perform. That list forms the heart of a questionnaire, and a large sample of professional developers will be surveyed in a formal job task analysis.

After the initial interviews, the Chipuka team developed a "pre-pilot" test to offer to software developers already working in their field. "That way, we can get some good data and see what are the tasks that actually discriminate between different skill levels, and what are the ones that are impossible for everybody," Miller says. The first pre-pilot exam was done in Nairobi on Aug. 28, when 10 highly regarded young software developers and advanced university students worked on six different tasks. Results ranged from one student who was unable to correctly write any tasks to others who went well beyond what Miller's team expected. Reactions to the test were "uniformly positive," Miller says, with test-takers reporting that they liked working on real problems, and the fact that they had time to prepare for the exam. 

A pilot exam is expected to take place in April 2013, with the full certification program up and running in Kenya by October 2013. Exam takers will be expected to prepare beforehand by downloading the code base and development tools and familiarizing themselves with the specifications. Miller envisions an exam taking about six hours ("I don't think you really ought to plan a lot of other things for that day," he says) in which a developer has to take an existing piece of software and add a new feature. For instance, one pre-pilot exam required users to write a program to read and write comma-separated-values to a spreadsheet. 

There's no timetable yet for when Chipuka might go international. To gather real-world problems and build trust, the Chipuka team is leaning heavily on collaboration with developers in the open-source software community. "I believe if what we're doing is successful, we will set a standard," Miller says. "It's a really different philosophy. Typically, in an examination, people spend a lot of resources hiding things. We're spending a lot of resources not only exposing them, but trying to get people to check them out."

-- Jason Togyer (DC'96) is editor of The Link.

For More Information: 

Jason Togyer | 412-268-8721 | jt3y@cs.cmu.edu