The 2016 AP Computer Science A Test Reviewed by an Old Curmudgeon

I did the AP Computer Science A free response test while watching basketball, baseball, and hockey games on the teevee.  It took me about 3.5 hours, but that included an overtime basketball game, an overtime hockey game, a triple-overtime hockey game, and a lot of channel-switching.

Realizing that it is much easier to be an armchair whiner than it is to write a test of this nature, this test still warrants criticism.  To the credit of the AP people, they listened to people and recognized that the multiple choice portion of the test was too long for the amount of time provided and they moved from 75 to 90 minutes this year.  They also recognized that the free response portion of the test was too short–it really was way too short for years–and they reduced the amount of time for that from 105 to 90 minutes.  So far so good.

The problem is that if you are going to write a 90 minute test, it should probably mean 90 minutes for a good computer science student, not 90 minutes for a good computer science teacher.

For the reader with some programming expertise, it probably makes sense to have a browser tab open with the test to follow along.

QUESTION: Did anyone give this test to high school teachers to find out how long they thought it would take?  I’m serious about this.  I just went through a thread on Facebook where top teachers at top schools were expressing concern about how long the test took them and the kinds of errors they made.

QUESTION: I thought I taught topics other than string operators this year.  Was this a mistake?  I’ve never written a test with this much emphasis on strings and I hope I never come close.

#1: My students should do pretty well on this problem.  (Maybe I’ll learn whether they internalized it based on the test results.)  I’ve actually been giving this problem as homework for about a decade, but returning random, unique numbers instead of Strings.  If you’re curious or want to use it yourself,

http://paleyontology.com/AP_CS/randp.html

#2: Good luck to the ELL student and the student who gets bogged down in terminology.  This problem could have been written in about 1/3 of the space used on this test.  Just get rid of the MessageLog reference.  It’s superfluous.

In case a test writer is reading this, PLEASE be nice to ELL students.  Make sure that what you are testing is whether they understand the big ideas and core topics, not your festive tangent that includes unfamiliar and irrelevant terms.

#3: I enjoyed this one, which may be because I like crosswords. I wonder how much my familiarity with crosswords saved me time trying to understand this problem.

It is much nicer doing this problem with Eclipse than it is to do it on paper. I lost a point on the first pass for failing to properly create an array, but it was an easy catch with the NullPointerException displayed by Eclipse.

// RANT ON //
CAN WE PLEASE GIVE STUDENTS AN ORGANIC TESTING EXPERIENCE THAT INCLUDES AN INTEGRATED DEVELOPMENT ENVIRONMENT?  YOU KNOW, LIKE HUMANS USE TO DO PROGRAMMING NOW THAT IT IS NO LONGER 1981 AND WE NO LONGER NEED PUNCH CARDS?  CHECK THIS OUT!  IT’S FREE!
// RANT OFF //

#4: Imagine being a student who is feeling time pressure when landing on that first page of this problem with all those explanations, instructions, and examples. Is that how settlers felt after traversing 2/3 of the United States only to encounter the Rocky Mountains?

QUESTION: Why do we think our standardized test is any better than all of the ones that people rail against?

12 thoughts on “The 2016 AP Computer Science A Test Reviewed by an Old Curmudgeon

  1. So, let me respond to a number of your questions above.

    * Yes, these tests are field tested before they’re administered.

    * Yes, high school teachers are consulted on these exams before they’re administered. Half of the committee that writes these exams are veteran APCSA high school teachers.

    * Online test administration has tons of problems. First of all: how do you make sure that you have enough computers in your high school so that every student gets their own computer during the exam? If you’re lucky enough to have multiple sections of APCSA in your school, you probably don’t have enough machine. Then, how do you secure the machines so that students don’t go online to collaborate on answers during the test — oh, but you *do* want them to go online to access the API, right? Then, what happens when a student’s machine mysteriously crashes in the middle of the exam, and all of the student’s work is lost? Finally … what happens when you get a student who spends most of their time on a stupid syntax error, and fails the exam because they can’t figure out they stuck an extra semicolon in the wrong place?

    As primitive as paper-and-pencil exams are, they’re much simpler to administer and score than the alternative.

    1. As far as the review process goes, it is unfortunate that the test seems to have been on the long side. Two pieces of feedback I offer are echoed elsewhere: (1) the emphasis on strings seems a bit much and (2) problems that introduce language that is not germane to computer science are going to take extra time to ingest, especially for ELL students.

      As to online testing, I think there are some answers and some questions.

      “First of all: how do you make sure that you have enough computers in your high school so that every student gets their own computer during the exam? If you’re lucky enough to have multiple sections of APCSA in your school, you probably don’t have enough machine. ”

      My school is probably a bad example, but we have two computer labs and several laptop carts. It is an affluent district, so I assume that this does not generalize. That said, how much of an issue is this? We now have a rule that says that a student has to do 20 hours of lab time to take AP CS A, so I would think that almost every school has at least one functional computer lab. I would be curious to learn how many schools have additional computing labs, mobile laptop carts, etc. Given enough time to plan, can’t laptops be rented well in advance in those places where there are insufficient machines?

      “Then, how do you secure the machines so that students don’t go online to collaborate on answers during the test — oh, but you *do* want them to go online to access the API, right?”

      There is online and then there is on the Internet. The API can be installed on local machines, which could then be disconnected from the Internet.

      “Then, what happens when a student’s machine mysteriously crashes in the middle of the exam, and all of the student’s work is lost?”

      I don’t have an answer for this one. I would be curious to learn what the probability of that happening is. In my classroom, I haven’t had a student lose a session due to a hardware crash in at least a decade. That said, my anecdote may not generalize and your point is well-taken.

      “Finally … what happens when you get a student who spends most of their time on a stupid syntax error, and fails the exam because they can’t figure out they stuck an extra semicolon in the wrong place?”

      Perhaps use Eclipse? One of the benefits of a good IDE is to get away from teaching syntax.

      That said, your point is salient. If a student produces a minor logic error and cannot figure it out, that can be costly. Perhaps that student would score 8/9 on a paper test and we do not want that student to spend too much time to get that last point. Couldn’t timeouts can be enforced by software to limit that downside?

      I suppose that, at the end of the day, there is something that must be measured and if people feel that paper captures it sufficiently well, that will rule the day. It’s just very hard for me to feel good about an assessment process that does not seem to reflect normal practice.

    2. MC questions are field tested. FRQ are not, as a general rule. If I remember correctly, I think some FRQ are field tested at times, but not on a yearly basis.

  2. On the content stuff, I’ve been responding to a lot of that elsewhere, but I can respond to the IDE question. Aside from a whole big pile of questions like which IDE allowed, locking down the computers, etc, etc, etc, the biggest question here is where are all the computers coming from? There was at one point some discussion of developing the APCSP exam—brand new—as computer-based, but they abandoned that because a lot of schools didn’t have the lab space and computer resources to give it. Even (maybe especially) in the well-off schools, the ones that are big and funded enough to have multiple sections of APCSA, might not have the lab space to house all those classes *concurrently* as would have to happen in a test administration. And you mentioned the ESL learners’ disadvantage on these exams; can you imagine the disadvantage of the students at schools with slower vs faster computers, or old versions of the IDE, or whatever?

    As artificial as it is (and it *is* artificial), I think paper-based administration is with us for a long time to come, and is the best solution to a thicket of thorny problems.

    1. Let me try to take on some of Don’s concerns.

      “the biggest question here is where are all the computers coming from? There was at one point some discussion of developing the APCSP exam—brand new—as computer-based, but they abandoned that because a lot of schools didn’t have the lab space and computer resources to give it.”

      Don’t schools have to have the lab space to meet the 20 hour requirement that is explicitly stated in the AP CS A course description? My understanding is that the discussion of an online test was before this requirement was in place. At this point, it is hard to imagine that many school that have AP CS A simultaneously lack sufficient computers.

      “Even (maybe especially) in the well-off schools, the ones that are big and funded enough to have multiple sections of APCSA, might not have the lab space to house all those classes *concurrently* as would have to happen in a test administration.”

      I think that is probably wrong. I would expect that most affluent schools have a fair number of laptops to go along with whatever they put in standard computing labs (desktops, laptops). In any case, an affluent school with sufficient advance notice can solve this problem. This is a matter of logistics and there are ways to manage it pretty easily.

      “And you mentioned the ESL learners’ disadvantage on these exams; can you imagine the disadvantage of the students at schools with slower vs faster computers, or old versions of the IDE, or whatever?”

      I just don’t agree with the premise here. Eclipse runs on old computers (10 years old, say) just fine. My experience with ESL students is that they do just fine when programming and the hard part is understanding what they are being asked to do. Old versions of Eclipse are fine, too. I would argue that we could make Eclipse part of the testing expectation. It’s really useful even if the overwhelming number of bells and whistles are ignored. Even if we don’t go this route, we could provide a list of supported IDEs that are commonly used. I do not see this as a reason to keep tests on paper.

      The process of locking down the testing computers could consist of putting the Java SE documentation and Eclipse (and other IDEs if desired) on the machines and then disconnecting them from the Internet until the test is over. In the absence of a less crude method, have students copy and paste their solutions to a text file and then have the test administrators upload that file once the students have left the test site. I’m oversimplifying, but it wouldn’t take that many directions and they could be written up in plain English so that someone with no systems administration skills could still implement them.

      In addition to the organic programming experience, the range of problems that could be tested could make for a *fun* test with creativity. Creativity should be a core CS concern as suggested by AP CS Principles. It should not, however, be confined to AP CS Principles. Have the kids complete the writing of a game, for example.

      Consider graphics. We do nothing with graphics, but the crossword problem on the 2016 AP CS test could have been presented in a GUI. Imagine if the student’s code actually produced something that *looked* like a crossword. The student could test their understanding of the problem with a visual. Good luck with that on paper.

      I would fully expect some speed bumps in the first year and the criticism that goes along with it, but this really should be doable. With a skilled team of techies, it could probably be done in 2-3 months. With a Google employee calling this a 20% project it might be done in 2-3 weeks. Or 2-3 days, maybe?

  3. Jim, I’m not so sure that the Free Response Questions are pretested by the CB, because they’re so memorable.

  4. A few thoughts here:

    First, doing the test on computer is really a no go for reasons stated here, in the Facebook thread and beyond. Doesn’t matter if it’s better or worse, it’s just not practical in so many places.

    One comment on your thought on timing — I’d go further. If an exam is 90 minutes, a good student, should finish with 5 to 10 minutes to spare and an average student should finish the exam with 3 to 5. Tests SHOULD NOT be races. Too often, higher grades go to the speediest not the most knowledgeable, creative, or thoughtful.

    Students should have adequate time to express what they know and double check for careless errors — it doesn’t mean everyone will ace the test – there will be wrong answers but it won’t be because a kid was rushing to finish before the bell rings.

    On vetting the test, I do wonder about the credentials of the test makers – 3 high school teachers and 3 college. I don’t care how long someone’s been teaching APCSA – that’s not a credential – I know a lot of people teaching for years that are good and a lot that are bad.

    This is yet another problem with AP – no accountability – it’s the same BS as with Pearson standardized tests – made in secret. Rarely release multiple choice. Who knows the quality and qualifications.

    This is kind of like “security through obscurity.”

    Release all the parts of all the tests information and results of field testing, engage teachers.

    You know that my program was responsible for 1% of all ABCS – AB test takers in its last year of its administration – you’d think that these AP test makers would have wanted to reach out to a single school responsible for 1% of their test takers for feedback, input, or even for them to analyze – nope, never a word.

    There are people in the high school CS Ed community who are open about their work and their beliefs – I agree with some and disagree with others but they’ve put themselves and their programs out there – not so much the college board.

  5. Mike, I’m really uncomfortable with some of that.

    It’s one thing to be critical of the exam. I think that is more than fair since thousands of students take it, crazy amounts of hours are dedicated to helping them prepare for it, and for lots of them, it’s effectively betting $115 to win $4000 (plus or minus some money) because of college credit. It’s serious stuff.

    It’s fair to ask about process. It’s fair to offer suggestions. Hell, it’s even OK to rant (although others may not agree with me on that) a bit as long as it’s not personal.

    Where I draw the line is when unambiguous accusatory language is used. If the College Board were that opaque, why do their people always seem willing to answer my questions? I’ve been doing this for 14 years, and they have always been willing to answer questions, sometimes when those questions have been phrased poorly. I’m not just talking about Jim and Don. It has been that way for as long as I have taught AP CS. Everyone has been friendly and tried to be as informative as they could when I have had tough questions.

    I’m not sure at what level of granularity you would demand transparency, but if every business were fully transparent, very little would get done out of fear that any minor error would end up blown out of proportion.

    I’d prefer to see what happens going forward before making any kind of assertions about accountability. This year’s test is done, and they’ve gotten a lot of feedback. Let them effect change.

    1. I’m talking institutionally – it’s not personal.

      Try going through channels when dealing with the curriculum approval process.

      I’ve had to navigate it for myself and with others multiple times – no transparency there.

      The College Board collects has collected large sums of money from NY taxpayers for the PSAT, now for the SAT and AP Exams as well yet exams aren’t open for public scrutiny as one example.

  6. I want to address the online testing idea. It’s hard for me to dismiss because we executed Smarter Balanced at my school last year, it seems to scale, and there is public accountability. Smarter Balanced is an online standardized testing tool and security is built into the system. Perhaps we could learn from them? Maybe license from them?

    Ideone (http://ideone.com) is proof of concept that a browser-based editor is doable.

    Tablets are way cheaper than laptops or desktops.

    An online test is feasible. The hard part will be to produce content. By the time that has been sorted out, we’re talking 3-5 years down the road and the tools for doing online testing will have been flushed out even more, presumably be easier to use, cheaper, etc.

    I urge the College Board to reconsider the possibility of an online AP Computer Science test. At least take a closer look?

  7. I’ve just read a few of the comments concerning online testing for the AP CS exam and the concerns for an insufficient amount of computers or computer labs. I am a high school student about a month from taking the AP CS exam. My school is a charter school, nearly all of our funding comes from the district. Every single student at our school, sans the graduating class, has received a brand new chromebook laptop. With the school WiFi, they are fast and definitely able to host an exam such as the AP CS exam.

    This is a school of 2,200, excluding the graduating class. I think it is possible for schools to have the proper amount of machines, it’s a matter of state funding (and whatever other funding the school can get) and allocating those funds to these relatively cheap computers.

    These computers (which are rather reliant on the cloud) offer a secure browser for online but non-internet testing, which we’ve done over the past week for the SBAC exams. So, I think moving the AP CS test online is completely doable and worth the “hassle” because it doesn’t benefit the student going into college as a CS major who will be coding and programming online. Correct me if I’m wrong but there are no longer any real life situations or occasions in which handwritten code will be needed.

  8. Could someone explain to me the rationale of the following regarding the fact when a student highly suspects he/she has not been scored fairly. I have been a system level sensor software developer, and also his CS instructor. It is NO way he could have just got 4; while he could easily write algorithms such as minimax, breath-first and depth-first search, dijkstra , etc.

    My contentions include:
    1) college board said, “You may request to have your multiple-choice answer sheet rescored by hand. That multiple-choice score and your free-response score are then appropriately weighted and combined. ..” Does it mean that they won’t rescore the free-response too !? There is almost no point to ask for rescore if they only score one section, but not another.

    2) The key is that we need to know what are the distribution of scores between the multiple-choice vs the free response. Why making it available to students becomes insecure?

    3) They gave him back the free-response.. ok.. then, how about the scores that they use !? Why not just give students their multiple choice work too? It is almost like they would try to come up with some sort combination so that you cannot say for sure that they have made a mistake.

    They refuse to let students to have neither the distribution of scores, nor the free response with scores sheet. How do they gauge the accountability there?

Leave a Reply

Your email address will not be published. Required fields are marked *