Variations in this system are in active use in the US as well.
Do you feel it is effective?
It seems to me that there is a massive asymmetry in the war here: proctoring services have tiny incentives to catch cheaters. Cheaters have massive incentives to cheat.
I expect the system will only catch a small fraction of the cheating that occurs.
> I expect the system will only catch a small fraction of the cheating that occurs.
The main kind of cheating we need them to prevent is effective cheating - the kind that can meaningfully improve the cheater's score.
Requiring cheaters to put their belongings in a locker, using proctor-provided resources, and being monitored in a proctor-provided room puts substantial limits on effective cheating. That's pretty much the minimum that any proctor does.
It may not stop 100% of effective cheating 100% of the time, but it would make a tremendous impact in eliminating LLM-based cheating.
If you're worried about corrupt proctors, that's another matter. National brands that are both self- and externally-policed and depend on a good reputation to drive business from universities would help.
With this system, I expect that it would not take much to avoid almost all the important cheating that now occurs.
Remote proctoring programs at least are pretty rough these days. Their environment conditions are pretty exacting and then they expect you to just stare at the screen and think for basically the whole exam. Minor normal webcam problems can invalidate the entire exam through no fault of the examee or if you look around or fidget a lot it can trigger their cheat detection as well. I'm glad I finished my test taking time before it became the norm.
I had to retake a multi-hour proctored test (and only got to do so after a ridiculous amount of back and forth with the school) because my cat jumped up on my computer table while I was taking it, and I looked over at her and gave her a few pets before looking back at the screen. Not joking in the least. It was maddening.
What do they do if you don't have a webcam? Or if your webcam is broken? Or if you don't feel comfortable sharing your video?
They tell you to come back when you’re ready to take the test. This can’t be surprising…
Most institutions use a remote test proctoring service. Before the test begins, the service verify all the requirements are met.
Anything short of all the requirements and they do not administer the test. In the case something is broken, there is usually a window rather than a specific time when you can take the test, so you have an opportunity to resolve it.
If a webcam is required, students are informed about this requirement before enrolling in the institution, and are usually required to purchase hardware capable of doing this as a condition of enrollment.
You get a working web cam. It's a requirement for many remote proctoring services and if you don't have access to one you're screwed.
I get why they use it, without it there's no way to know you're not on your phone or another device cheating since they can only really see what's on the device you've installed the proctor software/rootkit on.
Sadly Linus Tech Tips video of him taking the CompTIA A+ exam has been taken down after threatening letters from CompTIA but they demanded a basically baren room, 360 photos and spotless web cams.
Webcam is broken is now pretty universally interpreted as I don’t want to be on video.
But it also only catches cheating on exams. For homework/projects, you can't really have that be in person.
My take:
- Make “homework” ungraded. Many college classes already do this, and it has been easy to cheat on it way before AI by sharing solutions. Knowledge is better measured in exams and competence in projects. My understanding is that homework is essentially just practice for exams, and it’s only graded so students don’t skip it then fail the exams; but presumably now students cheat on it then fail exams, and for students who don’t need as much practice it’s busywork.
- Make take-home projects complex and creative enough that they can’t be done by AI. Assign one large project with milestones throughout the semester. For example, in a web development class, have students build a website, importantly that is non-trivial and theoretically useful. If students can accomplish this in good quality with AI, then they can build professional websites so it doesn’t matter (the non-AI method is obsolete, like building a website without an IDE or in jQuery). Classes where a beyond-AI-quality project can’t be expected in reasonable time from students (e.g. in an intro course, students probably can’t make anything that AI couldn’t), don’t assign any take-home project.
- If exams (and maybe one large project) aren’t enough, make in-class assignments and projects, and put the lectures online to be watched outside class instead. There should be enough class time, since graded assignments are only to measure knowledge and competence; professors can still assign extra ungraded assignments and projects to help students learn.
In summary: undergraduate college’s purpose is to educate and measure knowledge and competence. Students’ knowledge and competence should be measured via in-class assignments/exams and, in later courses, advanced take-home projects. Students can be educated via ungraded out-of-class assignments/projects, as well as lectures, study sessions, tutoring, etc.
> My understanding is that homework is essentially just practice for exams
There are a LOT of people that don't take exams well. When you combine that with the fact that the real world doesn't work like exams in 90% of cases, it makes a lot of sense for grades to _not_ based on exams (as much as possible). Going the other direction (based on nothing _but_ exams) is going to be very painful to a lot of people; people that do learn the material but don't test well.
I made another comment on this thread about that. Exams should be test important knowledge (not computation or trick questions) so they should be easy for students who learned the material, even those who traditionally have trouble with exams. Most of the grade should be frequent in-class assignments or long take-home projects, which test almost if not the same skills students would use professionally (e.g. debug a simulated server failure in-class; develop a small filesystem with a novel feature at home).
The in-class assignments should also be easier than the take-home projects (although not as easy as the exams). In-class assignments and exams would be more common in earlier classes, and long projects would be more common in later classes.
> proctoring services have tiny incentives to catch cheaters. Cheaters have massive incentives to cheat.
If they don’t catch them they don’t have a business model. They have one job. The University of London, Open University and British Council all have 50+ years experience on proctoring university exams for distance learning students and it’s not like Thomson Prometric haven’t thought about how to do it either, even if they (mostly?) do computerised exams.
The problem is that the business model is that when you outsource compliance (in this case that might be catching cheaters), the important thing is to be able to say that everyone did their best, and you don't necessarily need to do your best to say that.
Well, if they don't catch someone. They don't have much incentive to avoid false positives. Catching someone who did not cheat but failed to follow all the draconian rules, is probably a lot easier than to catch an actual cheater.
And I daresay most of the corporate certs from companies like Microsoft and Red Hat are probably have pretty well-proctored exams too. To what degree their processes are applicable to a University environment I don't know.
I took one last year. Microsoft offer a choice of remote proctoring or in-person at a Pearson VUE test centre. I chose the latter.
You put your stuff in a locker. They compare your face to some official photo ID and take your photo. You sit the test. They print out your results along with your mugshot. That's it. It was very painless.
Teachers typically also have years, sometimes decades, of experience running exams. Yet I've never seen a teacher that is any good at stopping cheating. And that's in person for the class that they are teaching.
> Teachers typically also have years, sometimes decades, of experience running exams. Yet I've never seen a teacher that is any good at stopping cheating. And that's in person for the class that they are teaching.
The difference is running exams is a small part of a teacher's job, and almost certainly not the part they're passionate about.
Also proctors demand things I've seen no teacher at any level demand (or be able to demand).
> I expect the system will only catch a small fraction of the cheating that occurs.
It'll depend a lot on who/where/how is doing the screening and what tools (if any) are permitted.
Remember that bogus program for TI8{3,4} series calculators that would clear the screen and print "MEMORY CLEAR"? If the proctor was just looking for that string and not actually jumping through the hoops to _actually_ clear the memory then it was trivial to keep notes / solvers ... etc on the calculator.
It's actually somewhat of a challenge to display "Mem cleared" without access to the lowercase font. You have access to any uppercase character, spaces, and BASIC functions. With stat vars, you also get lowercase "a" "b" "c" "d" "e" and "r". And you can display text at a specific row and column.
I ended up displaying "M" "e" "min(" "c" "log(" "e" "a" "r" "e" "d". Then covered up the "in(" with spaces.
Then you lower your contrast for the full effect.
Was I at university in a small window in time when a TI-89 and TI-92 was allowed?
In the years since, I’ve only ever heard mention of older models, not newer ones which makes me wonder if this is a special case and situation where technology is frozen in time intentionally to foster learning.
I was in such a window. TI-89 was allowed by mistake, we were allowed to keep using it since it was expensive. Next year they were back on TI-83s.
Oh yes, they're frozen in time, but since the people who pay for them are not the same people who demand they must be used, they're not frozen in price. It's the most expensive kilobytes you'll ever buy.
They are not "older models" just lower end. TI-92 came out in 1995 and discontinued in in 1998. TI-83 was introduced in 1996 and discontinued in 2004. TI-89 came out in 1998 and was discontinued in 2004.
At my high school we were allowed to have TI-83s but not TI-89s, because 89s had built in CAS (computer algebra system) and could do your algebra homework for you. When I went to college I already had an 83 so I didn't feel the need to upgrade.
I wasn’t allowed anything more complex than a Casio FX-300ES. Even my 991ES wasn’t allowed, let alone something like a TI83/4. This (from what I’ve heard) is pretty standard in Canadian universities for calc 1-3, linear algebra, discrete, etc.
Supposed to be the same thing in the UK but no one cares. In fact most of our students (undergrad mathematics) appear to have HP Prime now which has a full CAS built in. The questions are designed to break the CAS sometimes. Try expanding (a-2b)^1000 on a calculator to get a coefficient out. It gets stuck and hoses the whole calculator.
You can't stop people hiring someone who looks similar from sitting the exam, or messages in morse code via Bluetooth. It's hard to stop a palm card.
But it stops a casual cheater from having ChatGTP on a second device.
You can.
I did a remote proctored exam for the NREMT last year. They had me walk the camera around the room, under the desk, etc. All devices had to be in my backpack. No earbuds. They made me unplug the conference tv on the wall, lift picture frames etc. I had to keep my hands above the table the whole time, I couldn't look down if I was scratching an itch. They installed rootkit software and closed down all of the apps other than the browser running the test. They killed a daemon I run on my own pcs that is custom. They are recording from the webcam the whole time and have it angled so they can see. They record audio the whole time. I accidentally alt tabbed once and muted the mic with a wrong keyboard, those were first and second warning within 5 seconds.
When you take the test in a proctored testing center location they lock all of your stuff in a locker, check your hands, pockets, etc. They give you earplugs. You use their computer. They record you the whole time. They check your drivers license and take a fingerprint.
Those methods would stop a large % of your attack vectors.
As do the repercussions:
A candidate who violates National Registry policies, or the test center's regulations or rules, or engages in irregular behavior, misconduct and/or does not follow the test administrator's warning to discontinue inappropriate behavior may be dismissed from the test center. Exam fees for candidates dismissed from a test center will not be refunded. Additionally, your exam results may be withheld or canceled. The National Registry of EMTs may take other disciplinary action such as denial of National EMS Certification and/or disqualification from future National Registry exams.
At a minimum you're paying the $150 fee again, waiting another month to get scheduled and taking another 3 hours out of your day.
I'd rather go take the test in person than subject myself to such extreme surveillance of my own premises.
I'd agree, but I did it at work in a conference room. And I was able to schedule a day out virtually instead of a month out for in person, and I didn't want them taking my fingerprint.
I used a spare laptop I wipe.
Making them surveil your employer instead is not a bad idea either.
I'd pit my megacorp's security against theirs any day of the week, but as I said I just used and wiped a laptop just for the test.
I took 3 CompTIA certification tests at a community college testing center. This was the procedure, more or less.
> When you take the test in a proctored testing center location they lock all of your stuff in a locker, check your hands, pockets, etc. They give you earplugs. You use their computer. They record you the whole time. They check your drivers license and take a fingerprint.
While attending there, I also took a virtual Calculus class. The instructor was based in the satellite campus, several miles away. The virtual class required a TI graphing calculator, used Pearson textbook & video lectures, and all the tests and quizzes were in Canvas. I worked from home or the main campus, where there was a tutoring center, full of students and tutors making the rounds to explain everything. I received tutoring every other week.
But then our instructor posted the details on our final exams. We were expected to arrive in-person, for the first time of the semester, on that satellite campus at specified times.
I protested, because everything I'd ever done was on the main campus, and I rode public transit, and the distance and unfamiliarity would be a hardship. So the disability services center accommodated me.
They shut me into in a dimly lit one-person room with a desk, paper, and pencil, and I believe there was a camera, and no calculator required. The instructor had granted an extended period to complete the exam, and I finished at the last possible moment. I was so thankful to be done and have good results, because I had really struggled to understand Calculus.
wow, that’s intense. i wonder how much actual cheating they must have caught to arrive at such a draconian model. it would be interesting if they published their statistics to make it clear whether all these things are truly necessary.
What stats would convince you? A woman was jailed in the UK last week for taking in person tests on behalf of others. She wore a variety of wigs to fool test centre staff. Where there's demand there's people who will try to supply it.
i guess i would expect them to publish some rates of disciplinary actions per sitting and the type of attempted behavior.
ex “1% of test takers were disciplined for attempting to contact someone for help using a disallowed electronic device surreptitiously”
minimally as deterrance
The remote proctored exam is a major invasion of privacy, but nevertheless, there's at least a dozen ways you could cheat despite all of that.
I fear that remote-proctoring can be liable to more false positives, if they are going to flag actions that "might" indicate a cheating sort of behavior, but they can't reach in and unveil your secret cheat sheet or identify your accomplice. I don't know the whole process after the remote proctor flags something, but it would seem more difficult for the student to defend innocence.
It's quite unfair of them to basically say "we're not competent enough as proctors to come up with evidence of guilt, so we'll use a guilty-until-proven-innocent system instead."
Both of those are so hard and so expensive that usually just learning the material is more practical.
LLMs and remote exams changed the equation so now cheating is incredibly easy and super effective compared to trying to morse code someone with a button in your shoe.
From what I've seen it works.
There is definitely a war between cheaters and people catching them. But a lot of people can't be bothered and if learning the material can be made easier than cheating then it will work.
You can imagine proctoring halls of the future being Faraday cages with a camera watching people do their test.
Local LLMs are almost here, no Internet needed!
Almost?
I've been running a programming LLM locally, with a 200k context length with using system ram.
Its also an abliterated model, so I get none of the moralizing or forced ethics either. I ask, and it answers.
I even have it hooked up to my HomeAssistant, and can trigger complex actions from there.
Way back like 25 years ago in what we call high school in the US, my statistics teacher tried her damndest to make final exams fair. I said next to someone I had a huge crush on, and offered to take their exam for them. I needed a ‘c’ to ace the class, and she needed an ‘a’ to pass. 3 different tests and sets of questions/scantrons. I got her the grade she needed, she did not get me the grade I needed.
So to your point, it’s easy to cheat even if the proctor tries to prevent it.
I am confused by your pronouns and other plot holes.
You wanted to "ace the class", which is an "A" on your final report card? But your crush's exam tanked your grade? You passed the class anyway, right?
Did you swap Scantrons, then, and your crush sat next to you, writing answers on the dgfitz forms?
She wouldn't pass without an "A" on the exams, so her running point total was circling the drain, and your effort gave her a "C-" or something?
In what ways did your teacher make the exams "fair"? What percentage of the grade did they comprise?
Were the 3 tests administered on 3 separate occasions, so nobody caught you repeatedly cheating the same way?
> Were the 3 tests administered on 3 separate occasions, so nobody caught you repeatedly cheating the same way?
I imagine that it would be utterly trivial for two people to nearly-undetectably cheat in this way, by both of them simply writing the other person's name on their exam.
My impression was that in high school, girls and boys had pretty distinct handwriting.
Not sure if that impression is accurate though, or if it's true of mathematical writing.
Yes, high school boys and girls have clearly distinct handwriting.
If you're just filling in bubbles on a scantron, then handwriting isn't very visible and each person can just write their own name on the scantron they're submitting as their own.
If you've been to one of these testing centers, you'd realize it's not easy to cheat, and the companies that run them take cheating seriously. The audacity of someone to cheat in that environment would be exceptionally high, and just from security theater alone I suspect almost no actual cheating takes place.
I did a proctored exam for Harvard Extension at the British Council in Madrid. The staff is proctoring exams year-round for their in-house stuff so their motivation notwithstanding they know what they’re doing.