For official courses, we go over the generated course with the creator to vet the content. Generally they're pretty impressed but have a few things they'd like to change/add before publishing.
For self-created courses, it's generally been quite accurate and we're playing around with some eval metrics to make it as good as possible, but it's definitely a concern.
Is the course creator being impressed the most important metric? Are there other more concrete metrics you are able to use to determine quality from the perspective of a student?
I am curious if you are using any methodologies from the digital learning space like knowledge tracing to help ensure that learners are actually retaining knowledge and improving over time or knowledge mapping to understand the gaps that might exist in your content?
Do you maintain your own skills taxonomy? Are you tagging your questions or assessment events with knowledge components or skills of any kind to understand what you are testing your students for?
All of this is really cool, I’m just curious at what level you’ve gotten to on some of this because there is a very fine line in online educational content between making the students life more difficult and actually helping them learn, especially when you get into auto-generating content, and especially if you aren’t following solid principles to verify your content. (I work for an online education company and particularly in the space of training LLMs and verifying their outputs for use in educational contexts)
At this stage it seems like a good metric since it's the creator's content. We're adding a feature for question feedback from users, so they can like/dislike/report questions, but very open to other metrics if you have any ideas.
Yep—also in the process of adding learning paths for certain subjects, so you can go from an introductory course to more advanced topics and fill in gaps in understanding. Agreed: our mission is to help students actually learn in the best way possible, we have individual courses now to start out but the goal is to integrate the learning experience.
Very curious to chat about what you guys do, and if you have recs for any literature in the space that we should look at.
I made a comment a few weeks back that goes over a lot of the research I’ve been reading in the area: https://news.ycombinator.com/item?id=43543335
Also happy to chat more, I find this space really interesting!
So in less promotional words:
- For official courses the creators are doing some quality control and do necessary fixes. - For self-created courses there is zero human supervision or quality control.
Is that correct?
Yes for the first, technically yes for the second? The user can go in and change the content as well (i.e. if it's a teacher generating a course for students). But not sure what other human supervision + quality control methods you're referring to that we could implement.