Proposal for new recommendation system

11 replies [Last post]
shaffer
shaffer's picture
Offline
White BeltYellow BeltGreen BeltRed BeltBlack Belt
Joined: 2009-05-28
Posts:
Points: 2019

A fundamental purpose to AlgoViz.org and the catalog system is to help people find AVs of value to them, and a key mechanism for doing so is recommendations about the AVs. Recommendations get used in some way to rank order AVs on searches, or to constrain searches. We have two systems in place for generating rating information: The "editorial" rating of Recommended/Has Potential/Not Recommended, and the "public" rating based on 5 stars. Unfortunately to date we have not had a lot of success at getting people to supply the "public" 5-star rating. That issue needs to be dealt with by various community-building activities that we will begin focussing on in the near future. The purpose of this post is to propose a change in how we handle the "editorial" recommendation.

A persistent concern for the "editorial" recommendation has been its reliability. Is it just the opinion of whoever does that evalation? Or would any "reasonable" evaluator give the same recommendation? One reason for using a very coarse metric (3 levels in this case) has been to increase the likelyhood that it would be consistent across evaluators. To actually test the reliability of the recommendation process, I asked a few volunteers from the steering committee to evaluate a set of specific AVs. That test is still ongoing, but an important result came out of initial feedback. That is, reviewers had trouble with the fact that a given AV might be really good for, say, supporting a lecture or letting students try something out on a topic that they have already heard a lecture on, but not good at all if it were the student's only exposure to the topic. That concept of being "good for" some purposes but not others was meant to be captured in the "good for" field. But it has never been well thought out what the field values mean, nor have they been consistently applied to all catalog entries. And there is no direct connection between the "good for" and the "Recommendation" fields.

Based on feedback and discussion with Tom Naps, Susan Rodger, and Steve Edwards, I have come up with the following proposal to resolve this issue. I am looking for feedback from the rest of the community about the proposal before we commit to implementing it.

 –––––––––––––––––––––––––––––––––––—

 Proposal:

The current "Recommendation" and "AV is good for" fields would be replaced by four independent recommendation fields, each of which can take on a value of "Recommended" "Has Potential" or "Not Recommended". The four fields would indicate the AV's suitability for the following:

* Lecture Aide
* Self-study supplement to lecture or text
* Standalone self-study of topic
* Debugging aide

"Lecture Aide" is meant to indicate that the AV is used by an instructor as a visual aide in a presentation, or otherwise used by students under an instructor's direct supervision.

"Self-study supplement to lecture or text" is meant to indicate that the student will have read material or heard a lecture on the topic before using the AV. This might be used in a non-directed self-study (free choice), or as "homework" (instuctor directed them to use it at home), or in the classroom/lab (instructor assigned time in-class to use it).

"Standalone Self-study of topic" is meant to indicate suitability of the AV to be the sole mechanism for providing the content of the topic. Whether assigned by the instructor, or discovered by the student on their own.

"Debugging Aide" is meant to indicate that the AV provides sufficiently detailed results such that those results could be used to help a student determine if their own implementation for the algorithm or data structure is correct.
 –––––––––––––––––––––––––––––––––––—

I would appreciate hearing feedback about this proposal. Do you think it would help you to better understand the potential value of a given AV? Do you think it is viable from a maintainabilty standpoint? Do the four proposed field appropriately "span" the space for recommendation?

                      — Cliff

archie
archie's picture
Offline
White BeltYellow BeltGreen Belt
Joined: 2009-06-10
Posts:
Points: 21
Re: Proposal for new recommendation system

I think it's a good idea to change the way these recommendations work. A couple of questions came to my mind, however.

Some of the four fields might be overlapping. For example, I have used some visualizations in a lecture situation even though they are meant to be used for assessment, i.e., self-study supplement. The visualization might not be perfect for that, but could have potential. On the other hand, some of the fields might not be applicaple at all for all the systems. These are not problems. However, is the idea that each time a) you evaluate all the four fields b) you evaluate only the most appropriate field or c) you evaluate all the fields that are applicaple.

If a), it could be a good idea to have one more choice "not applicaple", which is somewhat different evaluation than "not recommended".

If b), in my opinion, selecting only one field is not always the whole truth about the system.

if c), there is a risk that some of us still evaluate the system also in fields that are not applicable (and give rating not recommended, which is still somewhat different than not applicaple).

— Archie

guido
Offline
White BeltYellow BeltGreen BeltRed BeltBlack Belt
Joined: 2009-06-03
Posts:
Points: 679
Re: Proposal for new recommendation system

I like the new evaluation system as proposed. I would prefer to rank the system on all four scales at the same time, since - as discussed previously - a visualization that may be "very recommended" as a lecture aide might be "not recommended" for standalone use without prior exposure to the topic area or supervision.

  

What I would also like to see is a search facility that will allow a user to find "top-rated" visualizations for the fitting use type: if I am looking for recommended materials for standalone use, I may not worry too much how they were rated for use as a lecture aide (and vice versa). Similarly, I would recommend to keep the different categories separate and not simply to "average out" the ratings to receive a final ration.

  

 Guido

shaffer
shaffer's picture
Offline
White BeltYellow BeltGreen BeltRed BeltBlack Belt
Joined: 2009-05-28
Posts:
Points: 2019
Re: Proposal for new recommendation system

Yes, the current idea is that a viewer would see a table or series of four fields. And for each field there would be its name (such as Lecture Aide) along with a rating (such as Recommended). Then would be the next field (say, "Self-study supplement") followed by a separate rating (maybe Has Potential). Any given AV is intended under this proposal to have four independent ratings.

  

There is some potential redundancy here, as I predict that the first three fields (lecture aide, self-study supplement, self-study standalone) will tend to be monotonically non-increasing in recommendation level. But not necessarily. Some AVs that are good as self-study tutorials might not have the necessary visual appeal to be a suitable lecture aide.

  

And yes, these fields should then get used by the search engine in a useful way. The search facility is our next major area of attention, as right now it is fairly primitive. Definitely, users should be able to filter/rank the results on one of those four fields. That is, you should be able to search for hashing AVs that are suitable for standalone self-study without the ordering of the results being corrupted by considerations of how useful the AV is for lecture aide.

  

I have not thought before of the idea that "not applicable" could be a possible value for these fields. I am reluctant to do that, since it would complicate the decision-making process for an evaluator. And does it really carry more information? I know that "not applicable" is a little different from "not recommended". But is it enough different to be worth the complication? I tend to think not, but then I am very sensitive to the workload issue! If there is a lot of support for the idea of separating "not applicable" from "not recommended" then we would implement it.

                                    — Cliff

Monika
Offline
White BeltYellow Belt
Joined: 2009-05-28
Posts:
Points: 7
Re: Proposal for new recommendation system

I am adding a screenshot to show how the Recommendation field will look like if we make the proposed changes. 

  

clancy
Offline
White BeltYellow BeltGreen Belt
Joined: 2009-06-05
Posts:
Points: 15
Re: Proposal for new recommendation system

I use visualizations as activities (often collaborative) in supervised closed labs. I'm having trouble deciding which of the four categories apply to me. For example, a facility that allowed the input and output for a visualization to be saved for later viewing by a partner or a t.a. would be useful for students in my labs. I think it wouldn't be applicable for any of the given four categories.

naps
Offline
White BeltYellow BeltGreen BeltRed Belt
Joined: 2009-06-11
Posts:
Points: 65
Re: Proposal for new recommendation system

I think that Monika's screen shot of what a user would see if the new recommendation system were implemented indicates that it would provide much more useful information regarding ways in which the AV could be effectively incorporated into a course I was teaching.

  

Regarding the addition of a "Not applicable" rating for each of the four fields, it seems to me this is covered already by the "Not recommended" rating.   That is, to say that an AV is "not applicable" for self-study certainly implies that the reviewer making this rating would "not recommend" it for self-study.   So as a reviewer, in certain circumstances, I would actually be confused whether I should rate something as "not applicable" or "nor recommended" in one of the categories.

  

Regarding Mike Clancy's comment on how to capture whether a particular AV might be suitable for a lab-based activity —  I hate to see another category introduced beyond the four we already have.   It also seems to me that the suitabilty of an AV for a lab-based activity is something that is highly dependent on the creativity and imagination of the instructor who uses the AV in their class.   That is, I could see almost any AV that was recommended for a self-study supplement, standalone self-study, or debugging aide leading to a lab activity of some sort if an instructor wanted to create such a lab activity that used the AV.   But the actual lab activity would be an artifact of what the instructor did with the AV in the context of their class instead of being intrinsically rolled into the AV itself.   It seems to me the field reports much more would capture how an AV led to a lab activity.   Perhaps in the screen shot Monika developed we could add a link to field reports that existed for a given AV?   This would not only give a user a broad-based review/recommendation on the AV, but where field reports existed, immediately let the user view actual ways in which some other instructor had used the AV.   If we wanted to empahsize a little more the notion of lab-based exercises, perhaps the category "Standalone self-study of topic" could be enhanced to "Standalone exploration of topic", with "exploration" conveying more a notion of activity-absed learning?   Granted our problem here is that most of the AV's don't yet have any kind of field report associated with them.:-(

shaffer
shaffer's picture
Offline
White BeltYellow BeltGreen BeltRed BeltBlack Belt
Joined: 2009-05-28
Posts:
Points: 2019
Re: Proposal for new recommendation system

Yes, we have discussed making a link from catalog entries to any associated field reports. But I don't see it on the active "todo" list, I'll make sure that this is taken care of.

  

Regarding labs and the recommendation catagories: One way to view it is that the first three catagories refer to how much external support is needed to make use of it. "Lecture Aide" means the most external support is available, because it is being used directly in a context where  supporting information is provided. "Self-study supplement" is intermediate in terms of supporting information: Some sort of external information about the topic in the form of lecture or text is required. While "Standalone" means that it contains all necessary explanation. HOW it gets used (self-directed by students, directed by instructors, in lab or for homework) seems to me to be dependent on how much support is required. If it includes its own tutorial, you can use it in various ways without additional content provided. If it is "supplemental" then you could use in various ways — but a text or lecture will be required. I think if it is "recommended" as "supplemental" then it should be viable in any of these contexts… provided the tutorial information is also provided.

  

I guess the key question is this: Let's say that something is "recommended" for standalone study. That is, it contains sufficient tutorial information to be self-explanatory. Is it possible that it would be "good" for a student to go learn the topic on their own, but NOT "good" for use in a lab? Or vice versa? At a coarse level, I tend to think that "good" AVs are suitable for multiple uses, with the caveat that some don't contain tutorial information to make them self-sufficient. But maybe I am being naive. Perhaps someone can give a counterexample? I guess to some extent, Mike did this with the example of saving the state for a partner. If that were a defining characteristic of being "lab worthy" then it wouldn't be captured by the categories. How important is this particlar ability? I imagine that it is rarely found in current AVs.

pilucrescenzi
pilucrescenzi's picture
Offline
White BeltYellow BeltGreen BeltRed BeltBlack Belt
Joined: 2009-06-08
Posts:
Points: 161
Re: Proposal for new recommendation system
Sorry if I am posting only now. Maybe I am too pessimistic, but I am afraid that the new recommendation system might turn out to be too time consuming for a reviewer and that the resulting four evaluations might turn out to be very similar. I think that we have very few examples of visualizations really useful from a debugging point of view and very few examples which are useful for self-study but not for lectures. Cannot we simply add to the original recommendation system something like “Requires supplement for self-study”? Yes I know, I am keeping things too simple, but I would like to avoid making a review a too much complicated task.
shaffer
shaffer's picture
Offline
White BeltYellow BeltGreen BeltRed BeltBlack Belt
Joined: 2009-05-28
Posts:
Points: 2019
Re: Proposal for new recommendation system

Pilu — Thanks for the comments!

  

Regarding "debugging aide": I estimate that 1/4 to 1/2 of the AVs are suitable for use as a debugging aide. That depends of course on what I mean by the term. I mean that the AV gives out detailed information that a student could compare to the output of their own program to determine if that output is correct. So for example, consider an implementation for Dijkstra's algorithm for single-source shortest paths in a graph. If the AV allows one to input one's own choice of graph, and if it gives back a distance table, then I would consider it worth recommending as a debugging aide. If it has a meaningful sample graph and gives back distances for that graph (but doesn't let you input your own graph) then I would consider it "has potential". If it doesn't have a mechanism to easily see the actual distances, then I would consider it "not recommended." On the other hand, I it might be that this category could be handled with a "yes/no" value instead of a 3-level ranking. Either it gives concrete values as output or not. But I do think that there are reasonable gradations in between, as in my example.

  

The proposal for the first three recommendation categories is a response to the real problems that reviewers were encountering. Consider again an AV for Dijkstra's algorithm. Imagine that it had everything that you could want in a presentation of the dynamic behavior: You could enter graphs easily, you could see a really nice dynamic display of the behavior of the algorithm, and you had some meaningful interaction. But it comes with no background text. That means no explanation for the problem being solved, no discussion of the cost of the algorithm, no discussion comparing the two typical implementations for finding the next-closest vertex, or anything else that might typically be part of a complete presentation for the topic. How should it be evaluated? A student sitting down to look at it with no background at all might well be lost since they get no context, and even if that might be considered self-evident, they certainly won't get an understanding of the analysis issues. But on the other hand, it might be a wonderful tool for understanding the details of how this algorithm works. So as a stand-alone presentation it is deficient, but as a supplement to a lecture, or an aide in a lecture, it might be great.

  

Reviewers complained that they didn't know how to handle cases like this. What is the single recommendation to make? That depended on the use scenario they envisioned, which had a lot to do with how they typically use AVs themselves.

  

What are you proposing for an alternative? If I understand you, perhaps you are suggesting that there remain a single recommendation value, and an additional yes/no field with a title like "requires supplemental tutorial instruction"? That might work for many AVs. I am concerned that reviewers would be stumped on at least some AVs where the AV provided good information for a student working on their own, but doesn't have sufficient visual appeal to be good in a lecture presentation.

pilucrescenzi
pilucrescenzi's picture
Offline
White BeltYellow BeltGreen BeltRed BeltBlack Belt
Joined: 2009-06-08
Posts:
Points: 161
Re: Proposal for new recommendation system
Thanks, Cliff! Certainly, the clarification about the term “debugging aide” makes many more AVs useful from a debugging point of view (I thought you were talking about the possibility of using the visualization system as a debugger). Indeed, my suggestion was (more or less) to add a yes/no field, but I also see your point. Clearly, I agree that the proposed new recommendation system is much more flexible: I hope that it will not turn out to be “too much flexible”. In any case, I think that it is worth trying to use it for the next review round.
Keith Moss
Offline
White BeltYellow BeltGreen BeltRed BeltBlack Belt
Joined: 2013-06-15
Posts:
Points: 99
Re: Proposal for new recommendation system
Now that I’ve found out how I can make a reply I’d like to add my voice to those who have replied but more in the way of being willing to act as an occasional reviewer.