Why don't more instructors use AVs?

6 replies [Last post]
shaffer
shaffer's picture
Offline
White BeltYellow BeltGreen BeltRed BeltBlack Belt
Joined: 2009-05-28
Posts:
Points: 2019
Thank you for your recommendation, I just finished read it, but still curious, * Why those tools (with vary of implementation technique and level of engagement) have not been adopted broadly (based on algoviz survey result) and have not been proved significantly yet to ease the program comprehension problem? * Why did they separate between AV and PV, in my opinion we need an integrated tool that can support us on learning programing in high level and low level, and also enable us to cross-referencing among the levels? I really appreciated for the answers or clues of my curiosity… ( or my stupidity?? :D ) many thanks.
–––––––––––—
 
This seems to be the major question surrounding AVs in recent years: With so many AVs available, and mounting evidence that they do help learning, why don't more instructors use them in classes? Here are a few observations.
 
Do they really help learning? There is some debate about that, which could be putting some instructors off of using them. The research literature contains many papers that report studies where an AV didn't improve learning. But maybe these tend to be older? I think there is much room for some good papers that summarize the more recent literature (since the 2002 "Metastudy" paper), though see here. Personally, I think the data are compelling in favor of AVs, whenever they serve to increase student engagement in the material. More to the point, faculty don't often express concerns on the effectiveness of AVs as a reason not to use them (see notes on our survey).
 
You mention PV (program visualization) vs. AV (algorithm visualization). I am not sure what "integrating" them would mean. Each given piece of software does a job, hopefully well (but too many do not do so well). There is the issue that so many artifacts exist that many instructors report difficulty in figuring out which are worth using. But I don't think that a broader tool that attempts to "do it all" will be the answer. My own observation is that the broader AV systems that try to cover too many topics end up having a lot of weak AVs even if they have some strong ones. It seems like any individual group has only so much creative energy, and only so much content expertise. I think that developing good AVs on difficult topics is somewhat like developing works of art. No artist can do everything!
 
I think that the real bottleneck is the sheer amount of effort involved in effectively integrating AVs into courses. This is one of the two major factors that instructors have reported in all of the surveys (the other is difficulty in finding a suitable AV to start with). Looking at the examples of relatively large-scale use of AVs that I know of, they appear to succeed best when some group makes a major effort at a major curriculum integration. Some examples follow.
 
(1) TRAKLA in Finland, which is used throughout that country. This is an integrated system in that there are many AVs, and it is integrated with a sophisticated quizzing system.
(2) Pilu Crescenzi and his collaborators in Italy. They have a full textbook (in their native language), along with AVs that fit into the textbook.
(3) To a lesser extent, JHAVÉ. Again, a large set of AVs integrated with a quizzing system. But I don't know how widely this is being used.
 
I think there is a real opportunity for breakthrough when somebody does an online Data Structures textbook with built-in AVs. The key point being that an instructor doesn't integrate AVs into their class — it gets done for them. Then the instructor only needs to adopt that curriculum. If someone can do a good job at that, then there might be a lot of adoption.
 
That said, the latest survey results are not so bad. It shows that a (bare) majority of the surveyed instructors have used something recently. But then, it is not a representative sample of CS instructors. So maybe it exaggerates AV use.
 
I hope that we can get some discussion going here!

ville
ville's picture
Offline
White BeltYellow BeltGreen BeltRed BeltBlack Belt
Joined: 2009-05-28
Posts:
Points: 559
Re: Why don't more instructors use AVs?

What you indicate in the last paragraph with the survey results is what I’ve been thinking lately: are AVs as little used as the developer community believes?

I know it is a good motivation for any innovative solution to AV both for the development and publishing of the new or improved system. Still, all the data gathered is based on surveys for rather small samples of teachers. It would be interesting to get some actual use statistics collected by AV systems. We might be positively surprised, as I believe that at least students find the visualizations and use them actively if they help their studies.

While most of my research has been focused on helping teachers create and use animations, a question worth discussing is whether it is even necessary for the teachers to use AV or is it enough if students use them? Engagement of students has been proven (?) effective with AVs. Any thoughts?

Ville Karavirta, Aalto University, http://villekaravirta.com/

guido
Offline
White BeltYellow BeltGreen BeltRed BeltBlack Belt
Joined: 2009-06-03
Posts:
Points: 679
Re: Why don't more instructors use AVs?

I agree that the existence of an online data structures textbook with integrated AVs would be very helpful - and indeed, I am working on just that! The textbook as it is is described in the publication "A Visualization-Based Computer Science Hypertextbook Prototype", see the portal of acm.org. It is essentially a prototype that works on top of Moodle, and thus knows a bit about its users. Each paragraph can be discussed, annotated, marked with a text marker, or flagged as "hard to understand", by each user. AV materials are directly incorporated into the pages.

  

Since creating sensible content is a fair amount of work, there is currently only one "book chapter", a German chapter of sorting algorithms. However, two additional chapters - on searching and data compression - are in the process of proof-reading, and should become available "soon". The prototype is available at this URL - but note that you have to register first in the Moodle system, using an arbitrary "Matrikelnummer" (it must be unique system-wide) and the "Studiengang" fields.

Feedback is welcome!

guido
Offline
White BeltYellow BeltGreen BeltRed BeltBlack Belt
Joined: 2009-06-03
Posts:
Points: 679
Re: Why don't more instructors use AVs?

ville wrote:

While most of my research has been focused on helping teachers create and use animations, a question worth discussing is whether it is even necessary for the teachers to use AV or is it enough if students use them? Engagement of students has been proven (?) effective with AVs. Any thoughts?

  

I think that giving the tools to students is helpful even if the teacher does not "adopt" them. However, students run the risk of familiarizing themselves with an algorithm or data structure with the AV system which differs in some (perhaps minor, but still important) aspect from the one used in the course. For example, there are several variations on how to partition data in the Quicksort algorithm: do elements equal to the pivot belong to the "left" or to the "right" subset? How is the pivot chosen - as the first, the last, the middle element of the entries to be sorted, or the median of the (first, middle and last) element? These differences may not affect the principal understanding of Quicksort, but certainly lead to intermediate "incorrect" answers that may be penalized in homework assignments or exams. At the same time, such small differences are hard to notice for students, who may assume that "since both the teacher and the resource use Quicksort, it must be the same".

  

I believe that a clear statement from the educator, to raise awareness of AV tools and of potentially critical differences between the "lecture content" and the "portrayed content", is important. Additionally, I know from personal discussions that some educators are reluctant to sponsor AV "because then students can simply copy the displayed solution or intermediate steps for their homework". While I have to agree that this is a good argument, I would also strongly state that AV makes it possible to place wholly different and more "advanced" questions, such as "specify an input set of length 10, consisting of a permutation of the numbers 1 to 10, so that Quicksort will show the following behaviour: …", or "Given the following adjacency matrix as an input for Dijkstra's Shortest Path Algorithm, determine values for the variables x, y, and z in the adjacency matrix so that node 3 will be reached with a total cost of 8, and the cheapest route to node 5 is only found in the last step". These tasks should be interesting and test deeper understanding - but without AV capabilities, simply "guessing" values and then manually tracing the algorithm (multiple times, most likely) would be far too much work.

  

Opinions on this would be welcome!

shaffer
shaffer's picture
Offline
White BeltYellow BeltGreen BeltRed BeltBlack Belt
Joined: 2009-05-28
Posts:
Points: 2019
Re: Why don't more instructors use AVs?

Ville wrote:

It would be interesting to get some actual use statistics collected by AV systems. We might be positively surprised, as I believe that at least students find the visualizations and use them actively if they help their studies.

Naturally, we track statistics at the website here about accesses through the catalog. We could publish statistics about which are the most frequently accesses AVs… at least so far as we can determine. All we can really tell you is how many times someone follows a link from the catalog to a given AV. But this might be of some value. We are nearly finished with migrating the old Wiki version of the catalog over to this site. We should have an official announcement abou that soon. At that point, we should be able to start working on tools to report access data. If anyone can suggest what they would like to see, perhaps we can provide it. One alternative is an automated system that periodocially posts selected statistics. Another possibility (a little harder to implement) would be some sort of query system that visitors to the site can generate data on demand.

  

Ville wrote:

Still, all the data gathered is based on surveys for rather small samples of teachers.

Yes, I think it would be a good project for someone to take on to do a proper survey of CS instructors about overall AV use. That would be a big job, though! Some sort of survey of the SIGCSE membership should provide a representative sample, if a reasonable response rate could be generated.

Guido wrote:

However, students run the risk of familiarizing themselves with an algorithm or data structure with the AV system which differs in some (perhaps minor, but still important) aspect from the one used in the course. […] These differences may not affect the principal understanding of Quicksort, but certainly lead to intermediate "incorrect" answers that may be penalized in homework assignments or exams. At the same time, such small differences are hard to notice for students, who may assume that "since both the teacher and the resource use Quicksort, it must be the same".

Doesn't that depend a lot on what sort of questions that you ask? One of the advantages of using AVs is that hopefully you can move students higher in the Bloom taxonomy on the type of understanding. I find that I don't tend to ask the type of question where students would run into trouble with this sort of thing, or when I do, I give them the relevant code with the question.

archie
archie's picture
Offline
White BeltYellow BeltGreen Belt
Joined: 2009-06-10
Posts:
Points: 21
Re: Why don't more instructors use AVs?

Hi everybody,

  

Shaffer: "I think there is a real opportunity for breakthrough when somebody does an online Data Structures textbook with built-in AVs. The key point being that an instructor doesn't integrate AVs into their class — it gets done for them. Then the instructor only needs to adopt that curriculum. If someone can do a good job at that, then there might be a lot of adoption."

  

One idea is to provide on-line tutorials such as this Binary Heap tutorial based on TRAKLA2 exercises. Those would be easy to integrate into any course as the link to the tutorial is just enough. We have a lot of similar exercises that could be embedded into such tutorials if I only we have somebody to write the tutorials… I would be happy to do collaboration in this kind of projects. We can easily provide a web site and techical support to do the rest, for example, if someone else can provide the plain text. Actually, Tom Naps has written a nice graph tutorial that we could publish next. The current version includes both TRAKLA2 exercises and JHAVÉ quizzes, but requires registration and server connection to work. In this stand alone Binary Heap tutorial, we used TRAKLA2 exercises that do not need a server connection (i.e., the points and marks got from the exercises are not stored anywhere). 

A good thing with stand alone tutorials is the fact that there is no need for registration. This seems to be one obstacle for both students and teachers. Everything should be as smooth as possible to take into use. Of course, if the teacher wants to use the system for summative evaluation, there must be a mechanism to collect the necessary data (i.e., points and marks related to a user account). To establish and maintain such a sever seems to be too much for most of us. Even the configuration can be too much. TRAKLA2 has succeeded because TKK has been hosting the service for most of the other universities in Finland. Currently, there is too many "customers" and TKK closed down the service. Another reason was that a commercial actor (By The Mark Corp.) started to provide the same service, i.e., to host and maintain a similar server (I'm a stakeholder in this as well, thus if you think that this is an inappropriate ad, please, just delete this message). Anyway, this kind of "turnkey" service could be one solution to the poor dissemination of AV tools.

One of the problems with commercial services is the question who should pay for it. Is it the university or the students? In Finland, university education is free, thus universities pay for the service. It gets more difficult if the payer should be the student. However, students already need to pay for the text books, so why not the online material as well? Actually, I like this idea to incorporate the online material to be a part of a text book. For example, the student could buy a text book, and in addition get an account to the learning environment. We already have text books, we already have good materia. I look forward to find out perfect matches…

  

 Archie

affandy
Offline
White BeltYellow Belt
Joined: 2009-08-20
Posts:
Points: 8
Re: Why don't more instructors use AVs?

After 2 years ago since my last comment, I’m so surprised that finally Algoviz gave some space for some PVs too. For me Jeliot3, Alvis live, Ville, are such a very recommended PV tools that can help student in understanding the behaviour of the program. Combined with the advantage of the AV, I’m still in progress in order to build a kind of synergetic visualization to reveal the programming as process to novice student. I believe that understanding the logical flow of the algorithm and follow with understanding the mapping of each step into the syntactical of particular programming language will give student the comprehension of whole process of programming. This strategy allows student to visually observe the higher-level, lower-level and intermediate-level of program abstraction as well and in the same time. For instructors, it is likely to help them in order to present or explain on a software development process.

I look forward for any suggestions..