Bielefeld Interactive Sonification Workshop 2004

January 7th - 10th - 2004

Report by Eoin Brazil


Day 1

 

Day 2 - Morning session

 

Day 2 - Afternoon session

 

Day 3

 

Day 4

The journey started late on the 7th when I meet Mikael at Shannon for the morning flight to Heathrow. We meet up at the checking desk and in typical fashion arrived in time for some quick refreshments before boarding the plane.

After a quick pint, we travelled onward to the departure gate. The flight was quite pleasant. Finally we reached Heathrow and were in for the typically rush across the terminals from 1 to 2 to check in at the cross transfer flight area. It was a bit of a walk as usual but it was quite painless as Mikael was travelling light and I'd checked my luggage through.

Leaving Heathrow we flew to Hannover airport and had a bit of confusion getting used to the sign layout of the airport before grabbing a S-Bahn (Suburban train - akin to Dublin's Dart) into Hannover Haupt Bahnhof (Main Train station). It was a bit chaotic initially in the main lobby of the station but we struggled and found the ticket office and an empty platform. The punctuality of a proper transport system was highlighted to us as the normal German knew that if they were at the platform with five minutes to spare that the train would be on time!


The workshop kicked with a letter from Greg Kramer (the founder of the ICAD community and its spiritual leader - at least in my opinion any ways) which was introduced by Andy Hunt and then a really excellent talk by both Andy and Thomas Herman. This began a interesting and enlightening day which helped me personally get a much better definition of both Interactive Sonification and its participants requirements.

Greg's letter really touched a chord with me as I can understand his interest in both Music and Science and how together they can be applied to address some of the challenging problems of complex data domains. He also stated that Sonification is like a solution in search of a problem which is another issue I can relate too. He also mentioned how we must now strive to find interfaces to control large sets of parameters and how he really saw this relationship between his newly developed keyboard (see MOOG website) and the current research in sonification. But I think it was his parting statement which I will include a brief portion of, that was the must important point that he made and drew us all back to the reality of our role and that of our research.

  • "Without methodical research, science and even commerce will afford your work little credibility and relegate it, however interesting, to the backwaters of pseudo-science. Likewise, it is not at all trivial to infuse the scientific endeavour with the vigour of creativity and inventiveness, and there is no way you can approach this task without those qualities." Greg Kramer, taken from a letter to participants of the Interactive Workshop 2004, Bielefeld, Germany
  • Next up was a double tag team effort by Andy and Thomas giving their definition for Interactive Sonification. Their talk can summarised by the following:

    1. Interactive Sonification is 'the discipline of data exploration by interactively manipulating the data’s transformation into sound'
    2. Previously the dominant techniques for analysing data were 2-dimensional graphical plotting and associated statistics
    3. Auditory displays here offer an interesting alternative by being able to show variation in a multitude of attributes (such as pitch, duration, envelope, spatial location, timbre, and brightness) simultaneously.
    4. Human learning-skills are the most important aspect that needs to be exploited in interactive systems. Andy gave two anecdotical situations, a faulty washing machine and a faulty car. The problem of the fault was first highlighted by sound and later was the primary influence in how the engineer and mechanic repaired each problem.
    5. Sound has many roles in everyday human interaction, from simply marking events (e.g. the sound of two objects coming into contact) to detailed source-related information (e.g. the sound which continuously indicates the fill level of the glass), to real-time feedback to assist the co-ordination of human activity.
    6. Musical instruments were talked about as being particularly good examples of interaction where the acoustic system feedback plays an important role for coordinating the user’s activities.
    7. The differing types of sonification were presented - earcons, auditory icons, audification, parameter mapping sonification, model-based sonification.
    8. Several examples from both Andy's and Thomas's research were then presented.
    9. A research agenda was the final area with a call for research in particular areas as directions for future work.

    The reasoning for the verbose description of the first talk was that it really set the scene for the entire day and was the one talk which offered the most in the way of future directions for research.

    The first paper session brought us presentations by John Williamson on "Granular Synthesis for Display of Time-Varying Probability Densities" and by Matthias Rath on "Informative sonic feedback for continuous human controlling a sound model of a rolling ball". I was particularly interested in the idea of Quickening (no - its not the same idea as mentioned in the Highlander films) which is were sampling is used to give a linear approximation of users actions so suggestive actions are presented to the user. In the case of my research I could offer an example from the Sonic Browser by taking a analysis of a users mouse movement that a if the direction is to the left of the screen across two successive movements that you should play sounds under the mouse but also open the sound files in the direction of the mouse movement. This opening of files is very focused and is anticipatory of the users future movements by their current movements. Matthias being an old SOb colleague and friend is always most interesting to hear and to discover what amazing sound model work he has achieved. We were treated to a further refinement of his research that he presented at DAFx 03 in London. This work deals with providing a continuous sonic feedback to a user from a sound object model, in this case a virtual ball upon a grooved piece of wood with sensors so that the tilt of the wood controlled the roll and also the sound of the ball within several experimental studies performed by Matthias.

    The second paper session brought us presentations on "POSER: Parametric Orchestral Sonification of EEG in Real-Time for the Self-Regulation of Brain States", "Using sonification to enhance perception and reproduction accuracy of human movement patterns" and "Sonification of Group Behaviour for Analysis and Training of Sports Tactics". These papers presented some interesting works and illustrated how sonification could be applied in fields from medicine to sports science. I particularly liked Höner et al's work on using sonification for sport tactics as it was an interesting domain and really helped highlight how sonification could be applied to the most unlikely of situations - that of assisting a coach in analysis handball group tactics and behaviour.


    Lunch was held at a restaurant within the University, one aside about Bielefeld's University is the main building's orientation and how it is focused on a large (I mean large!) central atrium which branches off to different departments but which has a shopping centre feel to the atrium with its abundance of shops and restaurants. Its quite a place and it seems to work very well on casual observation. A nice touch was that even though the S-Bahn station was about 3 minutes walk that in the centre of the atrium was a electronic timetable board, similar to those found trackside on the platforms. If you didn't take the public transport there was ample parking in several large but closely accessible multi-story car parks.

    The third paper session consists of the following talks "Integrating Data Sonification with Synthetic Speech for Unified Data Access ","Interactive Sonification in two domains: helicopter flight analysis and physiotherapy movement analysis","Model-based Target Sonification on Mobile devices","Interactive Sonification for exploring Single Molecule Properties with AFM based Force Spectroscopy" and "Sonified Interactions with Mobile Devices". Tony's paper was a practical examination of applying data sonification and navigation for assisting visually impaired users as always another interesting and worthwhile application of these techniques. Sandra Pauletto then presented some work which Andy Hunt had touched upon in the opening talk of the day. This dealt with applying interactive sonification to analysing data from the domains of helicopter flights and from physiotherapy movement and then proceeded to introduce the Interactive Sonification toolkit for data analysis. Andrew Crossan's presentation on model-based target sonification was quite interesting and again applied the idea of quickening using a PDA and a 3DOF accelerometer. The presentation discussed experimental results from several studies which explored the use of quickened audio displays for localisation and selection based on tilt control of mobile devices. Martini's et al paper discussed how audification can be applied to a complex mechanical procedure to provide informative and dynamic feedback to the operator of a Force Spectroscopy machine. It was an interesting paper and a complex domain which showed another interesting but varied application of sonification techniques. The last presentation in the session was by Michal Rinott and presented some of her work for her thesis project on the Masters in Interaction Design program at the Ivrea Design Institute. It was an audio-tactile interface based on the QuikWriting software (handwriting recognition software, similar purpose to Graffiti) for use in mobile devices and was called the "KeyBong". It was an interesting interface and I have shown it to our students on our Masters in Interactive Media as exemplar for their thesis projects.

    Saving the best for last! We had the old SOb project co-ordinators of Davide Rocchesso and Mikael Fernström presenting. Davide presented the talk "Contact Sounds for Continuous Feedback" while Mikael presented "Actions and Agents -- Examining Factors for the Identification and Classification Of Everyday Sounds". Davide championed the use of continuous audio feedback in embodied interfaces and presented several pieces of work derived from the SOb project and also elaborated upon the idea of cartoonification and its usefulness in sound models. Mikael's talk focused on expressing a design process for using sound within the interface and highlighted how our results have indicated that in interaction design using auditory representations we should focus on mapping human activity to actions rather than objects. He also presented so work on studies using auditory representation of soft-buttons to allow users to have a pseudo-haptic experience supporting the development a mental model of the layout of a device.

    The last part of the day focused on some seeding ideas and a roundtable. These generated a lively debate and resulted in several useful ideas and a rough definition of the area of Interactive Sonification. The resulting definition was sketched on the white board and you can see a copy below. I don't think that it is the comprehensive definition of the area but I do think that it covers most of the area and related issues. It was then off to dinner which consisted of a quick hop across the campus to a very nicely appointed function room for dinner and conversation.


    Friday started with the COST287-ConGAS management committee meeting, chaired by Nicola Bernardini. The COST287-ConGAS action intends to contribute to the advancement and development of musical gesture data analysis and to capture aspects connected to the control of digital sound and music processing. Various issues were on the agenda for the meeting:

  • Discussion on the Short Term Mission (STM) to be held by the ConGAS action
  • A description and discussion on the COST287-ConGAS Symposium on Gesture Interfaces for Multimedia Systems to be held in the University of Leeds, UK 29 March 2004
  • The reports from the various sub groups within the action
  • Discussion on various topics related to the action regarding its website, review and design of gestural databases to the possibility of a special issue in IEEE Multimedia
  • The scandalous state of the current financial arrangements with control being flung from the EC to the new and possibly still born European Science Foundation
  • This last point was a major point of discussion as it highlighted the difficulty in running an action when the organisers have absolutely no idea where there finances will be going from due to the intractable Brussels bureaucracy again changing organisational structures mid-stream without due regard to existing projects and the related transfer of control. The result is that COST action projects are in disarray with previous funding nearing an end and the new European Science Foundation being not yet functional or able to assist in these matters.

    After the most depressing report on the financial arrangements, the meeting improved with some interesting suggestions for STM's and on how to apply for them as well as on how they would be selected by the committee. Daniel Arfib presented working group (WG) one's report on Gesture, Kia Ng presented WG two's report on Input Technologies and also spoke briefly on the symposium to be held in Leeds which he is organising. Andy Hunt presented WG three's report on Integrated Applications and discussed the review of multimedia databases which will be reviewed at a side session for COST287-ConGAS members during the Leeds symposium. The next management committee was scheduled for Barcelona around the 28th - 29th of June 2004 with the possible inclusion of a Lab tour.

    The final part of interest for me during the meeting was the discussion on the website for the action. It was developed by Nicola Bernardini with some suggestions from Thomas Hermann. The result is a GForge site with a Wiki for the action. GForge is a fork of the popular Source forge software collaborative development system which can allow for uploading and sharing for source code or documents, discussion groups and mailing lists. The Wiki (WikiWeb) is a tool for collaboratively writing Patterns, which allows anyone to edit and add new Patterns to a Pattern collection very simply using an Internet Browser. Nicola stated that any action delegate can get a logon to add material to the site or could just simple e-mail him to have him upload the material.

    The last stage of the meeting concentrated on structural changes to the working groups which I felt a bit overwhelmed by and so went back to the hotel to catch a couple of hours of sleep before being called by Mikael for dinner and one last dinner with the rest of the delegates.


    Another early morning! As we had to travel to the train station, then to Hannover and then to Hannover Airport to catch an early morning flight we got up just before six to ensure we'd make check-in at the airport. Mikael and I had ordered a taxi so it was a short journey to Bielefeld train station where to our surprise we met Davide and Amalia from Uni di Verona. This allowed us to have a very nice conversation with some friends as we chugged smoothly onto Hannover. Davide and Amalia had another nine or ten hours for a train trip back to Verona so we parted at the station as they rushed to catch their next train. A short S-Bahn ride and we're were at the airport but not before having sampled some fine wurst (German sausage) in the train station (You've always got to try the local cuisine!). After check-in and a cup of coffee we were greeted by a stranger at the departure gate (See the picture!) and then travelled onward to Heathrow where we killed several hours in happy conversation and reading the papers. After this it was a short hop back to Shannon and a return home in time to see a defeated Munster finish their rugby match in the European rugby cup.