Mining Online Software Tutorials: Challenges and Open Problems by Adam Fourney
- tutorials are a rich resource for procedural informatio
- want to extract structured semantic data to identify timing, tasks, etc
- can automatically detect commands and interface elements, but nowhere near automated
- at least a dozen problems: coreference resolution, spatial reasoning, purpose clauses, parameters and values, etc
- parameters and values: underspecified, constrained
- anti-patterns: some instructions are what not to do
- titles and topic phrases: narrative content beyond steps, not operations
- just enumerating the problems advances the work: common vocabulary, sub-problems for tailored approaches (using machine learning or crowd-sourcing)
- Q: why is automation interesting? not a goal in itself, but leads to enough understanding to use in other ways
- Q: anti-patterns, negated can't be detected? an example where identifying it may help address the problem
HCI Over Multiple Screens by Caroline Jay
- collaboration with U Manchester and BBC
- people watch TV with more than one device; broadcasters would like to develop content and experiences for companion devices
- most research has focused on social context, but not cognitive details of attention
- lean forward: web and social media; lean backward: newspaper, film, TV; multiple screens allows combinations
- observation and uncontrained interaction with eye tracking
- challenges: track movement over two screens, ecological realistic/valid; found good tracking calibration, eye tracking and video analysis matched well but hands may occlude eye tracker, good match with and without eye tracking
- split attention across two screens: tablet fixation when topic changes, updates and action (to tablet “where's the dolphin? to TV “there, there, there, there!”)
- additional challenges: eye tracking only usable in lab, investigating logging on device; many other factors like interaction, content, environment; even if we can do this, what about privacy?
- Q: is this primarily for filling in gaps in attention? individual interests influence the attention but that's not the primary goal
A Funny Thing Happened on the Way to the Website: Telling About Browsing by John Fass
- how do we derive meaning from our browsing history?
- people average 30s per page; tolerate 450ms for a web site to load, same as human blink; many tabs; multiple devices
- browser history is a list like many other elements of web content; easy to implement but life is more of an interrupted journey; many attempts at visualizing the journey over the last 20 years but no real change in browsers
- “machine” (browser and cloud services) is learning about your behavior, much more than you; don't really know where it goes, and it's algorithmic
- human meaning making: narrativity, continuous and episodic time cross-over, framing effects
- narrative: story (events over time), versions (texts), fabula (affect of the characters)
- people think of their browser history experientially
- people understand the richness of multiple aspects; need to be able to forget things
- visual comic-like layouts, curation, and collections
- also discovered conductive ink; maybe we could interact physically with our browser history
- Q: what motivated you? do the research in public with diverse participation; learned that it's very hard to get a computer to do a human story
- Q: did you see people imagining the future when reflecting on their past? internet is so fast it's hard to look far back in time to effectively imagine future
- Q: curation may delete unpredictable serendipity? yes, serendipity is another area we may want to facilitate
- Q: how might visual communication inform our work? this conference is very technical, visual design comes from a very different place; hybrid systems are very interesting
Translation from Text to Touch: Touching a “Japanese Old Tale” by Susuki et al
- tactile sense is non-verbal communication: signal (single tap on shoulder vs double), language (massag)'
- text has tactile impression: how can we extract tactile senses from text? text/words + emotion = tactile impression; eg strong/angry words are short and clipped, weak leads to many words
- create tactile score with notation like musical score
- translate long story: used natural language processing to break into chunks and coun syllables; created tactile score from counts; English translation was very different; used tactile score to translate into dance, haptic device, music and graphics; did performance based on all those translations; have also created tactile books
- “The Massage is the Medium.”
- Q: does translating to the tactile score leave out some of the richness of touch? massage therapists have a 46 element language that is used for the tactile score, so it is quite rich
Discussion: How does your research mesh with each other?
- Semantic analysis with a machine is difficult, but humans understand the narrative easily.
- HCI is traditionally so focused on tasks, and we're now starting to deal with so many new types of interactions, and we don't know what it may lead to.
- Maybe this is about bringing meaning back into our interactions with computers and algorithms, searching for ways to include and retain meaning.
