What in this case would be considered a multimodal environment?

Basically we're looking at using different modes of communicating. In terms of the technology, different ways in which we can handle input and output or feedback - it might be visual, it might be touch or sound.

Language is a problem because we're dealing with children at Key Stage 1 (5-7 years old) - very young. The idea is that we tackle this age group because this is the point at which social interaction comes in very strongly. Children may have problems with communication or interacting with peers; there may be situations they find intimidating - there are a number of different factors that may come into children failing to engage with peers in social interaction and verbal communication.

Particularly here in London where you have so many cultures and accents.

Yes, indeed. Plus this would be even harder if you considered all the regional variations such as South and North English, Scottish and Welsh variations, which are the areas of the UK in which we will be working.

The project is spread across various institutions in the Scotland, England and Wales, led from Edinburgh. We identified various areas of expertise which we didn't have when we conceived the idea of ECHOES. So we've got developmental psychologists, people who deal specifically with Autism Spectrum Disorder (ASD), we have media people who have worked in television on programs for kids. We've got computational linguists, cognitive scientists, and also artificial intelligence people. And accessibility specialists as well.


What is your specific role?

Do you have some ideas about the technology, and how you're going to implement this?

We're very cautious, and realise the difficulties we're facing. So we devised a plan for technology development which is iterative and divided into specific stages. We're also very keen to use existing technologies, and trying to integrate them into a new environment that we can exploit in this particular context.

So we will be starting with a system developed by the University of Wales Institute Cardiff, called Reactive Colours. It's quite well known - and becoming more and more known, especially in the ASD community. It's very simple, rudimentary from the point of view of AI in Education. It involves an interactive whiteboard and different stimuli such as sound and touch.

Basically the idea is that children stand in front of the whiteboard and play with it. There is no language involved, no structured pedagogy; it's not driven by pedagogical considerations, it's driven primarily by the need to engage children in an activity, and then discover what they do with that activity, where they take it, and what kind of impact that has on them.

So far it's really encouraging, for all sorts of reasons. One is how autistic children engage - the fact that they can engage with this kind of environment. There is some evidence - though not formally tested - that children who engage with the Reactive Colours system manifested some intentionality. We're talking here about the low end of ASD - these are kids who don't normally communicate, even with their own parents, who find the real world environment quite disturbing. There's just too much noise for those kids - that's a typical problem for people with ASD.

Is the idea to work with them in groups or pairs, or does the system act as a precursor or stimulant for social interaction?

It's both, being a very exploratory project. It's all observational - what hardcore psychologists call anecdotal, which is fair enough. But if you're in the business of designing technology for kids - especially educational technology to help children with specific difficulties - you have to come from both directions. You have to try to give as much freedom as possible to the kids, to give them the opportunity to tell us what they need. Because you cannot predict how children will react in specific environments. That's true of any environment, not just digital.

We as adults and researchers and therapists and teachers and parents have our own ideas of what might work, what we would like to work for our kids. We have theories and biases. But children tend to find their own ways of interacting. This is one thing that is really nice about Reactive Colours. What we're trying to do in ECHOES is embrace that, but also provide some hard evidence, and give pedagogical underpinning - so the interaction is more structured.

So one aspect that is really crucial to the project is participatory design with the kids and with practitioners. We take it extremely seriously that children are involved in the design of the environment right from the start.

You've worked with kids in research before?

No, actually - I've worked with adults. But there are people, thankfully, on the project who have worked with kids - particularly kids with ASD.

Is this related to the fact that English is not your first language?

It's my third. I came from Poland to the UK 18 years ago. I did my first degree in Warsaw, completely unrelated to this - though actually now I can see it coming in to what I do. It was in acting and drama. But I decided I didn't want to be an actress - I didn't enjoy it as much as I thought I would.

It's an interesting way into linguistics.

It is, though linguistics runs through my family. You spend much of your childhood, especially as a teen, trying to escape from your parents - they are both philologists. But then you do a full circle and end up where they started!

I came to the UK to do linguistics, in Edinburgh, a joint honours degree in linguistics and French, French being my second language. But through an absolute coincidence I spoke to some people who did AI. I knew nothing about it, had no training in computer science whatsoever. In fact up to that point I didn't consider myself in any way a scientist. But I thought it sounded really interesting - the study of intelligence, not only human but in general. And how you could bring in the study of language into it, and formalise it as well - try to capture it in order to understand it. Which is what traditional linguistics does anyway. But computer science brings an extra dimension to it, further requirements of specificity and precision - this was something that really fascinated me.

I still struggle with this thought of trying to capture something so organic as language into a formal system. Nevertheless it's something that we humans try to do all the time, whether we call it philosophy or linguistics or social science, or indeed AI. And I think it has a really useful side to it, this formal approach. You might not end up capturing the whole of the organic system of communication, but you may get snippets that, in themselves, will be informative as to who we are and how we communicate and how we think. The possibilities for practical applications is how I came to be in education.

How do you find your acting experience integrating into ECHOES?

What we're trying to do involves creativity - how you devise learning scenarios, for example, in a way that engage. Not just for ECHOES but everything I have done during and after my PhD in relation to education. Any pedagogical activity involves an aspect of performance - not just in terms of how well you do, but putting yourself in a hypothetical situation and being able to perform in that situation.

In ECHOES it's completely visible because children will be engaging in this enhanced reality environment with different objects, with animated agents, in interactions of various sorts. In that sense I think it's just a dimension that opens up that aspect. I'm not sure how formal this will become, to what extent I will use it. But it's there at the back of my mind, always.

Do you still divide your time between London and Edinburgh?

I do still have an honorary fellowship there, at the Human Communication Research Centre, and I do like keeping that connection - I think it's an important connection to have, for me personally but also for LKL as well because they're very formal there, computationally. Social science is kind of chaotic by comparison, though not altogether unpleasant. I was really worried coming here because I'm not an educationalist by any definition.

But the thing I discovered is that it's not really how you categorise yourself, it's what inspires you at different points. And this place really made me free again - actually seeing things from different perspectives, which is what interdisciplinarity should be. So I think at the moment this is the perfect place for me.

Keeping the connection with Edinburgh is important to me because it is a constant reminder of the need for rigorous methodology in order to understand complex, and often hidden from the naked eye, processes. I think there is so much potential in combining the rigour of computer science and the openness and creativity and social aspects of education - that's something that really inspires me.

Find details on the previous project, ECHOES, here.