Q: What are your current research interests?

A: I'm interested in how different forms of communication, as they are configured in different places - the primary classroom, the secondary classroom, the museum or gallery - how those things provide different possibilities for thinking and engaging with knowledge. And how people's use of those resources can produce new kinds of meanings. The design of meaning in different spaces.

Being at the Knowledge Lab, I've stuck a big frame around that, saying 'All of this in the realm of technology.' And every now and again I panic because I'm not quite in the realm of technology.

But you are, because the multimodal research you do [see here for example] seems to be - or could be, or should be - central to how we learn with technology.

I think it is, but it's weird though. You know when academics suddenly realise something exists, that artists and designers and typographers have known forever? An academic suddenly realises, 'Do you know what? When you write, it's not just about the words you're using.'

Someone who has been typesetting and thinking about this for ages knows that. Everything has always been multimodal. But when we did the English project and the science project, we didn't see any new technologies in those classrooms at all.

What was the English project?

It was between 2000 and 2003, called The Production of School English. The book from it was called English in Urban Classrooms. It asked the question, What is English as a school subject? In a sense, that question is brought about by the changing communicational landscape. In the past, it was quite clear what English was about - literature, language, these kinds of things. There was a complete connection between 'Englishness' - the sensibility of English - and literature. Literature as a way for the working class to become morally good.

You've got the official curriculum. But in classrooms, in schools that are ethnically and racially diverse, that are working class - what is that Englishness? How does the subject and the sensibility of studying it get realised today, and for what purposes?

So we looked at how English was being realised, not just through what teachers and students said and wrote but also the design of the classroom, what kinds of texts, what kinds of materials came in, whether or not images were in the classroom, how they were used, how gestures, body posture and position in the classroom of the teacher and students created meanings as well. So English - which common sense would just see as just about language - we looked at that as a site of multimodal communication.

Where does multimodal theory come from?

It comes from different places. In part from the Hallidayan functional systemic linguistics. That says that languages developed to fulfill social purposes, the representational needs of people, to represent the world - creating social relations and coherence.

reading images grew out of that desire to try and understand images in that way.

Other people working in the area started to say that It's just a kind of linguistic imperialism to take these ideas from languages and impose them on these other modes. People brought in ideas from art history or cinema, sociology of space and so on. And some carried on with a relatively pure linguistic approach- so there's a whole load of people doing multimodality who are very strictly systemic functional grammar linguists. And they apply those categorisations to images, buildings, whatever.

There's another lot who talk more about discourse analysis. They're working more with social interactionist ideas of turn taking, context - these kinds of things. Then there are other people still - like Charles Goodwin in the US, who does conversation analysis and ethnomethodology - who have expanded their work to look not just at what people are saying but what they're doing as well.

So it is kind of happening everywhere. Technology does have something to do with it. People say that conversation analysis came about because with tape recorders you could hold on to speech. And video cameras enable you to hold onto and look at interactions in different ways. So it's a combination of theorising, the technology to capture and look at it, and also the social landscape that changes the uses of technology as well. I think it's all of those things.

As texts - the materials that linguists were looking at - started to have more images in them - well, can you apply those ideas about meaning to the visual as well? So Gunther and Theo's work on

I always thought of it as looking at more than just language, but as you explain it, it's applying linguistic principles to other modes.

When Halliday developed social semiotics, and other theories of meaning - they dealt with language because that's what they dealt with. That's what the technology enabled looking at in a clear way. When he talks about ideational meaning - representing what's in the world, and interpersonal meaning - representing how human relations are constructed and the textual giving that all coherence, he develops that in relation to language.

But I think that if that is a higher order of social meaning, and if - as I think is the case - gesture and image and all these other things develop also to realize social functions, as people use them for their social purposes, alongside or independent of language, then why can't you say that all of those things are involved in realising similar kinds of meanings? I think they are.

What Gunther and Theo did was to say, 'Let's take these realisational images of language and put them onto the visual and see what happens.' That can be criticised because images aren't words. So it's not going to entirely work. But it is a way of opening up meaning across a range of modes and beginning to theorise how these modes and modal configurations contribute to meaning and learning.

book about multimodal discourse really acknowledges that. Rather than focus on individual modes it attempts to get at meanings across modes. For instance, 'framing' happens across all of these modes, and focusing on semiotic elements across modes is another way of getting at multimodal meaning.

I have tried to combine multimodality more strongly with other social theories of learning. For me multimodality is really useful, but it's more of a descriptive tool and without an additional theory I think what it can show is limited – as all theories are.

And that's what they realised in doing it. Their later

So as a tool, is there a particular way of using it, a particular methodology?

In the science project, that's what we tried to develop. This was from 1997 to 2000, also funded by the ESRC. It was called The Rhetorics of the Science Classroom. The book is called Multimodal Teaching and Learning [2001].

Everyone would acknowledge that action has a role in science education - the manipulation of objects, for example. And images have a place in the classroom. But a lot of science educators would say that those things just support what is being said. Because ultimately, doing science is about talking science, and learning science is about being able to articulate that verbally or in writing.

The project looked at how science is realised in the classroom. When an experiment is going on, is it just illustrating or does it give additional information? What gets done in speech and what gets done in the other modes? And how are speech and writing and image and action all being orchestrated by the teacher in the classroom to make meaning?

What we showed was that - unsurprisingly - all of these modes are central to learning and teaching, and what is central goes across modes. So what we argue is that if you look at language alone, you get a very partial picture of what is happening. You need to embed language in all these other modes.

Another point was to develop methodological tools. So we tried to think about how to transcribe this stuff, map it, analyse it.

There's a kind of rush on at the moment about how to pin down the 'right' methodology for doing multimodality. Personally I'm trying to avoid that, because I think it really depends what you're looking at, why you're looking at it, what kinds of questions you're trying to answer, what theory you've got of learning.

The way that Gunther and I do multimodality, I think there's a clear set of conceptual tools, and there is a clear set of processes for how you transcribe a piece of data and analyse it. But within that, the role of other social theory will change that.

So in my PhD I put together multimodality with activity theory. And in the English project we brought in Bourdieu and other social theories to look at larger-scale issues. In the interactive whiteboards project with Gemma Moss, we brought in her work around new literacy studies, and ideas of texts in action. In another project with colleagues in Delhi and Johannesburg, we brought in ideas around culture and language - kind of multiliteracies with multimodality, to contrast the differences in the three countries.

The whiteboards project has gotten a lot of press, though it has tended to be - as it usually is - simplistic. Whiteboards don't work, essentially.

It's really problematic because a particular strand of the tender was to look at the statistical impact of interactive whiteboards on student attainment. And we didn't think that you'd actually see an impact of any technology, no matter how fantastic, on pupils' outcomes, in just one year. Because the technology is still getting embedded in the classroom and teachers' practices.

So it's a shame that that's what was picked up in the press because that was, in a sense, a strand that we, and the funders, knew wasn't going to show a great change, but that it would provide baseline data for the future. That was why it was there.

What was more interesting? Was anything surprising?

One thing that was surprising was that over 70 percent of teachers make their own materials for the interactive whiteboard. In a way it's nothing new to say that the materials used in a classroom shape the pedagogy of the classroom. But what was interesting was that the things that got displayed on the interactive whiteboard - and this is true with a lot of commercial materials as well - tend to be designed around familiar practices, familiar kinds of texts and materials. So the worksheet, for example, migrates to the interactive whiteboard.

For people making commercial resources, that happens because they're designing for a familiar set of practices so that people will buy their tools. For teachers, their design principles and experience are based on what they've done before. So it's kind of unsurprising, actually, that a lot of the interactive facilities of the whiteboards are not used. Because the texts used on them don't demand interactivity that often.

We did see some very interesting uses of the boards with peripherals. Anything that moved the teacher away from the front of the board, and anything that changed whose texts appeared on the whiteboard, seemed to make an interesting difference.

So in English, we saw teachers scanning students' texts that were made on the spot, to display them on the interactive whiteboard, then annotate them. That couldn't happen before, that a student's text would become the focus of a whole class' attention.

It gives it a kind of digital authority.

It also creates a new kind of textual space. It's not the teacher's text anymore and it's not the student's text anymore. It's a hybrid text, which undoes some of the notions about ownership of the board. And I think those spaces that can get opened up are really interesting. The digital whiteboard is a space where new kinds of texts can come into the classroom.



If you envision them as three knobs on a graphic equalizer - pace, multimodality and interactivity - at the moment, the idea is that you turn them all up to full. But that's not a very useful message. What is more appropriate is to think about when you would turn interactivity up, when would you turn it down - to see it as a much more subtle interaction between those three elements, in relation to the pedagogic intentions. How can a teacher know when using an image is apt or not?

Our main finding was that the kind of rhetoric of how the boards are positioned - commercially, through the government, and also through teachers' own discourses about technology - is that there is an emerging idea of good pedagogic practice as being fast, multimodal, and interactive.

What was the project with Delhi and Johannesburg?

It's was really small but a lovely project, and I think it will blossom into something else.

After the English project, a colleague, Pippa Stein at Witwatersrand University, asked if I thought that the things we'd found in the English classroom in England would resonate with the kinds of things they'd find in Johannesburg - or if the methodologies could be transported.

They're really interested in South Africa in developing south-south relationships - so working with Delhi and other southern hemisphere partners. So we looked at the English findings and methodologies in relation to post-colonial contexts - how English as a subject is so connected to Englishness.

So we got a small British Academy grant to fund a collaboration between us, basically for me to go over and work with the partners in Johannesburg and Delhi. And we shared video data, we went on visits to schools.

And we saw how English as a multimodal subject got configured differently by colonial forces in those three places.

I's very interesting to see what role talk and writing have, and what role performance has. We looked at how teachers work with particular texts, but also how textbooks come into these places. In India, all teachers are teaching from the same textbook at the same time. Each week what is to be taught, and how it's to be taught, is uploaded to a web site. And all teachers teach to that program, unless they're an elite private school. In Johannesburg there's a much looser notion of the curriculum.



In the Delhi classroom, you didn't see all those modes. The mode of gesture or body posture was silence. What does it mean for the production of your identity?

On the other hand, in the Johannesburg classroom we worked with there was a series of lessons that had all this multimodal kind of pedagogy, and lots of high-quality discussion. But absolutely no writing. At the end of the day, the teacher didn't get students to write.

In the UK for example, we looked at this lesson where they're teaching Macbeth. There's loads of different ways it can be taught - making up comic book narratives, animated films, dramatic films, performing scenes from the play. The assignment was an essay about staging the play.

You're still co-editing the journal Visual Communication. How is that going?

We're in our sixth year, and next year we go to four issues a year instead of three. We have had three special issues - the first on typography, the second on screens, and the third, which I'm just reading the proofs for, is on immersion.

For example, we've got a visual essay by Sharon Gould who is a gold medal Olympic swimmer, talking about what it means to be in the water. She's got this technique where she takes photographs underwater to understand swimming strokes.

Another piece is about intelligent fabrics - how the skin is a kind of boundary, how you get immersed in different kinds of experiences through technologies.

We're still trying to broaden what a journal article can be, with the visual essays, with practitioner pieces, stuff like that.

Thinking multimodally.

Yeah, but it throws up lots of questions about the relation between image and writing. So sometimes we'll have a visual essay, and while one of us might really like it, the other would say, 'Well, what's it really saying? Is the image illustrating the word or vice versa?' There are real difficulties in trying to find new genres of making texts – especially in a printed form.

Are you writing anything else?

I've just got a contract to edit a handbook called The Routledge Handbook of Multimodal Analysis. It'll be a very effective doorstop, if nothing else - it'll be 24 chapters, in four parts.

The first part will sketch different theoretical concepts quite fully - language, materiality, that kind of thing. The second section is on particular social forces that shape modal configurations - things like culture, time, technologies.



There will be a final section of case studies showing the application of multimodal research in different contexts - to show how it can be used to answer particular kinds of questions.

So that's a big project.

Another section is around how different disciplinary perspectives engage with multimodality - sociology, semiotics, psychology - to map the fluidity of what multimodality is, not to tie it down, because I think it's too young to be tied down. It's just not a clear thing, and I don't want to help create a discipline.

You're teaching as well.

I'm just doing a bit of teaching. We've got this idea to establish a Centre for Multimodal Research here, and it will be launched as a network in September. So we're trying to rationalise the courses that link to multimodality, to roll them out as a kind of programme over the year, so that when students have an interest in multimodality they can develop it through the year. It'll make it easier for people who are working in that area, or who are just interested in it, to develop some kind of ongoing community. Because at the moment it's just too idiosyncratic, having grown up organically.

I'm teaching a couple of courses on video analysis. These will be expanded next year to focus on the practicalities of using video, because many people have never even used a video camera before. And then something around data management, storing, and the use of computer technologies, and something on transcription and maybe some kind of extended data analysis workshops where people bring in their own stuff.

And I'm doing a course around visual communication and learning. I try to work with students' data. A bit like the interactive whiteboards project - why should I keep sticking my data up there? I should use the students who are in the course, and get people to work with their data in groups.