Now, it often seems that this classification and recognition of these
signals that come in is already a solved problem.
In many lectures, we have now talked about what needs to be done, or
at least addressed that.
But I would like to talk a little bit about it today, where there are
actually still research tasks and challenges.
You see language learners here who work and translate simultaneously.
The impression among you as students could now arise that this is all
solved.
I prefer to look for something else to research.
And I want to get rid of this idea completely, because we still have a
lot of tasks that just don't work yet.
And we would like to address that briefly.
Well, first of all, this is a slide that I like to show to indicate
that communication can take on different forms when we involve the
machine here.
The communication between man and machine, which we usually always see
traditionally, is the interaction of man with machine.
And if that means language, it would be, so to speak, speaking into
the computer.
The computer recognizes that and types it out.
Now, that alone is simply not possible.
Or you can solve one or the other problem with it, like the
transcription here.
But in many situations, like now with the tasks with the robot, there
is also a need for interaction, namely the dialogue, so that the
computer can also answer a spoken statement.
That means you also have to be able to interpret this statement and
then formulate a response.
We have also briefly addressed that.
And that ultimately means that I understand the content of this
statement and then understand what the human desire is, so that I can
then formulate an answer accordingly.
But that doesn't stop there either, because there are also many tasks
for us in these research areas where the computer interacts with a
data source.
We are part of a large project that is underway in cooperation with
our French colleagues, where we are trying to interpret multimedia
data on the Internet, for example.
Imagine you want to search for television news.
You want to see what's new in football today.
What does the weather forecast look like?
And instead of looking at the weather forecast or the whole show, you
want to have an answer to your question quickly or be able to enter
the material quickly in order to find the corresponding points.
In such data sources, the information is usually contained, but very
often the information is in spoken form or visual form and not
necessarily in textual form.
Here, too, we need perception methods on how we understand the
content, both language recognition and image recognition.
And then the processing, the search of these materials from the
computer.
Another problem is, of course, these two arrows here.
First of all, that the computer can also be a link between two people.
In this case, it is not a human-machine communication, but a human
-human communication, which, however, is transmitted through the
computer.
In other words, people do not understand each other for some reason
and the computer produces the communication.
In our case, this means that two people want to communicate with each
other through translation, but do not understand the same language.
That means we use the computer to perform interpretation services.
Another possibility, however, is that we imagine that two people
communicate with each other and they are even in the same room and
communicate with each other, but the computer now observes this
communication and tries to provide additional assistance.
Imagine this as a butler who floats in an intelligent room in the
background and tries to provide them with the appropriate information
or even services at the appropriate place.
For example, a robot could not only be a human-machine communication,
but a humanoid robot could possibly stand here in the room and observe
two people communicating with each other.
If the appropriate word falls or the appropriate things are addressed,
they suddenly become active and say, Now I'm doing something.
I would now like a cup of coffee.
Then the robot should get up and bring me one.
Unfortunately, coffee was not delivered today.
So we have these different forms of communication.
Human-computer-human, human-human with computer observation and
computer with human data.
All of these now produce certain challenges.
Between 2004 and 2007, we had a project called CHILL, where we
especially studied such tasks and dealt with all these forms of
interaction that are not just human-machine, but human-human, where
the machine has an impact on them.
We had a few interesting applications.
You can imagine, if you wanted to build a computer butler who observes
in the background what people are doing, what could you do with it?
We have developed a number of such services.
One service was, for example, this situation that we all know.
We sit in a room, it's a discussion, it's a lecture, and somewhere a
cell phone rings.
Wouldn't it be nice if you could avoid that?
Because if the cell phone already knew that I was in the lecture and
then doesn't ring at all, but only communicates with two people when
they are accessible.
That means we want to avoid these interference influences and build a
butler who essentially knows when people can be communicated with each
other.
Another application would be the so-called Memory Jog, which we had
developed.
A device that I carry with me and when I'm in a conversation with
other people I can decide or know who these other people are and then
whisper into my ear, I forgot the name and then I say, what was his
name again?
And I sit across from him and I can't remember it and the device
whispers into my ear, that's your good friend, Rüdiger Dillmann,
you've known him for 20 years, how can you forget his name?
That means such a Memory Jog, with all kinds of additional
information, we all have it, it's all in your address books, but to
bring it into contact with the environment, with the world we live in,
is essentially the challenge here.
The other application of such management services, you see here, that
a device constantly listens and delivers automatic translation
services to each of us at every point.
Now, if you think about it all, you suddenly realize that it's
enormously difficult.
We'll be researching this for years, decades.
Why?
When I go to a meeting and come out of the meeting and you couldn't be
in the meeting and you ask me, why did Joe get angry at Bob about the
budget?
Why is that difficult?
It's a simple question, we as humans could simply answer it, but we
realize that for a machine that should answer this, it would need a
whole range of skills.
First of all, you have to realize that Joe and Bob were in the room,
that Joe addressed Bob, that the emotion was angry, that the content
of the conversation was the budget, and now you ask why that was so,
that means you still have to know something about the course of this
whole conversation.
That means we have to do a whole series of research tasks that all
have to work together.
Namely, you have to recognize language, you have to recognize words,
you have to identify the speaker, you have to determine the emotion,
you have to determine what style of conversation we are in.
Is this a lecture or a coffee talk?
We want to identify the language, we want to convey content,
handwriting recognition, visually we have to recognize identity,
gesture recognition, body language, follow faces and people, facial
expressions and also determine who is talking to whom.
Now these are all exactly the topics that we are doing in cooperation
with our friends in robotics, namely to decide what is going on here,
what do people do together, no matter if the information is only in
the acoustic signal or if it is possibly in the visual signal or in
the handwriting.
That is all that we discuss with the title Multimodal Perception.
We have to extract this information from different modalities.
We therefore often call it who, what, where, why and how.
Who is there, what is being done here, where are the people, why are
they doing something and how are they doing it?
This means that this interaction among people really needs such a
multimodal approach.
We actually equipped the project with rooms with many cameras,
microphones and sensors and then dealt with the description of such
events.
We have a lecture in which there is a speaker and now we want to
determine where he is, what he means, where he goes, what he does and
what he says.
All these questions suddenly play a role in identifying this scene.
When I have extracted this information, I can infer what the intention
or the course of these human interactions is.
We then converted this into all possible research tasks that we
systematically worked out and implemented in this project in
evaluations and made amazing improvements.
Mr. Stiefelhagen, whom you may know from the image processing,
worked on the focus of attention during his promotion.
How can I determine who is paying attention to whom, who is talking to
whom?
This was one of the tasks that we then incorporated in CHILL.
Now, in the language itself, in the acoustics, in the recognition of
spoken language, we have already seen that not all problems are
solved.
Because, as we have seen, acoustic signals are actually...
I don't know if this works.
Does it still work?
This was a language test to show that not everything is solved That
is, the recognition of spoken language is quite good when someone
speaks clearly but as soon as it becomes more spontaneous, the words
simply disappear from the acoustic signal and we now have to guess
what the person actually said.
This sentence was recorded in a meeting in the USA and the speaker
said it in conversation with the others.
We then asked the speaker to read the sentence she spoke from the
manuscript with the same microphone That is, we transcribed this
signal and presented it to her and asked her to say it again.
If you spontaneously say it in a meeting and then read your own
sentences afterwards,
it is the same sentence, the same speaker, the same microphone, read
from the manuscript and in the other case spoken spontaneously.
You see that the second sentence is clearer, easier to identify and
better to recognize.
The big surprise is that there is a doubling of the error rate.
That is, from spoken to spontaneous twice the error rate.
That is, much more difficult.
This is also a problem for international students.
You can speak the language well or have learned it at school and then
you come to another country and the people speak fluently and suddenly
it is very difficult to understand.
That is partly the reason for this.
We have also worked a lot on image processing in this project and, as
I said, Mr. Stiefelhagen continues to work on this.
We have to do with the recognition of faces.
We say low quality, a small picture in a camera section.
The lighting can sometimes be terrible.
People turn their heads.
It's not like you always see a nice photo of the person looking from
the front.
People turn their heads, they look from all possible sides and then it
is suddenly not so easy to recognize.
And then cover-up.
I always do that when I look in the room in such a lecture.
Then there are about 30% of the people who have their fingers in their
face.
Everywhere fingers in the face, pointer in the face, so that you don't
get a good look or a good picture of the face at all.
That means that the recognition of people in video material is not
easy because at first you don't even know where the face is and how to
recognize it before you can classify it at all.
When we talk about classification, you see that it suddenly becomes
much more difficult when you have to classify opportunistically,
namely at a moment when you see the face at all.
If you want to do the whole thing multimodally, it becomes even more
difficult.
Sometimes you can recognize the person acoustically because she
speaks, but somehow has her hands in front of her face.
Or at another moment the person says nothing and you can still see the
face visually.
All this leads us to such multimodal systems where you can combine the
whole thing.
I have a few videos here.
You see videos of the visual classification of people in our
laboratory at the time.
Down here you see this focus of attention to determine what people are
paying attention to and here on the left a gesture recognition that
combines the language with the visual gesture.
That means I point at something, say turn off this light, turn on this
loudspeaker and this clock should be set again.
Someone is sitting over here, could you answer this question?
All these gestures combined with the language then bring us
information or we need if we want to interact with a robot in a
meaningful way.
We have the whole thing in robot platforms.
In our area the robots don't look as sexy as the robots with the
robots.
Because they only need cameras and ears Otherwise they don't do much.
But in this context we studied dialogues and interaction mechanisms
between the systems and the device.
Now I would like to show you a video from this work.
We have a great video but if we have time at the end I will show you
this chill system.
The other problem we discussed today is the translation.
Here we work on many systems which are not only about the
classification of words but also about the recognition and the
translation in another language and the synthesis in the other
language so that the person on the other side can answer.
We need a language recognition in the other language a translation
from the other language and a text-to-speech synthesis in the own
language.
These wearable systems which translate words or sentences are now
available for purchase.
These are commercial systems where you can translate sentences from
one language to another.
This is a very nice interaction.
How are you today?
How long is it that we have not seen each other?
You can develop such systems on the phone and we use the same
statistical algorithms which we have already learned.
We use the base rules for language recognition in such systems for
translation where we try to guess what the original language was and
we do this with two models the translation model and the language
model.
The translation model replaces the acoustic model because we use a
statistical lexicon to do the translation.
All this research has gone through several cycles or several phases.
We have worked on research prototypes and in the last decades mainly
on practical systems which are used in the field as well as in these
simultaneous translators as we present them to you today.
This happened in four countries where there was constant advertising
with our app.
We were very happy because this led to the worldwide spread of these
systems.
In Japan, this system was launched in the first week as the best
-selling app in Japan when it was released in early 2010.
In the meantime, it is available on Android devices and iPhones and as
I said it is used in humanitarian missions.
Here you see a variant which works with an iPad so you have a nurse in
Thailand and an American doctor who interacts with Thai patients.
You have sick people and patients who come here to be treated.
This is a humanitarian mission which we carry out every year at the
beginning of the year in Thailand.
At the end of the summer we will go to Cambodia for another mission
this year and other missions are currently running in Honduras and
South America.
With this system we can help people in medical missions.
You see a happy Thai patient with his first glasses in his life.
These are remote areas where people never go to a doctor because they
can't afford it.
Such missions offer
enormous help.
This is a doctor from Pittsburgh who does charity work in remote
villages in the mountains of Honduras every year in South America.
This is a great area for our technology.
Now we talked about consecutive and simultaneous translation.
I don't have to say much about it because we talked about it all
morning and you see our simultaneous translators here.
These works started when we tried to translate and our first attempts
were between English and Spanish.
Here we have a video which unfortunately doesn't work but you can see
a parliamentary debate where we did the transcriptions and the
translations.
Now to the results.
This area is driven by good results and by decreasing error rates and
by permanently improving the performance of these systems.
In one of these projects you can see how the error rates improved
enormously by improving the algorithms.
We can't talk about many of the algorithms in detail here but I can
point out the lectures we hold every year.
Here you can see how the error rates decreased from 30% to 7%.
The so-called BLEU scores improve dramatically over the course of the
project.
We always start at the beginning as it should be.
But this result was more interesting.
People were asked to determine if they could understand what was
discussed in the content.
People were asked certain questions about the lecture and if this
question was answered correctly it was counted.
At the end of this survey about 62% of the questions could be answered
correctly if the translation was done
by a human being.
This was very interesting for me because it shows that the machine is
not so far away from the performance of the human translator.
But why is the human translation not better?
Why is it only 74%?
The answer is that it is very difficult to listen to new material and
at the same time to deliver translations.
Human translation is therefore difficult.
Human translators are exchanged every 20 minutes.
They forget some things or skip some things.
The fact that this is not 100% is due to the fact that a lot of
material is skipped in such a translation.
The translator did not come with the material but we can assume that
with better language recognition and translation we can increase this
number.
The omission of information is not improved because of cognitive
stress.
We cannot record more therefore we cannot translate better.
It could be assumed that at some point, possibly, even the human, the
machine translation could be the same or could even be in competition
with humans.
Now, here, too, there are evaluations.
I would like to briefly point out other evaluations that are carried
out under a competition at QUERO or at IWSLT or at WMT.
All these are competitions where we participate and always try to
improve our translation systems and the recognition systems.
All of this went a bit quickly.
This is an example of such translation scores, where the error rates
of the translators are counted.
And you see, for different language pairs, English-French, German
-English, English-German, our error rates are actually very nice
compared to other groups.
Now, in lectures, we have already seen that they are, of course, more
complicated.
This is a translation from a lecture by Mr. Dillmann, who, for
example, puts on the microphone at the beginning.
That means, you see how it starts with, aha, and that in the pocket,
and so on and so forth.
That means, you see how a kind of meta-dialogue actually takes place
at the beginning, where the user first only talks about the act of the
speech and then enters the actual lecture and actually talks about the
content of the material.
Now, these are automatic translation results.
You see, they are very nice, but they came from a transcript.
That means, in reality, we still have to add the error rates of the
language recognition, so that the result you observe here is no longer
quite as good.
That means, of course, we have to continue to work on both fronts, to
improve the language recognition algorithms as well as the translation
techniques and then, of course, to make the combination of the two
better and better.
Another point I would like to talk about is how we present the whole
thing.
You have the one presentation form in front of you.
Of course, we can simply broadcast it here on the screen in
translation.
But then we have the problem that we have to do this for ten languages
in the future, if we want to deliver the whole thing in English,
Chinese, Russian, Japanese and some other languages.
Then we would need ten screens in the room and that would no longer be
practicable.
What we do is, we beam everything over the Internet, over the servers,
so that you can all watch the issue as subtitles on your iPad or on
your personal phones and devices.
And that's already possible.
You can do that now.
You can go to the website and look at this subtitling.
Another method is to make these two slides accessible to you via
speakers, where the speakers are aligned.
We have also experimented with the fact that we practically install
very strongly radiating speakers.
They work with ultrasonic, so that a synthetic output, only in one
corner or in one corner of the room, is audible.
The vision here would be that we all sit in a meeting, each of us
speaks in our own language and everyone hears it in their own
language, in translation.
Because we get the translation, so to speak, whispered to by these
strongly aligned speakers.
That means, I sit in the room, I speak German, someone else speaks
Chinese, I get German whispered to and vice versa.
In this way, we believe that it will be possible in the future to hold
multilingual meetings where everyone speaks in their own language.
We have these speakers with us at the institute and are experimenting
with them.
Alternatively, you could, of course, instead of an iPhone or an iPad,
also think about simply inserting the text into the glasses.
That means, we have also worked with heads-up displays, with such
glasses, where you ultimately insert the screen into the glasses.
You can see an example of such glasses here, where there is a prism or
a mirror in it, where you then see the text in the picture.
That means, I can listen to my speaker and then look at the subtitle
under his face.
Now I would like to briefly report on the completion of a project that
we are currently actually working on.
This is the so-called U-Bridge project.
Whoever was there this morning, has one of the gentlemen here on the
podium, Mr. Cincioni, is responsible for this program of the RG, which
is supporting us here.
This is a big project.
This is the biggest project in this area at the moment, which is about
achieving three practical goals by improving the algorithms and
methods that lie underneath.
The first is to produce automatic subtitles under television news.
That means, we are working together with a company called Red Bee
Media.
This is a company from the BBC, the famous television company in
England.
They partially produce all subtitles for BBC news and films in a human
way.
That means, they have a lot of people who do it semi-automatically,
but that is of course an enormous cost to produce this subtitling and
at the moment no translation is delivered yet.
They are partners in this project.
So we work with them together.
We actually get news and data that come directly to us and then we try
to produce the subtitles and translations automatically under this
news.
In July we will also be in Berlin, where the company Titelbild
produces the translations to talk to us about how they normally do it,
mainly manually.
Here is the big chance to partially automate this.
The second point is multilingual lectures.
You can see an example of this right here, that we can and will
simultaneously translate lectures.
Of course, we use this in KIT, but other areas are these famous web
lectures.
You may have seen them before.
There are these TED lectures on the Internet, which are broadcast in
English, but also translations are possible here.
And then of course the European Parliament.
Ms. Altenberg, who was also here this morning in this panel, is
responsible for these human translators and has also confirmed to us
that it is no longer possible for the European Community to translate
all lectures and all conversations at the EG humanly.
This also means the possibility and the opportunity to partially
support translations mechanically.
Now in the long term, multilingual mobile telephone systems.
Here, too, we work together with companies to say that we want to
develop systems where people can ultimately communicate with each
other, where we can do the translation partially with individual
sentences, but in the future, of course, also make more fluid
teleconferences or meetings.
We have already presented this service, that the whole thing runs on a
server architecture and in this way we also have different use cases
or can support different services where users can simply join the
corresponding services.
Many of the problems we have today ...
Who was there this morning?
No shame.
Who was not there?
Okay.
Many of you have not noticed where the problems are with such a
simultaneous translation system.
I should perhaps mention a few of them.
One is, of course, the vocabularies.
That is, we have many vocabularies where special words occur that then
have to be built into the recognizer accordingly.
The word Newton, for example, in physics, in English, that will not
work here, of course, in the German recognizer.
But if you say this word in English, then it can of course be this
combination of two tones or new tones or the name of this famous
physicist.
That is, we have to distinguish here and if this word has not yet
appeared in the vocabulary, it will of course recognize this word
incorrectly and probably exchange it with something else.
In German lectures, we often use words that come from abroad.
We found this word laser in a lecture by Mr. Dillmann, who used a
laser as an optical part.
Here, my laser pointer.
Does that work?
My laser pointer.
So I have a laser pointer here that I use to point at the ceiling.
This word laser pointer went into the vocabulary, but of course our
system thought that the word laser in German should be pronounced L-A
-S -E-R, like laser.
That means, of course, laser is not recognized correctly.
In this case, we have to deal with errors if the wrong information is
in the vocabulary.
That means the question here, how do I fill in my vocabulary correctly
when it comes to foreign words?
You see, many of these words only work because they are either
corrected or because we actually work with English pronunciation
rules.
The word iPhone, for example.
How would you pronounce iPhone in German?
iPhone according to German pronunciation rules.
Iphone.
But that's not how we pronounce it.
That means we have to represent it with English pronunciation rules.
The last point is, of course, this stammering and spontaneous language
that occurs everywhere.
So if I cough or laugh in the middle of my sentence, then the
recognizer should actually produce nothing instead of some wrong
words.
I don't know if that always works right, but that is of course also
one of the challenges.
In addition, of course, we have speech dependence and independence
when different speakers speak.
But then there is also emphasis when someone makes a strong background
noise.
Or you may have heard my lecture on the topic of vocal tract.
When I simulate the vocal tract, I sometimes sing into the microphone.
And in such cases, what do I do with it?
How do I recognize it?
How do I represent it?
Microphones.
A huge problem.
If you are more in the direction of signal processing, this is also an
excellent challenge.
To switch from close-up microphones to distance microphones.
Distance microphones produce additional noises that make this room
echo.
You can hear that in this room the signal drops back and forth on the
walls.
And when it reaches the microphone, you get a disturbance source that
is a delayed copy of your own signal.
This is the most complicated way to remove the disturbance because we
are ultimately in competition with our own language.
This is also a huge open problem.
Speaking styles and style questions are also included here.
And then there are the metadialogues that I have already mentioned.
When Mr. Dillmann puts on his microphone and says, hmm, and yes, and
that, then this is a metadialogue.
This is not part of the lecture, but part of the preparation for the
lecture.
This is similar to questions in between.
When a question comes from you and I say, yes, hmm, then what do I do
with it?
How should it be classified?
How do we want to convey it so that it is clearly readable in the
lecture?
For many things we have initial approaches, but there is also a lot of
research that you can still imagine.
For example, we have carried out a diploma work in the last year,
where we have been able to achieve an improvement in the content of
the vocabulary.
We have already implemented this this summer semester.
That means we take the PowerPoint slides of the speakers, we do an
Internet search with them, we look for all the words from the Internet
that the system cannot yet, then we automatically make pronunciation
rules so that we have the pronunciation of these words and
automatically paste them into the vocabulary of our reader.
That means, based on these PowerPoint slides, we can exchange and
improve vocabulary rules.
That alone will not be enough, that means we also have additional
tools or editors with which a person, possibly you as a student
yourself, can then also correct words in the future.
Does that help?
The work here in this diploma work, as you can see very nicely, is
when you do this vocabulary adaptation, then you can actually see how
these missing words can be dramatically reduced.
That means you start with 5.6% words that are not in the vocabulary.
If 5.6% words are not in the vocabulary, then every missing word
produces an average of 1.5 false words in the recognition.
That means 5% missing words produces 7% higher error rates for us.
So it is very, very important to deal with this word problem in these
lecture recognizers and translators.
Because if we take a very wild lecture from another field, let our
recognizers run, they always start with an error rate of 40%.
And it is only when you then carry out this vocabulary adaptation that
you may come down to 20% or 20% to 25%.
And if you then also let the system adapt to the speaker, you come
down to about 15% to 18%, which we think we need to create a
translation.
So we need about 15% error rates in the worst case, 15% error rates to
be able to translate at all.
Now one last problem I would like to point out.
Not all problems are already done here in the area of language
recognition.
Over the years, we have made many dissertations and diplomas on the
topic of how we deal with the diversity of languages.
Of course, we can put a lot of years of research into German and
English, because you can get a lot of money there for research work.
Now, as you can see, the world is not only influenced by three
languages, but we have a total of about 6,000 languages.
And if we have to put a million dollars and a year of research work
into each of these languages, then for all linguists in the world that
is simply an astronomical sum that is completely unthinkable to
develop these systems accordingly.
So what do we do?
We have a whole series of tasks here.
You see, of course, for the very, very many people Chinese and
Spanish.
There are, of course, also a lot of work.
But you can see that even among the many spoken languages of the world
they also meet very quickly on languages where there are not many such
systems.
Very often this is then simply always a question whether there are
enough research resources available to do the work.
We therefore try to improve this problem with various approaches and
also research work.
We have to reduce the costs of building these systems.
And we do this partly through different strategies in which we either
make the modules language-independent or that we make the process
cheaper, that we work with so-called intermediate languages when
translating or that we reduce the performance of the systems also with
less data, that we may be able to select and prepare data better, that
we use human interpreters or translators and use the data from them or
that we interactively or through so-called crowdsourcing techniques
generate the data cheaper.
That means, we hope that at some point you will simply correct our
systems in the lecture and the system will then learn by itself
interactively and incrementally.
Only in this way do we get the problem under control so that those who
consume the output then also improve the systems through their own
improvement.
Another problem is the project that we are currently working on in the
USA.
This is a new project called Babel, which has just started.
This also offers opportunities for you in research The challenge is to
quickly design language recognition for a new language.
How long do we need to build a language recognition in a new language?
Even if the data is available.
We see that it actually takes a year until you get reasonable error
rates and even if the data is available.
What we are trying to do in this project is to dramatically reduce the
time of building such a system.
By automatically generating lexical morphology and automatically
finding words.
There are also many partners here.
We are in charge of this project.
The Babel project at Johns Hopkins University is part of our group in
Karlsruhe.
Mr. Stücker is the head of this part and the colleagues in Saarland
and Mobile Technologies.
This system also involves building several language recognition
machines that then produce corresponding word hypothesis graphs that
are then indicated so that a language indication can be carried out on
this in this project.
The critical part is this automatic language recognition, automatic
speech recognition, to create machines quickly when a new language is
attacked.
That was all I wanted to say on these topics.
As you can see, we still have many tasks in language recognition and
translation and then of course in combination with other modalities so
that the perception with machines or the interaction with machines is
actually better and more flexible so that it can be made more portable
so that it can work better for different domains and different
languages.
Of course, there are research problems Of course, we are constantly
working on new studies, formal studies, diploma studies and
dissertations and of course we would be happy if we could interest you.