So, we are first of all very grateful for the panelists who have
joined us here and have come from afar to discuss with us the future
potential of language technologies.
I will briefly say their names.
Professor Dillman is a colleague of mine here at KIT.
Dr. Cincioni is in charge of language technologies of the European
Commission and has a long history and experience with managing
projects in the European Commission in language technologies.
Professor Carbonell is my colleague at Carnegie Mellon University and
also the department head of the Language Technology Institute, in
which Interact is also housed at Carnegie Mellon.
Ms. Altenberg, we are also very happy that you made the long trip from
the European Parliament.
She is engaged in the human aspect of translating speakers in the
European Parliament, so she intimately knows about the problems that
we are facing.
Professor Asfor, another colleague of ours here in KIT, who has
participated with us this summer in delivering these lectures in
simultaneous translation by computer.
And Mr. Karnadi, a student at KIT from Indonesia, who you have seen in
the videotape.
So one of the victims of both the language problems that we have here,
as well as the technologies that we are imposing on the students who
have these language problems.
So thank you, first of all, for coming.
We would like to begin this podium discussion, this panel discussion,
by giving each of the speakers an opportunity to introduce themselves
and speak a little bit about their own role and their own relationship
to language and language problems.
Perhaps what we can do is we can just go from left to right in
sequence.
Without further ado, I ask Professor Dillmann to begin.
Good morning.
My name is Rüdiger Dillmann.
I am the speaker from the Institute of Anthropomatics.
And we are studying mainly how men can interact with different
machines.
I personally am interested in humanoid robots and interaction between
humans and humanoid robots.
Here we have, in terms of language, a lot of difficulties.
One is to show a robot and to explain a robot what has to be done, how
it has to be done.
Then to ask the robot, for example, how it perceives the environment.
For example, say what you see.
The other one is that a robot can also explain why it's doing
something in a certain way.
The problem we have here is that it's more than transcription.
It's language understanding.
It's generating representations which can be understood by both humans
and the robot, so that they can exchange information and that they can
interact.
The first step has been done already, that an interface between humans
and machines can be established.
But the problem to be solved is to go deeper, to have in machines some
kind of mental model, which allows to have meaningful exchange of
information and to speak meaningful things between machines and
robots.
At the end, I think the robot should also be able to have maybe a
cultural bridge, for example, that people can communicate, for
example, over long distances via humanoid robots.
For example, we in Karlsruhe speak German with our robot.
Then the robot transmits that to a Japanese robot, for example, at
Waseda University.
And that robot can translate what we said in German, in Japanese, and
also doing that, what we showed the robot.
So this is very far a vision on cultural bridging where we have human,
machine, humans, maybe with cultural barriers.
Thank you.
It has been tough, four hours to get here from Luxembourg.
Okay, my name is Roberto Ciancioni.
I'm the head of the unit within the large department of the European
Commission, the Information Society Department, which provokes,
promotes, sponsors research and technology work into speech,
technology, and natural language processing.
So the Commission has been doing that since the early 80s, with a
number of ups and downs.
The last up happened around 2008, 2009, when, as a consequence of the
enlargement, as you know, I mean, quite a few, I mean, new countries,
I mean, entered the community, due to the perception that there were a
number of interesting, I mean, advances in scientific and
technological terms, linked, I mean, to statistical data-driven
approaches and the like.
Due to the explosion of online content, in particular, I mean, social
networks and the like, there was, I mean, a sort of a new, a new
rewound, I mean, program in the area of language technology, which has
yielded, I mean, some nearly 60 projects, totally around €150 million.
Not much, if you just consider the sheer number of languages and
language peers, and the sheer number of people, I mean, who apply and
aspire, I mean, to receive some sort of financial.
Now, projects, our projects, just one word, I mean, may be different
from what most of you are used to.
So we try, and we have to, that's the role of European Commission in
this particular area, to stimulate a collaboration between entities
from different countries, from different, I mean, scientific technical
quarters, whenever possible.
We try to stimulate alliances, be it, I mean, within a certain project
framework, between researchers, between commercial technology
providers and leading users.
And, in a way, over and beyond, I mean, scientific technical work, we
try, I mean, to connect the dots, i.e., create, I mean, some little
path, taking, I mean, these results, I mean, towards exploitation.
So making even imperfect, but usable, I mean, technologies, I mean,
operationally and commercially used.
And this is one of the reasons why I welcome, I mean, the work of
Professor Weiber and his collaborators, because, I mean, there is an
element of exploitation, whether it's, I mean, internal application,
commercial, or whatever.
So, one obvious element for us is that we tend to expect all our
projects, I mean, to address multiple languages.
So it's not the English plus 3% of in German.
We try, I mean, to have a more balanced, I mean, coverage of various
languages.
Although, of course, let's recognize that English is still, I mean,
even in scientific terms, dominating largely, I mean, the market of
public spending programs.
And just to finish off, I mean, since I'm neither a scientist nor a
bureaucrat, we are now launching, I mean, another call, which is 30,
40 million euro available, not much, again, which addresses, I mean,
issues that what you've been demonstrating, I mean, touch upon.
The conversion between translation and speech recognition, and maybe a
bit of understanding.
Recognition for sure.
Dialogue where possible.
Understanding, I mean, you know, eventually.
And that's it.
Thank you.
Hi.
As Alex said, I'm Jaime Carbonell, the Director of the Language
Technologies Institute at Carnegie Mellon University.
Before I say something about the technical aspects of language and
translation, I simply want to emphasize the difficulty of simultaneous
interpretation.
I always thought it was hard, and then at some point I was forced to
do it myself.
I was in All America's conference, scientific conference in Buenos
Aires, Argentina, and the guest speaker was Professor Colmarauer from
France.
The simultaneous translator never showed up.
And he was going to make his presentation in French, and so you needed
a translator from French to Spanish.
I was the closest approximation.
And so I tried.
It was extremely difficult.
My output was not as good as the machine you saw today.
Some things I could not think fast enough to translate.
You cannot appreciate until you do it on the spot in front of even
more people than are here.
So I really welcome the machines, so I don't have to do it again.
Maybe the other professionals are better than I at doing that.
Let me now say a few words about translation.
My focus area is human-to-human communication via machine-enabled
translation, primarily by clarification, by automatic summarization,
by automatic search of information in other languages.
At the Language Technologies Institute, in preparation for this panel,
I had my group of people enumerate the number of languages that we had
worked on, including a large part of them in the Interact laboratory.
It was 55.
That's a tiny fraction of the 6,000 languages in the world, and even a
tinier fraction of the language pairs.
And one of the languages we have worked with is the world's
southernmost language.
Does anybody here know what that language is called?
Well, it's Mapudungun.
It's spoken by the Mapuche people in the southern part of Chile.
There are no natives in Antarctica, so it cannot go further south.
We also worked in the world's northernmost language.
Anybody know what that one is?
There are two strong contenders.
One is Greenlandic.
The other one is Inupiaq.
Inupiaq has the settlement that is furthest north.
So I suppose that counts as the world's northernmost language.
Now we're trying to figure out which is the eastern and westernmost
language.
We do not know that yet.
But if there is an official definition, perhaps we will work on them
too.
We are working on translation of difficult and rare languages.
Mapudungun, for example, is called highly agglomerative.
That means it has complex morphosyntax.
One word in Mapudungun maps into seven words in Spanish in some of
these examples.
Spanish and English have approximately the same number of words.
German sometimes puts words together, but not as much as Mapudungun,
not nearly as much.
If you think that segmentation is difficult in German, it's more
difficult for some other languages.
The reason that we're doing this is that we're trying to understand
the DNA of all the languages in the world as much as we can.
We can only sample.
We don't have anywhere near the capacity to do them all.
So we're sampling African languages, Native North American and South
American languages, European languages, and Eastern languages, of
course.
This is a small step towards true language universal understanding.
Thank you.
Thank you very much.
Good morning.
I'm very much impressed by all the scientific excellence in this room
today and in this institute and worldwide.
I come more from the practical side of things because I'm a human
conference interpreter.
I do simultaneous interpreting in the European Parliament.
I do no longer interpret myself, but I used to be.
I'm a trained interpreter.
And I work from English, French, Spanish, Dutch and Polish into German
in the meetings of the European Parliament.
And after being responsible for all the German interpreters in the
European Parliament, I took over their unit for multilingualism
support.
Which very much also deals with this aspect of language technology.
And I'm quite impressed by what my machine colleague did this morning.
But just to give you a little bit of background of the constraints we
are working in.
First of all, the European Parliament together with the European
Commission is the biggest language service worldwide.
And the European Parliament is a real multilingual institution.
And so far as all our meetings, all our meetings are interpreted in
all the official languages.
On a demand basis, of course.
If in a meeting a certain language is not represented, we don't have
interpretation.
Which makes 23 official languages, soon with Croatian 24.
Just to give you an overview of the challenges we're dealing with.
And multilingualism in a Parliament, I think, is very much the basis
of democracy.
You were talking about a university without barriers.
Language knowledge should not be a prerequisite to learning.
But language knowledge should also not pose a barrier to democratic
representation.
So the members of the European Parliament are elected by the citizens
because they represent their ideas.
And not because they're fluent in several foreign languages.
And that's why we have to offer communication and translation and
interpretation to our members.
Now, just about the terminology.
Translation is written translation.
And that, in the European institutions, is very much separated from
interpretation, which is oral translation, so to say.
And whereas our colleagues, the translators who did written
translation, are already confronted and deal on a regular basis with a
computer -assisted translation.
They work with machine translation.
They use these tools.
It hasn't yet so much gone into the interpretation side of things.
And that's why I'm very much interested in the results of this work.
Until now, interpreters used to say, well, that's not for us.
It's something that the translators can deal with.
Because there is, of course, also a lot of resistance.
People think, oh, will we soon all be replaced by machines?
I think one has to approach it from a different way.
But I think we will talk about this later.
Just to give you one more idea is that we were talking a lot about use
of English this morning.
The way we work in interpretation, we don't use one hub or spoke
system or one language.
So the aim is to provide direct interpretation.
For the moment, we have 506 language combinations.
Now, with Croatian, this will be multiplied.
So, for example, in a plenary session of the European Parliament, the
aim is to interpret all languages directly into the other languages.
The aim.
It does not work for all language combinations for obvious reasons.
We can provide direct interpretation for the more common languages.
But to find the interpreter who can interpret from Greek into Finnish,
although we have one, but she cannot be in all meetings at the same
time, then we use a PIVO language.
So Greek would first go into German, doesn't necessarily have to be
English or into Spanish, and would then be interpreted into the other
language.
And finally, maybe to give you some idea of the human dimensions, in
the European Parliament, we have roughly 400 staff interpreters.
In the European Commission, there are also many more, even than 400.
We work with a pool of about 2,000 freelance interpreters.
And after this meeting today, I will travel to Strasbourg, which is
not very far away, so it was convenient on the way between Brussels
and Strasbourg, for the plenary sessions of the European Parliament,
where for one day, roughly 1,000 interpreter staff and freelance are
working to interpret in the different meetings.
And I'm sure the English Booth colleagues who have to interpret from
German into English, I will tell them how difficult their work is.
Because they keep saying German is so difficult, and I keep convincing
them it's not.
But if I can now tell them that the machine translator has exactly the
same problems as they have, they will be very much relieved, I'm sure.
Thank you very much.
So, my name is Tamim Asfour.
I am here at KIT working on humanoid robots since 10 years.
And one of the challenges, or our goal, is to make these robots
available for humans in daily environment.
And of course, communication and interaction with the robots is a
very, very important challenge.
We were working with Alex's group in the last 10 years, and of course,
on this topic, on how to make the robots, or how to provide the robots
with natural dialogue components where a human can interact with these
robots in the same way as humans do.
And of course, I participated, it was a pleasure for me to participate
in the lecture translated this year, with the lecture Rishna
Organisation.
And from scientific point of view, I see several parallels or
interests from my side to the natural language or language
technologies and speech.
On the one side, my goal or my dream is to generate actions on robots
from a limited set of elementary actions.
So this is the fact how it is done in language.
So we have a limited set of letters from which we can combine or build
words and combine or build semantically and syntactically correct
sentences.
And the question is how to translate these aspects to generate complex
actions of a robot.
On the other side, I think what a very important aspect nowadays is
skill transfer between different embodiments and different robots.
And a very important aspect, what Professor Dillman mentioned, is how
a multicultural skill transfer can work based on the translation
software or translation technologies we have nowadays.
And the last thing I would like to mention is the capability of or how
we can endow robots with the capability of generating semantic scene
representations.
So from perceptual point of view, how to use technologies which are
none and used in natural language or in translation to use that in a
very concrete scenario where a robot has to tell a human what's on the
table, where is the red cup, in a natural way.
How to generate these natural representations based on the
methodologies and techniques similar to language.
Thank you.
So, hello everyone.
My name is Karl Karnadi.
I'm a computer science student here in CASO Institute of Technology.
And I come from Indonesia.
So I'm speaking my native language, my Bahasa Indonesia, of course.
And I'm speaking English quite well, but I'm speaking also German, but
in a more limited extent, I should say.
So, I'm here to provide some perspective from the student point of
view.
So, when I came here to Germany, I came here in 2006, I actually got
one of the best grades in the German language examination that I have
to pass in order to be able to study here in Germany.
So I got Eins, I got one in that DSH examination.
But, even with that good grades, that doesn't count at all when I come
to the lecture hall and actually try to understand the lectures.
It's a different thing, you know, to pass an exam and to understand
the lectures.
So I'm having a lot of difficulty to understand the German lectures.
Well, it could be sometimes because of technical terms, sometimes
because of the new German words that I've never heard before, and
sometimes it's also simply because of the way the lecturer is speaking
it, so the pronunciation of the lecture that isn't clear enough for
me.
So then I just simply given up and I often find myself in the library
searching, hunting for English books, or German books, or any books
that I can get to be able to study by myself.
And it's a really difficult thing to do, I should say.
So my grades went up and down pretty extremely, and it's not only
because of the difficulty of the subject, but it's even more because
it depends on how good I can understand the lecture, not depend on the
subject itself.
So that's actually picturing the difficulty that I get.
So therefore I'm really thankful for the opportunity to be here, to be
a part, to experience a great technology that will give me a second
chance to go back to the lecture and try to understand it.
Thank you.
Thank you very much.
Yeah, we get to the second question, and at this point I'd like to
also invite people from the audience or the press to ask questions.
Raise your hand and we have microphones here.
What we'd like to cover in the second question is the state of the
technology.
You've seen where we are now.
It's been a tremendously difficult road to get to where we are, and
still we're all convinced and we see that it's by no means perfect, so
there is still continuing work that's necessary.
And so I'd like to have the panelists perhaps discuss and make an
assessment of where they think we are, where we're still not adequate
enough, where we are actually have made great progress, where we
should be going and what kinds of new initiatives we may push forward
with in order to have this technology improve and have impact in
society.
So, again, in this vision of Europe of the future, one of our common
dreams, I should say, is probably to make this language problem
transparent, even though we enjoy the differences of our cultures and
our languages.
But how do we make it transparent?
What kinds of aids can we use here at the current state of technology?
If no one volunteers, I will ask Professor Carbonell, since he
commented already yesterday by email on this question, perhaps to take
the lead.
So,
first, let me try to paint a global picture and then go into a couple
of details with respect to the state of the technology and the state
of language technologies.
There is a saying in the Quechua language, Quechua is what they speak
in the Andes region or part of the Andes region in South America, I do
not have the Quechua words memorized, but the meaning in English
translates approximately as, we can see the summit, it is clear, but
the path between where we are and the summit is long, it has unknown
turns, it heads up and it heads down, and eventually it will reach
there to those who persevere.
That, I think, is a very accurate description of language
technologies.
The path is long and we cannot see all the turns until we embark upon
it.
The reason is that human languages present difficulties for machine,
many of which we appreciate and more of which we appreciate further as
we study it.
For example, at first it was thought that translation was just
equivalent to substitution of words and phrases from one language to
another.
That was in the 1950s.
In the 1960s, we realized that there was such a thing as syntax.
The order of the words and phrases matter significantly for
translation.
In the 1970s, we discovered that there is such a thing as semantics.
The words do not always mean the same thing.
One word in one language can translate into many words in other
languages.
Of course, linguists and translators knew this, but the extent to
which that problem really is a stumbling block for machines was not
realized until we tried to reduce it to algorithms and methods that
work mechanically.
In the 1980s and 1990s, we discovered other methods, statistical
machine translation, interlingual-based translation, and so on.
All of these methods helped us to overcome the limitations of the old
and also discover some new challenges.
These challenges range from voice recognition to voice synthesis to
mapping one language to another to inferring the meaning of language
so that we can communicate to the robot and it does what we mean, not
necessarily what we say.
This enables misinterpretation and we understand what the robots tell
us about how they perceive the world of the situation in a way that
that explanation is clear and we can take action based on it.
I'm not going to enumerate the specifics of the current technical
challenges that we're dealing with, but you can imagine what some of
them are.
When we're dealing with rare languages, I mentioned already the very
complex morphology.
Also, when you're dealing with rare languages, you have less data.
You don't have large amounts of pre-translated text that you can apply
standard statistical algorithms to.
You have to combine statistics with other forms of structural machine
learning, with forms of linguistic analysis, and you still don't get
it quite right, to say the least.
So the state is in flux and the summit is clear.
The summit is essentially to have machines that can be true
interpreters and translators and understanding of the human languages.
We have not seen yet all the turns in the path.
Can you hear me?
From the standpoint of our sponsor, just a few bits and pieces.
I'm not going to comment on science.
Throughout the 90s, the level of public investment in Europe was on a
par with U.S.
federal agencies, at least the white offices, not other environments
in and around Washington.
Today, I estimate that this particular field, large language and
speech, has lost 50% of the current level of public funding throughout
Europe.
Europe is struggling.
Germany is in a much better state than other countries, and yet, as a
body, as an entity, we are struggling.
I'm not saying we are sinking, that may happen, but right now we are
struggling.
Which means that politicians and businesses and citizens alike do need
practical, usable results.
And although I agree with all the statements that have been made so
far, I suggest that the scientific community, rather than public,
because there are a number of public labs as well, should find ways to
try and deliver things which are slightly more usable.
There are many components in place, there are many modules,
everywhere.
But one of the problems of this field is that when it is researched,
it has got a name, either speech or natural language processing, when
it is sort of solved, it is embodied within a larger application, a
larger component, and therefore disappears.
So the success stories don't carry the name, you never recognize a
successful inventor or developer.
So that's one of the things.
This affects a number of things.
It affects public investors, private companies, because what you do is
not a Gartner category, there is no software category for processing
human language.
And therefore, if I need to invest 50 or 500 million dollars, I don't
know how to compare, how you perform, how you score, how to compare
your technology results to others.
So that's putting a face, an identity, a bit of branding can only
help.
The other element is, in Europe, too many teams are too small, and
therefore necessarily have a limited ambition.
I think that there should be ways to try and concentrate, at a certain
level at least, when it comes to multi-party, cross-national
collaborations, to sort of concentrate efforts on a few objectives,
and to struggle until you crack the problem, or at least the next big
breakthrough.
There is a lot of local research, which is, so to say, publication
-driven, which doesn't scale up and doesn't offer a pathway towards
more engineering, more business, more application-related
developments.
Thirdly, although the technology may be mature in many areas, in many
respects, I think one of the conditions, and this is one of the things
we have been trying very modestly to do the last few years, is to
connect the dots.
In other terms, I mean, create a market pool.
In other terms, I mean, try and understand the pathway from the lab,
through engineering of a piece of technology, to actually building and
integrating the technology within a product, to actually deploying the
product.
Because that's the only pool which matters at the end of the day.
So making it useful, applied, and deployed.
And that we don't have yet, I mean, for many domains, segments of this
field.
And the danger I see over the last few years is industry
consolidation.
If we wait any longer, there are already a number of megaliths, and
you know them all.
But even, I mean, at the lower level of small corporations, I mean,
there is a sort of bar to excise, I mean, going on.
And if we wait any longer, at the end of the day we'll be left with a
search and no access whatsoever to the market.
Ours, and the worldwide global market.
So what I'm calling for is the fact that computers can transform
information.
I mean, I fed my family very successfully thanks to, you know,
informatics.
It was basically, I mean, you get out what you put in for 30 years.
It's only last, what, 20 years maybe that we've seen, I mean, software
actually understanding, interpreting, doing things, transforming, I
mean, the thing.
I mean, more than just, I mean, the representation, I mean, the
substance of it.
So the fact that the technology mature, I think, is understandable.
What makes the difference and can, you know, just trigger either
success or failure is, I believe, I mean, trying and promoting some
sort of ecosystem whereby researchers, technologists, and also
business -oriented people, I mean, work together, understand each
other, is a dialogue.
And then you can build, I mean, you know, access to market and create
opportunities for more and more ambitious research.
Yeah, thank you very much.
I couldn't agree more with that comment because we're obviously
passionate in this environment to turn basic science into something
that people get to use.
And as I know many of the team will attest to, it's not only painful,
it's actually an order of magnitude harder than just to do a
prototype.
And it's also sometimes difficult to do at a university because it
doesn't necessarily immediately, some aspects of it doesn't
necessarily immediately lead to publications, but is nevertheless a
key component of actually making the technology useful and happen.
The same applies, I mean, to commercial technology developers in this
field.
And I understand that, I mean, it's not me, most of the time I quote
more respectable, I mean, influential characters.
It's largely due to the fact that the valuation of technology
companies in Europe is much lower than in the States.
So basically, I mean, going around and shopping technology, I mean, in
Europe is a very good proposition for even relatively small U.S.
groups.
I mean, I say U.S., tomorrow maybe Chinese or another one.
One of the questions is basically our technology costs a lot, I mean,
to develop, but it's very cheap to buy, you see what I mean.
Now, since we cannot go, I mean, to Frankfurt and triple, I mean, the
valuation of all the companies, but there is an element, how do we
actually add value to the technology which has, you know, taken so
long to mature.
I mean, how do you put, I mean, a price on it, okay, which cannot be
the cost, I mean, it's a different element, right?
Okay, I have a short history on speech interfaces to machines.
In the early 80s, we had the first single word spoken recognizers
which did allow to program a robot saying stop, move a joint number
three to the left.
And I think about 200, more than 250 words could be recognized, but it
was individual, single word.
So in the 90s, there has been development towards continuous speech
with simple descriptive sentences to machines describing objects,
maybe to have some.
Then, in the 90s was a strong development on image processing to
describe what is on the image, and this can be translated to speech.
So that the approach of say what you see is one thing where speech
technology is to be applied.
This means objects, moving objects, relations.
And on the other side, there is the other development where spoken
speech has to be translated into representation.
We found out that in robotics, for example, to explain a robot how to
do it, currently we do that with a camera and we show a task.
And this is processed by the computer, then segmented, and the program
is generated.
But we found out that language is very important to explain, to show,
but also to explain why to do, for what it is necessary to be careful,
what has to be observed, and what has to be done in which situation.
This is work about 2000-2005, the first works came out.
And then in the last years, there is increasing interest on haptics
and contact and forces.
So if you do assemblies or if you do interaction, we can detect
additional sensors to touch, to derive forces.
And this can also be translated into words.
For example, if you touch to an object, it's a plane, it's a sphere,
or some features as a whole, or operating with a tool.
And since 2010, there is increasing activity where semantics on
interaction is discussed, so that situations and the meaning of what
is observed and what is demonstrated can be represented.
The problem is to go up from signals, from actuator signals to
symbolic representation and then to abstract levels, and then we are
at the speech level.
And the other one is going back from the symbolic level, from speech,
from demonstration to the actuator and the signal.
The problem is how to speak with a controller, with a control loop.
If you say, do something careful, you need some parameters, you need
some sensing, some observations, and this has to be translated, only
to give some example.
I think a very big challenge is human-robot-human interaction.
One is for translation, one is for telepresence, to perform
activities.
In the past, in the Gulf of Mexico, or in a nuclear disaster scenario
of the past, we have seen that it's very important to be present in
dangerous environments.
This is a core activity for robots, and to interact directly with the
scenario in both directions, which is necessary.
Thank you.
Thank you very much.
I will not comment on the details of the technological development,
which I think is very impressive.
But maybe one thing that I also mentioned in the beginning and that I
found very interesting in your presentation, Professor Weiber, was
that, starting from speech recognition to the actual translation
process, that the machine deals exactly with the same problems,
obviously, as human interpreters and translators would.
And I think seeing how speech recognition is progressing, I think it's
quite comforting.
But what are the challenges in an environment like the European
Parliament, for example, or any discussion forum, is of course that
our members speak freely, fast, they mumble, they have very different
accents.
Would the machine recognize Irish, Scottish, American, English, let
alone some of the German regional variations?
So, these are challenges.
But also, the first stage speech recognition could also be very
interesting in our context, for example, when we do testing of
interpreters.
We, of course, make the interpreters interpret the speech in a testing
environment and then afterwards the selection board needs to...
Did he actually say this?
Did she say that?
And if you would have a nice speech recognition on our computer and we
could then follow what the interpreter said, that would also be
interesting for testing and examinations of interpreters.
And then one difference is, of course, that we cannot rework, whereas
our colleagues in translation, they get the machine translation and
they can work and revise and change it.
Whereas in interpretation, the product is, of course, simultaneous and
instantaneous.
And we would like to rework sometimes what we said, because it's now
web -streamed live, the plenary session.
And as we are not machines, we make mistakes and human errors, and
sometimes we would like to rework.
So this is maybe an interesting idea of how you could work with that.
And then, of course, all the technical development which is going on
and which is very exciting also leads to high expectations from our
users.
Because, of course, members of parliament now, if they have to wait
two weeks for a document to be translated, they get very impatient and
they say, well, just put it through Google Translate and we get it
soon.
And also for interpretation, it leads to expectations and I think it's
very important to inform about the realistic possibilities as they are
now, what would be possible in the future, and also to stay in the
loop and follow what is going on.
Okay, I can also say something from a robotics point of view.
I think we would love to have these perfect systems, recognition and
dialogue systems for robotics, because this would really push the
field of service robotics or human-centered robotics very, very much.
Not only in the form of robots which are operating in a kitchen or
daily environment, but in all applications or medical applications,
robot assistance systems for surgery, for example, and others.
But, unfortunately, from my own experience, not only with the software
from KALISO, but from the other groups working on building natural
dialogue systems for robots, all the systems are still very... or
dialogue systems which are presented by the community or by the
robotic community are well-designed so that somebody can perform a
demonstration, which is nice, but they cannot deal with noisy
environments, with different speakers, and I think these challenges
have been addressed by the previous colleagues here.
So, from my point of view, I think the robustness of the systems is
very, very important, and the flexibility to really talk with the
system, a robot or an iPhone or I don't know what, about very, very
simple things.
Just to perform a small talk with your robot and where you have a
system which really can answer your questions.
How was the match yesterday between Italy and Spain, for example, and
did you go to the beer garden after that?
And not only these well-designed and structured dialogues where more
or less everybody is expecting what you can do or how you can talk to
your robot or to your system to get the best results.
So, I think robustness and having systems which can deal with
different aspects, noisy environments, different speakers, of course,
different languages, and so on, is a very challenging and important
issue.
Okay.
So, it's kind of hard to follow these high-quality speeches from the
other speakers.
So, I'm probably going to provide a little bit of a positive note,
especially on the lecture translator system and the lectures.
So, when I come to the lectures, I saw this, many of my friends, the
foreign students, have these little devices, the electronic
translation devices, a little bit similar to a calculator where you
type something, you type some German words, and you get the
translation in the language that you chose.
So, then I think, well, would it be nice if we have a hands-free
automatic translation, you know, without the need of typing anything
to it?
But that is actually exactly what this lecture translator technology
is.
This is a hands-free.
You don't need to type anything.
You don't even need to buy a new device because it's online.
It's already accessible in the laptop and tablet.
It's pretty cool, I would say.
So, I'm sure there are still a lot of rooms for improvement.
There are still a lot of challenges to fix.
But, even in the current state, I think it's far, far better than
those ugly electronic translation devices.
It's better than none, obviously.
Yeah, thank you very much.
Perhaps I might add one comment since the question was raised, you
know, how to deal with accent and spontaneous speakers and so on.
It's perhaps surprising that some of the things aren't so much of a
problem at all, while others are huge problems.
And some of them are counterintuitive, and some of them agree with the
human experience.
So, some of the counterintuitive things are that accents turn out not
to be a big problem for us.
We have adaptation algorithms.
A classic example is our colleague Asfor, who can't pronounce the
umlauts very well in German.
So, we had this in a big Audimax lecture hall here in Karlsruhe.
Maybe some of the students here attended that lecture, where he said
the word that gives him the most trouble, which I showed, is
Würfelkalkül, because it has two umlauts and it's a complicated word.
And it recognized it fine.
And so there was a thundering applause in the audience that it could
handle Tanim Asfor's pronunciation of Würfelkalkül.
But if you think about it, what it does is it learns very quickly that
there's a transformation between the typical accents and the original
speaker, and it adapts, and it does a reasonably good job.
Two other problems, though, that are really difficult is this mumbling
that you mentioned.
Mumbling is something that the machines agree with humans is very,
very difficult.
When people mumble and stutter, and some of our colleagues do that
too, it is very hard to transcribe and even harder to translate.
Because if you see the transcript of a mumbled lecture, a stuttered
lecture, it's, even when you read it in the original language, very
hard to follow.
So converting things in understandable, readable text, I think, is a
big challenge.
And then in the range of hard-to-understand challenges that are
impossibly difficult, you all see us here with closed-speaking
microphones, and there's a good reason for it.
If you put microphones at a distance, it's still an open challenge
that no one can address with computer systems, because the echoes and
the noises, etc., are actually a genuine problem.
And so the comment on robustness is well taken, because I think
that's, to me, an area that still requires a lot of work to get these
things robustly.
So spontaneous speech, noisy environments, they're all challenges that
still are open to be taken.
I have one question.
A good lecture is an interactive lecture.
So one is to have a monologue, and you train the machine, but the
other one is to have questions with students, to have a dialogue which
is not predictable what happens.
Can you say something to that?
Yeah, it's not a problem if the responses are, again, sensible
sentences.
I can show you recordings of teleconference conversations or meetings
between people that are so fragmentary that they're not intelligible,
even if you read them perfectly transcribed in the original language.
Because very often, as humans, it's like a ping-pong game.
We say a little fragment, and the other person adds another fragment,
and so on.
And we reconstruct the meaning of it without it ever having been said.
And that remains a big problem.
So really open, fast exchange, and so on, is troublesome.
Now the typical lecture interaction, when there's a question from
someone in the room, and the lecturer responds, tends not to be a
problem, because then again, you get full sentences by and large.
So I think for lectures, I'm pretty sanguine and pretty optimistic.
I think what's still a challenge is when it's really a fast exchange
in a seminar, let's say.
So seminars will still concern us in moving forward, because there you
have a lot of interactive exchange, which is also important, of
course, for foreign students.
But let me move to the third question in the interest of time.
We have the last one, which might interest many of you, is what's the
symbiosis between machine and humans?
Ultimately, we're here to build a tool to help people do another task.
Technology here is a means to an end.
And so we also don't want to replace humans.
It would be foolish.
It's going to be always a work together.
And all successful technologies in the history of humankind have been
technologies where ultimately it's been a collaboration between humans
and machines.
So I'd like to have the panelists maybe comment on how they see
technology gradually moving into their environments and how they see
the combination or interaction between human services and machine
services to play out.
So give us a view of your predictions of the future.
Maybe I can ask Frau Altenburg perhaps to comment first, since she is
intimately involved in that question.
I think one important aspect of this is to ease this fear that some of
the human language workers, be it translators or interpreters, have
that they will be replaced by a machine.
I think for translation it's working well now because there is an
excellent cooperation, as I said.
They use computer-assisted tools.
There is machine translation that sometimes is then reworked and
especially all the translation memories that are extremely useful.
I think none of the translators would want to go back to the pencil
and piece of paper method of the past, let alone all the terminology
work and terminology extraction that can be done with the assistance
of a computer.
Now for interpreters, the situation is of course different and I think
the first step is really to inform also the human interpreter about
advances in technology and that it still has to be seen how this can
help them with their work, apart from all the meeting preparation
work, where this is of course helpful.
But also, whereas in a big, as I said, parliamentary environment like
a plenary session or committee meeting with political discussions, I
suppose probably, I'm not sure, but in my lifetime I think the human
interpreters will not be replaced there.
But there are also other ways when, for example, the members of the
European Parliament go on delegations, on missions, and they can be
ill like any other citizen and the interpreter is not always
available.
And as you said, although humans are still better, it's maybe better
than having no translation or interpretation at all.
So I think, but what I still see is how, or what is not entirely clear
to me now is how could this really, or how we can make this interact
with the human interpreter.
I think, as I said, for the translator it's quite clear how this
works, but as we have no possibility afterwards to rework the output,
how can this work?
And maybe first in a bilingual setting, of course, before starting to
look at huge multilingual meetings, which have, of course, other
challenges.
Thank you.
All
right.
So, before addressing this symbiosis, let me make a connection between
robustness and this symbiosis.
I'll tell you briefly about a system at Carnegie Mellon called Let's
Go.
Let's Go is a dialogue speech recognition system to help people figure
out bus routes.
It's a very simple task.
How do I get from X to Y?
When does the bus come?
Do I need to change buses?
And so forth.
So you make the simplification by task.
But you make, it's completely open-ended otherwise.
People call with trucks driving by, with a kid demanding mommy's
attention, and mommy's trying to find out about the bus, and the kid
wants something else.
I'd like to go home without realizing that the bus is the way you're
going to get home.
And the problems there are not, as Alex already said, are not
different accents.
In the U.S.
we have the equivalent of the Bavarish here, or Schweizerdeutsch, or I
should say Zwitserdutsch, I suppose.
That problem is not a hard one.
The hard one is threefold.
It's the background, other noise.
It is the fact that the humans do not provide sufficient information.
I want the bus to go home.
Okay, well, where do you live?
You have to engage in a dialogue in order to extract enough
information to carry the dialogue.
Sometimes the extraction of information is because you did not hear
clearly what the other person said.
Building these systems, even for simple applications, is hard.
At 5 o'clock p.m., it switched the system from the human to the
machine, because Pittsburgh did not have enough money to pay humans to
man the telephones after 5 p.m., and people were very upset because
the quality of the machine help was not the same as the quality of the
human help.
They wanted to talk to a human.
A year later, they still wanted to talk to a human.
The system had gotten somewhat better.
Two years later, the requests to talk to a human were fewer.
Either people got used to it, or the system got better.
We did not know which one.
Four years after it was installed, we had the first time somebody who
called at 3 p.m.
while there was still a human.
The human had some trouble understanding what the person wanted.
The person said, I want to talk to the robot.
We had one more example of the request to talk to the robot.
So maybe it's getting there.
So, come back to the larger picture of this symbiosis.
I don't think symbiosis is necessarily the correct word, because the
machines will always be in service of the humans, rather than the
other way around.
A word processor and a human are not exactly in a symbiotic
relationship.
An automobile and a human are also not exactly.
These are extremely useful mechanical electronic devices.
I think most of language technologies will be the same way.
It will translate.
It will provide a service.
The quality of that service will improve.
It will make people more efficient.
It will lower costs overall.
It will make society better.
But it will never really be a true symbiosis, in my opinion.
Roberto?
Orthogonal, but maybe restating one or two concepts that you have just
heard.
First, I think it's a well-known fact that the most successful, the
usually successful applications cannot be predicted.
And texting is one such thing, but there have been many more.
So, number one, I mean, this leads to me to, I believe very much, the
interaction between machines, software, and humans.
And I would define it as almost a form of respect.
When you develop a piece of software, which is what I've done for more
than 20 years, we treat basically users like nuisances.
They are the robots.
Software is the real thing.
And I've got, I think, six Android smartphones, like this, from one
and the same maker.
And the most irritating thing in life is when I use it to type in text
in French, Italian, and English, mostly in English, and it keeps
changing my words, which are perfectly okay.
Because it cannot digest some English words, and of course it refuses
to learn French and Italian expressions.
So, I think there must be a software engineer somewhere between Taiwan
and California who should be taught to show ways of respecting the
user's intentions.
This is deeper than it sounds.
Third element, most of you, all of you, I hope, are aware of MOSIS,
which is the Text Machine Relation System, which was partly or largely
developed thanks to European funding.
Now, what you can't possibly know is that since then, the last three
or four years, all the projects we've been putting money into are
about putting it to work, its own use of SM&T.
Basically, by combining two MT engines, by putting TM next to MT, by
doing MT and post-editing, by looking at the iris of an editor to
guess the intention, the purpose, and the stress.
So, there is, and there should be more, research on actual use of
technology than there is in certain corners.
So, I repeat, when you design your software, just bear in mind that
there is, on average, an intelligent person sitting in front of the
screen, the phone, trying to gather the intentions and respect them.
Number two, don't try to predict what will happen to your software
because it will fail.
WhatsApp was designed by two guys.
Viper has become hugely successful.
It's no bigger than that.
So, there are small little things that change everybody's life.
Thirdly, user-related research is, by now, today, as important as
hardcore technology-based research.
I think we live in a world where we need to combine technological
innovation and innovation in the way we use these things and how we co
-evolve with these things.
Some additional situation on service.
There are noisy and fuzzy requests for service.
For example, if a person says, help me, humans very immediately
negotiate and know how to help, and I make a decision, and I can help
you or you.
If you have a machine, a robot, and you ask, help me, a long
discussion, a dialogue has to be started.
So, to look to such a situation, I think it's important, of course, to
know more about the people.
Maybe the preferences, maybe to recognize the intention of the person.
Maybe the discourse, if we are on a railway station, on an airport, in
the supermarket, or at home, or if it's a handicapped person at home.
And I think there is a language body necessary to describe the
environment and what may happen, what are typical situations in that
environment, and coming back to robustness.
I think the best way to go towards robust systems is to make the
discourse small and very clear, and then I think this fuzzy situation,
for example, help me or support me or give me some information.
Of course, there are cognitive capabilities necessary, but again, I
think to use language technology, also gesture is important, maybe
body language, maybe emotion, I don't know if this is possible.
And more important is the intention, to recognize that.
Thanks very much.
Perhaps just to interject one, insert one comment.
Of course, with lectures, we are at the opposite extreme of limiting
domains.
That's the absolute opposite extreme where the topic can be anything,
and it's very specialized in that topic, right?
But the robustness issue and the practical issue and the usability
issue is just the same in the sense that we need to then think about
how does it scale?
How do we quickly insert it in that particular situation and make it
automatically adapt?
So I think that's a gigantic challenge of, in fact, handling such
broad domain problems.
I have a question to Dr. Cenzioni.
What we discussed actually is that there is one side speech, on the
other side there is vision, on the other side there is robotics,
haptics, but then also cognitive science.
I have the feeling that in a lot of programs, these domains are
separated.
And to have there maybe a challenge towards the future, I think it's
important to build bridges between the communities, starting from
representation and then grounding that to sensors, to actuators, and
to speech.
Do you have any idea?
Are there programs coming up?
So, number one, I mean, it's happening.
It has been happening for some time already.
And in any case, we as a commission, as a sponsor of this type of
research, we are going multi-modal.
So I don't expect to see speech-only projects in the coming months and
years, number one.
Number two,
we as a commission, after extensive consultations, discussions,
experts and the like, we publish a work program.
Then we get it out in the wild, and you people, I mean, you come back
with your project idea.
So, rightly or wrongly, I've come to the decision, I mean, in my
little mind that a public spending program addressing also speech, has
basically, I mean, one priority, which is dialogue, and maybe a sub
-priority, which is open domain.
All the rest, I mean, is gladly done by many people in many different
ways, I mean, left, right and center.
So we ask, I mean, with a number of features and qualifiers, we ask
for dialogue.
And we get sophisticated command and control systems, we get, I mean,
speech recognition, noisy environment, this and that and so forth, we
get acoustics, we get, I mean, lots of transcription and some of them
coupled with translation.
So, many things, all equally respectable, useful and indeed necessary,
everything but dialogue.
So just to show that, you know, you can, I mean, try and pull and push
and, you know, set a certain agenda, but at the end of the day, the
ball is very much in your camp.
Third comment.
I'm old enough, I mean, to have seen, I mean, three main phases in
community programs, research programs.
First phase was basically, I mean, multinational.
Get together, do whatever you want, but do it on a multinational
basis.
And that was, I mean, the 80s.
Sometime, I mean, in the course of the 90s, probably around the mid
-90s, we bumped into cross-disciplinary, interdisciplinary approaches.
We want people to get together and basically combine, I mean,
advances, thoughts, approaches from different disciplines.
And that's been, I mean, the big news, I mean, for 10 years.
Now, I believe that the Commission, in its future programs, I mean, is
moving towards what I call address the network failure.
Build a value chain and look at all the players and sectors and
stakeholders that will make that research successful in the, you know,
in the real sense of the word.
So, basically, I mean, from research through, I mean, technology
engineering, to transfer, to early deployment, through user partners
and so forth, until when they say, okay, now it's all yours, I mean,
you know, make money out of it.
So, from countries through disciplines to the various players in the
value chain, which, again, I guess, in the same direction.
So, the answer is yes.
I don't think there will be, at the European level, any programs, and
possibly not even projects, exclusively centered on speech.
It will be speech with all these affective, cognitive, multimodal,
visual, haptic, and so on, elements that you mentioned, okay?
Because that's the only way, I mean, to make them work, eventually.
It may be seen as a fit-en-avant, so you can't even, I mean, recognize
in normal speech you want to build this complex, I mean, factory.
And it is a fit-en-avant, but on the other hand, I mean, it's the only
recipe, I mean, to get somewhere one day, okay?
So, that's happening, and will happen even more.
So, there will be, necessarily, these regrouping, I mean, of people,
ideas, protocols, and the like.
So, if you ever have a chance to tell this to our American funding
colleagues, please do.
It's been very difficult to get the cognitive, the dialogue, the
multimedia, the speech, funded together into a program that truly
integrates it.
The PAL program at DARPA was an exception.
That one was successful, it led to Siri, for example, as an
application, but that was an exception.
That was the only one.
Clearly, there is need for more.
Yeah, thank you very much.
Tanim and Karel Kanadi, very quick final comment.
Very, very brief comment.
So, for me, it was a pleasure to participate in the lecture translator
with this lecture with a huge number of students.
As you noticed, they didn't listen to me.
They followed the translation on the screen, your translation, so it
was a great experience.
And I think, really, we need this technology.
This will be a service for everybody worldwide.
I believe we can enhance the capabilities of our mobile devices and
computers by such translation capabilities, where one could select
whether he preferred to listen to the speaker with the accent or would
prefer to listen to another German speaker or to an English or French
speaker.
So, I think this is a great achievement and which must be continued in
the future.
Thank you.
Also, very short remarks.
So, one of the questions that pops out would be, will language
technology be an obstacle for learning a new language?
And I think the answer would be no.
Because one thing about learning a new language, the biggest obstacle
is when you have frustrating moments.
And I think language technology has a huge potential to smoothen the
learning curve in learning new languages, so it won't be a problem.
And secondly, what I think is one of the most important purposes of
this technology would be how to attract foreign students to come to
Germany, right?
As you have presented in your presentation.
So, yeah, of course they will come and just hearing about it will
attract them.
Yeah, that's one thing we are here talking about technical stuff,
talking about technology.
We sometimes forget that this lecture translator technology is more
than just a technology.
It's a technology that shows care.
It shows a special care, special attention to us, foreign students.
The whole system built to accommodate our needs, to help us to solve
our problems.
So I see it as a special gift to us, the foreign students.
And I'm very, very thankful for the team and to the country, Germany,
that welcomed us and made the whole thing possible.
So, thank you.