The virtual made real – new technology for the media of the future

MTC is pursuing a very practical goal with its project for AR glasses in Zurich city centre. Anyone who looks at Paradeplatz through AR glasses will see local information such as tram departure times, advertising posters for the surrounding shops – or newspaper articles about the two major banks located there. The focus is on questions of practical legibility: Where and when is an advertising poster displayed? Where are the articles displayed? Which surfaces are suitable for the displays?

A dinosaur strolling through Zurich

 “Above all, MTC is concerned with how these new types of digital content can be generated for the media in the simplest way possible,” says Klingler. Using MTC’s tool, it is easy to augment Paradeplatz with virtual characters – turning it into a zoo or an underwater world, for example. “Everything that someone creates in virtual reality appears automatically on the real Paradeplatz,” explains Klingler. “Our tool means users no longer have to grapple with the difficult question of how to get their virtual content into the right real place. They can simply concentrate on the design.”

One of the technical challenges, for which there is still no simple solution, is “occlusion handling”: unlike a film made for the cinema, which is edited in a studio, virtual characters moving through a real situation in real time often fail to disappear from view immediately when a user turns a corner in the street. This disrupts the credibility of the visual experience.

The ETH researchers have devised an approach that makes this work better – for example, a virtual dinosaur wandering around near the tram stop in Stampfenbachplatz now disappears from view immediately as in real life as it turns the corner by the external pagecanton of Zurich’s media centrecall_made (see video). The ETH researchers used a 3D model of the city of Zurich to achieve this.

Local weather in local dialects

Besides virtual moving pictures, voice assistants are likely to become increasingly important in our daily use of media. The media technology trend indicates that media content such as videos, radio news or texts will be retrieved less by hand – through clicking, typing or swiping – and more frequently by voice. We are already familiar with talking intelligent assistants from fitness apps or traffic forecasts on our smartphones.

“But none of the assistants available today speak or understand Swiss dialects, which makes communication with them rather unnatural for Swiss people,” says Klingler. This is why MTC is developing a voice assistant that speaks Swiss German. It has been speaking fluent “Bärndütsch”, the dialect of Bern, since 2019, and is currently being expanded to include other dialects.

As is characteristic of the work of MTC, the researchers are developing the Swiss German assistants in partnership with a Swiss media company. In this case, the partner is the main Swiss radio and television network, SRF. SRF publishes large amounts of content in local dialects on its channels. Technology that translates High German to Swiss German or that can announce the local news and weather forecast in the local dialect is therefore highly practical and gives the automatically spoken text a regional authenticity.

Support for digital transformation

MTC’s development projects are generally implemented in one to two years and offer practical benefits to editorial offices and media companies. The aim is to support Swiss media companies in their digital transformation. This applies to both the generation of journalistic content and the production of various media formats and methods that bring digital content to users and personalise media consumption.

The Media Technology Center is part of the ETH media technology initiative. The centre was established in 2019 with partners from the media industry. It is supported by the media companies Ringier, TX Group (previously Tamedia), SRG SSR, NZZ, Verband Schweizer Medien, the association of employers and entrepreneurs in the Swiss printing industry viscom and the ETH Zurich Foundation. All partners - like ETH Zurich - have contributed financially to make the MTC possible. The centre is still open to expand its partnerships. The scientific head of the centre is ETH computer science professor Markus Gross, who is joined by experts from the business and scientific sectors.

Open for media project ideas

“We are open to any project ideas from media companies and editorial offices,” says Klingler. MTC began its work in March 2019 and comprises eight researchers, supported by students and doctoral students, in the fields of computer vision, computer graphics, natural speech processing, recommendation systems, user modelling and machine learning. Ryan Cotterell has also been working in Zurich since February 2020. The computer science professor was appointed as part of the media technology initiative, and combines linguistics, automated speech processing and artificial intelligence.