Subtitles in 360° videos
On this page we will explain
- the subtitle presentation modes developed and tested in ImAc and
- key aspects of the technical realisation.
Please follow the links provided (in the text and at the end of this page) to get more detailed information on the particular topics.
Subtitles are already supported on various 360° platforms. The ImAc project investigate “how” the subtitle service is presented to the user. In a 360° video, the “speaker” may not be visible in the current field of view. The user needs to know where to navigate so he can see the speaker.
Many of the features described here can be tested in the ImAc portal
An arrow is placed left or right next to the subtitle. The arrow is shown when the speaker is not visible within the picture. It is pointing into the direction of the speaker. The arrow disappears as soon as the speaker becomes visible.
The below picture shows an example. A reporter is in the current field of view but the subtitle is related to the instructions of the director (who is not visible in the picture). An arrow at the second line of the subtitle points to the direction of the speaker (in the case the director). The subtitle text is “Nicht in die Kamera gucken und bitte”. This translates into “Do not look into the camera and action!”
A “radar system” is rendered next to the subtitle and is always active. It indicates the current viewing direction. For each active subtitle, a coloured dot is rendered into the radar system. It points to the horizontal position of the related speaker within the 360° scene.
The below picture shows an example. Two crew members of the film team are shown, but the subtitle relates to the reporter who is not shown in the picture. A circle in the picture simulates a “radar”. A white arrow in the “radar circle” indicates that the reporter is right of the field of view.
You can find a detailed explanation of the tested presentation modes in document Deliverable 5.2 chapter 18.104.22.168
The final evaluation report will be provided at a later stage after all user tests have been performed. But you can find preliminary results in document Deliverable 5.4 chapter 4 and 5.
We see a great need for personalisation options especially when using head mounted displays (HMDs). The quality range of the image in these devices requires to use larger font sizes than for traditional subtitle services. Additionally, users reported that they prefer the subtitles moved further into the center of the picture. The ImAc player allows to adjust presentation to personal taste and to the displaying device with the following settings:
- The preferred option for the subtitles background: outline, or a semi-transparent
- background box.
- The preferred position: top, bottom.
- Easy-to-Read or Traditional ST.
Easy-to-read subtitles are an alternative subtitle service / track that may be present in addition to the standard subtitle track (in the same language).
Detailed information about the personalisation options and the user interface can be found in document Deliverable 3.5, chapter 7.5
Alternative subtitle presentation
We also tested other new approaches for presenting subtitles where subtitles are fixed to the speaker in the video.
In this mode, subtitles will not always be visible as it is the case when fixed to the screen. When the viewer looks away from the speaker, an arrow appears in the field of view, pointing towards the speaker’s location. IRT, one of the ImAc partners, has created a live demo of an early implementation.
Another focus was on the representation of sound events with emojis. We wanted to get a preference either for the traditional representation with text or for emojis.
The alternative presentation modes are not included in the latest player version but were only experimental implementations.
The final evaluation report on these will be provided (and linked here) at a later stage after all user tests have been performed.
Contrarily to other existing platforms, the ImAc platform has been developed by exclusively relying on standard web components and technologies, without requiring any specific hardware or the installation of additional software. The use of web-based components involves low computational load and guarantees universal (i.e., multi-network, multi-device and multi-platform) support.
We developed an end-to-end workflow in ImAc including a 360° web-player, production tools for access services as well as a packaging and delivery framework.
Subtitles are delivered using the IMSC subtitle format which we have extended to carry additional metadata needed for the presentation modes shown above. One essential information is the location of a related speaker within the scene for each subtitle.
As an example, here’s a snippet from an IMSC file including the custom attribute “equirectangularLongitude” describing the horizontal angle from the center of the video where the speaker is located:
The document Deliverable 4.4, sections 2 contains a detailed description of all relevant signalling and metadata for subtitles as we used it in ImAc. We are currently propose to W3C to standardise a way to indicate speaker locations in 360° environments for subtitles.
Displaying subtitles in the ImAc player is realised by adding an “accessibility layer” to the player’s scene. This layer is the projection plane for rendering subtitles and guiding elements. The 3D framework used for setting up the scene is three.js. The player architecture including information on all open libraries used is explained in document Deliverable 3.5, section 3.
Mozilla has a good explanation about how to add IMSC subtitle support to a webplayer in general using imscJS.This solution is not specific to 360° videos. Thus, the ImAc player implements its own rendering algorithm for subtitles in 360°. But the imscJS library is used in ImAc for parsing the IMSC file and resolving subtitle start and end times.
Subtitles as presented in ImAc require additional metadata to be sent along with the subtitles. This metadata needs to be authored.
In the project, an editor for the creation of 360° subtitles was developed. With this web-based editor, the ImAc extensions can be edited along with the rest of the subtitle properties.
Key features of the editor are:
- Web based and responsive design for universal use
- Embedded 360° video player
- Creating and editing 360° metadata for subtitles
- Support for various video codecs and formats
- Support for subtitle format IMSC1.1
More information on the ST web editor can be found in this factsheet.
If you’re interested, please request a test account for the web editor here: firstname.lastname@example.org
References & Links
Subtitle related ImAc news articles:
- Workshop: Subtitling in Immersive Environments
- Test of new concepts for subtitles and sign language service
- Fixed positioned subtitle demo
- Experimental: Responsive subtitles for 360° video
- Subtitles for 360° media – alternative presentation for immersive experiences (part 1)