A Neural, Interactive-predictive System for Multimodal Sequence to Sequence Tasks

Abstract

[EN] We present a demonstration of a neural interactive-predictive system for tackling mul- timodal sequence to sequence tasks. The system generates text predictions to different sequence to sequence tasks: machine translation, image and video captioning. These predictions are revised by a human agent, who introduces corrections in the form of characters. The system reacts to each correction, providing alternative hypotheses, compelling with the feed-back provided by the user. The final objective is to reduce the human effort required during this correction process. This system is implemented following a client server architecture. For accessing the system, we developed a website, which communicates with the neural model, hosted in a local server. From this website, the different tasks can be tackled following the interactive-predictive framework. We opensource all the code developed for building this system.