Automatic story illustration
Problem being addressed
Given a sequence of sentences that tell a short story, generate a consistent sequence of related images that illustrate the story
The researchers have developed a clever neural network architecture comprising a sequence of Generative Adversarial Networks (GANs) that learns to generate not only a plausible image that illustrates an individual sentence, but where each image naturally follows from the previous one and tracks the progression of the narrative - in a similar way to a traditionally illustrated story book, cartoon or comic.
Advantages of this solution
Previous approaches have only been able to generate a series of unrelated images for each sentence, or to generate a short video from an initial description. As well as significantly improving on previous approaches using automated evaluation metrics, teams of observers consistently rated the generated images dramatically more highly for quality, coherence and relevance.
Possible New Application of the Work
This new approach has applications in all areas of advertising, marketing and promotion, where visual assets need to be created and assembled to fit a narrative. In science communication, graphical abstracts and, more recently, video abstracts, are becoming an increasingly popular way to promote and illustrate the main methods and findings of the latest research. Potentially, this neural network approach could be used to automatically generate these visual assets, or at least provide a starting point for them.
Source URL: #############