Disaster tweet text and image analysis using deep learning approaches

Abstract

Fast analysis of damage information after a disaster can inform responders and aid agencies, accelerate real-time response, and guide the allocation of resources. Once the details of damage information have been collected by a response center, the rescue resources can be assigned more efficiently according to the needs of different areas. The challenge of the information collection is that the traditional communication lines can be damaged or unavailable in the beginning of a disaster. With the fast growth of the social media platforms, the situational awareness and damage data can be collected, as affected people will post information on social media during a disaster. There are several challenges to analyzing the social media disaster data. One challenge is posed by the nature of social media disaster data, which is generally large, but noisy, and comes in various formats, such as text or images. This challenge can be addressed by using deep learning approaches, which have achieved good performance on image processing and natural language processing. Another challenge posed by disaster related social media data is that it needs to be analyzed in real-time because of the urgent need for damage and situational awareness information. It is not feasible to learn supervised classifiers in the beginning of a disaster given the lack of labeled data from the disaster of interest. Domain adaptation can be applied to address this challenge. Using domain adaptation, we can adapt a model learned from pre-labeled data from a prior source disaster to the current on-going disaster. In this dissertation, I propose deep learning approaches to analyze disaster related tweet text and image data. Firstly, domain adaptation approaches are proposed to identify informative images or text, respectively. Secondly, a multimodal approach is proposed to further improve the performance by utilizing the information from both images and text. Thirdly, an approach for localizing and qualifying damage in the informative images is proposed. Experimental results show that the proposed approaches can efficiently identify informative tweets or images. Furthermore, the type and the area of damage can be localized effectively.