Multi-Scale Auralization for Multimedia Analytical Feature Interaction

Nguyen Le Thanh, Hyunhwan “Aiden” Lee, Joseph Johnson, Mitsunori Ogihara, Gang Ren, and James W. Beauchamp, “Multi-Scale Auralization for Multimedia Analytical Feature Interaction”

Modern human-computer interaction systems use multiple perceptual dimensions to enhance intuition and efficiency of the user by improving their situational awareness. A signal processing and interaction framework is proposed for auralizing signal patterns for augmenting the visualization-focused analysis tasks of social media content analysis and annotations, with the goal of assisting the user in analyzing, retrieving, and organizing relevant information for marketing research. Audio signals are generated from video/audio signal patterns as an auralization framework, for example, using the audio frequency modulation that follows the magnitude contours of video color saturation. The integration of visual and aural presentations will benefit the user interactions by reducing the fatigue level and sharping the users’ sensitivity, thereby improving work efficiency, confidence, and satisfaction.

Geo-Influence: Modeling Location-Specific Effects of Social Influence on Brand Preferences

Hyunhwan “Aiden” Lee, Joseph Johnson, and Gerard J. Tellis. “Geo-Influence: Modeling Location-Specific Effects of Social Influence on Brand Preferences”

  • Preparing for submission at Journal of Marketing Research
  • The 41st ISMS Marketing Science Conference, University of Rome. June 20 – 22, 2019
  • The 2019 Haring Symposium, Indiana University. April 19 – 20, 2019
  • The 1st Interdisciplinary Research Cluster Day, University of Miami. April 1, 2019
  • The 3rd Annual Three Minute Thesis Competition, University of Miami. February 6, 2019
  • Behavioral Insights from Text Conference, University of Pennsylvania. January 17, 2020
  • Artificial Intelligence in Management Conference, University of Southern California. March 14, 2020
No brand is the market leader uniformly across geographic locations. Brands have strong and weak markets. Therefore, analysts need to track brand preferences by location. However, current methods of tracking geographic brand preferences (e.g., surveys, company sales, scanner panel data, Google Analytics, etc.) have limitations: limited sampling of brands, stores, and categories with time and region aggregation. We address these limitations through a spatial, nonparametric approach to track brand preferences at the micro-temporal and micro-geographic level using Twitter. Our approach inherits the intrinsic strengths of social media: micro-temporal, micro-geographic, faster than news media, higher frequency than sales, low cost, competitive monitoring, sentiment inclusion, and social network inclusion. A key feature of our approach is modeling the temporal geographic influence of social connections based on the followers of each post. We estimate our model for carbonated soft drinks brands. Our validation uses Pepsi sales, an apparel fashion brand, and the 2016 U.S. presidential election (a domain where marketing is critical). We find that geographic influence adds predictive power to micro-temporal and micro-geographic brand measures, leads online sales and predicts elections. Our study implies that the micro-temporal and micro-geographic social media measures provide actionable insights for marketing brands and political candidates.

Keywords: micro-temporal brand preference, micro-geographic brand preference, sentiment analysis, Twitter, geographic influence model, election prediction

Multimodal Content Analysis for Effective Advertisements on YouTube

Nikhita Vedula, Wei Sun, Hyunhwan “Aiden” Lee, Harsh Gupta, Mitsunori Ogihara, Joseph Johnson, Gang Ren, and Srinivasan Parthasarathy. “Multimodal Content Analysis for Effective Advertisements on YouTube”

The rapid advances in e-commerce and Web 2.0 technologies have greatly increased the impact of commercial advertisements on the general public. As a key enabling technology, a multitude of recommender systems exists which analyzes user features and browsing patterns to recommend appealing advertisements to users. In this work, we seek to study the characteristics or attributes that characterize an effective advertisement and recommend a useful set of features to aid the designing and production processes of commercial advertisements. We analyze the temporal patterns from multimedia content of advertisement videos including auditory, visual and textual components, and study their individual roles and synergies in the success of an advertisement. The objective of this work is then to measure the effectiveness of an advertisement, and to recommend a useful set of features to advertisement designers to make it more successful and approachable to users. Our proposed framework employs the signal processing technique of cross-modality feature learning where data streams from different components are employed to train separate neural network models and are then fused together to learn a shared representation. Subsequently, a neural network model trained on this joint feature embedding representation is utilized as a classifier to predict advertisement effectiveness. We validate our approach using subjective ratings from a dedicated user study, the sentiment strength of online viewer comments, and a viewer opinion metric of the ratio of the Likes and Views received by each advertisement from an online platform.

Sequential Pattern Based Temporal Contour Representations for Content-Based Multimedia Timeline Analysis

Gang Ren, Joseph Johnson, Hyunhwan “Aiden” Lee, Mitsunori Ogihara. “Sequential Pattern Based Temporal Contour Representations for Content-Based Multimedia Timeline Analysis”

Temporal contour shapes are closely linked to the narrative structure of multimedia content and provide important reference points in content-based multimedia timeline analysis. In this paper, multimedia timeline is extracted from content as time varying video and audio signal features. A temporal contour representation is implemented based on sequential pattern discovery algorithm for modeling the variation contours of multimedia features. The proposed contour representation extracts repetitive temporal patterns from a hierarchy of time resolutions or from synchronized video/audio feature dimensions. The statistically significant contour components, depicting the dominant timeline shapes, are utilized as a structural or analytical representation of the timeline. The modeling performance of this proposed temporal modeling framework is demonstrated through empirical validation and subjective evaluations.