What is SuggestBot?
SuggestBot is a software component that identifies the user's needs and suggests an action, which is the key aspect of "Smart Interaction". Two key elements of SuggestBot are "context awareness" and "real-time interaction".
Support
This research was supported by Next-Generation Information Computing Development Program through the National Research Foundation of Korea(NRF) funded by the Ministry of Science, ICT (2017M3C4A7065955)
이 연구는 정부(과학기술정보통신부)의 재원으로 한국연구재단-차세대정보・컴퓨팅기술개발사업의 지원을 받아 수행된 연구임(2017M3C4A7065955).
Period : 2017.09.01 - 2021.03.31
Goals
- Development of a context inference engine to support SuggestBot's context awareness in various wearable computing scenarios
- Building labeled big data for researching context reasoning for context inference engine design
- Development of SuggestBot UI and service software subsystem to guarantee the real time requirement of SuggestBot
Contents
| DataSet | |
| A dataset for the analysis of interest and affective state in exhibition visiting scenario | |
| We present a multimodal dataset for the analysis of human interest and affective states. Physiological, motion, and video signals of 100 participants were recorded as each visited the exhibition room for at least 20 minutes. | |
| K-EmoCon, a multimodal sensor dataset for continuous emotion recognition in naturalistic conversations | |
| We present a multimodal dataset for the analysis of human emotion. EEG, physiological, motion, and video signals of 30 participants were recorded as each participated in the debate for at least 15 minutes. | |
| Online discussion and evidence dataset | |
| From Internet Argument Corpus (IAC) 2.0 dataset, this dataset covers 10,000 posts containing claims that can be verified with facts. The dataset contains annotations on verifiable sentences for each post and factual information that supports or refutes each annotation. | |
| K-EmoPhone, students' emotion and stress in daily life | |
| We present a sensor dataset collected by wearable sensors and smartphones. This dataset contains physiological signals, motion/location-related records, behavior features with the corresponding emotion and stress labels. | |
| Context-related Hashtag Classification Dataset | |
| This dataset contains the classification of hashtags from social media posts into eight context categories ("Emotion", "Mood", "Location", "Time", "Object", "Activity", "Event", "Other"). | |
| Psychological Safety Dataset | |
| This dataset contains annotations on the utterances from AMI Corpus that would affect the psychological safety of the group. Each annotation describe a) whether the utterance would positively or negatively affect the psychological safety of the group and b) how the annotator would explain their annotation to the meeting participants. | |
| Data Collect Program | |
| Sensor and interactive data logger for mobile and wearable devices | |
| The application collects sensor and interactive data from mobile and wearable devices, store the data in the device, and sends to predefined servers. | |
| Heart rate information logger based on Ant+ protocol | |
| The application detects and collects heart rate information via BLE (Bluetooth Low Energy) connection based on ANT+ protocol. ANT+ is a Wireless Personal Network protocol. | |
| Data logger library/application for Android devices | |
| This is a simple, but powerful logger that allows Android apps to log both to the internal log, as well as an external file (i.e. in sqlite format). This can also be used as a stand-alone application. | |
| ABC Logger, for the collection of a variety of smartphone data | |
| This application collects various types of sensor data from an Android smartphone and the connected wearable sensors (e.g., Polar H10 chest-band), and the user's own status logs by using the experience sampling method (ESM). | |
| Smart Speaker ESM, for the collection of user contexts with using a voice-based experience sampling method | |
| This application collects the opportune moments in the domestic context for proactive conversational interactions with smart speakers. | |
| Crowdsourcing Task for Collecting Hashtag-Context Mapping | |
| This crowdsourcing interface is designed to collect hashtags likely to be used in social media posts and classification of hashtags into context categories. The interface collects hashtags from worker's own social media posts with explicit agreement or hypothetical social media posts created by the worker with their own photos. | |
| Crowdsourcing Task for Collecting Annotations on Psychological Safety [Frontend] [API server] | |
| This crowdsourcing interface is designed to collect annotations on the utterances from meeting logs. The current version of the interface is targetted to collect utterances affecting the psychological safety of the group, and it could be applied to various other annotations. | |
| Deep Learning Module | |
| Denoising Convolutional and Recurrent Neural Network for Multimodal Emotion Recognition | |
| This program incorporates multiple heterogeneous sensor data to recognize human emotion. The main machine learning model of the program pipelines convolutional neural networks and a recurrent neural network to effectively extract features from multimodal sensor data and recognize human emotion using the extracted features. To improve the model performance, various optimization techniques including the denoising gated recurrent unit (DGRUD) and the convolutional block attention module (CBAM) are applied. The repository also provides pre-trained models with a set of sensor configurations. | |
| Search Engine for Claim-supporting Content with Deep Learning | |
| We build a module that identifies the contents that can support(e.g. related, strengthen) a given claim. We use Wikipedia and news articles as main resources. | |
| Context-Aware Tag Prediction Model | |
| This tag engine is devised for recommending relevant tags. This exploits various context information such as image, location, and time to suggest a list of tags. The tags to be output are pre-defined based on how often they occur in the collected Instagram posts. | |
| Platform System | |
| TFLite based benchmark program for multiple DL model running situation on mobile | |
| The benchmark tool measures DL model execution performance (e.g., latency) while multiple CNN models running simultaneousl. This program has been confirmed to work on a Pixel 2 device with Android 8.0 or higher. The benchmark currently offers two popular DL models, MobileNet_V1(quantized) and Inception_V3(quantized), which are officially supported by TensorFlow Lite. | |
| Privacy-preserving Deep Learning System | |
| This repository contains an implementation of a privacy-preserving deep learning inference system for remote clouds or edge servers. The system prevents data disclosure and modification from malicious attackers by leveraging a trusted execution environment(TEE). The system is built upon Caffe, a popular deep learning framework, to port into Intel SGX enclaves. Since using shared libraries are not allowed in the enclaves, the repository includes all source files of Caffe and its dependencies. | |
| Wearable Text entry system for smart glasses with a smartwatch as an input device | |
| The wearable text entry system decomposed into 2 parts, Smart glasses as an output device (Display) and Smartwatch as an input device (Touch gestures). Both devices are connected with Bluetooth connection and both of the devices should be powered by Android OS (min SDK ver. is 19). There are embedded two different keyboards, one is SwipeBoard and the other is HoldBoard. In the case of HoldBoard, it is needed to use an edge-touch sensitive smartwatch, but it is more effective to enter a text. | |
| Gaze-Assisted Typing Interface for Smart Glasses | |
| We develop gaze-assisted typing (GAT) interface, which uses both a touchpad and eye tracking, for more efficient smart glass text entry. | |
Keywords
Smart Interaction, SuggestBot, Context Big Data, Context Deep Learning, Wearable Computing Platform