Add to Chrome
Add to Firefox
Add to EdgeZhongliang Yang
ReSee: Responding through Seeing Fine-grained Visual Knowledge in Open-domain Dialogue
May 23, 2023
Incorporating visual knowledge into text-only dialogue systems has become a potential direction to imitate the way humans think, imagine, and communicate. However, existing multimodal dialogue systems are either confined by the scale and quality of available datasets or the coarse concept of visual knowledge. To address these issues, we provide a new paradigm of constructing multimodal dialogues as well as two datasets extended from text-only dialogues under such paradigm (ReSee-WoW, ReSee-DD). We propose to explicitly split the visual knowledge into finer granularity (``turn-level'' and ``entity-level''). To further boost the accuracy and diversity of augmented visual information, we retrieve them from the Internet or a large image dataset. To demonstrate the superiority and universality of the provided visual knowledge, we propose a simple but effective framework ReSee to add visual representation into vanilla dialogue models by modality concatenations. We also conduct extensive experiments and ablations w.r.t. different model configurations and visual knowledge settings. Empirical, encouraging results not only demonstrate the effectiveness of introducing visual knowledge at both entity and turn level but also verify the proposed model ReSee outperforms several state-of-the-art methods on automatic and human evaluations. By leveraging text and vision knowledge, ReSee can produce informative responses with real-world visual concepts.
PCAE: A Framework of Plug-in Conditional Auto-Encoder for Controllable Text Generation
Oct 07, 2022
Controllable text generation has taken a gigantic step forward these days. Yet existing methods are either constrained in a one-off pattern or not efficient enough for receiving multiple conditions at every generation stage. We propose a model-agnostic framework Plug-in Conditional Auto-Encoder for Controllable Text Generation (PCAE) towards flexible and semi-supervised text generation. Our framework is "plug-and-play" with partial parameters to be fine-tuned in the pre-trained model (less than a half). Crucial to the success of PCAE is the proposed broadcasting label fusion network for navigating the global latent code to a specified local and confined space. Visualization of the local latent prior well confirms the primary devotion in hidden space of the proposed model. Moreover, extensive experiments across five related generation tasks (from 2 conditions up to 10 conditions) on both RNN- based and pre-trained BART [26] based auto-encoders reveal the high capability of PCAE, which enables generation that is highly manipulable, syntactically diverse and time-saving with minimum labeled samples. We will release our code at https://github.com/ImKeTT/pcae.
FairVFL: A Fair Vertical Federated Learning Framework with Contrastive Adversarial Learning
Jun 07, 2022
Vertical federated learning (VFL) is a privacy-preserving machine learning paradigm that can learn models from features distributed on different platforms in a privacy-preserving way. Since in real-world applications the data may contain bias on fairness-sensitive features (e.g., gender), VFL models may inherit bias from training data and become unfair for some user groups. However, existing fair ML methods usually rely on the centralized storage of fairness-sensitive features to achieve model fairness, which are usually inapplicable in federated scenarios. In this paper, we propose a fair vertical federated learning framework (FairVFL), which can improve the fairness of VFL models. The core idea of FairVFL is to learn unified and fair representations of samples based on the decentralized feature fields in a privacy-preserving way. Specifically, each platform with fairness-insensitive features first learns local data representations from local features. Then, these local representations are uploaded to a server and aggregated into a unified representation for the target task. In order to learn fair unified representations, we send them to each platform storing fairness-sensitive features and apply adversarial learning to remove bias from the unified representations inherited from the biased data. Moreover, for protecting user privacy, we further propose a contrastive adversarial learning method to remove privacy information from the unified representations in server before sending them to the platforms keeping fairness-sensitive features. Experiments on two real-world datasets validate that our method can effectively improve model fairness with user privacy well-protected.
AdaVAE: Exploring Adaptive GPT-2s in Variational Auto-Encoders for Language Modeling
May 20, 2022
Variational Auto-Encoder (VAE) has become the de-facto learning paradigm in achieving both representation learning and generation for natural language. However, existing VAE-based language models either employ elementary RNNs, which is not powerful to handle complex situations, or fine-tunes two pre-trained language models (PLMs) for any downstream task, which is a huge drain on resources. In this paper, we introduce the first VAE framework empowered with adaptive GPT-2s (AdaVAE). Different from existing systems, we unify both the encoder\&decoder of VAE model using GPT-2s with adaptive parameter-efficient components. Experiments from multiple dimensions validate that AdaVAE is competent to better organize language in generation task and representation modeling, even with less than $15\%$ activated parameters in training. Our code is available at \url{https://github.com/ImKeTT/adavae}.
Pixel-Stega: Generative Image Steganography Based on Autoregressive Models
Dec 21, 2021
In this letter, we explored generative image steganography based on autoregressive models. We proposed Pixel-Stega, which implements pixel-level information hiding with autoregressive models and arithmetic coding algorithm. Firstly, one of the autoregressive models, PixelCNN++, is utilized to produce explicit conditional probability distribution of each pixel. Secondly, secret messages are encoded to the selection of pixels through steganographic sampling (stegosampling) based on arithmetic coding. We carried out qualitative and quantitative assessment on gray-scale and colour image datasets. Experimental results show that Pixel-Stega is able to embed secret messages adaptively according to the entropy of the pixels to achieve both high embedding capacity (up to 4.3 bpp) and nearly perfect imperceptibility (about 50% detection accuracy).
Provably Secure Generative Linguistic Steganography
Jun 03, 2021
Generative linguistic steganography mainly utilized language models and applied steganographic sampling (stegosampling) to generate high-security steganographic text (stegotext). However, previous methods generally lead to statistical differences between the conditional probability distributions of stegotext and natural text, which brings about security risks. In this paper, to further ensure security, we present a novel provably secure generative linguistic steganographic method ADG, which recursively embeds secret information by Adaptive Dynamic Grouping of tokens according to their probability given by an off-the-shelf language model. We not only prove the security of ADG mathematically, but also conduct extensive experiments on three public corpora to further verify its imperceptibility. The experimental results reveal that the proposed method is able to generate stegotext with nearly perfect security.
Graph-Stega: Semantic Controllable Steganographic Text Generation Guided by Knowledge Graph
Jun 02, 2020
Most of the existing text generative steganographic methods are based on coding the conditional probability distribution of each word during the generation process, and then selecting specific words according to the secret information, so as to achieve information hiding. Such methods have their limitations which may bring potential security risks. Firstly, with the increase of embedding rate, these models will choose words with lower conditional probability, which will reduce the quality of the generated steganographic texts; secondly, they can not control the semantic expression of the final generated steganographic text. This paper proposes a new text generative steganography method which is quietly different from the existing models. We use a Knowledge Graph (KG) to guide the generation of steganographic sentences. On the one hand, we hide the secret information by coding the path in the knowledge graph, but not the conditional probability of each generated word; on the other hand, we can control the semantic expression of the generated steganographic text to a certain extent. The experimental results show that the proposed model can guarantee both the quality of the generated text and its semantic expression, which is a supplement and improvement to the current text generation steganography.
Master-Auxiliary: an efficient aggregation strategy for video anomaly detection
May 24, 2020
The aim of surveillance video anomaly detection is to detect events that rarely or never happened in a specified scene. Different detectors can detect different anomalies. This paper proposes an efficient strategy to aggregate multiple detectors together. At first, the aggregation strategy chooses one detector as master detector, and sets the other detectors as auxiliary detectors. Then, the aggregation strategy extracts credible information from auxiliary detectors, which includes credible abnormal (Cred-a) frames and credible normal (Cred-n) frames, and counts their Cred-a and Cred-n frequencies. Finally, the aggregation strategy utilizes the Cred-a and Cred-n frequencies to calculate soft weights in a voting manner, and uses the soft weights to assist the master detector. Experiments are carried out on multiple datasets. Compared with existing aggregation strategies, the proposed strategy achieves state-of-the-art performance.
A Promotion Method for Generation Error Based Video Anomaly Detection
Dec 17, 2019
Using the generation error (GE) of a generative neural network (GNN) to detect video anomalies exhibits excellent performance. However, there are two problems when using the trained GNN models to detect anomalies. First, utilizing the frame-level GE to detect anomalies reduces the anomaly saliencies, because anomalies usually occur in local areas. Second, when multiple discriminants (a discriminant is an anomaly score sequence) are available, using the weighted sum method to aggregate multiple discriminants does not always perform effectively, and the weights are hard to tune. To address these problems, we propose an approach consists of two modules. Firstly, we replace the frame-level GE with the maximum of the block-level GEs in the frame to detect anomalies. Secondly, assuming that the higher the anomaly threshold, the more reliable the anomaly detected, we propose a reliable-anomaly (R-anomaly) based strategy to aggregate multiple discriminants. We use the R-anomalies in the auxiliary discriminants to enhance their anomaly scores in the main discriminant. Experiments are carried out on UCSD and CUHK Avenue datasets. The results demonstrate the effectiveness of the proposed method and achieve state-of-the-art performance.